What does the KL-divergence penalty do in RLHF PPO, and if zeroed?

What's really being asked

This question probes whether you understand the fundamental misalignment between proxy reward models and true human preferences in RLHF. Interviewers want to see that you recognize the reward model is not ground truth but a learned classifier, and that unconstrained reinforcement learning against this classifier leads to exploitation rather than genuine improvement. It also checks if you know why the reference policy matters as a trust region and how PPO is adapted for language model fine-tuning.

The full answer

First, state clearly that the KL-divergence penalty acts as a trust-region constraint that keeps the PPO policy from drifting too far from the supervised fine-tuned reference model. Second, explain that without this term the policy will over-optimize the reward model, a phenomenon known as reward hacking, where outputs become adversarial sequences that score highly on the proxy but are incoherent, repetitive, or hallucinated to human evaluators. Third, mention that the reference model preserves fluency, diversity, and prior knowledge because it was trained on broad human text, so staying close to it prevents mode collapse. Fourth, note that setting the coefficient to zero removes this safety boundary, causing training instability and a measurable drop in human preference ratings even as the proxy reward rises.

The mistakes people make

A red flag is describing the KL term as generic regularization or a mere training stability trick without connecting it to the reference policy or reward hacking. Another weak answer is claiming the penalty prevents overfitting to the training data; in RLHF the danger is overfitting to the reward model, not the training corpus. Some candidates confuse the KL penalty with entropy bonus, saying it encourages exploration, which is incorrect because the KL term specifically penalizes deviation from a fixed reference rather than encouraging action diversity.

What usually comes next

An interviewer might ask how you would tune the KL coefficient in practice, so be ready to discuss grid search or adaptive schemes based on the magnitude of the KL divergence during training. They may also ask how the KL penalty interacts with the clipping objective in PPO, or whether you can replace the KL term with a constraint on the reward model itself such as a pessimistic or ensemble-based reward. Another follow-up is comparing KL penalization in RLHF to the KL term used in variational inference or DPO.

A concrete example

Imagine a summarization model where the reward model prefers concise bullet points. Without a KL penalty, the PPO policy might collapse into generating endless short fragments like Buy now or Click here because the reward model accidentally scores terse text highly. The KL term penalizes this drift because the reference model assigns extremely low probability to such repetitive spam, forcing the policy to stay in a region where summaries remain grammatical and on-topic while still becoming more concise.