Reward models in RLHF and PPO

What's really being asked

The question checks whether you can separate the three models in RLHF and explain how subjective human preference is converted into an optimizable reward.

The full answer

The reward model is a separate network, usually initialized from the same base LLM, with the language head replaced by a scalar output. It is trained on comparison data: for a given prompt, humans rank two or more candidate responses, and the model learns via a pairwise loss so the preferred response receives a higher score. In the PPO loop, the policy generates a completion, the reward model assigns a scalar reward, and PPO uses that reward to compute advantages and update the policy toward higher-scoring outputs. Critically, a KL-divergence penalty against the frozen reference model is added to the reward so the policy does not drift into degenerate text that games the reward model.

The mistakes people make

Saying humans score each PPO rollout in real time, confusing the reward model with the policy being trained, or omitting the KL penalty. Without the KL term, PPO exploits weaknesses in the reward model and produces incoherent or repetitive reward-hacking outputs.

What usually comes next

Expect questions on reward hacking, on why DPO removes the explicit reward model, on the Bradley-Terry preference loss, and on how reward-model overoptimization is detected.

A concrete example

Humans see two summaries of an article and pick the better one thousands of times. The reward model learns to predict that preference as a number. During PPO, the policy writes a new summary, the reward model rates it, and the policy nudges its weights to earn higher ratings, while the KL penalty keeps the summary fluent and on-topic rather than collapsing into reward-maximizing gibberish.