Direct Preference Optimization explained

WHAT THIS TESTS The interviewer wants to see whether you understand why DPO can align a model directly on preference data without the reward-model and PPO machinery of classic RLHF.

A GOOD ANSWER COVERS DPO starts from the same objective as RLHF: maximize reward subject to a KL constraint against a reference model. The key insight is that this constrained problem has a closed-form optimal policy, and the reward can be rewritten as a function of the optimal policy and the reference policy. Substituting that expression into the Bradley-Terry preference model yields a loss defined purely over the policy and reference, with no explicit reward network. The loss is classification-like: it increases the log-probability ratio of the chosen response over the rejected response, scaled by a temperature beta that plays the role of the KL strength. Because there is no online sampling, no reward model, and no PPO, training is far simpler and more stable.

COMMON WRONG ANSWERS Calling DPO ordinary supervised fine-tuning on chosen responses; it explicitly contrasts chosen against rejected and depends on the reference-model probability ratio. Another error is claiming DPO always matches or beats RLHF; on some tasks careful PPO still wins, and DPO can overfit or push probability mass oddly.

LIKELY FOLLOW-UPS Expect questions on the role of beta, on why a reference model is still needed, on failure modes like decreasing likelihood of both responses, and on variants such as IPO or KTO.

ONE CONCRETE EXAMPLE Given a prompt with a preferred and a rejected answer, DPO computes each answer's log-probability under both the current policy and the frozen reference, then takes a gradient step that pushes the policy to favor the preferred answer relative to the reference. Thousands of such pairs align the model with one stable offline loss, replacing the three-model RLHF pipeline.