Explain Supervised Fine-Tuning, RLHF, and DPO

### What this tests This tests your understanding of the evolution of LLM alignment techniques, moving from basic instruction following (SFT) to complex preference optimization (RLHF, DPO). The interviewer wants to see if you grasp the mechanisms, data requirements, and trade-offs of each, not just the acronyms.

### A good answer covers * **Supervised Fine-Tuning (SFT):** This is the first step after pre-training to teach the model specific skills, formats, or styles. It's behavior cloning. You use a high-quality, often human-curated dataset of `(prompt, ideal_response)` pairs and fine-tune the model using a standard cross-entropy loss to imitate those responses. * **Reinforcement Learning from Human Feedback (RLHF):** This is a preference alignment technique to make the model's outputs more helpful, harmless, and aligned with human values. It's a complex, multi-stage process: 1. Generate multiple responses to a prompt. 2. Have humans rank these responses (e.g., Response A > Response B). 3. Train a separate **reward model** on this preference data to predict which response a human would prefer. 4. Use a reinforcement learning algorithm (like PPO) to fine-tune the LLM, using the reward model's score as the reward signal. The goal is to maximize the expected reward. * **Direct Preference Optimization (DPO):** This is a more recent, simpler, and more stable alternative to RLHF. It achieves the same goal of preference alignment but **bypasses the explicit reward model and RL training**. It uses the same preference data (`chosen_response`, `rejected_response`) but optimizes the policy (the LLM) directly using a novel loss function. This loss function effectively increases the relative probability of the chosen response over the rejected one. * **The Relationship:** A typical state-of-the-art pipeline is Pre-training -> SFT -> Preference Tuning (RLHF or DPO). SFT teaches the model what to do, and preference tuning refines *how* it does it.

### Common wrong answers * **Confusing the stages:** Treating SFT and RLHF/DPO as interchangeable choices rather than sequential steps in a pipeline. SFT teaches capability; RLHF/DPO aligns behavior. * **Misunderstanding RLHF:** Describing it as simply 'fine-tuning with a thumbs up/down' signal. This completely misses the critical role of the separately trained reward model and the complexity of using an RL algorithm like PPO. * **Hand-waving DPO:** Saying "DPO is just a newer RLHF." The key insight to mention is that it *avoids* the explicit reward modeling and RL steps, making it a direct optimization that is more stable and computationally cheaper.

### Follow-up the interviewer might ask * "Given its complexity and instability, why was RLHF used in the first place? Why not start with something like DPO?" * "You have a budget to create a dataset for preference tuning. Would you collect 10,000 high-quality preference pairs or 50,000 lower-quality pairs? Why?"

### One concrete example Imagine the prompt: "Write a thank you note for a gift." * **SFT Data:** A single, perfectly written thank you note: `(prompt, "Dear [Name], Thank you so much for the wonderful [gift]...")`. The model learns this format. * **RLHF/DPO Data:** The model generates two notes. Note A is polite but generic. Note B is more personal and warm. A human labeler marks `B > A`. * **RLHF Process:** This `(B > A)` data point helps train a reward model. The LLM is then updated via PPO to generate notes that would score higher on this reward model. * **DPO Process:** The `(chosen=B, rejected=A)` pair is used directly in the DPO loss function to update the LLM, increasing the probability of generating responses like B and decreasing the probability of responses like A.