What RLHF is and the safety problem it solves

WHAT THIS TESTS The interviewer wants to know if you understand why a model trained only to predict the next token is not automatically helpful or safe, and how RLHF closes that gap. It is the canonical alignment technique behind modern assistants.

A GOOD ANSWER COVERS The three stages. First, often a supervised fine-tuning step on curated demonstrations to get a reasonable starting policy. Second, preference data collection: sample multiple responses to a prompt and have humans rank or compare them, then train a reward model to predict which output humans prefer. Third, reinforcement learning: optimize the language model policy to maximize the reward model's score, commonly with PPO, while a KL-divergence penalty keeps it from drifting too far from the original model and degenerating. The safety problem it addresses: the pretraining objective, maximize likelihood of internet text, does not encode human values, so a raw model may produce unhelpful, dishonest, toxic, or unsafe completions even when fluent. RLHF aligns behavior with human intent, the helpful, honest, harmless goals, because human preferences are easier to provide than a hand-written objective.

COMMON WRONG ANSWERS Describing RLHF as just more supervised fine-tuning, omitting the reward model and RL loop. Saying it improves factual accuracy directly, it shapes preferences, not ground truth. Forgetting the KL penalty that prevents reward hacking and collapse.

LIKELY FOLLOW-UPS What is reward hacking and why the KL term, the policy can exploit reward model flaws. How does DPO differ, it skips the explicit reward model and optimizes preferences directly. What are RLHF limitations, annotator bias and sycophancy.

ONE CONCRETE EXAMPLE Asked a harmful question, a base model may helpfully comply. After RLHF, humans consistently ranked refusals higher, the reward model learned that preference, and the fine-tuned policy now declines, demonstrating value alignment the pretraining loss never specified.