Walk through RLHF's three stages, outputs, and purposes.

WHAT THIS TESTS: This question tests whether you understand RLHF as a multi-model training pipeline rather than a monolithic algorithm. Interviewers want to see that you know which artifact is produced at each stage, why that artifact is necessary, and how the stages connect. At the senior level, they also care whether you recognize the design space, such as the fact that the initial LM can vary in size and fine-tuning history, and that the reward model can be an end-to-end language model or a modular ranking system.

A GOOD ANSWER COVERS: First, pretraining the language model. The output is a base LM, potentially already instruction-tuned, that can generate diverse completions. Its purpose is to provide a capable policy to initialize the RL loop. Second, training the reward model. The output is a model that accepts a text sequence and returns a scalar reward. Its purpose is to capture human preference numerically so automated optimization can occur. Third, RL fine-tuning. The output is the aligned language model. Its purpose is to maximize the learned reward while still generating coherent text, effectively baking complex human values into the policy. A strong candidate mentions that stage two requires expensive human comparison data and that stage three typically uses an algorithm like PPO.

COMMON WRONG ANSWERS: A common red flag is describing RLHF as just supervised fine-tuning on human-written answers. Another is conflating the reward model with the final policy, for example saying the reward model is what gets deployed to users. Some candidates also omit the scalar nature of the reward, instead saying the reward model outputs text or rankings without clarifying the scalar conversion. Saying there is only one model trained end-to-end also signals confusion.

LIKELY FOLLOW-UPS: Interviewers often ask why you need a separate reward model instead of using human labels directly in the RL loop. They may ask how you prevent the policy from over-optimizing the reward model, known as reward hacking. Another follow-up is what data is collected in stage two, typically pairwise or ranked comparisons of model outputs. They might also ask which RL algorithms are used, such as PPO, and why.

ONE CONCRETE EXAMPLE: In InstructGPT, OpenAI started with a smaller GPT-3 base model. They then collected human comparisons of multiple model outputs for the same prompt and trained a reward model to predict those preferences as a scalar value. Finally, they used PPO to fine-tune the base LM against that reward model, producing the final InstructGPT model that better followed instructions and aligned with human intent.

Source: huggingface.co