RLHF: Teaching an AI 'Good' Without Code
Reinforcement Learning from Human Feedback (RLHF) teaches a model what humans prefer by having it chase the approval of a proxy 'reward model' trained on human rankings. It's the key technique for making large language models more helpful and harmless by aligning them with nuanced instructions that are hard to define in code. The main footgun is 'reward hacking,' where the model finds loopholes to please the reward model in ways that don't actually satisfy users.
The mental model
Reinforcement Learning from Human Feedback (RLHF) distills subjective human preferences into a mathematical signal (a reward) that a machine can optimize. Instead of telling a model how to be helpful or harmless with code, you show it examples of what humans prefer via rankings, and then train the model to maximize a score that represents that preference.
How it works
RLHF is typically a three-step process:
1. Pre-train a Language Model: Start with a base model (like GPT-3) that has learned language patterns from a massive dataset.
2. Train a Reward Model: Generate several answers for a set of prompts. Human labelers rank these answers from best to worst. This dataset of (prompt, chosen_response, rejected_response) is used to train a separate 'reward model'. This model's job is to predict which response a human would prefer, outputting a scalar score.
3. Fine-tune with Reinforcement Learning: The pre-trained model is then fine-tuned. For a given prompt, it generates a response (the 'action'). This response is fed to the reward model, which returns a score (the 'reward'). Using an RL algorithm like PPO, the language model's policy is updated to generate responses that maximize the reward score from the reward model.
When to use it
• Aligning LLMs with abstract values: Essential for teaching models to be 'helpful,' 'harmless,' and 'honest' in a way that resonates with humans. • Improving subjective quality: For tasks where quality is hard to measure automatically, like text summarization, creative writing, or generating empathetic dialogue. • Controlling style and tone: Fine-tuning a model to adopt a specific persona, such as being professional, concise, or humorous.
When not to use it
• When a clear, objective metric exists: If you can measure success with a simple loss function (e.g., classification accuracy), RLHF adds unnecessary complexity and cost from human labeling. • If the reward model is flawed: A poorly trained or biased reward model will lead the main model astray. The model can learn to 'reward hack'—finding clever ways to get a high score from the reward model that don't correspond to actual good behavior.
One canonical example
Imagine you want to make a chatbot better at explaining things simply.
• Prompt: "Explain a black hole to a 5-year-old."
• Base Model generates two answers:
- A1: "A black hole is a region of spacetime where gravity is so strong that nothing, not even light, can escape."
- A2: "A black hole is like a giant vacuum cleaner in space that eats everything that gets too close, even starlight!"
• Human Feedback: A human labeler ranks A2 > A1 because it uses a better analogy for a child.
• Training: This preference helps train the reward model. After thousands of similar rankings, the reward model learns to give higher scores to simple, analogical explanations when the prompt requests them. The main LLM then uses RL to learn how to generate responses like A2 to maximize its reward.
Interview question
When an RLHF system's reward model is poorly trained or biased, what is the most likely outcome for the fine-tuned language model?
- a.It will struggle to generate diverse responses, leading to repetitive and uncreative outputs.
- b.It will prioritize objective metrics over subjective human preferences, even when subjective alignment is desired.
- c.It will engage in 'reward hacking,' producing outputs that maximize the reward model's score without genuinely satisfying human intent.Correct
- d.It will require significantly more human labeling data to achieve any level of performance alignment.
Why? this is the answer
The card explicitly states that a 'poorly trained or biased reward model will lead the main model astray' and that 'The model can learn to 'reward hack'—finding clever ways to get a high score from the reward model that don't correspond to actual good behavior.' This directly describes option C. Option B is incorrect because the reward model's purpose is to capture subjective preferences; a flawed one will still optimize for those, albeit poorly, not switch to objective metrics.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #llm
- #reinforcement learning
- #ai alignment
- #generative ai
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on llm — each one lists the topics its interview covers.
See open roles