RLHF: Teaching an AI 'Good' Without Code
Reinforcement Learning from Human Feedback (RLHF) teaches a model what humans prefer by having it chase the approval of a proxy 'reward model' trained on human rankings. It's the key technique for making large language models more helpful and harmless by aligning them with nuanced instructions that are hard to define in code. The main footgun is 'reward hacking,' where the model finds loopholes to please the reward model in ways that don't actually satisfy users.
### The Mental Model
Reinforcement Learning from Human Feedback (RLHF) distills subjective human preferences into a mathematical signal (a reward) that a machine can optimize. Instead of telling a model *how* to be helpful or harmless with code, you show it examples of what humans prefer via rankings, and then train the model to maximize a score that represents that preference.
### How It Works
RLHF is typically a three-step process:
1. **Pre-train a Language Model:** Start with a base model (like GPT-3) that has learned language patterns from a massive dataset. 2. **Train a Reward Model:** Generate several answers for a set of prompts. Human labelers rank these answers from best to worst. This dataset of `(prompt, chosen_response, rejected_response)` is used to train a separate 'reward model'. This model's job is to predict which response a human would prefer, outputting a scalar score. 3. **Fine-tune with Reinforcement Learning:** The pre-trained model is then fine-tuned. For a given prompt, it generates a response (the 'action'). This response is fed to the reward model, which returns a score (the 'reward'). Using an RL algorithm like PPO, the language model's policy is updated to generate responses that maximize the reward score from the reward model.
### When to Use It
* **Aligning LLMs with abstract values:** Essential for teaching models to be 'helpful,' 'harmless,' and 'honest' in a way that resonates with humans. * **Improving subjective quality:** For tasks where quality is hard to measure automatically, like text summarization, creative writing, or generating empathetic dialogue. * **Controlling style and tone:** Fine-tuning a model to adopt a specific persona, such as being professional, concise, or humorous.
### When NOT to Use It
* **When a clear, objective metric exists:** If you can measure success with a simple loss function (e.g., classification accuracy), RLHF adds unnecessary complexity and cost from human labeling. * **If the reward model is flawed:** A poorly trained or biased reward model will lead the main model astray. The model can learn to 'reward hack'—finding clever ways to get a high score from the reward model that don't correspond to actual good behavior.
### One Canonical Example
Imagine you want to make a chatbot better at explaining things simply.
- **Prompt:** "Explain a black hole to a 5-year-old." - **Base Model generates two answers:** - A1: "A black hole is a region of spacetime where gravity is so strong that nothing, not even light, can escape." - A2: "A black hole is like a giant vacuum cleaner in space that eats everything that gets too close, even starlight!" - **Human Feedback:** A human labeler ranks `A2 > A1` because it uses a better analogy for a child. - **Training:** This preference helps train the reward model. After thousands of similar rankings, the reward model learns to give higher scores to simple, analogical explanations when the prompt requests them. The main LLM then uses RL to learn how to generate responses like A2 to maximize its reward.
Read the original → en.wikipedia.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.