Reward Modeling: Teaching an LLM What 'Good' Means

Why it exists

Standard language model training uses simple objectives like next-token prediction, which doesn't capture subjective human values like helpfulness, creativity, or truthfulness. A mechanism was needed to translate complex, subjective human preferences into a numerical signal that a model can be optimized against, making it 'aligned' with user intent.

The mental model

A reward model is a separate AI model that acts as a judge. It's trained to evaluate the output of another, larger language model. It takes a prompt and a generated response as input, and outputs a single number—a scalar reward—that represents how much a human would likely prefer that response. It essentially learns to quantify 'goodness'.

How it works

The process starts with a pretrained language model. For a given set of prompts, this model generates multiple different responses. Human labelers are then shown these responses and asked to rank them from best to worst. This creates a dataset of human preferences (e.g., for a specific prompt, response B was better than A, which was better than C). The reward model is then trained on this dataset. Its sole job is to predict the human preference score for any given prompt-response pair.

When to use it

Reward modeling is the core component of Reinforcement Learning from Human Feedback (RLHF). Use it to fine-tune a language model to align with complex, subjective goals that are hard to define with a simple loss function. This is essential for making chatbots helpful, honest, and harmless, as seen in models like ChatGPT and Claude.

When not to use it

Avoid reward modeling when a clear, objective, and cheaper metric already exists. For instance, if you're fine-tuning a model to generate code where the only goal is executability, a simple pass/fail test is more direct and less expensive than collecting human preference data. It's overkill for tasks where standard metrics like BLEU scores are sufficient.

One canonical example

The training of InstructGPT, a precursor to ChatGPT, is a classic example. OpenAI took a pretrained GPT-3 model, generated responses to various prompts, and had humans rank those responses. They then trained a reward model on this preference data. This reward model was then used in the reinforcement learning stage to guide the main GPT-3 model toward producing outputs that humans would rate more highly, making it significantly better at following instructions.