Reward Modeling: Teaching an LLM What 'Good' Means
A reward model is a judge that scores an LLM's outputs based on human preferences. It learns to assign a numerical 'goodness' score to text, turning subjective quality into an optimizable signal for training models like ChatGPT.
A reward model acts like a judge, scoring an LLM's outputs based on human preferences. It learns to assign a numerical 'goodness' score, turning subjective quality into an optimizable signal. This is the critical step in RLHF, used to align models like ChatGPT to be more helpful and harmless, guiding them toward outputs humans actually prefer. The footgun is a flawed reward model that can be gamed, leading the LLM to find bizarre, high-scoring but useless outputs.
Read the original → huggingface.co
- #llm
- #rlhf
- #ai alignment
- #generative ai
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.