Value Learning
Value learning is the AI-safety approach of having a system infer what humans actually value, rather than optimizing a hand-coded proxy, so that capable agents pursue goals aligned with human intent even in novel situations.
WHY IT EXISTS We cannot write down everything humans care about. Any explicit reward function is a proxy, and a sufficiently capable optimizer will exploit the gap between the proxy and what we truly meant, producing reward hacking or harmful specification gaming. Value learning exists to close that gap by having the agent learn the objective itself rather than receiving it as fixed code.
THE MENTAL MODEL Treat the true human objective as unknown and represent the agent's belief about it as a probability distribution. The agent's job is not to maximize a given reward but to maximize expected satisfaction of the values it is still learning. Because it holds uncertainty over what humans want, it has an incentive to gather more information, ask, and avoid irreversible actions that might violate values it has not yet pinned down. This uncertainty is what makes it corrigible and willing to be corrected.
HOW IT WORKS Preferences are inferred from evidence about human behavior and choices. Inverse reinforcement learning recovers a reward function from observed expert behavior. Reinforcement learning from human feedback trains a reward model on human comparisons of outputs, then optimizes a policy against it. Cooperative inverse reinforcement learning frames it as a joint game where the human and agent both act to help the agent learn the human's reward. All share the idea of learning the target rather than assuming it.
WHEN IT MATTERS It matters most as systems become more capable and operate in open-ended environments where a static objective inevitably fails to anticipate every case. It underpins the alignment of modern large language models and any autonomous agent whose decisions carry real consequences.
ONE CONCRETE EXAMPLE Modern chat assistants are tuned with reinforcement learning from human feedback: humans rank candidate responses, a reward model learns to predict those rankings, and the policy is optimized against that learned model. The model never receives an explicit formula for helpfulness or harmlessness; it infers an approximation of those human values from the comparison data, which is value learning in practice.
Interview question
Why does value learning prefer to infer a model of human preferences instead of optimizing a fixed, hand-written reward function?
- a.Learned rewards eliminate the need for any human feedback after deployment
- b.A fixed proxy objective can be gamed and fails in situations its author never anticipatedCorrect
- c.Hand-written rewards train faster but use more memory at inference
- d.Inferred preferences are always provably optimal once training converges
Why? this is the answer
Human values cannot be fully specified in code, so any fixed proxy leaves a gap a capable optimizer can exploit; learning the objective adapts to novel cases. Learned rewards are approximations, not provably optimal, and still depend on human feedback.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #ai-safety
- #alignment
- #value-learning
- #rlhf
- #reward-modeling
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles