Value Learning

WHY IT EXISTS We cannot write down everything humans care about. Any explicit reward function is a proxy, and a sufficiently capable optimizer will exploit the gap between the proxy and what we truly meant, producing reward hacking or harmful specification gaming. Value learning exists to close that gap by having the agent learn the objective itself rather than receiving it as fixed code.

THE MENTAL MODEL Treat the true human objective as unknown and represent the agent's belief about it as a probability distribution. The agent's job is not to maximize a given reward but to maximize expected satisfaction of the values it is still learning. Because it holds uncertainty over what humans want, it has an incentive to gather more information, ask, and avoid irreversible actions that might violate values it has not yet pinned down. This uncertainty is what makes it corrigible and willing to be corrected.

HOW IT WORKS Preferences are inferred from evidence about human behavior and choices. Inverse reinforcement learning recovers a reward function from observed expert behavior. Reinforcement learning from human feedback trains a reward model on human comparisons of outputs, then optimizes a policy against it. Cooperative inverse reinforcement learning frames it as a joint game where the human and agent both act to help the agent learn the human's reward. All share the idea of learning the target rather than assuming it.

WHEN IT MATTERS It matters most as systems become more capable and operate in open-ended environments where a static objective inevitably fails to anticipate every case. It underpins the alignment of modern large language models and any autonomous agent whose decisions carry real consequences.

ONE CONCRETE EXAMPLE Modern chat assistants are tuned with reinforcement learning from human feedback: humans rank candidate responses, a reward model learns to predict those rankings, and the policy is optimized against that learned model. The model never receives an explicit formula for helpfulness or harmlessness; it infers an approximation of those human values from the comparison data, which is value learning in practice.