Policy Gradient: Teach an Agent What to Do, Not What's Valuable

WHY IT EXISTS: Value-based reinforcement learning methods, like Q-learning, struggle in environments with continuous or very large action spaces. They must compute a value for every possible action to find the best one, which is computationally infeasible. Policy gradient methods were invented to solve this by learning the policy directly.

THE MENTAL MODEL: Imagine teaching a robot to throw a dart. A value-based method would involve calculating the expected score for every possible angle and force (the 'value'). A policy gradient method is like having the robot throw, and you just tell it "a little more to the left" or "a bit less force". You are directly adjusting the action-taking policy itself, not calculating intermediate values.

HOW IT WORKS: The core is a parameterized policy, often a neural network, that takes a state and outputs a probability distribution over actions. The agent samples an action from this distribution and executes it. After running for a while (an 'episode' or 'trajectory'), the algorithm looks at the rewards received. It then uses gradient ascent to increase the probability of actions that led to good outcomes and decrease the probability of actions that led to bad ones. For this to work, the policy function must be differentiable.

WHEN TO USE IT: Policy gradient methods are the go-to for problems with continuous action spaces, like controlling a robot's joint angles or a car's steering and acceleration. They also work well when the optimal policy is stochastic, meaning there isn't one single best action in a state (like in rock-paper-scissors).

WHEN NOT TO USE IT: For problems with small, discrete action spaces, value-based methods are often more sample-efficient and stable. The biggest footgun of policy gradients is their high variance; the gradient signal can be very noisy, leading to slow or unstable training. They often require large batches of experience and advanced techniques (like actor-critic methods) to work reliably.

ONE CANONICAL EXAMPLE: The REINFORCE algorithm is the simplest policy gradient method. An agent plays a full game of Pong. If it wins, all actions it took during that game are considered 'good' and their probabilities are increased. If it loses, all actions are considered 'bad' and their probabilities are decreased. This simple approach directly reinforces entire sequences of actions based on the final outcome.