RL components and how Q-learning works

WHAT THIS TESTS This verifies that you can articulate the reinforcement learning framework precisely and explain how a value-based method bootstraps its way to an optimal policy.

A GOOD ANSWER COVERS The agent is the learner and decision maker. The environment is everything it interacts with. At each step the agent observes a state, the current situation, chooses an action, and the environment responds with a new state and a scalar reward. The objective is to learn a policy, a mapping from states to actions, that maximizes the expected cumulative discounted reward, where a discount factor gamma between 0 and 1 weights future rewards less than immediate ones. Q-learning learns an action-value function Q(s,a) estimating the expected return of taking action a in state s and acting optimally thereafter. After each transition it applies the update: new Q equals old Q plus a learning rate times the temporal-difference error, where the target is the reward plus gamma times the maximum Q over actions in the next state. Because the target uses the max, Q-learning is off-policy and converges to the optimal Q under broad conditions.

COMMON WRONG ANSWERS Confusing the reward, an immediate signal, with the return, the discounted sum. Forgetting the discount factor or the max over next actions. Calling Q-learning on-policy; it is off-policy. Describing the policy as fixed rather than derived greedily from Q.

LIKELY FOLLOW-UPS Why is Q-learning off-policy while SARSA is on-policy. How does the agent explore during training. What breaks when the state space is continuous or huge.

ONE CONCRETE EXAMPLE Consider a grid where an agent must reach a goal cell. States are grid cells, actions are up, down, left, right, and the reward is plus ten at the goal and minus one per step. The agent starts with a zero Q-table, explores using epsilon-greedy moves, and after each move updates Q toward the reward plus the discounted best next-cell value. Over many episodes the Q-values propagate backward from the goal, and the greedy policy eventually traces the shortest path.