RL components and how Q-learning works
foundational RL vocabulary and the Q-learning update.
agent acts on the environment, observes state and reward, seeking to maximize cumulative discounted reward; Q-learning iteratively updates Q(s,a) toward reward plus discounted best…
What's really being asked
This verifies that you can articulate the reinforcement learning framework precisely and explain how a value-based method bootstraps its way to an optimal policy.
The full answer
The agent is the learner and decision maker. The environment is everything it interacts with. At each step the agent observes a state, the current situation, chooses an action, and the environment responds with a new state and a scalar reward. The objective is to learn a policy, a mapping from states to actions, that maximizes the expected cumulative discounted reward, where a discount factor gamma between 0 and 1 weights future rewards less than immediate ones. Q-learning learns an action-value function Q(s,a) estimating the expected return of taking action a in state s and acting optimally thereafter. After each transition it applies the update: new Q equals old Q plus a learning rate times the temporal-difference error, where the target is the reward plus gamma times the maximum Q over actions in the next state. Because the target uses the max, Q-learning is off-policy and converges to the optimal Q under broad conditions.
The mistakes people make
Confusing the reward, an immediate signal, with the return, the discounted sum. Forgetting the discount factor or the max over next actions. Calling Q-learning on-policy; it is off-policy. Describing the policy as fixed rather than derived greedily from Q.
What usually comes next
Why is Q-learning off-policy while SARSA is on-policy. How does the agent explore during training. What breaks when the state space is continuous or huge.
A concrete example
Consider a grid where an agent must reach a goal cell. States are grid cells, actions are up, down, left, right, and the reward is plus ten at the goal and minus one per step. The agent starts with a zero Q-table, explores using epsilon-greedy moves, and after each move updates Q toward the reward plus the discounted best next-cell value. Over many episodes the Q-values propagate backward from the goal, and the greedy policy eventually traces the shortest path.
Interview question
In the Q-learning update, what does the target term reward plus gamma times max next-state Q capture?
- a.The total reward summed over the entire training run
- b.The immediate reward only, ignoring future steps
- c.The estimated best achievable return from the next state onwardCorrect
- d.The probability of the next action under the current policy
Why? this is the answer
The target bootstraps the best estimated future return by taking the max Q over next-state actions, which makes Q-learning off-policy. It is not just the immediate reward, since gamma discounts and adds future value.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #reinforcement-learning
- #q-learning
- #policy
- #bellman-equation
- #machine-learning
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles