How gradient descent and the learning rate work
optimization fundamentals.
gradient descent steps downhill along the negative gradient to minimize cost; the learning rate sets step size; too high diverges or oscillates, too low converges painfully slowly.
What's really being asked
This checks your grasp of the core optimization algorithm behind most machine learning training and the practical tradeoffs of its key hyperparameter.
The full answer
Gradient descent minimizes a cost function by iteratively adjusting parameters. At each step it computes the gradient, the vector of partial derivatives that points in the direction of steepest increase, and then moves the parameters in the opposite direction, the negative gradient, to decrease the cost. Repeating this walks the parameters downhill toward a minimum. The learning rate is a scalar that controls how big each step is. Picking it well is central: it determines whether training converges quickly, slowly, or not at all.
The mistakes people make
Saying parameters move along the gradient rather than against it. Claiming a larger learning rate always speeds training; beyond a point it harms it. Confusing the cost minimum with always being the global minimum, ignoring that non-convex losses have local minima and saddle points. Forgetting that the gradient magnitude shrinks near a minimum, naturally slowing steps.
What usually comes next
What is the difference between batch, stochastic, and mini-batch gradient descent? They differ in how much data each step's gradient uses, trading noise for speed. How do optimizers like momentum or Adam help? They adapt or smooth steps to escape slow regions. How do you choose the learning rate? Tune it, use schedules that decay it, or warm it up. What about convex versus non-convex? On convex losses gradient descent reaches the global minimum; on neural networks it finds good local minima.
A concrete example
Picture descending a foggy hill by feeling the slope under your feet and stepping downhill. If the learning rate is too high, you take giant strides and leap across the valley, landing higher on the far slope and bouncing back and forth or even climbing away, which is divergence or oscillation. If it is too low, you shuffle in tiny steps and take an impractically long time to reach the bottom, possibly getting stuck on a nearly flat plateau. A well-chosen rate takes steady, sizable steps that settle efficiently into the valley floor, which is the minimum of the cost function.
Interview question
What happens when the learning rate in gradient descent is set too high?
- a.Steps overshoot the minimum and the loss may oscillate or divergeCorrect
- b.The gradient is computed in the wrong direction
- c.The model converges to the global minimum guaranteed
- d.Training always reaches the minimum faster with no downside
Why? this is the answer
Too large a step size overshoots the minimum, causing oscillation or divergence instead of convergence. A high rate does not guarantee faster or correct convergence, and it does not change how the gradient is computed.
Just read this? Test yourself on what you have been reading.
Read the original → en.wikipedia.org
- #machine-learning
- #optimization
- #gradient-descent
- #learning-rate
- #training
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on machine-learning — each one lists the topics its interview covers.
See open roles