How gradient descent and the learning rate work

WHAT THIS TESTS This checks your grasp of the core optimization algorithm behind most machine learning training and the practical tradeoffs of its key hyperparameter.

A GOOD ANSWER COVERS Gradient descent minimizes a cost function by iteratively adjusting parameters. At each step it computes the gradient, the vector of partial derivatives that points in the direction of steepest increase, and then moves the parameters in the opposite direction, the negative gradient, to decrease the cost. Repeating this walks the parameters downhill toward a minimum. The learning rate is a scalar that controls how big each step is. Picking it well is central: it determines whether training converges quickly, slowly, or not at all.

COMMON WRONG ANSWERS Saying parameters move along the gradient rather than against it. Claiming a larger learning rate always speeds training; beyond a point it harms it. Confusing the cost minimum with always being the global minimum, ignoring that non-convex losses have local minima and saddle points. Forgetting that the gradient magnitude shrinks near a minimum, naturally slowing steps.

LIKELY FOLLOW-UPS What is the difference between batch, stochastic, and mini-batch gradient descent? They differ in how much data each step's gradient uses, trading noise for speed. How do optimizers like momentum or Adam help? They adapt or smooth steps to escape slow regions. How do you choose the learning rate? Tune it, use schedules that decay it, or warm it up. What about convex versus non-convex? On convex losses gradient descent reaches the global minimum; on neural networks it finds good local minima.

ONE CONCRETE EXAMPLE Picture descending a foggy hill by feeling the slope under your feet and stepping downhill. If the learning rate is too high, you take giant strides and leap across the valley, landing higher on the far slope and bouncing back and forth or even climbing away, which is divergence or oscillation. If it is too low, you shuffle in tiny steps and take an impractically long time to reach the bottom, possibly getting stuck on a nearly flat plateau. A well-chosen rate takes steady, sizable steps that settle efficiently into the valley floor, which is the minimum of the cost function.