Gradient Descent: Finding the Bottom of the Hill
Think of finding the lowest point on a foggy hill by taking steps in the steepest downward direction. It's how machine learning models learn, by iteratively minimizing a cost function. The footgun is the step size: too large overshoots, too small is too slow.
WHY IT EXISTS: Many complex problems, especially in machine learning, boil down to finding the best parameters to minimize an error or cost. For functions with millions of parameters, solving for this minimum analytically is impossible. Gradient descent provides a way to find this minimum iteratively.
THE MENTAL MODEL: Imagine you are on a vast, hilly landscape in thick fog, and your goal is to reach the lowest point. You can't see the whole landscape, only the ground right under your feet. The best you can do is feel which direction is steepest downhill and take a step that way. You repeat this process: check the slope, take a step, and check again. The landscape is your cost function, your position is the set of model parameters, and the lowest point is the optimal set of parameters that minimizes the cost.
HOW IT WORKS: Gradient descent is an iterative optimization algorithm. First, you start with an initial guess for the parameters (a random point on the landscape). Second, you compute the gradient of the cost function at that point. The gradient is a vector that points in the direction of the steepest ascent. Third, you update your parameters by taking a small step in the opposite direction of the gradient (downhill). The size of this step is controlled by a parameter called the learning rate. This process is repeated until the parameters converge to a minimum, where the gradient is close to zero.
WHEN TO USE IT: Use it to find the minimum of any high-dimensional, differentiable function. It's the default optimization method for training most machine learning models. This includes linear and logistic regression, support vector machines, and especially deep neural networks, where it's used with backpropagation to update millions of weights.
WHEN NOT TO USE IT: It is not suitable for non-differentiable functions. For functions with many local minima, basic gradient descent can get stuck in a suboptimal valley and fail to find the global minimum. For simple, convex problems where an analytical solution exists (like ordinary least squares), solving it directly is more efficient.
ONE CANONICAL EXAMPLE: Training a simple linear regression model. The goal is to find the line of best fit for a set of data points. The cost function is the Mean Squared Error (MSE), which measures the average squared vertical distance from each point to the line. Gradient descent is used to find the slope and intercept of the line that minimize this MSE. It starts with a random line and iteratively adjusts its slope and intercept by stepping down the gradient of the MSE landscape until it finds the best possible fit.
Read the original → en.wikipedia.org
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.