Regularization: Penalizing Complexity to Prevent Overfitting

Why it exists

Machine learning models can become too good at fitting the data they were trained on. They learn not just the underlying signal, but also the random noise, a problem called overfitting. When an overfit model sees new data, it performs poorly because the noise it memorized isn't present. Regularization was created to solve this by discouraging overly complex models.

The mental model

Think of regularization as a 'complexity tax' added to a model's training process. A model's primary goal is to minimize its prediction error. Regularization adds a second goal: minimize the magnitude of its own internal parameters (weights). This forces a trade-off, pushing the model to find simpler explanations that are more likely to generalize to unseen data, rather than a complex explanation that perfectly fits the training set's quirks.

How it works

During training, a penalty term is added to the model's loss function. This penalty is based on the size of the model's weights. The two most common forms are L1 (Lasso) and L2 (Ridge) regularization. L1 penalizes the absolute value of the weights, which can force some weights to become exactly zero, effectively selecting the most important features. L2 penalizes the squared value of the weights, which keeps all weights small but rarely zero. A hyperparameter, often called alpha or lambda, controls the strength of this penalty—the 'tax rate'.

When to use it

Regularization is a default strategy when training most parametric models, especially with high-dimensional data (many features) or when the dataset is not massive. It is standard practice for linear and logistic regression, support vector machines, and neural networks to improve their generalization to new data.

When not to use it

If your model is already underfitting—performing poorly on both training and test data—it is already too simple. Adding regularization will only worsen the problem by simplifying it further. It is also less directly applicable to non-parametric models like k-Nearest Neighbors, though similar concepts like pruning exist for decision trees.

One canonical example

In standard linear regression, the model finds coefficients that minimize the sum of squared errors. With L2 regularization (Ridge Regression), the objective changes. The model now minimizes the sum of squared errors PLUS a penalty: lambda times the sum of the squared coefficients. If a coefficient grows large to fit a noisy point, the penalty term increases sharply, forcing the model to accept a slightly higher error in exchange for smaller, more stable coefficients. This results in a less erratic, more generalizable model.