What is overfitting and how does Dropout prevent it?

WHAT THIS TESTS: Whether you understand generalization as the core goal of supervised learning and can articulate the gap between training performance and test performance. It also checks if you can explain dropout as a stochastic regularization technique that modifies the training process rather than a simple pruning or architecture search method.

A GOOD ANSWER COVERS four things in order. First, define overfitting as the situation where a model achieves small training error but large test error because it memorizes noise and spurious patterns in the training data rather than learning the underlying distribution. Second, explain that dropout addresses this by randomly setting a fraction of hidden unit activations to zero during each training forward pass according to a Bernoulli mask, which creates a different thinned subnetwork on every update step. Third, note that this randomness prevents hidden units from co-adapting too strongly to specific other units, forcing each neuron to learn robust features that are useful in many different contexts and not dependent on the presence of any single partner neuron. Fourth, mention that at test time dropout is turned off and the full network is used with activations scaled down by the keep probability so that the expected output matches training behavior, effectively approximating an ensemble average over all possible subnetworks.

COMMON WRONG ANSWERS include saying that dropout permanently deletes neurons from the architecture, that it simply reduces the number of parameters like pruning, or that it adds noise to weights rather than to activations. Another red flag is forgetting that dropout requires scaling at inference or claiming that dropout should be applied during testing to obtain multiple predictions.

LIKELY FOLLOW-UPS are how dropout relates to bagging and model averaging of exponentially many subnetworks, why we must scale outputs by one minus the dropout rate at test time or equivalently scale up during training, how dropout interacts with batch normalization, what happens if you set the dropout rate too close to zero or too close to one, and whether dropout is still necessary in very large datasets.

ONE CONCRETE EXAMPLE is training a multilayer perceptron with two hidden layers on a small image classification task. Without any regularization the network quickly reaches ninety nine percent training accuracy but stalls at seventy five percent validation accuracy, a clear overfitting signature. After inserting dropout with probability zero point five after each hidden layer, training accuracy falls to ninety two percent while validation accuracy climbs to eighty two percent. The twenty point gap between training and validation error shrinks to ten points, demonstrating that dropout has suppressed memorization of label noise and improved generalization by breaking harmful co-adaptations among feature detectors.