Why CNNs need nonlinear activations like ReLU

WHAT THIS TESTS Whether you understand the single most important reason deep networks work: nonlinearity between layers. This separates people who memorize architectures from those who understand them.

A GOOD ANSWER COVERS ReLU, defined as the maximum of zero and the input, is applied element-wise after convolutions. Its role is to make the overall function nonlinear so that stacking layers actually increases the class of functions the network can represent. ReLU is favored because it is cheap, produces sparse activations, and its gradient is one for positive inputs, which mitigates the vanishing gradient problem that plagued sigmoid and tanh in deep nets. If you removed all activation functions and used only convolutions and linear layers, the entire network would collapse: the composition of linear maps is itself a single linear map. So a deep stack would be mathematically equivalent to one linear layer and could only learn linear decision boundaries, no matter how many layers you added.

COMMON WRONG ANSWERS Thinking depth alone provides capacity; without nonlinearity it does not. Saying ReLU is needed mainly for speed; speed is a bonus, expressiveness is the point. Forgetting the dying ReLU issue where neurons stuck at zero stop learning.

LIKELY FOLLOW-UPS What is the dying ReLU problem and how do leaky ReLU or GELU address it. Why did ReLU largely replace sigmoid. Can you prove the linear collapse algebraically.

ONE CONCRETE EXAMPLE Two linear layers with weight matrices W1 and W2 applied in sequence give W2 times W1 times x, which equals a single matrix W3 times x. Adding more such layers just multiplies more matrices into one equivalent matrix, so the model can never separate classes that are not linearly separable, like points arranged in concentric rings. Inserting ReLU between the layers breaks this collapse and lets the network carve nonlinear boundaries.