Inductive biases of ViT versus CNN

WHAT THIS TESTS Whether you understand inductive bias as built-in assumptions that reduce data needs, and can reason about why a ViT's weaker bias changes its training regime.

A GOOD ANSWER COVERS Inductive biases are assumptions baked into an architecture that constrain the hypothesis space toward solutions likely to fit the domain. A CNN has strong, image-appropriate biases: locality, because each filter only sees a small neighborhood; translation equivariance and weight sharing, because the same filter slides everywhere, so a feature learned in one place transfers to all places; and a hierarchical multi-scale structure from pooling. These priors match natural-image statistics, so CNNs learn well from limited data. A standard Vision Transformer has very weak image-specific bias. Self-attention is global and inherently permutation invariant; the only structure imposed is splitting into patches and adding learned positional embeddings, and even those positions must be learned rather than guaranteed. There is no built-in locality or translation equivariance.

PRACTICAL CONSEQUENCES Because the ViT must learn spatial relationships, locality, and invariances from data rather than getting them for free, it is data hungry. On small or medium datasets a CNN typically wins. Only with very large-scale pretraining, for example JFT or ImageNet-21k, or strong augmentation and regularization, do ViTs match or exceed CNNs, since enough data lets them learn the right inductive structure and even surpass fixed convolutional priors.

COMMON WRONG ANSWERS Saying ViTs have the same locality bias as CNNs. Claiming ViTs always beat CNNs regardless of data. Ignoring positional embeddings entirely.

LIKELY FOLLOW-UPS How do hybrid models or convolutional stems reintroduce bias? Why do DeiT-style distillation and augmentation help small-data ViTs? How does Swin add locality back? What happens to attention maps as data scales?

ONE CONCRETE EXAMPLE Trained from scratch on ImageNet-1k, a ViT trails a comparable ResNet; pretrained on ImageNet-21k then fine-tuned, the same ViT surpasses it, illustrating that its missing bias is compensated only by large-scale data.