How ViT and CNN process an image differently

WHAT THIS TESTS Whether you can describe, step by step, how each architecture turns raw pixels into the first layer's input, highlighting local convolution versus global tokenized attention.

A GOOD ANSWER COVERS A CNN starts from the raw image as a height-by-width grid with color channels. Its first layer is a set of small learnable filters, for example 3 by 3, convolved across every spatial location with shared weights. Each filter produces a feature map by computing local weighted sums, so the very first layer captures local patterns like edges, and translation equivariance and locality are built in. A Vision Transformer takes a different route. It divides the image into a grid of fixed-size non-overlapping patches, say 16 by 16 pixels. Each patch is flattened into a vector and passed through a single linear projection to produce a patch embedding, a token. A special learnable class token is prepended, and positional embeddings are added so the model knows where each patch sat, since attention is otherwise order-agnostic. This sequence of tokens is fed to a standard transformer encoder, where self-attention lets every patch interact with every other from the first layer, giving global context immediately.

COMMON WRONG ANSWERS Saying ViT convolves over pixels; its patch embedding is a linear projection (sometimes implemented as one strided conv, but it is non-overlapping, not a sliding filter bank). Forgetting positional embeddings, without which a ViT is permutation invariant. Claiming CNNs see global context in layer one.

LIKELY FOLLOW-UPS Why does ViT need positional embeddings but a CNN does not? What is the role of the class token? How does patch size trade resolution against cost? Why does ViT need more data?

ONE CONCRETE EXAMPLE A 224 by 224 image: a CNN convolves 3 by 3 filters across all positions producing edge maps. A ViT cuts it into 196 patches of 16 by 16, linearly embeds each to a 768-dim token, adds a class token and positional embeddings, and sends 197 tokens into attention.