Adapting ViT for dense semantic segmentation

WHAT THIS TESTS Whether you understand that a ViT outputs a sequence of tokens, not a spatial map, and can describe how to recover dense per-pixel predictions plus the cost and inductive-bias issues this creates.

A GOOD ANSWER COVERS A ViT splits the image into non-overlapping patches, embeds each as a token, and processes them with self-attention; for classification only the class token is read out. For segmentation you instead use the patch tokens. Reshape the token sequence back into a two-dimensional grid corresponding to patch positions, then attach a decoder. Options include simple progressive upsampling, a Mask Transformer head as in Segmenter that uses learnable class tokens attending to patches, or a DPT-style design that reassembles tokens from several encoder depths into multi-scale feature maps and fuses them. The output is upsampled to full resolution.

COMMON WRONG ANSWERS Using only the class token, which discards all spatial information. Assuming the ViT gives a feature pyramid like a CNN; plain ViT keeps a single resolution. Ignoring that attention is quadratic in the number of patches, so high-resolution images explode in cost and memory.

LIKELY FOLLOW-UPS How does Segmenter's mask head differ from a convolutional decoder? Why does ViT need more pretraining data than a CNN? How do hierarchical variants like Swin restore a pyramid? How do you keep memory tractable at high resolution?

ONE CONCRETE EXAMPLE Segmenting a 512 by 512 image with 16-pixel patches yields a 32 by 32 token grid. You reshape the 1024 patch tokens into that grid, fuse tokens from layers 6, 12, 18, and 24 with a DPT decoder to add scale, and upsample to 512 by 512, recovering crisp class boundaries the single-scale ViT alone would miss.