How Swin Transformer achieves linear attention

WHAT THIS TESTS Whether you grasp why global self-attention is quadratic and how localizing it plus shifting windows recovers linear cost without losing cross-region communication.

A GOOD ANSWER COVERS In a standard ViT, every patch token attends to every other token, so for N patches the attention cost grows as N squared. Doubling image resolution quadruples the patch count and roughly multiplies attention cost sixteen-fold, which is intractable for high-resolution images and dense tasks. Swin restricts self-attention to small, fixed-size non-overlapping windows, say 7 by 7 patches. Within each window patches attend only to each other, so the cost per window is constant and the total cost scales linearly with the number of windows, hence linearly with the number of patches. The problem is that pure window attention never lets information cross window boundaries. Swin solves this with shifted windows: in alternating transformer blocks the window grid is shifted by half a window so the new windows overlap the previous boundaries, connecting neighboring regions over successive layers. Swin also merges patches at deeper stages to form a hierarchical, multi-scale feature pyramid like a CNN backbone.

WHY IT MATTERS Linear complexity makes attention affordable on megapixel images, and the pyramid yields multi-scale features, so Swin serves as a general backbone for detection and segmentation, not just classification.

COMMON WRONG ANSWERS Saying Swin keeps global attention but is just faster. Forgetting the shifted-window step, leaving windows isolated. Claiming windowing alone gives a global receptive field in one layer.

LIKELY FOLLOW-UPS How does the shift get implemented efficiently with cyclic shifting and masking? How does patch merging build the hierarchy? How does Swin compare to dilated or linear-attention approximations? What window size trades cost against context?

ONE CONCRETE EXAMPLE On a 1024 by 1024 image, global ViT attention is prohibitive. Swin partitions patches into 7 by 7 windows, attends locally for linear cost, then shifts the window grid in the next block so a feature near a window edge can reach its neighbor, building global context across layers while staying tractable.