Attention in diffusion U-Nets for text conditioning

WHAT THIS TESTS: Whether you grasp how a convolutional U-Net is augmented with attention so it can model long-range spatial dependencies and absorb a text prompt during denoising.

A GOOD ANSWER COVERS: Self-attention is inserted at one or more resolution levels, usually the deeper, lower-resolution blocks where the feature maps are small enough to make the quadratic cost tolerable. Each spatial position becomes a token; queries, keys, and values all come from the image features, so every location can attend to every other location and capture global coherence the local convolutions miss. Cross-attention is the conditioning mechanism: the queries come from the image feature map, but the keys and values come from the encoded text tokens produced by a text encoder. This lets each spatial region pull in the relevant words from the prompt. In practice both live inside a transformer block (self-attention, then cross-attention, then a feed-forward layer) interleaved with the residual convolutional blocks, and the same blocks appear on the encoder and decoder halves around the bottleneck.

COMMON WRONG ANSWERS: Saying text is concatenated directly to the pixel input; swapping the roles so that text supplies the queries; claiming attention runs at full image resolution everywhere, which would be prohibitively expensive; forgetting that the timestep embedding is injected separately, typically via additive or modulated scale-shift, not through cross-attention.

LIKELY FOLLOW-UPS: Why put attention only at low resolution? How does classifier-free guidance interact with the cross-attention path? What changes when you move to latent diffusion operating on a VAE latent rather than raw pixels? How would you edit a region by manipulating cross-attention maps?

ONE CONCRETE EXAMPLE: In Stable Diffusion the U-Net denoises a 64x64 latent. CLIP text encoder output, a sequence of 77 token embeddings, feeds the keys and values of every cross-attention layer, while the noisy latent feeds the queries. Self-attention blocks at the 16x16 and 8x8 stages enforce global structure, so a prompt like a cat wearing a hat keeps the hat on the head rather than floating elsewhere.