How does text guide Stable Diffusion via U-Net cross-attention?

WHAT THIS TESTS: This question probes whether you understand the architecture of latent diffusion models at a component level, specifically how a text modality conditions a generative image model. Interviewers care if you can distinguish the roles of the text encoder, the VAE, and the U-Net, and whether you grasp the attention mechanism that actually binds words to pixels. Senior candidates should explain cross-attention without hand-waving.

A GOOD ANSWER COVERS: First, the prompt is tokenized and passed through a frozen text encoder such as CLIP to produce a sequence of text embeddings, typically 77 tokens by 768 or 1024 dimensions. Second, these embeddings are injected into the U-Net denoiser via cross-attention layers that are interleaved with the standard convolutional and self-attention blocks. Third, in each cross-attention layer, the Query matrix is derived from the flattened intermediate image latent features, while the Key and Value matrices are derived from the text embeddings. Fourth, the attention scores map spatial image regions to semantic text tokens, allowing the denoising process to strengthen or suppress visual features based on the prompt. Fifth, this happens at multiple resolutions across the U-Net down-blocks, mid-block, and up-blocks, ensuring both global composition and local detail are guided.

COMMON WRONG ANSWERS: Claiming the text vector is concatenated to the image latent or noise vector before entering the U-Net. Describing cross-attention but getting the Q and K/V sources backwards. Calling it self-attention instead of cross-attention. Forgetting that the process operates in latent space rather than pixel space. Saying the text encoder is fine-tuned with the U-Net when in many Stable Diffusion variants it is frozen.

LIKELY FOLLOW-UPS: How does classifier-free guidance modify this cross-attention pathway? What happens if you manipulate the cross-attention maps during inference, such as in prompt-to-prompt editing? Why is the text encoder kept frozen rather than trained end-to-end? How does the model handle multiple prompts or negative prompts?

ONE CONCRETE EXAMPLE: Consider the prompt a red cube on top of a blue sphere. In the U-Net cross-attention layers, the spatial region corresponding to the upper center of the image will show high attention scores to the tokens red and cube, while the lower region will attend strongly to blue and sphere. If you were to visualize these attention maps, you would see that the model disentangles the two objects by routing different spatial queries to different text keys, which is why it can respect both color and positional adjectives simultaneously.

Source: huggingface.co