Temporal consistency in video diffusion

WHAT THIS TESTS The interviewer wants an architecture-level answer to why naive per-frame generation flickers and how a temporal dimension fixes it.

A GOOD ANSWER COVERS An image diffusion U-Net only has spatial layers, so applying it independently to each frame yields temporal inconsistency: textures, identities, and backgrounds jitter between frames. The standard solution inflates the model with temporal layers. You keep the spatial convolutions and spatial self-attention, then insert temporal modules that operate along the time axis: either temporal attention, where each spatial location attends to the same location across all frames, or temporal 1D convolutions across frames, sometimes combined into pseudo-3D blocks. Crucially, all frames are denoised jointly as one tensor with a time dimension, so information flows between frames at every step. A common engineering pattern initializes the spatial layers from a pretrained image model and freezes them, training only the new temporal layers, which transfers strong image priors and saves compute. Conditioning on the first frame or optical-flow cues further stabilizes motion.

COMMON WRONG ANSWERS Generating each frame fully independently and post-hoc smoothing, which cannot fix identity drift or content jitter. Another error is using only a single shared noise seed across frames, which reduces but does not eliminate inconsistency because the model still has no cross-frame attention.

LIKELY FOLLOW-UPS Expect questions on factorized spatiotemporal attention cost, on training data and frame counts, on long-video generation via autoregressive chunks, and on classifier-free guidance for motion.

ONE CONCRETE EXAMPLE Generating a clip of a walking dog frame by frame makes the fur and collar flicker. Adding temporal attention so each pixel attends to its counterpart across frames, and denoising the whole clip together, keeps the dog's appearance stable and the motion smooth, while frozen spatial weights preserve image quality.