Multimodal video understanding architecture

WHAT THIS TESTS This evaluates whether you understand how multimodal models bring vision and language into one representation, and the practical pipeline for handling video rather than a single image.

A GOOD ANSWER COVERS The pipeline begins with frame sampling: a video has too many frames to process all, so you sample at a fixed rate or use keyframe selection to capture the salient moments. Each sampled frame passes through a vision encoder, typically a Vision Transformer, producing patch embeddings. A projection module, often a small MLP or a resampler, maps those visual features into the same embedding space and dimensionality as the language model's text tokens, turning the image into a sequence of visual tokens. These visual tokens are concatenated with the tokenized text prompt and fed into the language model. Inside the transformer, self-attention or cross-attention lets text tokens attend to visual tokens and vice versa, so the model fuses both modalities and generates a description conditioned jointly on what it sees and what was asked. For video specifically you also need to encode temporal order, often by adding temporal position information so the model can reason about sequence and motion, not just isolated stills.

COMMON WRONG ANSWERS Running a separate captioner then feeding captions to a text model with no shared representation; processing every frame and ignoring sampling cost; forgetting the projection layer that aligns modalities; ignoring temporal information for video.

LIKELY FOLLOW-UPS How do you handle long videos and token budgets? Why is the projection layer necessary? How is temporal order encoded? How do you reduce hallucinated visual details?

ONE CONCRETE EXAMPLE For a thirty-second clip you sample one frame per second, encode each with a ViT, project the patches into language-model tokens, prepend them to the user's prompt asking what happens, and let attention fuse them. The model produces a description grounded in the visual tokens, referencing actions across frames because temporal positions were encoded.