LLaVA versus Flamingo vision-LLM design

WHAT THIS TESTS The interviewer wants a precise comparison of how each design connects a vision encoder to an LLM and the resulting engineering trade-offs, not just naming the models.

A GOOD ANSWER COVERS LLaVA takes the approach of adapter-as-input. A frozen vision encoder produces image features, then a small trainable projection, originally a linear layer and later an MLP, maps those features into the LLM's token embedding space. The projected visual tokens are prepended to the text tokens, so the unmodified LLM processes vision and text in one sequence. Training is cheap: you tune the projection and optionally fine-tune the LLM, with no architectural surgery. Flamingo instead keeps the pretrained LLM frozen and inserts new gated cross-attention dense layers interleaved between the existing transformer blocks. A Perceiver resampler compresses variable numbers of image features into a fixed set of tokens, and the gated cross-attention lets text tokens attend to these visual features, with a tanh gate initialized to zero so the pretrained LLM behavior is preserved at the start. This design natively supports interleaved sequences of multiple images and text.

COMMON WRONG ANSWERS Saying both feed raw pixels to the LLM, or that LLaVA also uses cross-attention layers; LLaVA's mechanism is input-token projection, not inserted attention. Another error is ignoring that Flamingo adds new trainable parameters inside a frozen LLM while LLaVA adds them only at the input.

LIKELY FOLLOW-UPS Expect questions on the Perceiver resampler, on why gated cross-attention is zero-initialized, on which design scales better to many images, and on training data and compute differences.

ONE CONCRETE EXAMPLE For a single-image caption, LLaVA simply projects the image into a few tokens and prepends them to the prompt, reusing the LLM as is. For a few-shot prompt interleaving several images and captions, Flamingo's cross-attention layers let each text span attend to the right image, handling the interleaving more naturally than a flat token sequence.