Classic image captioning architecture

WHAT THIS TESTS The interviewer wants the classic encoder-decoder mental model and the role of attention, confirming you see captioning as conditional sequence generation, not classification.

A GOOD ANSWER COVERS The two core components are a visual encoder and a language decoder. The encoder is typically a convolutional network pretrained on image classification, with its final layers used to produce either a single global feature vector or, more powerfully, a grid of spatial feature maps. The decoder is a sequence model, classically an LSTM and in modern systems a transformer, that generates the caption autoregressively, predicting one word at a time conditioned on the image features and the words generated so far, starting from a start token until an end token. The bridge between them is attention: at each decoding step the model computes weights over the encoder's spatial features so it can focus on the relevant region, attending to a dog when emitting the word dog and to grass when emitting grass. Training minimizes cross-entropy against reference captions, often with teacher forcing.

COMMON WRONG ANSWERS Describing captioning as a single multi-class prediction, which cannot produce variable-length fluent sentences. Another error is omitting attention and assuming a single fixed image vector suffices; that works but produces vaguer captions because the decoder cannot localize.

LIKELY FOLLOW-UPS Expect questions on beam search versus greedy decoding, on evaluation with BLEU and CIDEr, on teacher forcing and exposure bias, and on how modern vision-language models replace this with a unified transformer.

ONE CONCRETE EXAMPLE Given a photo of a dog catching a frisbee, the CNN encodes spatial features, the decoder emits a, then dog, attending to the animal, then catching and frisbee while attending to the disc, producing a dog catching a frisbee, a fluent variable-length sentence rather than one class label.