Designing a baseline Visual Question Answering model

WHAT THIS TESTS: Whether you can specify a coherent two-stream multimodal pipeline and the practical choice of treating answering as classification.

A GOOD ANSWER COVERS: The inputs are an image and a free-form question about it. For the image stream, use a pretrained CNN such as a ResNet and take a global feature vector from a late layer; freezing or fine-tuning depends on data size. For the text stream, tokenize the question, embed the tokens, and summarize them with an RNN such as an LSTM final state, or for an even simpler baseline average the word embeddings. Project both modality vectors to the same dimension and fuse them into a joint representation; common simple fusions are concatenation or elementwise multiplication, the latter encouraging interaction between modalities. Pass the joint vector through a small multilayer perceptron and a softmax over a fixed answer vocabulary, treating VQA as multi-class classification over the few thousand most frequent answers rather than open-ended generation. Train with cross-entropy on answer labels. This baseline is intentionally minimal; attention over image regions conditioned on the question is the natural next upgrade.

COMMON WRONG ANSWERS: Treating a simple VQA baseline as free-form text generation instead of classification; forgetting to fuse the two streams and only using one modality; using raw pixels with no CNN; ignoring that strong language priors can let a model answer without looking at the image, inflating accuracy.

LIKELY FOLLOW-UPS: Why might the model exploit language priors and how do you detect that? How would adding region-level attention help? Concatenation versus elementwise product for fusion? How do you handle rare or open-ended answers the fixed vocabulary cannot represent?

ONE CONCRETE EXAMPLE: Given a kitchen photo and the question what color is the mug, a ResNet produces a 2048-dimensional image vector and an LSTM encodes the question into a 1024-dimensional vector. Both are projected to 1024 dimensions and multiplied elementwise; the joint vector goes through an MLP and softmax over the top one thousand answers, outputting red. A purely text-driven model might guess a common color without the image, which is the bias you must guard against.