Early versus late modality fusion

What's really being asked

The interviewer wants a clear grasp of where modalities meet in the network and the practical consequences of fusing early versus late.

The full answer

Modality fusion is the strategy for integrating signals from different modalities, such as text, image, and audio, into a joint representation or decision. Early fusion combines the modalities at or near the input, concatenating raw features or low-level embeddings so a single network jointly models cross-modal interactions from the start. This captures fine-grained interplay, like how a word relates to a specific image region, but it is sensitive to alignment, requires all modalities present, and scales awkwardly. Late fusion runs a dedicated model per modality and only combines their high-level outputs, for example averaging or concatenating final embeddings or predictions. This is modular, easy to train per modality, and robust when a modality is missing, but it forfeits low-level cross-modal interaction since each stream is processed in isolation. Intermediate or hybrid fusion combines features at multiple depths to balance the two.

The mistakes people make

Reversing the definitions, calling output-level combination early fusion. Another error is claiming one is universally better; the right choice depends on how tightly the modalities interact and whether inputs may be missing.

What usually comes next

Expect questions on cross-attention as a fusion mechanism, on handling missing modalities, on alignment requirements, and on how models like CLIP or LLaVA fuse vision and text.

A concrete example

For visual question answering, early fusion mixes image-region and word features so the model can reason that the word red refers to a specific object, yielding precise answers. A late-fusion sentiment system, by contrast, scores text and audio separately and averages them, which is robust if audio is missing but cannot tell that a sarcastic tone flips the literal words.