Batching strategy for multimodal training

WHAT THIS TESTS The interviewer wants a thoughtful data-engineering plan that prevents the failure modes specific to mixed-modality training, not just generic data-loading advice.

A GOOD ANSWER COVERS The central lever is sampling ratio. Rather than concatenating datasets and sampling proportionally to their raw size, which lets a huge image-text corpus swamp scarce interleaved data, define explicit per-source mixing weights and sample batches accordingly, tuning the ratios as a hyperparameter. To prevent catastrophic forgetting of language ability, keep a substantial share of pure text data in the mix so the model does not drift away from its pretrained linguistic competence, an effect often called language degradation in VLMs. To prevent modality collapse, where the model ignores one modality, ensure each modality is well represented and monitor per-modality loss curves, optionally enforcing modality-balanced or mixed batches. On the systems side, variable sequence lengths, differing image counts per sample, and the presence or absence of images make batching tricky, so group samples by similar shape, pad consistently, and use a custom collate function. Curriculum strategies, such as starting with alignment then adding harder interleaved data, also help.

COMMON WRONG ANSWERS Concatenating all sources and shuffling once with size-proportional sampling, which lets the largest dataset dominate and starves rare modalities. Another error is dropping text-only data entirely, which accelerates forgetting of language skills.

LIKELY FOLLOW-UPS Expect questions on how to tune mixing ratios, on temperature-based sampling across sources, on detecting modality collapse from loss curves, on freezing components to limit forgetting, and on efficient packing of variable-length sequences.

ONE CONCRETE EXAMPLE A team mixes a large captions dataset, a small interleaved-document set, and a text corpus. Sampling by raw size makes captions ninety-five percent of batches, so the model overfits short captions and its chat ability degrades. Switching to fixed weights, say half captions, a quarter interleaved, a quarter text, restores balanced multimodal and language performance.