Aligning text and image representations

WHAT THIS TESTS The interviewer wants concrete techniques for cross-modal alignment and the understanding that alignment must be explicitly learned, not assumed.

A GOOD ANSWER COVERS The first and most influential technique is contrastive learning, as in CLIP. Separate image and text encoders produce embeddings, and a contrastive loss over a large batch maximizes cosine similarity between matched image-text pairs while minimizing it for all mismatched pairs. After training, both modalities inhabit a shared latent space where similarity is directly comparable, enabling zero-shot classification and retrieval. The second technique is learned projection or adapter layers: take a strong pretrained encoder from one modality, freeze it, and train a small linear or MLP projection that maps another modality's features into its space, which is how vision-language models like LLaVA feed image features into a frozen LLM's embedding space. Other approaches include cross-attention bridges, where one modality attends into another, and shared-encoder or fusion transformers trained jointly. The common thread is that alignment is a learned objective.

COMMON WRONG ANSWERS Assuming two independently pretrained encoders already share a latent space; they do not, and their embeddings are not comparable without an alignment step. Another error is conflating contrastive alignment with generative training; CLIP does not generate, it aligns.

LIKELY FOLLOW-UPS Expect questions on the role of large batch sizes and negatives in contrastive learning, on the temperature parameter, on how projection layers are trained while encoders are frozen, and on zero-shot transfer enabled by a shared space.

ONE CONCRETE EXAMPLE CLIP is trained on hundreds of millions of image-caption pairs so that the embedding of a dog photo sits near the embedding of the text a photo of a dog and far from a photo of a car. This shared space lets you classify a new image zero-shot by comparing its embedding to text embeddings of candidate labels.