CLIP's contrastive objective and zero-shot classification

WHAT THIS TESTS Whether you can state CLIP's contrastive training objective precisely and explain the mechanism that lets it classify unseen tasks with no task-specific fine-tuning.

A GOOD ANSWER COVERS TRAINING OBJECTIVE. CLIP has two encoders, one for images and one for text, trained jointly on a large set of image-caption pairs scraped from the web. For a batch of N pairs there are N-by-N possible image-text combinations; only the N diagonal pairs are true matches. The contrastive objective pushes the cosine similarity of matched image-text embeddings high and the N-squared minus N mismatched combinations low, using a symmetric cross-entropy loss over image-to-text and text-to-image similarities, scaled by a learned temperature. The result is a shared embedding space where an image and its describing text land near each other. Note it is not trained to predict a fixed label set; it learns alignment between arbitrary text and images. ZERO-SHOT. To classify, you do not fine-tune. You write each candidate class as a natural-language prompt, for example a photo of a dog, embed all class prompts with the text encoder, embed the image with the image encoder, and pick the class whose text embedding has the highest cosine similarity to the image embedding. Because the space aligns free-form text with images, any new label expressible in words becomes a classifier on the fly.

COMMON WRONG ANSWERS Calling CLIP a standard softmax image classifier with fixed classes. Saying it predicts captions autoregressively. Forgetting the shared embedding space and cosine similarity. Confusing the contrastive loss with a reconstruction or generative objective. Thinking zero-shot requires fine-tuning.

LIKELY FOLLOW-UPS Why does prompt wording (prompt templates) affect zero-shot accuracy? What role does the temperature parameter play? How do mismatched in-batch pairs act as negatives? What are CLIP's limitations on fine-grained or compositional tasks?

ONE CONCRETE EXAMPLE To classify an image as cat, dog, or car with no training, you embed a photo of a cat, a photo of a dog, a photo of a car and the image itself, then compute cosine similarities. If the image embedding is closest to the cat prompt, the prediction is cat. The same trick works for any new label simply by writing it as text, which is the heart of zero-shot transfer.