CLIP's contrastive objective and zero-shot classification
Contrastive vision-language pretraining.
Train image and text encoders to align matched pairs and repel mismatched ones in a shared space; classify zero-shot by comparing an image to text prompts of class names.
What's really being asked
Whether you can state CLIP's contrastive training objective precisely and explain the mechanism that lets it classify unseen tasks with no task-specific fine-tuning.
The full answer
TRAINING OBJECTIVE. CLIP has two encoders, one for images and one for text, trained jointly on a large set of image-caption pairs scraped from the web. For a batch of N pairs there are N-by-N possible image-text combinations; only the N diagonal pairs are true matches. The contrastive objective pushes the cosine similarity of matched image-text embeddings high and the N-squared minus N mismatched combinations low, using a symmetric cross-entropy loss over image-to-text and text-to-image similarities, scaled by a learned temperature. The result is a shared embedding space where an image and its describing text land near each other. Note it is not trained to predict a fixed label set; it learns alignment between arbitrary text and images. ZERO-SHOT. To classify, you do not fine-tune. You write each candidate class as a natural-language prompt, for example a photo of a dog, embed all class prompts with the text encoder, embed the image with the image encoder, and pick the class whose text embedding has the highest cosine similarity to the image embedding. Because the space aligns free-form text with images, any new label expressible in words becomes a classifier on the fly.
The mistakes people make
Calling CLIP a standard softmax image classifier with fixed classes. Saying it predicts captions autoregressively. Forgetting the shared embedding space and cosine similarity. Confusing the contrastive loss with a reconstruction or generative objective. Thinking zero-shot requires fine-tuning.
What usually comes next
Why does prompt wording (prompt templates) affect zero-shot accuracy? What role does the temperature parameter play? How do mismatched in-batch pairs act as negatives? What are CLIP's limitations on fine-grained or compositional tasks?
A concrete example
To classify an image as cat, dog, or car with no training, you embed a photo of a cat, a photo of a dog, a photo of a car and the image itself, then compute cosine similarities. If the image embedding is closest to the cat prompt, the prediction is cat. The same trick works for any new label simply by writing it as text, which is the heart of zero-shot transfer.
Interview question
How does CLIP perform zero-shot classification on classes it was never explicitly trained to predict?
- a.It fine-tunes a softmax head on the new classes before predicting
- b.It embeds class names as text prompts and selects the class whose embedding is closest to the image embeddingCorrect
- c.It generates a caption and string-matches it to the class list
- d.It retrains the image encoder for each new label set
Why? this is the answer
CLIP aligns images and text in a shared space, so any label written as text becomes a classifier via cosine similarity, no fine-tuning needed. It does not add a softmax head or retrain per task.
Just read this? Test yourself on what you have been reading.
- #llm
- #clip
- #contrastive-learning
- #multimodal
- #zero-shot
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on llm — each one lists the topics its interview covers.
See open roles