Zero-shot classification with CLIP

WHAT THIS TESTS The question checks that you understand contrastive pretraining and why it lets a model classify into arbitrary, novel label sets without retraining. The key insight is the shared image-text embedding space.

A GOOD ANSWER COVERS Pretraining: CLIP has separate image and text encoders trained on hundreds of millions of image-caption pairs with a contrastive loss that pulls matching pairs together and pushes mismatched pairs apart in a shared embedding space. Inference: take the target class names, wrap each in a prompt template such as a photo of a label, and encode them with the text encoder to get one vector per class. Encode the input image with the image encoder. Normalize and compute cosine similarity between the image vector and each text vector, scale by temperature, and softmax to get probabilities; the highest-similarity label is the prediction. No gradient updates or target-set training occur.

COMMON WRONG ANSWERS Saying CLIP needs a linear classifier trained on the target classes, which is the linear-probe variant, not zero-shot. Or claiming a fixed output vocabulary, when labels can be any text at inference. Or ignoring prompt engineering, which materially affects accuracy.

LIKELY FOLLOW-UPS Why does prompt wording change accuracy and what is prompt ensembling. How does the contrastive loss shape the space. What are CLIP's failure modes, such as fine-grained or counting tasks. How does zero-shot differ from a linear probe.

ONE CONCRETE EXAMPLE To classify a photo as cat, dog, or rabbit, you encode the prompts a photo of a cat, a photo of a dog, and a photo of a rabbit with the text encoder, encode the image with the image encoder, and compare cosine similarities. The label with the highest similarity wins, all without training on any cat-dog-rabbit dataset. If accuracy is borderline you can ensemble several prompt templates per class and average their embeddings, which often lifts performance, illustrating why prompt wording materially affects zero-shot results even though no weights are updated.