CNN features for image retrieval

WHAT THIS TESTS The interviewer wants to see that you understand transfer learning, the hierarchy of features inside a CNN, and how to convert activations into a usable descriptor.

A GOOD ANSWER COVERS A CNN pretrained on a large labeled dataset has learned a rich hierarchy of visual features that transfer to other tasks. To use it as a descriptor extractor you remove the final classification layer and run an image forward through the network, then read out the activations from an intermediate or late layer as a feature vector. Convolutional feature maps from a late block can be globally pooled into a compact vector, or you can take the penultimate fully connected layer. You then L2-normalize the vector and compare images by cosine or Euclidean distance. The choice of layer matters because depth corresponds to abstraction: early layers respond to edges, colors, and textures, mid layers to parts and motifs, and deep layers to high-level semantic content.

COMMON WRONG ANSWERS Using the final softmax output, which collapses the image to class probabilities and loses the descriptive richness needed for matching. Ignoring normalization. Assuming the earliest layers are best for semantic retrieval.

LIKELY FOLLOW-UPS Why normalize the descriptor. When would earlier layers be preferable. How does fine-tuning compare with using frozen features.

ONE CONCRETE EXAMPLE Suppose you build a visual product-search index. You load VGG16 with its pretrained weights, remove the classifier head, and for each catalog image you take the activations of a deep layer, such as the last fully connected layer before the output or a globally pooled final convolutional block, giving a fixed-length vector. You L2-normalize each vector and store them in a nearest-neighbor index. At query time you push the query photo through the same network, get its normalized descriptor, and retrieve the catalog vectors with smallest cosine distance; these are the visually and semantically most similar products. Deep-layer features work here because two photos of the same handbag from different angles produce nearby deep descriptors even though their raw pixels differ a lot, since the deep layer encodes the object's high-level appearance rather than exact pixels. If instead you wanted to match fine surface texture, like fabric weave, you might draw from an earlier layer that preserves local detail. Fine-tuning the network on in-domain data can sharpen retrieval further when labeled data is available.