The role of anchor boxes in detectors

What's really being asked

The interviewer checks that you understand anchors as a design device that makes dense detection tractable, plus the practical pain they introduce.

The full answer

Problem solved: objects vary in position, scale, and aspect ratio, and a location may contain several objects. Anchors address this by placing a fixed set of reference boxes of various scales and aspect ratios at every spatial location of the feature map. The network then, per anchor, predicts an objectness or class score and regresses small offsets to adjust the anchor toward the true box. This reframes detection as classification plus bounding-box refinement relative to known priors, which is easier to learn than regressing boxes from scratch and naturally handles multiple, overlapping objects. Challenges: the scales and aspect ratios are hyperparameters that need tuning per dataset; most anchors are background, creating severe class imbalance addressed by focal loss or hard negative mining; performance suffers if anchor shapes mismatch the data, such as very tall or tiny objects; and dense anchors produce many candidate boxes requiring NMS. These pains motivated anchor-free detectors like FCOS and DETR.

The mistakes people make

Treating anchors as the final predicted boxes rather than priors that get refined. Or ignoring class imbalance. Or assuming one universal anchor set works for any dataset.

What usually comes next

How do you choose anchor scales and ratios, perhaps via clustering ground-truth boxes. How does focal loss tackle anchor imbalance. How do anchor-free detectors avoid these issues. How is an anchor matched to a ground-truth box via IoU.

A concrete example

For a pedestrian dataset, you cluster the ground-truth boxes with k-means and find most are tall and narrow, so you set anchor aspect ratios accordingly rather than using generic squares. At each feature-map cell the model scores its tiled anchors for objectness and class, and regresses offsets that nudge the best-matching tall anchor to tightly fit each person. An anchor is assigned to a ground-truth box when their IoU is high enough, which defines the positive training targets. Finally NMS removes the overlapping duplicate boxes that the dense anchors inevitably produce, leaving one tight box per pedestrian.