Anchor Boxes: Pre-defined Guesses for Object Detection

WHY IT EXISTS Early object detection methods were often slow, requiring multiple passes or complex proposal stages. Anchor boxes were introduced to make detection a single, efficient process. They reframe the problem from "where in this image could an object be?" to a simpler question: "for these fixed reference boxes, is there an object nearby and how do I adjust the box to fit it?"

THE MENTAL MODEL Imagine laying a grid of pre-sized stencils over an image. At each point on the grid, you have a few different stencils: a tall one for people, a wide one for cars, a big square, a small square. An object detection model learns to look at each location, pick the best-fitting stencil for any object present, and then tell you exactly how to nudge and resize that stencil to perfectly outline the object.

HOW IT WORKS At each spatial location in a model's feature map, a set of anchor boxes is generated. These boxes have different scales (sizes) and aspect ratios (e.g., 1:1, 1:2, 2:1). For every single anchor box in the image, the network predicts two things: first, a classification score for each possible object class (plus 'background'), and second, four regression values that define the offset (x, y, width, height) needed to transform the anchor box into a precise bounding box for the detected object.

WHEN TO USE IT Anchor boxes are a core component in many influential object detection architectures, including single-shot detectors like SSD and YOLO (v2-v5), and in the Region Proposal Network (RPN) of two-stage detectors like Faster R-CNN. They are excellent for detecting multiple, potentially overlapping objects of varying sizes and shapes in a single pass.

WHEN NOT TO USE IT A growing number of modern architectures are "anchor-free" (e.g., FCOS, CenterNet). These models predict object centers or keypoints directly to avoid the complexity and hyperparameter tuning associated with anchor boxes. If you are using one of these newer models, you will not need to configure anchors.

ONE CANONICAL EXAMPLE A detector processes an image containing a person standing next to a bus. In a feature map location corresponding to both objects, the model considers multiple anchors. It will likely use a tall, thin anchor to predict the person and a very wide anchor to predict the bus. For these two anchors, it will output high confidence scores and precise coordinate adjustments, while the remaining anchors (e.g., small square ones) at that location will be classified as background.