Non-Maximum Suppression: One Box Per Object

Why it exists

Object detection models are designed to be sensitive and often propose multiple bounding boxes for the same object, with slightly different positions and confidence scores. Without a cleanup step, you would get a messy, redundant output where a single car might be identified by five overlapping boxes. NMS solves this by reducing redundancy and selecting the single best box per object.

The mental model

Think of NMS as a competition where bounding boxes for the same object vie for the top spot. The box with the highest confidence score wins. All other boxes that are "too similar"—meaning they overlap the winner significantly—are eliminated. This process repeats until every box has either won its local competition or been eliminated.

How it works

The standard algorithm, called Greedy NMS, follows a simple procedure. First, it takes all proposed bounding boxes and sorts them from highest to lowest confidence score. Second, it selects the box with the highest score and adds it to the final list of detections. Third, it calculates the Intersection over Union (IoU)—a measure of overlap—between this winning box and all other remaining boxes. Any box with an IoU above a predefined threshold (e.g., 0.5) is suppressed. This process is repeated with the next-highest-scoring box that hasn't been suppressed, until no boxes are left. A variant called Soft NMS doesn't discard overlapping boxes but instead reduces their confidence scores, offering a more gradual suppression.

When to use it

NMS is a fundamental step in almost all modern object detection pipelines, such as YOLO, SSD, and Faster R-CNN. It is applied after the model has generated its initial set of raw predictions and before the final results are displayed or used. It is crucial for improving precision and reducing false positives.

When not to use it

The main weakness of Greedy NMS is when multiple distinct objects are very close together. If the bounding box for one object has a high confidence score, it might suppress the legitimate (but lower-confidence) box for a nearby object if their overlap exceeds the threshold. In these cases, variants like Soft NMS might be more appropriate as they penalize rather than eliminate overlapping boxes.

One canonical example

An object detector analyzing a photo of a crowded street might generate ten overlapping boxes for a single pedestrian. NMS takes these ten boxes, sorts them by confidence, and selects the one with the highest score (say, 98%). It then checks the other nine boxes. If any of them have an IoU greater than 0.5 with the 98% box, they are discarded. This leaves a single, clean detection for that pedestrian.