Intersection over Union (IoU): How Good is Your Bounding Box?

Why it exists

In computer vision, knowing an object is a 'car' isn't enough; we need to know where it is. Simple classification accuracy doesn't measure spatial precision. Intersection over Union (IoU) was created to quantify how well a predicted bounding box aligns with the actual, human-labeled ground truth box.

The mental model

Think of IoU as a Venn diagram for two rectangles: a predicted box and a ground truth box. IoU is the ratio of their overlapping area (the intersection) to the total area they both cover (the union). The resulting score is always between 0 (no overlap) and 1 (a perfect, pixel-for-pixel match).

How it works

The formula is simple: Area of Intersection / Area of Union. This ratio is scale-invariant, meaning it assesses performance fairly whether the object is a massive building or a tiny insect. During evaluation, engineers set an IoU threshold, commonly 0.5. If a prediction's IoU score is above this threshold, it's a 'True Positive'; if not, it's a 'False Positive'. This decision is a prerequisite for calculating aggregate metrics like Mean Average Precision (mAP).

When to use it

IoU is critical in three main areas. First, for model evaluation: it's the standard for scoring object detection accuracy. Second, during model training: advanced loss functions like GIoU and CIoU use this principle to help the model learn to place boxes better. Third, during inference: a technique called Non-Maximum Suppression (NMS) uses IoU to find and remove duplicate, overlapping detections of the same object, cleaning up the final output.

When not to use it

IoU measures spatial overlap, not classification correctness. A high IoU score is meaningless if the box is labeled incorrectly (e.g., a box around a dog is labeled 'cat'). Always use IoU in conjunction with classification metrics. It is not applicable for tasks without spatial localization, like pure image classification.

One canonical example

A ground truth box for a person is at coordinates [100, 100, 200, 200]. A model predicts a slightly offset box at [110, 110, 210, 210]. The area where they overlap is 8100 pixels (90x90). The total area covered by both is 11900 pixels. The IoU is 8100 / 11900, or approximately 0.68. If the evaluation threshold was 0.5, this would count as a correct detection.