Instance Segmentation: Counting and Outlining Objects

Why it exists

Computer vision often needs to understand not just what objects are in an image, but where they are and how many there are. Object detection draws a rough bounding box, while semantic segmentation labels pixels by category, like 'car' or 'person'. Instance segmentation solves the problem of needing to know the precise shape and count of each individual object, separating one car from another.

The mental model

Imagine coloring a picture of a crowd. Semantic segmentation is like using one color for every person. Instance segmentation is like using a different color for each individual person in the crowd. It combines the "what" (it's a person) and "where" (these are the pixels) with the "which one" (this is person A, and that is person B).

How it works

Modern instance segmentation models, like Mask R-CNN, typically follow a two-stage process. First, an object detection network proposes regions of interest (ROIs) where it thinks objects are located, drawing a bounding box around each. Second, for each proposed box, a separate, smaller network runs to predict a pixel-level mask. This "mask head" effectively colors in the pixels within the box that belong to the object, separating it from the background. The final output is a set of pairs: a class label, a bounding box, and a unique pixel mask for each object instance detected.

When to use it

Use instance segmentation when you need to count, measure, or track individual objects with high precision. This is critical in autonomous driving for tracking each vehicle and pedestrian, in medical imaging for measuring distinct tumors or cells, in retail for automated inventory counts, and in robotics for grasping specific items from a cluttered bin.

When not to use it

Avoid instance segmentation if a simpler representation is sufficient. If you only need to know that a car is present and roughly where, object detection is faster. If you only need to identify amorphous regions like 'sky' or 'road', semantic segmentation is more appropriate. Instance segmentation is computationally expensive, so don't pay the cost if you don't need to distinguish between individual objects of the same class.

One canonical example

A robotic warehouse system needs to pick a specific product from a bin full of identical items. A simple object detector would draw boxes around all of them. Semantic segmentation would label all of them as 'product'. Instance segmentation provides a distinct mask for each item, allowing the robot to plan a grasping motion for one specific instance without colliding with the others.