Semantic, instance, and panoptic segmentation

What's really being asked

Whether you understand the precise output of each segmentation paradigm and can pick the right one for a task.

The full answer

Semantic segmentation classifies every pixel into a category, producing a label map; all cars share the car label with no notion of how many cars there are, and it handles amorphous stuff like road or sky naturally. Instance segmentation detects each distinct object and outputs a separate mask plus class per object, so two adjacent cars get two masks; it focuses on countable thing classes and traditionally does not label background stuff, and instances can overlap. Panoptic segmentation unifies the two: every pixel gets a class, stuff regions get a single semantic label, and thing classes additionally get a unique instance identifier, yielding a complete, non-overlapping scene parse with both background and counted objects. Choose semantic segmentation when you only need region categories, such as drivable area; instance segmentation when you must count or track individual objects, such as separate pedestrians; and panoptic when you need a full scene understanding that names every pixel and still distinguishes individual objects, as in autonomous driving perception.

The mistakes people make

Saying semantic segmentation distinguishes separate instances; claiming instance segmentation labels every background pixel; treating panoptic as just running the other two without resolving overlaps into one consistent map; confusing segmentation with bounding-box detection; thinking panoptic permits overlapping masks.

What usually comes next

How does the panoptic quality metric work? What is the thing-versus-stuff distinction? How do mask prediction architectures differ across the three? How would you handle occlusion and overlapping instances in a panoptic output?

A concrete example

For a self-driving scene with a road, two pedestrians, and three cars, semantic segmentation outputs pixels labeled road, person, car with no counts. Instance segmentation outputs five separate object masks, two persons and three cars, but may leave the road unlabeled. Panoptic segmentation labels road and sky as single stuff regions and gives each of the two persons and three cars its own instance id, producing one coherent map covering every pixel.