Semantic versus instance segmentation

What's really being asked

Whether you can crisply distinguish two related dense-prediction tasks, which determines which model family you would reach for.

The full answer

Semantic segmentation assigns a class label to every pixel in the image, partitioning it into regions like road, sky, person, and car. Critically, it does not distinguish between separate objects of the same class; all person pixels get the same person label regardless of how many people there are. Instance segmentation goes further: it labels pixels by class and also separates each distinct object instance, producing an individual mask for each one. So three people would be one merged region under semantic segmentation but three separate masks under instance segmentation. The fundamental difference is instance awareness, whether objects of the same category are kept distinct. Panoptic segmentation unifies both, giving stuff classes semantic labels and thing classes instance masks.

The mistakes people make

Saying semantic segmentation outputs boxes; it outputs pixel labels. Thinking semantic segmentation can count objects of a class; it cannot separate them. Confusing instance segmentation with object detection, which gives boxes not pixel-precise masks. Forgetting panoptic segmentation as the combined view.

What usually comes next

What model architectures suit each, for example FCN or U-Net versus Mask R-CNN. What is panoptic segmentation. How is instance segmentation evaluated. Why is instance segmentation harder.

A concrete example

Picture a photo of three overlapping cows in a field. Semantic segmentation colors every cow pixel with the single label cow and the rest as grass and sky, so you cannot tell there are three animals. Instance segmentation produces three distinct cow masks, cow one, cow two, cow three, each a separate region, so you can count and isolate each animal. That instance-level separation is the defining difference between the two tasks.