Mask R-CNN: Region-Based Detection

WHY IT EXISTS: Object detection demands two things at once: finding where things are in an image and naming what they are. R-CNN was invented to solve this jointly by taking an input image and producing a set of bounding boxes, where each box contains an object and its category. The architecture achieves this by leveraging the feature maps outputted by a CNN, applying selective search over those maps to identify regions worth turning into final predictions.

THE MENTAL MODEL: Think of the CNN as a cartographer that converts a raw image into a dense, information-rich map where similar textures and shapes cluster together. Selective search acts like a surveyor reading that map, drawing rough rectangles around likely objects without yet knowing their exact names. Only after these regions are isolated does the system assign a category to each box. The insight is that the heavy lifting of visual understanding happens once, in the shared feature map, while the search merely points to where the interesting structures live.

HOW IT WORKS: An image is fed through a convolutional neural network, which outputs a stack of feature maps capturing hierarchical visual patterns. Rather than scanning the raw image blindly, the R-CNN family performs selective search over these feature maps to generate candidate object locations. Each candidate becomes a bounding box that encloses an object and carries a predicted category. Because the search operates on convolutional features instead of pixels, the model reuses the same deep representations for both localization and classification.

WHEN TO USE IT: Reach for an R-CNN architecture when your task requires object detection and localization, specifically producing bounding boxes with category labels. It is appropriate whenever you already have a CNN backbone and need to extend it from image-level classification to region-level understanding. Any pipeline that benefits from selective search over learned feature maps to isolate discrete objects is a natural fit.

WHEN NOT TO USE IT: Do not use this family if your problem does not require bounding box outputs with object categories. The architecture is built for detection and localization via selective search over CNN feature maps, so tasks that lack these requirements are outside its scope. If your pipeline does not involve generating candidate regions from shared convolutional features, a different approach will serve you better.

ONE CANONICAL EXAMPLE: Imagine an autonomous driving perception stack processing a camera frame. The CNN backbone transforms the scene into feature maps highlighting edges, shapes, and textures. Selective search runs over those maps to propose bounding boxes around a cyclist, a traffic light, and three parked cars. The model then labels each box with its category, giving the planning system precise spatial and semantic information about every relevant object in view.