Image Thresholding: Separating Foreground from Background

WHY IT EXISTS Computers analyze images most easily when the subject is clearly separated from the background. A grayscale image with millions of shades is complex. Thresholding is a fundamental technique to simplify an image into its most essential parts—foreground and background—making it easier for other algorithms to process.

THE MENTAL MODEL Think of a grayscale image as a 3D terrain map where pixel intensity is altitude. Thresholding is like flooding this terrain up to a specific water level (the threshold value). Everything above the water is classified as one thing (e.g., white 'foreground'), and everything submerged is another (e.g., black 'background').

HOW IT WORKS Thresholding operates on a grayscale image. The simplest method is Global Thresholding: you pick one value for the entire image, say 127 on a 0-255 scale. Any pixel with a value greater than 127 is set to 255 (max value, or white), and any pixel with a value less than or equal to 127 is set to 0 (black). This creates a binary image.

However, a single threshold fails if the image has different lighting conditions. Adaptive Thresholding solves this. Instead of a single global value, it calculates a different threshold for each pixel based on a small region of neighboring pixels. This can be the mean brightness of the neighborhood (ADAPTIVE_THRESH_MEAN_C) or a weighted sum (ADAPTIVE_THRESH_GAUSSIAN_C), allowing it to adapt to local lighting changes like shadows.

WHEN TO USE IT Use simple (global) thresholding for high-contrast images with very uniform lighting, like a cleanly scanned document. Use adaptive thresholding for almost everything else, especially images with gradients, shadows, or uneven illumination, like a photo of a page taken with a phone.

WHEN NOT TO USE IT Do not use simple thresholding on images with varied lighting, as it will either lose foreground details in dark areas or include background noise in bright areas. Thresholding is also a poor choice when color is the key differentiator between objects, as the conversion to grayscale discards this information.

ONE CANONICAL EXAMPLE To find the numbers on a sudoku puzzle in a photo, a global threshold of 127 would fail. A shadow might make the white paper in one corner darker than 127, while bright light makes the black ink in another corner lighter than 127. An adaptive threshold, however, calculates a local threshold for each part of the image, successfully isolating the printed numbers from the paper background everywhere, regardless of shadows.