Generative Inpainting: Filling in the Blanks with AI

WHY IT EXISTS Images often have imperfections we want to fix. This could be physical damage to an old photograph, an unwanted object in a picture, or simply the desire to extend an image beyond its original borders. Inpainting was developed to solve this by filling in missing or unwanted areas to present a complete, clean image.

THE MENTAL MODEL The concept originates from art conservation, where a restorer meticulously fills in damaged parts of a painting. In generative AI, the model acts as an automated restorer. It doesn't just smudge or clone nearby pixels; it analyzes the surrounding image context—and often a text prompt—to generate entirely new, semantically consistent pixels to fill the gap.

HOW IT WORKS A user provides an image and a 'mask,' which is a layer that specifies the exact region to be replaced. The generative model sees the unmasked parts of the image as a condition. It then generates new image data for the masked area that it predicts would plausibly exist there, based on the billions of images it was trained on. This process is a form of conditional image generation.

WHEN TO USE IT Use inpainting for tasks where a plausible fill is sufficient. Three common uses are: first, object removal, like deleting a person from a background; second, image restoration, like fixing scratches or tears in a scanned photo; and third, creative extension (often called 'outpainting'), where you expand the canvas and have the AI generate the new areas.

WHEN NOT TO USE IT Do not use inpainting when you need to recover the exact, original data. The AI is fabricating pixels, not uncovering what was truly there, making it unsuitable for forensic analysis. It can also struggle with complex, non-repeating textures or preserving fine details like text, often producing blurry or nonsensical results in those areas.

ONE CANONICAL EXAMPLE A user wants to remove a distracting logo from a t-shirt in a portrait. They would use an editing tool to draw a mask over the logo. The inpainting model would analyze the fabric's color, texture, and the way light hits the shirt to generate a new piece of fabric that seamlessly replaces the logo, making it appear as if it was never there.