Vision Framework: Computer Vision on Apple Devices

WHY IT EXISTS: Before Vision, implementing features like face detection or text recognition required deep expertise in image processing and machine learning, or reliance on third-party libraries. Apple created the Vision framework to provide a standardized, high-performance, on-device solution optimized for its hardware.

THE MENTAL MODEL: Think of Vision as a team of expert analysts for images and video. You don't teach them their job; you just hand them an image (a CIImage or CVPixelBuffer) and a task (a VNRequest). For example, you give them a photo and ask them to find all the faces. They do the complex work and return a simple list of observations (VNObservation), like the locations of the faces they found. It's an abstraction layer over complex Core ML models.

HOW IT WORKS: The workflow has three steps. First, you define what you're looking for by creating one or more request objects, like a VNDetectFaceRectanglesRequest or a VNRecognizeTextRequest. Second, you create a request handler, either a VNImageRequestHandler for a single image or a VNSequenceRequestHandler for a video stream. Third, you execute the requests using the handler. The framework processes the image and returns an array of observation objects, which you then interpret to get your results, like bounding boxes or recognized text strings.

WHEN TO USE IT: Use Vision for common, built-in computer vision tasks. This includes detecting faces and facial landmarks (eyes, nose), recognizing text (OCR), tracking objects across multiple video frames, finding barcodes and QR codes, and identifying rectangular objects like documents. It's the default choice for adding these features to an iOS, iPadOS, or macOS app.

WHEN NOT TO USE IT: Vision is not for identifying custom objects out of the box. If you need to detect a specific dog breed or car model, you must first train a custom Core ML model; Vision can then help you run that model, but it doesn't provide the classification logic itself. Avoid running intensive Vision requests on every single frame of a high-framerate video feed without throttling, as this will cause severe performance issues and battery drain.

ONE CANONICAL EXAMPLE: Building a document scanner. Your app uses the live camera feed. You create a VNDetectRectanglesRequest and run it on frames from the camera. Vision returns a VNCoreMLRequest with the four corner points of the largest detected rectangle (the document). Your UI can then draw an overlay using these points to guide the user. After the user captures a photo, you use the same corner points to apply a perspective correction filter, creating a perfectly flat, scanned image.