AVFoundation: Apple's Toolkit for Audio and Video

WHY IT EXISTS Apple needed a powerful, unified framework for developers to work with time-based audiovisual media. Higher-level APIs like UIImagePickerController are too simple for custom needs like building a unique camera interface or performing non-trivial video editing, so AVFoundation provides the necessary granular control.

THE MENTAL MODEL Think of AVFoundation as a professional film crew in a box. You are the director. You manage an AVCaptureSession (the production), which coordinates AVCaptureDevices (your actors and equipment, like cameras and microphones), AVCaptureInputs (what the devices capture), and AVCaptureOutputs (where the final product goes, like a file or a live preview screen).

HOW IT WORKS The core of media capture is the AVCaptureSession. You configure it by adding inputs and outputs. For example, to build a camera, you find an AVCaptureDevice (the phone's back camera), create an AVCaptureDeviceInput with it, and add it to the session. Then you create an AVCaptureOutput, like a AVCaptureMovieFileOutput for saving a video, and add that too. Calling startRunning() on the session begins the flow of data from input to output. Playback and editing use different components like AVPlayer and AVComposition.

WHEN TO USE IT Use AVFoundation when you need deep control over media. This includes building a custom camera UI, recording video with specific resolutions or frame rates, trimming or merging video clips programmatically, playing multiple audio tracks simultaneously, or analyzing video frames in real-time for machine learning tasks.

WHEN NOT TO USE IT Avoid AVFoundation for simple tasks. If you just need to let the user pick a photo from their library or record a standard video, use the higher-level UIImagePickerController or the modern PhotosPicker. These APIs are far simpler and less prone to configuration errors.

ONE CANONICAL EXAMPLE A common use case is building a custom camera view. A developer would create an AVCaptureSession, add inputs for the back camera and microphone, and then add an AVCaptureVideoPreviewLayer (a type of output) to their view hierarchy. This displays the live camera feed. To enable recording, they would add an AVCaptureMovieFileOutput and trigger it to start and stop saving to a file based on user interaction.