More in AI & ML — page 52
Cross-Attention: How Models Fuse Text and Images
Cross-attention lets a model fuse different data streams, like asking 'what in this image corresponds to this word?'. It's key for text-to-image generation, where text queries attend to image features. The footgun is confusing it with self-attention.
Swin Transformer: Efficient Vision with Shifted Windows
Swin Transformer makes Vision Transformers practical by processing images in local "windows" instead of all at once. It's a powerful backbone for object detection and segmentation where scale varies.
DETR: Object Detection as Direct Set Prediction
DETR reframes object detection from a filtering task to direct set prediction. It uses a Transformer to output a fixed set of object predictions in one pass, eliminating complex post-processing.
Multi-Head Attention: Seeing Data From Multiple Angles
Multi-head attention lets a model analyze a sequence from multiple perspectives at once. It runs several "attention heads" in parallel, each focusing on different relationships, like syntax vs. semantics.
Patch Embedding: Turning Images into Words for Transformers
Patch embedding chops an image into a grid of squares, turning each into a vector. This lets sequence-based models like Transformers "read" images. It's the core of Vision Transformers (ViTs), but it discards the fine-grained detail inside each patch.
Generative Adversarial Networks (GANs): A Forger and a Detective
A GAN pits two neural networks against each other: a Generator that creates fakes and a Discriminator that spots them. This adversarial game forces the Generator to produce highly realistic outputs, like photorealistic faces. The main footgun is mode collapse.
Attention in Vision: Teaching Models Where to Look
Attention teaches a model where to look in an image by dynamically weighting important pixels or features. It's used in object detection to focus on relevant regions. The footgun is assuming it's free; attention adds computational cost and complexity.
Temporal Segment Networks: Seeing the Whole Video Story
Temporal Segment Networks (TSN) understand video actions by sampling sparse snippets across the entire timeline. This gives a model long-range context to distinguish complex actions.
DeepSORT: Adding Visual Memory to Object Tracking
DeepSORT adds a 'visual memory' to object tracking, using a deep learning model to re-identify objects after they're hidden. It's used in surveillance and autonomous driving to maintain consistent IDs across frames.
Particle Filters for Object Tracking
A particle filter tracks an object by maintaining a cloud of possible states ('particles'), not one best guess. It's used to follow objects through clutter and occlusion, like tracking a face in a crowd.
Correlation Filters: Fast and Robust Object Tracking
Correlation filters track objects by learning a template that gives a peak response at the object's location. This makes them extremely fast for real-time visual tracking. The footgun: basic versions fail when the object changes scale or is occluded.
Two-Stream ConvNets: Seeing What vs. How It Moves
Two-Stream ConvNets split video analysis into two paths: a spatial stream sees *what* is in a frame, and a temporal stream sees *how* it moves via optical flow. This is used for action recognition, like telling 'typing' from 'waving'.
Mean Shift: A Mode-Seeking Algorithm
Mean shift is a technique for finding the peaks, or modes, of a density function. It's a non-parametric method used for cluster analysis in domains like computer vision and image processing to locate the densest areas in a feature space.
Kalman Filters: Predicting Through Noise
A Kalman filter predicts an object's next position, then corrects that prediction with noisy real-world measurements. It's used in video tracking to smooth an object's path between frames or in robotics to fuse imperfect sensor data.
Horn-Schunck: Assuming Smooth Motion to See Clearly
The Horn-Schunck method estimates video motion by assuming neighboring pixels move similarly. This global 'smoothness' constraint solves the aperture problem, where local views give ambiguous motion data.
Lucas-Kanade Method: Tracking Pixel Patches, Not Points
The Lucas-Kanade method tracks motion by assuming a small patch of pixels moves as a single unit. This overcomes the ambiguity of tracking one pixel. It's fundamental to video stabilization and feature tracking.
Action Recognition: Teaching Machines to Understand 'Doing'
Action recognition teaches computers to understand *what* is happening in a video by analyzing motion over time. It's used for fall detection in healthcare, sports analysis, and security. The footgun is confusing it with single-frame object detection.
Optical Flow: Tracking Motion by Watching Pixels
Optical flow tracks motion by watching how brightness patterns move between video frames. It's used for video compression and robot navigation. The main footgun is that it can mistake lighting changes, like a shadow moving, for actual object motion.
Background Subtraction: Finding What's Moving in Video
Think of a security camera that has memorized the empty room. Background subtraction digitally 'subtracts' this static view from the live feed, leaving only moving objects. It’s used to spot intruders or count cars, but fails if the background itself changes.
Frame Differencing: The Simplest Way to See Motion
See motion by subtracting one video frame from the next; what's left over is what moved. This is used for simple motion detection but is easily fooled by lighting changes or camera shake, which it misinterprets as movement.