DeepSORT: Adding Visual Memory to Object Tracking

Why it exists

Basic object trackers like SORT struggle when objects are occluded or move unpredictably. They rely on motion prediction and bounding box overlap to associate objects between frames. If an object disappears and reappears later, a simple tracker often assigns it a new ID, breaking the track's continuity. DeepSORT was created to solve this identity-switching problem by adding appearance information.

The mental model

Think of DeepSORT as a security guard watching a crowd. The guard uses motion to follow people (like SORT's Kalman filter). But if someone walks behind a pillar and re-emerges, the guard doesn't just rely on where they should be. They also use visual memory of the person's appearance—their coat, their hair—to confirm it's the same individual. DeepSORT's 'deep association metric' is that visual memory, powered by a neural network.

How it works

DeepSORT is a two-stage process. First, like its predecessor SORT, it uses a Kalman filter to predict the future position of existing tracks and matches them with new detections from an object detector. This matching is based on motion and spatial proximity. Second, for any tracks and detections that remain unmatched, DeepSORT calculates an 'appearance descriptor' (a feature vector) using a pre-trained convolutional neural network. It then attempts to match objects based on the similarity of their appearance features. This second check allows it to bridge long occlusions.

When to use it

Use DeepSORT when you need robust, long-term tracking in scenarios with frequent occlusions or complex movements. It is a standard for multi-object tracking (MOT) applications like pedestrian tracking for surveillance, vehicle tracking for traffic analysis, and sports analytics where players frequently cross paths.

When not to use it

Avoid DeepSORT if you need extreme real-time performance on low-power hardware, as the neural network adds computational overhead compared to SORT. If all your tracked objects look nearly identical, the appearance feature may not be discriminative enough to prevent identity switches. Most importantly, it is not a detector; you must pair it with an upstream object detector like YOLO or Faster R-CNN.

One canonical example

A system tracks pedestrians on a busy street. An object detector provides bounding boxes for all people in each video frame. DeepSORT takes these detections and assigns a stable ID to each person. When a person is temporarily hidden by a bus, the simple motion prediction fails. However, when they reappear, DeepSORT's CNN recognizes their appearance and re-associates them with their original track ID instead of creating a new one.