Designing a multi-object tracker

WHAT THIS TESTS The interviewer wants a coherent system design that links detection, motion prediction, and association, plus explicit handling of the hard cases that break naive trackers.

A GOOD ANSWER COVERS The dominant paradigm is tracking-by-detection. Components: a per-frame object detector produces boxes; a motion model, typically a Kalman filter, predicts each existing track's next position; a data association step matches detections to predicted tracks using the Hungarian algorithm with a cost combining IoU overlap and appearance embedding distance, as in DeepSORT. A track manager handles lifecycle: initialize new tracks for unmatched detections after confirmation, and delete tracks unseen for several frames. Occlusion handling: keep occluded tracks in a coasting state predicted by the motion model, and when the object reappears use appearance re-identification to reattach the same ID rather than spawning a new one. This directly reduces ID switches, the metric where one identity is wrongly relabeled.

COMMON WRONG ANSWERS Matching purely by IoU with no appearance cue, which fails badly under crossing paths and occlusion. Or detecting independently each frame with no temporal association at all. Or deleting tracks the instant a detection is missed, causing fragmented IDs.

LIKELY FOLLOW-UPS Why add appearance embeddings on top of IoU. How does the Kalman filter help during occlusion. What metrics evaluate tracking, such as MOTA and IDF1. How do crossing trajectories cause ID switches and how to mitigate them.

ONE CONCRETE EXAMPLE Two pedestrians cross paths in a crowded scene. A pure-IoU tracker, relying only on box overlap, swaps their IDs at the moment their boxes coincide, producing an ID switch. With DeepSORT, the Kalman filter predicts each person's motion through the brief occlusion and appearance embeddings encode what each person looks like, so when they separate the matcher reattaches the correct identities by appearance rather than by overlap alone. The track manager keeps the temporarily occluded track in a coasting state instead of deleting it, which is what lets the original ID be recovered on reappearance.