Design a tracking-by-detection tracker

WHAT THIS TESTS Whether you recognize that an object detector outputs independent boxes per frame with no identity, so tracking is fundamentally a data-association problem across time.

A GOOD ANSWER COVERS The pipeline is: on each frame run the detector to get boxes, then associate those detections with the set of existing tracks. Association needs a similarity measure, commonly intersection-over-union between predicted and detected boxes, optionally augmented by an appearance embedding so visually similar objects match even after a gap. The matching itself is an assignment problem solved by the Hungarian algorithm to get the optimal one-to-one pairing. A motion model, often a Kalman filter, predicts where each track should be in the next frame, making IoU matching robust to fast motion. Unmatched detections spawn new tracks; tracks unmatched for several frames are deleted. This is essentially the SORT algorithm.

COMMON WRONG ANSWERS Assuming the detector returns consistent ids, so no association is needed. Matching purely by IoU and ignoring appearance, which fails under occlusion. Never deleting stale tracks, causing ghost tracks to accumulate.

LIKELY FOLLOW-UPS How does the Hungarian algorithm guarantee optimal assignment? What does adding a Kalman filter contribute? How does DeepSORT use appearance features? How do you tune the track deletion threshold?

ONE CONCRETE EXAMPLE Two pedestrians walking, the detector finds two boxes each frame; you predict each track forward with a Kalman filter, build an IoU cost matrix between predictions and detections, run the Hungarian algorithm to assign, and keep ids 1 and 2 stable, until they cross and overlap, where IoU alone may swap their ids, an identity switch.

The main failure modes are occlusion losing a track, identity switches when objects cross, missed detections fragmenting a track, and dense crowds overwhelming association.