Design real-time multi-object tracking for AV

WHAT THIS TESTS: It checks whether you can architect an online tracker that balances detection accuracy, identity stability across frames, and a tight per-frame time budget on embedded automotive hardware.

A GOOD ANSWER COVERS: Use tracking-by-detection. A single-stage detector such as a YOLO variant gives boxes per frame within the latency budget. Each confirmed track carries a Kalman filter with a constant-velocity state that predicts its next position, giving a motion prior even when detection is momentarily noisy. For data association, build a cost matrix combining geometric overlap, IoU or Mahalanobis distance to the predicted box, with an appearance similarity from a lightweight re-identification embedding, then solve the assignment with the Hungarian algorithm, exactly the SORT and DeepSORT recipe. Manage a track lifecycle: tentative tracks need several hits to confirm, matched tracks update the filter, and unmatched tracks coast on prediction for a few frames before deletion.

COMMON WRONG ANSWERS: Pure per-frame detection with greedy nearest-box matching and no motion state, which is fragile. Ignoring appearance features, so two crossing pedestrians swap IDs. Running a heavy two-stage detector that blows the latency budget and makes the whole system unusable in real time.

LIKELY FOLLOW-UPS: How does the appearance embedding reduce identity switches during crossings. What state does the Kalman filter actually hold. How do you tune the max-age before a coasting track dies. How would you fuse LiDAR or radar to improve depth and robustness in poor lighting.

ONE CONCRETE EXAMPLE: A pedestrian walks behind a parked van for half a second. Detection drops entirely, but the Kalman filter coasts the predicted box forward across the gap; when the person reappears, IoU with the prediction plus the cached appearance embedding re-associates them to the same track ID instead of spawning a brand-new one, which avoids an identity switch and keeps downstream prediction stable.