Design real-time multi-object tracking for AV
tracking-by-detection under latency.
fast detector, Kalman motion model, Hungarian association on IoU plus appearance, track lifecycle for occlusions.
per-frame detection with no temporal state or ID management.
What's really being asked
It checks whether you can architect an online tracker that balances detection accuracy, identity stability across frames, and a tight per-frame time budget on embedded automotive hardware.
The full answer
Use tracking-by-detection. A single-stage detector such as a YOLO variant gives boxes per frame within the latency budget. Each confirmed track carries a Kalman filter with a constant-velocity state that predicts its next position, giving a motion prior even when detection is momentarily noisy. For data association, build a cost matrix combining geometric overlap, IoU or Mahalanobis distance to the predicted box, with an appearance similarity from a lightweight re-identification embedding, then solve the assignment with the Hungarian algorithm, exactly the SORT and DeepSORT recipe. Manage a track lifecycle: tentative tracks need several hits to confirm, matched tracks update the filter, and unmatched tracks coast on prediction for a few frames before deletion.
The mistakes people make
Pure per-frame detection with greedy nearest-box matching and no motion state, which is fragile. Ignoring appearance features, so two crossing pedestrians swap IDs. Running a heavy two-stage detector that blows the latency budget and makes the whole system unusable in real time.
What usually comes next
How does the appearance embedding reduce identity switches during crossings. What state does the Kalman filter actually hold. How do you tune the max-age before a coasting track dies. How would you fuse LiDAR or radar to improve depth and robustness in poor lighting.
A concrete example
A pedestrian walks behind a parked van for half a second. Detection drops entirely, but the Kalman filter coasts the predicted box forward across the gap; when the person reappears, IoU with the prediction plus the cached appearance embedding re-associates them to the same track ID instead of spawning a brand-new one, which avoids an identity switch and keeps downstream prediction stable.
Interview question
In a DeepSORT-style tracker, what primarily prevents two pedestrians who cross paths from swapping track identities?
- a.Increasing the detector's confidence threshold
- b.Running detection at a higher frame rate only
- c.Using a larger non-max-suppression IoU threshold
- d.Combining motion prediction with an appearance embedding in associationCorrect
Why? this is the answer
Appearance embeddings distinguish individuals that geometric overlap alone would confuse during a crossing, so combining them with motion fixes ID swaps. Higher frame rate or NMS tuning does not encode who is who.
Just read this? Test yourself on what you have been reading.
- #object-tracking
- #system-design
- #kalman-filter
- #autonomous-vehicles
- #data-association
Put your scrolling time to good use
Learn one idea, try a quiz and save useful cards for revision. Tezvyn makes it easy to learn and stay current in your tech field, a few minutes at a time.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles