Feature choice for real-time mobile SLAM

WHAT THIS TESTS The interviewer wants engineering judgment: choosing and defending a feature pipeline against the conflicting demands of accuracy, latency, memory, and power on an embedded device.

A GOOD ANSWER COVERS Visual SLAM must, every frame, detect features, describe them, match against the map or previous frame, and feed pose estimation, all within milliseconds and a small power budget. The pragmatic choice is ORB: the FAST detector finds corners very cheaply, the oriented BRIEF descriptor is a short binary string giving rotation invariance and a multi-scale pyramid giving partial scale invariance, and matching uses fast Hamming distance. ORB is also free of patent constraints, which matters for shipping products. The justification is that ORB hits the real-time and low-memory budget while keeping enough robustness for frame-to-frame tracking, which is exactly why it underpins widely used real-time SLAM systems.

COMMON WRONG ANSWERS Defaulting to SIFT for its accuracy without acknowledging it is too slow and memory-heavy to sustain the frame rate on a constrained device. Ignoring the matching metric, float descriptors need slower Euclidean distance. Forgetting that SLAM also relies on temporal continuity, which relaxes the per-frame robustness demand.

LIKELY FOLLOW-UPS How do you handle scale changes that ORB struggles with. How does loop closure change the descriptor requirements. Would a learned descriptor or hardware acceleration change your choice.

ONE CONCRETE EXAMPLE For an augmented-reality app on a phone that must track the camera at thirty frames per second while keeping battery drain low, you build the front end on ORB. Each frame you extract a few hundred FAST corners across a scale pyramid, compute their oriented binary descriptors, and match them to the local map using Hamming distance accelerated by bit operations, then estimate pose with RANSAC to reject outliers. This whole loop fits the millisecond budget on a mobile CPU. ORB's weaker robustness to very large viewpoint or scale jumps is mitigated because consecutive frames change only slightly, so matches stay reliable, and the pyramid handles moderate zoom. For occasional loop closure, where appearance can differ more, you can use the same ORB descriptors with a bag-of-words place-recognition database. Choosing SIFT instead would improve raw matching robustness but break the frame rate and drain the battery, which is unacceptable for the real-time, on-device requirement.