Deploying real-time detection on edge devices

WHAT THIS TESTS Whether you can balance accuracy, latency, memory, and power on real hardware rather than just naming a model. This is a systems question.

A GOOD ANSWER COVERS Start from the constraints: define the latency budget in frames per second, the memory ceiling, and the minimum acceptable accuracy. For model selection, prefer an efficient single-stage detector, such as a compact YOLO variant or SSD with a MobileNet or EfficientNet-lite backbone, since two-stage detectors are usually too slow for edge. Use depthwise separable convolutions and choose input resolution deliberately, as it strongly affects both speed and accuracy. For training, apply heavy augmentation, transfer learning from pretrained weights, and consider knowledge distillation so a large teacher transfers accuracy to the small student. For deployment optimization, quantize to int8 with post-training or quantization-aware training, prune redundant channels, fuse convolution-batchnorm-activation, and compile to a hardware-accelerated runtime like TensorRT, LiteRT, or a vendor NPU SDK. Finally, profile actual on-device latency and memory, not theoretical FLOPs, and tune the speed-accuracy tradeoff to the budget.

COMMON WRONG ANSWERS Picking a heavy two-stage detector for real time. Optimizing only mAP and ignoring the latency and memory budget. Assuming FLOPs predict on-device speed; memory bandwidth and operator support matter more. Skipping quantization or not measuring on the real device.

LIKELY FOLLOW-UPS Why does int8 quantization speed inference and what accuracy risk does it carry. When use quantization-aware training over post-training. How does distillation help small models. How does input resolution trade speed for accuracy.

ONE CONCRETE EXAMPLE To run thirty frames per second on a mobile NPU, you select YOLO-nano with a backbone of depthwise separable convolutions at a three-twenty by three-twenty input. You train with mosaic augmentation and distill from a larger YOLO teacher, then apply quantization-aware training to reach int8, prune low-magnitude channels, and export to LiteRT with NNAPI delegation. Profiling on the device shows latency dropping from sixty milliseconds in float to twenty in int8 with only a one to two point mAP loss, comfortably hitting the real-time target within the memory budget.