Deploying segmentation on edge devices

WHAT THIS TESTS The question probes whether you treat edge deployment as a system trade-off across architecture, numerics, and training, not a single magic switch. Phone cameras have tight memory, thermal, and latency budgets.

A GOOD ANSWER COVERS Architecture first: use efficient backbones with depthwise-separable convolutions such as MobileNet or efficient transformer-lite variants, and a lightweight decoder; segmentation also benefits from lower input resolution with learned upsampling. Quantization: convert FP32 weights and activations to INT8, preferring quantization-aware training when post-training quantization drops accuracy too far, since integer kernels are faster and smaller. Distillation: train the small student to match a large teacher using soft logits or feature alignment so it recovers accuracy lost from shrinking. Add structured pruning and operator fusion, then export to a mobile runtime like TFLite or Core ML.

COMMON WRONG ANSWERS Just lower the resolution, which alone destroys segmentation boundary quality. Or claim quantization is free, ignoring that activation outliers and per-channel scaling matter. Or distill without checking the teacher is actually better than the student baseline.

LIKELY FOLLOW-UPS Why is QAT often needed for segmentation versus classification. How do you handle quantizing the final softmax or argmax. What distillation loss works for dense prediction, and the answer involves per-pixel KL plus feature mimicking. How do you measure real on-device latency rather than FLOPs.

ONE CONCRETE EXAMPLE A heavy U-Net teacher with strong mIoU is too large for the phone. You design a MobileNetV3 encoder with a small decoder, distill it from the U-Net using per-pixel soft targets plus intermediate feature matching, then apply INT8 QAT so the student learns to tolerate rounding. The result runs in real time on the device with only a few points of mIoU lost and fits the memory budget. Crucially you profile actual on-device latency under sustained load, not just FLOPs, because thermal throttling and unsupported operators can erase the theoretical speedup.