Challenges deploying a model on edge hardware
edge deployment constraints.
limited memory and compute cause latency, thermal and power limits, accuracy loss from compression, operator support gaps.
only mentioning accuracy while ignoring latency, memory, and power.
WHAT THIS TESTS The interviewer checks whether you understand deployment as a multi-constraint engineering problem, since a model that is excellent on a server can be unusable on a Jetson Nano.
A GOOD ANSWER COVERS Memory: the device has limited RAM, so large weights and activation buffers may not fit, forcing smaller models or tiling. Compute and latency: far fewer FLOPS than a server GPU mean inference is slow, threatening real-time targets and throughput. Power and thermal: edge devices run on tight power budgets and throttle clocks when hot, so sustained performance differs from a quick benchmark. Accuracy trade-offs: to fit and speed up, you apply quantization to INT8, pruning, or distillation, each risking accuracy loss that must be validated. Software and tooling: the runtime such as TensorRT may not support every operator or precision, requiring model surgery, layer fusion, or fallback kernels.
COMMON WRONG ANSWERS Saying the only issue is a small accuracy drop, ignoring latency, memory, power, and thermal throttling. Or assuming FLOPs alone predict latency, when memory bandwidth and operator support dominate. Or believing the trained model runs as-is without conversion.
LIKELY FOLLOW-UPS How do you measure real on-device latency including thermal throttling. Why can FLOPs mislead. How do you handle unsupported operators. What compression order do you apply and how do you validate accuracy afterward.
ONE CONCRETE EXAMPLE A ResNet-152 that runs in tens of milliseconds on a server GPU can take many times longer on a Jetson Nano, exceeding a real-time budget and nearly exhausting the limited RAM. You convert the graph to TensorRT with INT8, swap to a lighter backbone, and fuse operators to cut memory traffic. Re-profiling on the device itself, you find sustained throughput drops once the chip heats up and throttles, and that one custom layer falls back to a slow CPU kernel because the runtime lacks support. Those measured findings, not server FLOPs, guide where you trim next.
Read the original → cvpr.thecvf.com
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.