Architectural challenges for deploying ML models on resource-constrained edge devices

WHAT THIS TESTS: Whether you can architect an end-to-end Edge MLOps system that respects hard constraints on power, compute, storage, and connectivity while still enabling continuous model improvement. Interviewers want to see you distinguish cloud MLOps from edge MLOps and understand the full lifecycle from field deployment through drift detection and retraining.

A GOOD ANSWER COVERS: First, model size reduction via quantization, pruning, and hardware-aware compilation so the model fits into limited memory and runs efficiently on an embedded accelerator or CPU. Second, resilient deployment patterns including delta over-the-air updates, A-B partitioning for atomic rollback, and update scheduling that respects battery state and intermittent connectivity. Third, power management through adaptive inference pipelines, dynamic frequency scaling, and offloading only when energy budgets allow. Fourth, a closed-loop data pipeline that handles unreliable networks by filtering and compressing edge-captured data locally, using observability large models in the cloud or edge hub to compare against edge inference results, detecting drift and false negatives, and routing only high-value samples for annotation and retraining rather than streaming raw media.

COMMON WRONG ANSWERS: Treating edge devices as tiny cloud servers by proposing full model redeployments on every update, ignoring the cost of bandwidth and flash wear. Suggesting continuous cloud inference for all edge data, which defeats latency and privacy goals. Failing to mention on-device validation gates that prevent a bad update from bricking thousands of remote cameras. Proposing synchronous training pipelines that assume stable backhaul.

LIKELY FOLLOW-UPS: How would you handle a fleet of ten thousand cameras where a new model causes a thermal shutdown on thirty percent of devices? What is your strategy when a device has been offline for weeks and misses multiple model versions? How do you balance privacy requirements against the need for annotated training data from the field?

ONE CONCRETE EXAMPLE: A smart camera running person detection on a battery-powered outdoor unit. The device uses an INT8 quantized model with a runtime memory footprint under fifty megabytes. Each night, when solar charging peaks, the camera uploads metadata and low-confidence thumbnails rather than full video. In the cloud, an observability large model reviews these samples against edge predictions to find false negatives caused by seasonal clothing changes. Annotated samples are augmented and used to fine-tune a smaller model, which is then pruned and quantized before being packaged as a delta update. The device downloads the delta only when Wi-Fi is available and battery exceeds fifty percent, validates the new model against a local golden dataset, and swaps the A-B partition only if accuracy passes the gate.

Source: SiMa.ai blog, Edge MLOps: Architecture, Challenges, and Platform, September 2024.