How do you monitor thousands of per-customer models as a fleet?

WHAT THIS TESTS: This tests whether you understand the difference between monitoring a single model and operating a high-cardinality fleet. At thousands of models, per-instance dashboards and alerts create operator fatigue and miss systemic patterns. The interviewer wants to see statistical thinking, tiered observability, and architectural patterns that scale sub-linearly with fleet size.

A GOOD ANSWER COVERS: First, cohort-based aggregation. Group models by schema, data domain, or customer tier and track distribution-level metrics like prediction entropy, feature covariance drift, and output histogram divergence rather than inspecting each model. Second, automated fleet anomaly detection. Use a meta-model or statistical process control to baseline each model against its peer cohort so outliers surface without hand-tuned thresholds per customer. Third, tiered telemetry. Separate infrastructure signals like latency and error rate from ML-specific signals like data drift and concept drift, using different retention and sampling policies. Fourth, hierarchical alerting. Fleet-wide degradation should page on-call immediately while single-model anomalies batch into daily digests unless the customer is tier one. Fifth, cost-aware sampling. Not every prediction needs full logging; use reservoir sampling or trigger-based capture to keep telemetry costs flat as the fleet grows.

COMMON WRONG ANSWERS: A red flag is proposing a Grafana dashboard per model with manual threshold setting. That does not scale past a few dozen customers. Another red flag is relying on a single global accuracy metric, which hides silent failures in small or non-stationary cohorts. Saying you will retrain all models on the same schedule also signals a lack of fleet-level operational maturity.

LIKELY FOLLOW-UPS: The interviewer may ask how you handle cold-start customers with no baseline, how you sample labels for ground-truth delayed feedback, or how you keep monitoring storage costs flat when prediction volume grows tenfold. They may also probe how you distinguish a true model failure from a customer-specific business event like a holiday sale.

ONE CONCRETE EXAMPLE: Imagine a fraud model per merchant. Instead of alerting on each merchant's daily fraud rate, you compute a Z-score of each merchant's score distribution against their vertical cohort like food delivery versus luxury retail. You emit a fleet health vector every five minutes showing what percentage of cohorts are drifting. If that percentage crosses ten percent, you trigger an investigation. Individual merchants only alert if they are statistical outliers and their transaction volume exceeds a tier threshold.

Source: MLOps Architecture: End-to-End Design for Production-Grade ML and LLM Systems, DEV Community