Mlops
265 bites tagged Mlops — interview questions with model answers, and 60-second explainers.
How would you architect dynamic training resource provisioning?
Routes jobs to right-sized instances, provisions managed jobs using spot, and auto-tears down on completion. elastic ML infrastructure and cost control. suggesting always-on clusters or serverless GPU training.
Design concept drift detection with automated retraining safeguards
Baseline monitors raise CloudWatch alarms; EventBridge triggers retraining with cooldowns; model registry gates promotion. MLOps design separating drift detection, triggers, and stability controls.
How do you programmatically promote a retrained model to production?
Compare on held-out data using significant metric uplift, schema, latency, and drift checks before shadow release. Gated promotion balancing statistics and safety. Using training accuracy without variance checks.
How would you implement automated data validation before training?
Enforce schema contracts, halt training on failure, quarantine bad batches, and alert owners. Pipeline gatekeeping and failure isolation in production ML. Manual reviews or soft warnings letting bad data into training.
What triggers automatic full retraining in an ML pipeline?
A strong answer lists four triggers: fresh data, code changes, model drift, and scheduled cadence. Your grasp of data, code, model, and schedule-driven automation in MLOps.
How do you monitor thousands of per-customer models as a fleet?
Tests fleet-level statistical aggregation versus per-instance alerting. Strong answers propose tiered telemetry, cohort baselining for drift, and hierarchical alerting to prevent fatigue.
Design an automated system to diagnose model performance drop root causes
Tests causal attribution between pipeline bugs and drift. Strong answers sequence schema/null audits, feature drift via PSI/KS, then concept drift via holdout decay. Red flag: skipping pipeline checks to retrain immediately.
Model output distribution shifts. What are root causes and next steps?
This tests covariate vs label shift vs concept drift when outputs shift. A strong answer checks features before labels, then feedback loops or staleness. A red flag is generic drift without separating P(X), P(Y), and P(Y|X).
Design a system to monitor a real-time prediction service for feature drift
Async feature logging, distribution comparison via PSI/KS against training baseline, and threshold-based anomaly alerts. production ML observability beyond accuracy checks.
Differences between monitoring a traditional REST API and a production ML model
Contrast latency/errors with ML signals like data drift and training-serving skew against baselines, noting ground truth delays. Awareness that ML fails via data decay, not code bugs.
Design a multi-model serving architecture for infrequently used models
Tests sparse-traffic cost efficiency via shared compute and dynamic loading. Strong answers: LRU cache on fast storage, scale-to-zero with async preload, pinned hot models, per-model quotas.
Design cost-effective inference for spiky traffic without idle GPUs
Tests designing inference that cuts idle GPU cost during troughs yet handles spiky peaks with low latency via SageMaker blue/green fleets, production variants, and CloudWatch baking periods. Red flag: always-on GPU pools with naive auto-scaling.
How would you design an A/B test for two live ML models?
Tests production experimentation rigor beyond random splitting. Strong answers cover: consistent user hashing for sticky assignment, isolated feature stores, guardrail metrics, and pre-calculated statistical power.
Explain model quantization, its benefits, drawbacks, and validation approach
Tests precision trade-offs in production. Answer: define lowering weights from fp32 to int8/int4; cite memory and latency gains versus accuracy loss; validate with downstream benchmarks and shadow A/B. Red flag: treating as lossless or skipping task metrics.
How would you systematically diagnose high latency in an online inference service?
Check p90/p99 and TTFT to split queuing from compute; inspect queue depth, batch size, GPU, and benchmarks; check cache. Systems reasoning across serving stack.
Deploy a trained model as a containerized REST API
This tests practical MLOps fluency. A strong answer covers loading the artifact, wrapping it in a web server, building a Dockerfile, and exposing health and predict endpoints. Red flag: conflating training with serving or omitting resource limits.
Describe the difference between online and batch inference.
Online uses autoscaling APIs for millisecond-to-second latency; batch uses scheduled compute for minute-to-hour latency. Your grasp of serving patterns and infra tradeoffs.
How would you systematically debug an inference API latency breach?
This tests structured debugging across the full inference stack. A strong answer traces the request path from ingress to GPU, splits TTFT from token-generation latency, inspects queuing and batching, then applies targeted fixes.
Design a system for batch scoring millions of customer records daily
Shard jobs via scheduler; partition storage; right-size CPU/GPU on spot; retry. Decoupling orchestration, storage, and compute for batch inference with cost/SLA in mind. One monolithic VM or real-time APIs for batch.
Where to place feature transformations: client, serving API, or upstream service?
Tests separation of concerns in ML systems. Client causes duplication and skew; serving API couples compute to requests; dedicated service adds a network hop but centralizes logic. Red flag: ignoring training-serving skew.
Describe the difference between online and batch inference.
Online serves single requests in ms on live endpoints; batch processes data offline with elastic compute. Latency vs throughput trade-offs in serving. Treating batch as slow online inference or ignoring SLA and cost.
How would you design safe, automatic schema evolution in CI?
Tests whether you separate schema evolution from semantic validation. Strong answer: versioned data contracts allowing additive enums, unknown-category model buckets, and automated contract negotiation. Red flag: manual allow-lists or disabling validation.
What infrastructure is needed for a Continuous Training pipeline?
Tests event-driven ML system design beyond CI/CD. Strong answers name orchestrators, feature stores, model registries, and validation gates, mapping triggers to retraining and promotion. Red flag: conflating CT with CI/CD or skipping model validation.
How would you implement shadow deployment and which metrics justify promotion?
Tests zero-impact validation when feedback loops are broken. Mirror traffic to a shadow variant, log predictions, and compare latency, errors, and drift against SLAs. Red flag: calling it A/B testing or claiming live business metrics from unserved responses.
Get Mlops bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.