Intermediate everything in AI & ML, page 12
Design a system to monitor a real-time prediction service for feature drift
Async feature logging, distribution comparison via PSI/KS against training baseline, and threshold-based anomaly alerts.
How would you design an A/B test for two live ML models?
Tests production experimentation rigor beyond random splitting. Strong answers cover: consistent user hashing for sticky assignment, isolated feature stores, guardrail metrics, and pre-calculated statistical power.
Explain model quantization, its benefits, drawbacks, and validation approach
Tests precision trade-offs in production. Answer: define lowering weights from fp32 to int8/int4; cite memory and latency gains versus accuracy loss; validate with downstream benchmarks and shadow A/B. Red flag: treating as lossless or skipping task metrics.
How would you systematically diagnose high latency in an online inference service?
Check p90/p99 and TTFT to split queuing from compute; inspect queue depth, batch size, GPU, and benchmarks; check cache.
Design a system for batch scoring millions of customer records daily
Shard jobs via scheduler; partition storage; right-size CPU/GPU on spot; retry.
Where to place feature transformations: client, serving API, or upstream service?
Tests separation of concerns in ML systems. Client causes duplication and skew; serving API couples compute to requests; dedicated service adds a network hop but centralizes logic. Red flag: ignoring training-serving skew.
What infrastructure is needed for a Continuous Training pipeline?
Tests event-driven ML system design beyond CI/CD. Strong answers name orchestrators, feature stores, model registries, and validation gates, mapping triggers to retraining and promotion. Red flag: conflating CT with CI/CD or skipping model validation.
How would you implement shadow deployment and which metrics justify promotion?
Tests zero-impact validation when feedback loops are broken. Mirror traffic to a shadow variant, log predictions, and compare latency, errors, and drift against SLAs. Red flag: calling it A/B testing or claiming live business metrics from unserved responses.
What events trigger automatic model retraining beyond code changes?
This tests whether you treat ML pipelines as event-driven systems, not just software CI/CD. A strong answer lists data drift, scheduled cron jobs, production metric degradation, schema changes, and upstream data pipeline completion.
Compare Airflow and Kubeflow for ML training pipelines
Tests orchestrator-to-workload fit. Strong answers contrast Airflow's data integration and Python DAGs with Kubeflow's K8s scaling, container reproducibility, and experiment tracking. Red flag: claiming one is always better without stage-specific reasoning.
Design training job submission to a shared Kubernetes cluster
Gateway with artifact caching; namespace quotas; GPU schedulers like Volcano; Prometheus metrics and cost attribution.
Robust checkpointing strategy for multi-day training jobs and seamless resumption
Tests production-grade distributed training reliability. Cover async atomic checkpoints, MTBF-based cadence, tiered storage, and recovery drills. Red flag: blocking synchronous writes that ignore silent corruption or straggler finalization.
How do you version and distribute Docker dev environments consistently?
Tests immutable dev environment distribution. Strong answers cover: versioned Dockerfiles in Git, immutable image tags pushed to a registry, and enforcing identical pulls for CI and developers. Red flag: using the "latest" tag or local Dockerfile rebuilds.
How do you build dev and production Docker images from one source?
Mastery of Docker multi-stage builds for isolating dev and production dependencies. Use a base stage, a dev target with linters and tests, and a lean production target copying only the build artifact.
How do you persist notebooks and artifacts in Docker?
Tests Docker storage abstractions. A strong answer distinguishes bind mounts for live notebook editing from named volumes for datasets and artifacts, and warns against docker commit for persistence. Red flag: treating containers as stateful VMs.
Design a CI/CD pipeline that automates model promotion from Staging to Production
Tests whether you treat model promotion as a gated software delivery workflow. Strong answers use registry state-change triggers, automated drift and performance checks, canary deployment gates, and rollback.
Why use a Model Registry over dated pickle files?
A strong answer covers versioning, lineage, promotion aliases, and governance.
How would you implement versioning for feature definitions in a feature store?
Tests separation of metadata schema versions from data snapshots for reproducible training. Strong answers cover: immutable schema versions on breaking changes, safe appends without backfill, and time-travel data reads.
Backfill a complex feature for millions of users without impacting production
Reuse the live pipeline on historical partitions, run bounded batches on separate compute, stage results, and validate before promotion.
Design a system to detect training-serving skew for a numerical feature
Tests ML monitoring design via statistical distribution comparison between training and live data. Strong answers cover PSI/KS tests, windowed thresholding, and tiered alerting. Red flag: comparing raw values instead of distributions or ignoring alert fatigue.
We are hiring for this. Every open role lists the topics its interview covers, so you can prepare for the real thing rather than guessing.
See open roles