Monitoring
115 bites tagged Monitoring — interview questions with model answers, and 60-second explainers.
SLOs Tied to User Journeys, Not APIs
A user-journey SLO measures the full flow a person experiences, not one microservice's health. If checkout is 99.9% up but payments fail, the metric lied. Teams drown in green per-service dashboards while users are furious.
How would you design drift detection for high-dimensional embeddings?
Tests distribution shift in latent spaces beyond per-feature stats. Strong answers use maximum mean discrepancy, k-NN two-sample tests, or domain-classifier AUC, plus windowing. Red flag: per-dimension KS tests or mean-difference thresholds.
How do you monitor model health with delayed ground truth labels?
Tests ML ops maturity beyond accuracy. A strong answer covers input drift via PSI or KS tests, prediction distribution shifts, proxy business metrics, and human spot-checking. A red flag is passively waiting for labels or retraining blindly without validation.
Compare Canary and Blue/Green ML deployments and model-specific metrics
Contrast Canary gradual shift vs Blue/Green instant swap; highlight silent failures, data drift, prediction distribution; cite accuracy and calibration. Model quality vs infra health in rollouts.
Design a near real-time cost visibility system for ML teams
Tests cost attribution across shared ML infrastructure and streaming pipeline design. Strong answers combine billing exports with resource labels, sub-hour aggregation, and anomaly detection for training spikes.
How would you programmatically monitor a deployed model for demographic bias?
Tests operationalizing fairness beyond static audits. Track group metrics like parity and equalized odds; slice by protected attributes; alert on drift; route violations to review. Red flag: treating fairness as a one-time check versus continuous monitoring.
How do you monitor thousands of per-customer models as a fleet?
Tests fleet-level statistical aggregation versus per-instance alerting. Strong answers propose tiered telemetry, cohort baselining for drift, and hierarchical alerting to prevent fatigue.
Model output distribution shifts. What are root causes and next steps?
This tests covariate vs label shift vs concept drift when outputs shift. A strong answer checks features before labels, then feedback loops or staleness. A red flag is generic drift without separating P(X), P(Y), and P(Y|X).
Design a system to monitor a real-time prediction service for feature drift
Async feature logging, distribution comparison via PSI/KS against training baseline, and threshold-based anomaly alerts. production ML observability beyond accuracy checks.
Differences between monitoring a traditional REST API and a production ML model
Contrast latency/errors with ML signals like data drift and training-serving skew against baselines, noting ground truth delays. Awareness that ML fails via data decay, not code bugs.
What events trigger automatic model retraining beyond code changes?
This tests whether you treat ML pipelines as event-driven systems, not just software CI/CD. A strong answer lists data drift, scheduled cron jobs, production metric degradation, schema changes, and upstream data pipeline completion.
How would you design a system to detect training-serving skew using model registry metadata?
This tests statistical monitoring between production data and registry training baselines. Strong answers: schema-bound metadata, incremental stats, drift metrics PSI, tiered alerting. Red flag: schema validation mistaken for drift or manual checks only.
How would you design automatic data drift detection for production inference?
This tests reference-vs-live monitoring architecture. A strong answer chunks data, runs univariate per-feature drift, adds multivariate PCA or domain classifiers, and ranks threshold alerts. A red flag is checking aggregate metrics instead of feature shifts.
What production metrics and auto-thresholds trigger model retraining?
This tests production monitoring maturity. A strong answer covers technical drift metrics, business KPIs tied to model decisions, and automated thresholds that page or trigger CI/CD retraining.
Design an automated rollback process when deployment error rates spike
Gate on error-rate and latency thresholds; use blue-green deploys to limit blast radius; require human approval for stateful rollbacks. judgment when automating rollbacks. instant full rollback from one noisy metric.
How do you diagnose a progressively slower CI pipeline?
Profile stage durations and critical path, audit runner CPU/memory/disk, flag flaky or late-failing tests. data-driven CI diagnosis across pipeline topology, runner resources, and test quality.
What are the four Golden Signals for service health monitoring?
Tests whether you can name the four essential metrics—latency, traffic, errors, saturation—and explain why each matters for detecting user-facing regressions after a release, rather than drowning in infrastructure noise.
APM: Turning System Metrics into Business Meaning
APM turns system metrics into business meaning by tracking software performance and availability. It matters most when slowdowns threaten service levels. The footgun is gathering data without translating IT metrics into what the business actually cares about.
How would you design an automated data quality monitoring system?
Tests turning data quality into tiered checks for exec dashboards. Strong answers combine freshness, volume, schema, and distribution validation with severity-based paging. Red flag: static thresholds without noise reduction or business-impact triage.
How would you design a data quality monitoring system?
This tests your systematic approach to data reliability. A strong answer defines quality dimensions (freshness, volume, schema), proposes specific checks, and outlines an alerting strategy. A red flag is listing checks without tying them to business impact.
Mean vs. Median for API Response Times?
Tests your understanding of non-normal distributions like latency. Choose median as it's robust to outliers that skew the mean. Then, state that even median is insufficient; percentiles (p99, max) are crucial for capturing the full user experience.
Design a Data Quality Monitoring & Alerting System
This tests translating business needs into a concrete data validation strategy. A good answer defines checks based on business impact (freshness, volume, schema), then outlines a tiered alerting system. A red flag is naming tools before defining the problem.
Post-Incident Review: Learning from Failure, Blamelessly
A Post-Incident Review (PIR) is a blameless process to learn from an outage, not to assign blame. Use it after a production incident to identify systemic flaws and create action items to prevent repeats.
Escalation Policy: When to Stop Shipping and Start Fixing
An escalation policy is a pre-agreed plan for when to divert engineers from feature work to fix reliability. When a service's error budget burns too fast, the policy's thresholds trigger specific actions. The footgun is thinking a quick rollback is enough.
Get Monitoring bites daily.
Five a day, five minutes, offline. With quizzes so it sticks.
Open testing — you’ll join as an early tester.