Design a system for batch scoring millions of customer records daily
Decoupling orchestration, storage, and compute for batch inference with cost/SLA in mind.
Shard jobs via scheduler; partition storage; right-size CPU/GPU on spot; retry.
One monolithic VM or real-time APIs for batch.
What's really being asked
This question tests whether you can design a production batch inference pipeline that separates concerns across three layers: orchestration, storage, and compute. The interviewer wants to see that you understand horizontal scaling, fault tolerance, idempotency, and cost optimization for ML workloads, not just that you can name AWS services.
The full answer
First, an orchestration layer that manages job submission, scheduling, tracking, and error handling. Candidates should mention services like Amazon MWAA or AWS Step Functions to coordinate multi-step pipelines and handle retries. Second, a storage layer using Amazon S3 with proper partitioning by date or customer segment so that workers can read subsets in parallel without I/O bottlenecks. Third, a compute layer using Amazon SageMaker Batch Transform or SageMaker Processing Jobs, with instance types matched to model complexity; for example, XGBoost or scikit-learn on CPU instances, while deep learning models may need GPU. Fourth, reliability mechanisms such as idempotent writes, checkpointing, dead-letter queues for failed batches, and CloudWatch alarms. Fifth, cost controls including spot instances for fault-tolerant workloads, right-sizing instances, and auto-scaling worker counts based on backlog.
The mistakes people make
A major red flag is proposing a single monolithic EC2 instance or a real-time API endpoint to score millions of records sequentially. Another is ignoring data partitioning, which forces all workers to contend for the same storage throughput. Candidates who neglect idempotency or retry logic will struggle to explain how the system recovers from mid-job failures. Suggesting GPU instances for simple linear models also signals poor cost awareness.
What usually comes next
Interviewers often ask how you would handle a missed SLA, how to scale from millions to billions of records, or how to version models and roll back a bad deployment. They may also probe how you would detect data drift in batch inputs or orchestrate A/B tests across different model versions in the same pipeline.
A concrete example
Imagine a marketing firm scoring 50 million customer records nightly with an XGBoost model. An MWAA DAG splits the dataset into 500 shards of 100,000 records each, writing input prefixes to S3. SageMaker Batch Transform runs on c5.2xlarge spot instances with five concurrent jobs, outputting predictions to a partitioned S3 prefix. A Step Functions state machine monitors each shard, retrying twice on failure and emitting metrics to CloudWatch. If the spot fleet is interrupted, the job resumes from the last completed shard because each output file is named by input shard ID, making the pipeline idempotent. The entire workload finishes within four hours at roughly 70 percent lower compute cost than on-demand instances.
Interview question
When designing a nightly pipeline to score millions of records with an XGBoost model, which approach best satisfies cost and reliability requirements?
- a.Partition input in S3 by shard ID, run SageMaker Batch Transform on CPU spot instances, and implement idempotent retries per shardCorrect
- b.Deploy a real-time inference endpoint and invoke it via API for each record individually
- c.Store all records in a single S3 object and process them sequentially on one large EC2 instance
- d.Partition input in S3 by customer segment but use GPU on-demand instances to maximize throughput for the XGBoost model
Why? this is the answer
The card recommends partitioning S3 data, using SageMaker Batch Transform on right-sized CPU spot instances, and designing idempotent shards so retries only reprocess failed batches. Option D is tempting because partitioning is correct, but GPUs are unnecessary for XGBoost and on-demand instances sacrifice the significant cost savings the card highlights.
Just read this? Test yourself on what you have been reading.
Read the original → aws.amazon.com
- #mlops
- #batch inference
- #aws
- #sagemaker
- #system design
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on mlops — each one lists the topics its interview covers.
See open roles