Design a system for batch scoring millions of customer records daily

WHAT THIS TESTS: This question tests whether you can design a production batch inference pipeline that separates concerns across three layers: orchestration, storage, and compute. The interviewer wants to see that you understand horizontal scaling, fault tolerance, idempotency, and cost optimization for ML workloads, not just that you can name AWS services.

A GOOD ANSWER COVERS: First, an orchestration layer that manages job submission, scheduling, tracking, and error handling. Candidates should mention services like Amazon MWAA or AWS Step Functions to coordinate multi-step pipelines and handle retries. Second, a storage layer using Amazon S3 with proper partitioning by date or customer segment so that workers can read subsets in parallel without I/O bottlenecks. Third, a compute layer using Amazon SageMaker Batch Transform or SageMaker Processing Jobs, with instance types matched to model complexity; for example, XGBoost or scikit-learn on CPU instances, while deep learning models may need GPU. Fourth, reliability mechanisms such as idempotent writes, checkpointing, dead-letter queues for failed batches, and CloudWatch alarms. Fifth, cost controls including spot instances for fault-tolerant workloads, right-sizing instances, and auto-scaling worker counts based on backlog.

COMMON WRONG ANSWERS: A major red flag is proposing a single monolithic EC2 instance or a real-time API endpoint to score millions of records sequentially. Another is ignoring data partitioning, which forces all workers to contend for the same storage throughput. Candidates who neglect idempotency or retry logic will struggle to explain how the system recovers from mid-job failures. Suggesting GPU instances for simple linear models also signals poor cost awareness.

LIKELY FOLLOW-UPS: Interviewers often ask how you would handle a missed SLA, how to scale from millions to billions of records, or how to version models and roll back a bad deployment. They may also probe how you would detect data drift in batch inputs or orchestrate A/B tests across different model versions in the same pipeline.

ONE CONCRETE EXAMPLE: Imagine a marketing firm scoring 50 million customer records nightly with an XGBoost model. An MWAA DAG splits the dataset into 500 shards of 100,000 records each, writing input prefixes to S3. SageMaker Batch Transform runs on c5.2xlarge spot instances with five concurrent jobs, outputting predictions to a partitioned S3 prefix. A Step Functions state machine monitors each shard, retrying twice on failure and emitting metrics to CloudWatch. If the spot fleet is interrupted, the job resumes from the last completed shard because each output file is named by input shard ID, making the pipeline idempotent. The entire workload finishes within four hours at roughly 70 percent lower compute cost than on-demand instances.

Source: aws.amazon.com