Design petabyte-scale distributed training

WHAT THIS TESTS This checks system-design breadth: at petabyte scale the data pipeline, not the model, is usually the constraint, and you must address storage, processing, training, and operations together.

A GOOD ANSWER COVERS Storage: keep raw data in cheap object storage such as S3, GCS, or Azure Blob, and write training-ready data in columnar or sharded formats like Parquet, TFRecord, or WebDataset so workers can stream contiguous chunks. Processing: run distributed ETL and feature engineering on Spark, Dataflow, or Ray, materializing features once rather than recomputing per epoch, ideally into a feature store to avoid train-serve skew. Training: use a data-parallel framework, PyTorch DDP, Horovod, or DeepSpeed, on a GPU cluster with high-bandwidth interconnect; shard the dataset across workers and prefetch so GPUs stay saturated. Infrastructure: orchestrate with managed Kubernetes or SageMaker/Vertex/Azure ML, use spot or preemptible instances with frequent checkpointing for cost and resilience.

COMMON WRONG ANSWERS Loading the whole dataset into memory. Ignoring file format and ending up I/O bound. Recomputing features every epoch. No checkpointing, so a single node failure restarts everything.

LIKELY FOLLOW-UPS How do you keep GPUs fed at this scale? Data versus model parallelism here? How do you handle node failures? How do you control cost?

ONE CONCRETE EXAMPLE Raw logs land in S3; a Spark job on EMR cleans and writes sharded Parquet back to S3. A SageMaker training job spins up dozens of GPU nodes running PyTorch DDP, each streaming its shard via a high-throughput dataloader. Checkpoints save to S3 every few minutes so spot-instance reclamation resumes cleanly, and the run is reproducible from versioned data and code.