Feeding large object-store data into training

WHAT THIS TESTS This evaluates whether you can keep an expensive accelerator fed by designing an efficient data pipeline from object storage rather than letting IO stall training.

A GOOD ANSWER COVERS The naive approach of downloading the entire terabyte to the instance's local disk before training wastes time, requires large attached storage, and delays the start. Instead, stream the data. Managed services offer modes for this: SageMaker provides pipe mode and fast file mode that stream objects from S3 on demand, and Vertex AI supports similar streaming and mounted access, so the job reads as it trains. Shard the dataset across data-loading workers and across distributed training nodes so reads happen in parallel and each worker pulls a distinct slice. Use prefetching and a data loader that overlaps IO and decoding with GPU compute, so the next batch is ready before the current one finishes. Crucially, address the many-small-files issue: a terabyte of individual image files means a huge number of small object reads, each with request overhead, so pack images into larger sequential record files such as TFRecord, WebDataset tar shards, or RecordIO, which lets the loader stream large contiguous reads and shuffle within shards. Also colocate the training job in the same region as the bucket to avoid transfer cost and latency.

COMMON WRONG ANSWERS Downloading all data to local disk first. Reading millions of tiny image objects one by one with no packing, drowning in request overhead. Forgetting to shard across workers, serializing IO. Ignoring prefetch so the GPU idles waiting on data. Training in a different region from the bucket.

LIKELY FOLLOW-UPS How do you shuffle when streaming. What file size to target for shards. How do you monitor whether IO is the bottleneck.

ONE CONCRETE EXAMPLE You repackage a terabyte of JPEGs into WebDataset tar shards of a few hundred megabytes each, stream them with fast file mode, and let multiple loader workers prefetch and decode in parallel. The GPU stays near full utilization instead of waiting on per-image S3 requests.