Describe an ML workflow with massive egress fees and re-architecture to mitigate

What's really being asked

This question tests whether you understand that cloud egress is not a networking footnote but a first-class architectural constraint in modern ML systems. Specifically, it probes your ability to identify boundary-crossing data flows, quantify their cost impact using real pricing tiers, and redesign workflows around data locality rather than treating transfer as an inevitable operational expense.

The full answer

First, name the boundary. Egress fees trigger when data leaves a provider network boundary, such as cross-region, multi-cloud, or hybrid on-prem to cloud paths. Same-region same-provider reads typically avoid per-GB charges. Second, paint a realistic scenario. A common trap is a team storing a 10TB training dataset in AWS S3 while running large-scale training on cheaper GPU instances in another cloud or region. Each epoch rereads the full dataset, and at roughly nine cents per GB that can exceed four thousand dollars per month before accounting for checkpoints, restores, and analytics exports. Third, propose architectural mitigations in order of impact. Move compute to the data by provisioning training clusters in the same region and provider as the storage. If multi-cloud is unavoidable, deploy a durable cache or replica near the compute and refresh it incrementally rather than streaming the full dataset on every run. Batch and compress exports, but only as a secondary optimization because compression does not eliminate the boundary crossing itself. Fourth, mention commercial and model shifts. Negotiate committed use discounts or private interconnects to lower rates, and evaluate storage providers or billing models that remove per-GB egress line items entirely for covered usage.

The mistakes people make

Suggesting generic compression or file format optimization as the primary fix without addressing locality. Proposing a CDN for training data, which is designed for low-latency small-object delivery rather than high-throughput sequential reads. Ignoring the repeated-read multiplier inherent in ML training epochs and checkpointing. Treating egress as a finance problem to be solved by billing alerts rather than an architecture problem to be solved by placement.

What usually comes next

How would you design a cache invalidation strategy for a near-compute replica without re-downloading the full dataset? What tradeoffs exist between duplicate storage costs and egress savings? How do you handle egress when training must span multiple geographic regions for compliance? What is your stance on data gravity versus compute portability?

A concrete example

A computer vision team stores 50TB of image data in Azure Blob Storage in East US. They discover a 40 percent cost savings on GPU spot instances in GCP us-central1 and launch a large training job there. The pipeline reads the full dataset once per epoch for 20 epochs, plus daily checkpoint restores and weekly analytics exports. At list rates, cross-cloud egress alone adds tens of thousands of dollars to the project. The re-architecture moves the primary training workload to Azure East US compute, co-locating with the data. For the GCP-specific experiments, they maintain a warmed NVMe cache of the current shard on the GCP nodes and only sync incremental data additions, cutting egress by over 90 percent while keeping storage duplication costs under five percent of the original transfer bill.