How would you version control a 50GB dataset in a CI/CD pipeline?

WHAT THIS TESTS: This question probes whether you understand that Git is a source control system, not a blob store, and that CI/CD runners have finite disk, bandwidth, and time. The interviewer wants to see you reason about storage architecture, immutability, caching strategy, and cost.

A GOOD ANSWER COVERS: A strong response compares at least two approaches with specific trade-offs. First, Git LFS: explain that it replaces large files with text pointers in Git and stores binaries on a separate LFS server. Pros include familiar git workflows and atomic code-plus-data commits. Cons at 50GB include massive clone times, LFS bandwidth charges, runner storage limits, and slow CI jobs because every run may re-download the full dataset. Second, DVC or versioned object storage: explain keeping small metadata files or URIs in Git while the actual dataset lives in S3, GCS, or Azure Blob. Pros include lightweight repos, fast git checkout, selective data pulls, and the ability to cache data on CI runners or use a shared DVC cache. Cons include adding another tool or dependency, managing remote storage permissions, and the two-step workflow of git checkout followed by dvc pull or aws s3 cp. A senior candidate should also mention cache invalidation strategies, such as pinning dataset versions via content hashes or S3 object versions, and discuss cost per gigabyte transferred in CI.

COMMON WRONG ANSWERS: The biggest red flag is suggesting Git can handle 50GB binaries directly, which reveals a misunderstanding of Git's design and performance characteristics. Another weak answer is proposing to zip or split the data without addressing the root problem that Git still stores every version of those bytes. Saying you will simply download the latest dataset from a network share without versioning or immutability is also a failure, because it breaks reproducibility. Finally, ignoring the CI/CD dimension entirely by discussing local development only shows a lack of production systems thinking.

LIKELY FOLLOW-UPS: The interviewer may ask how you would handle incremental updates to the dataset rather than full replacements, or how you would verify data integrity if a remote storage object is modified. They might also ask about cost optimization, such as using S3 Intelligent-Tiering or keeping a warm cache on self-hosted runners, or how you would version control a dataset that is updated nightly by an ETL pipeline.

ONE CONCRETE EXAMPLE: Suppose your team uses GitHub Actions with ephemeral Ubuntu runners. With DVC, you commit a data.dvc file and a .dvc/config pointing to an S3 remote. The CI job runs dvc pull, which downloads only the exact version referenced by the current Git commit. You configure the runner to persist a DVC cache directory between jobs, so subsequent builds fetch only changed chunks. If the dataset is 50GB and changes by 2GB daily, the typical CI pull drops from 50GB to 2GB, cutting transfer time and cloud egress costs significantly compared to Git LFS, which would re-download the entire pointer file set or large objects depending on implementation.

Source: doc.dvc.org