How would you version control a 50GB dataset in a CI/CD pipeline?
Code and data versioning without breaking CI/CD speed.
Contrast Git LFS (simple, but 50GB chokes CI clones) with DVC (git metadata plus S3; enables selective pulls and CI cache).
Storing 50GB binaries in Git.
What's really being asked
This question probes whether you understand that Git is a source control system, not a blob store, and that CI/CD runners have finite disk, bandwidth, and time. The interviewer wants to see you reason about storage architecture, immutability, caching strategy, and cost.
The full answer
A strong response compares at least two approaches with specific trade-offs. First, Git LFS: explain that it replaces large files with text pointers in Git and stores binaries on a separate LFS server. Pros include familiar git workflows and atomic code-plus-data commits. Cons at 50GB include massive clone times, LFS bandwidth charges, runner storage limits, and slow CI jobs because every run may re-download the full dataset. Second, DVC or versioned object storage: explain keeping small metadata files or URIs in Git while the actual dataset lives in S3, GCS, or Azure Blob. Pros include lightweight repos, fast git checkout, selective data pulls, and the ability to cache data on CI runners or use a shared DVC cache. Cons include adding another tool or dependency, managing remote storage permissions, and the two-step workflow of git checkout followed by dvc pull or aws s3 cp. A senior candidate should also mention cache invalidation strategies, such as pinning dataset versions via content hashes or S3 object versions, and discuss cost per gigabyte transferred in CI.
The mistakes people make
The biggest red flag is suggesting Git can handle 50GB binaries directly, which reveals a misunderstanding of Git's design and performance characteristics. Another weak answer is proposing to zip or split the data without addressing the root problem that Git still stores every version of those bytes. Saying you will simply download the latest dataset from a network share without versioning or immutability is also a failure, because it breaks reproducibility. Finally, ignoring the CI/CD dimension entirely by discussing local development only shows a lack of production systems thinking.
What usually comes next
The interviewer may ask how you would handle incremental updates to the dataset rather than full replacements, or how you would verify data integrity if a remote storage object is modified. They might also ask about cost optimization, such as using S3 Intelligent-Tiering or keeping a warm cache on self-hosted runners, or how you would version control a dataset that is updated nightly by an ETL pipeline.
A concrete example
Suppose your team uses GitHub Actions with ephemeral Ubuntu runners. With DVC, you commit a data.dvc file and a .dvc/config pointing to an S3 remote. The CI job runs dvc pull, which downloads only the exact version referenced by the current Git commit. You configure the runner to persist a DVC cache directory between jobs, so subsequent builds fetch only changed chunks. If the dataset is 50GB and changes by 2GB daily, the typical CI pull drops from 50GB to 2GB, cutting transfer time and cloud egress costs significantly compared to Git LFS, which would re-download the entire pointer file set or large objects depending on implementation.
Interview question
For a 50GB dataset in a CI/CD pipeline, which approach best balances version control with runner performance?
- a.Commit small metadata files to Git while hosting the dataset in S3, then pull only the version referenced by the current commit and cache data between runsCorrect
- b.Compress the dataset into smaller zip files and commit them directly to the repository to avoid single large blobs
- c.Store the dataset directly in Git and rely on shallow clones to keep CI checkout fast
- d.Use Git LFS to track the dataset so only text pointers are stored in Git and the full data is fetched on demand
Why? this is the answer
Keeping lightweight metadata in Git while storing the actual dataset in S3 enables selective pulls and runner caching, avoiding massive repeated downloads. Git LFS is a common distractor because it still forces CI to download the full 50GB dataset on every run, which exhausts bandwidth and disk.
Just read this? Test yourself on what you have been reading.
Read the original → doc.dvc.org
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on mlops — each one lists the topics its interview covers.
See open roles