How would you version control a 50GB dataset in a CI/CD pipeline?
WHAT IT TESTS: Code and data versioning without breaking CI/CD speed. ANSWER OUTLINE: Contrast Git LFS (simple, but 50GB chokes CI clones) with DVC (git metadata plus S3; enables selective pulls and CI cache). RED FLAG: Storing 50GB binaries in Git.
WHAT IT TESTS: Whether you can keep Git lightweight while versioning multi-gigabyte training data in CI. ANSWER OUTLINE: First, evaluate Git LFS: native git semantics, but 50GB files balloon clone times, exhaust runner storage, and rack up bandwidth costs. Second, propose DVC: store lightweight metadata in Git while keeping data in S3; CI pulls only the needed version and leverages remote or runner caching. RED FLAG: Recommending raw Git for 50GB binaries, ignoring immutable artifacts, or overlooking CI cost and cache strategy.
Read the original → doc.dvc.org
- #mlops
- #ci/cd
- #data-versioning
- #dvc
- #git-lfs
Get five bites like this every day.
Tezvyn delivers a daily feed of 60-second tech bites with quizzes to lock in what you learn.