DVC: Git for Data and ML Models

WHY IT EXISTS Git is phenomenal for versioning code but fails with large binary files. Machine learning projects depend on large datasets and multi-gigabyte models that need versioning just like code. DVC was created to solve this by managing large file versions outside of Git while linking them directly to Git commits.

THE MENTAL MODEL Think of DVC as a library's card catalog for your project's data. Your Git repository is the catalog itself, holding small, lightweight "cards" (.dvc pointer files). The actual "books" (large datasets, models) are stored on massive, efficient shelves (like Amazon S3, Google Cloud Storage, or a local cache). DVC manages checking books in and out, while Git manages changes to the card catalog.

HOW IT WORKS When you run dvc add my_model.pkl, DVC moves the file to a local cache and creates a small text file named my_model.pkl.dvc. This new file contains a hash and other metadata pointing to the original file in the cache. You then git add and git commit this tiny .dvc file. To share, you run dvc push, which uploads the actual large file from your cache to remote storage. A teammate can then git pull to get the pointer file and dvc pull to download the corresponding large file.

WHEN TO USE IT Use DVC for any project where you need to version files too large for Git, which is standard in machine learning. It's ideal for versioning datasets, trained models, and data processing pipelines. It ensures that anyone who checks out a specific Git commit can retrieve the exact data and model associated with that version of the code, guaranteeing reproducibility.

WHEN NOT TO USE IT DVC is overkill for projects with only small text files that Git handles natively. If your project doesn't involve large data or model artifacts, using DVC adds unnecessary complexity. It's not a replacement for Git, but an extension for a specific problem.

ONE CANONICAL EXAMPLE A team trains a computer vision model. The code is in Git, but the 50GB image dataset is tracked with DVC. A developer runs dvc repro to execute the training pipeline. DVC sees the data hasn't changed but the training code has, so it re-runs the script, versions the new model artifact, and updates the corresponding .dvc pointer file. The developer commits this pointer file to Git, ensuring the new model is tied to the exact code and data that produced it.