Versioning a 10TB dataset as code

WHAT THIS TESTS Whether you understand reproducible, space-efficient dataset versioning and can name a concrete tool, rather than naively copying data.

A GOOD ANSWER COVERS Data-as-code treats datasets as first-class, versioned artifacts tied to your code commits, so any experiment can be reproduced from an exact code plus data pair. The key to avoiding duplication is content addressing. You hash each file or chunk; the hash becomes its identity. Git stores only tiny metafiles, the pointers and hashes, while the multi-terabyte binaries live in object storage such as S3. A new dataset version that changes a few thousand images stores only those new objects; every unchanged file is shared by reference across versions. Some tools chunk files so even partial edits dedupe.

NAME A TOOL DVC is the canonical answer: it keeps a .dvc metafile in Git, pushes data to a remote like S3, and reconstructs any version on checkout. Alternatives include LakeFS, which gives Git-like branching over object storage, and Git LFS for smaller binary cases.

COMMON WRONG ANSWERS Copying the full 10TB for each version, committing binaries straight into Git, which bloats the repo, or relying only on timestamped folders with no hashing or reproducibility link.

LIKELY FOLLOW-UPS How checkout reconstructs a version, how you tie a model run to its exact data hash, and how you garbage-collect orphaned objects.

ONE CONCRETE EXAMPLE You run dvc add on the image directory; DVC hashes the files, writes a small .dvc pointer you commit to Git, and runs dvc push to S3. A teammate adding ten thousand new images creates a new version where only those objects are uploaded, and checking out an old commit plus dvc pull restores that exact prior dataset.