Imagine you are writing a school essay, and every single time you make a change, you save it as a brand-new file: “essay.doc”, “essay_final.doc”, “essay_final_v2.doc”, “essay_final_v2_REAL.doc”. A week later, you cannot remember which file actually has the good ending you wrote on Tuesday. Now imagine a team of people all editing that same essay at once, on their own computers, and somehow needing to combine everyone’s changes into one correct version. That chaos is exactly the problem version control was invented to solve.

Version control is a system that automatically keeps a complete history of every change made to a project — who changed what, when they changed it, and why — so that anyone can go back in time to any earlier version whenever they need to. Instead of dozens of confusingly named files, there is just one project with a full, searchable timeline of everything that ever happened to it.

In ordinary software projects, the most famous version control tool by far is Git — a free system that tracks changes to text-based files like program code. Git has been the backbone of software engineering for years. But machine learning projects are not just code. They also involve enormous datasets, trained model files, and a constant stream of experiments — and that is where the story gets more complicated.

Why Machine Learning Makes This Harder

A typical ML project has several moving parts that all change at different speeds and in different ways:

The central challenge of this entire guide boils down to one sentence: a machine learning result is only as reproducible as the weakest-tracked piece behind it. If you can perfectly version your code but have no idea which exact dataset trained last month’s best model, you cannot truly reproduce, trust, or improve on that result.

Before going further into DVC, it helps to remember exactly what Git does and why it became the default version control tool for almost all software on Earth.

Git works by taking little “snapshots” of a project every time you tell it to, called commits. Each commit records exactly what every tracked file looked like at that moment, along with a message describing what changed and why. Over time, this builds a complete, browsable timeline of a project’s entire life.

The Everyday Git Vocabulary

Git is brilliant at this job specifically because the files it was designed for — source code — are small, plain text, and easy to compare line by line. That is precisely the assumption that starts to break down the moment machine learning enters the picture, which the next section explores.

Git was built in 2005 to track the Linux kernel’s source code — millions of lines of small text files. Machine learning throws three things at Git that it was never designed to handle gracefully.

Problem 1 — Binary Files Bloat the Repository

Git is exceptionally good at storing the difference between two versions of a text file — it can store just the few lines that changed, not the whole file again. A trained model file (such as a .h5, .pt, or .pkl file) or a dataset is usually binary, meaning Git cannot meaningfully compute a small difference between two versions. Every tiny change forces Git to store an entirely new multi-gigabyte copy, and the repository balloons in size with every single commit.

Problem 2 — Hosting Services Cap File and Repo Sizes

Most Git hosting services impose hard limits precisely because of this binary-file problem. GitHub, for example, strongly discourages files over 100MB and enforces an upper limit on total repository size. A single modern training dataset or a large language model’s checkpoint file can blow past these limits in an instant.

Problem 3 — Reproducibility Needs More Than Code

Even if file size were not an issue, there is a deeper problem: a model’s behavior depends on its code and on the exact dataset it was trained on and on the hyperparameters chosen for that run. Git alone can tell you which version of train.py was used last Tuesday — it cannot tell you which version of the 40GB dataset folder sitting on someone’s hard drive was used at the same time, unless that link is captured somewhere too.

Before DVC existed, the software world’s first attempt to fix Git’s large-file problem was a tool called Git LFS (Large File Storage), and it is still worth understanding because many teams still use it today.

Git LFS works by replacing a large file inside the repository with a small text pointer — essentially a note that says “the real file lives somewhere else, and here’s its unique fingerprint.” The real, heavy file gets stored on a separate LFS server, while Git itself only ever has to track that tiny pointer file. This keeps the main repository lightweight even when the project includes large trained models.

# Turn on LFS tracking for this repository
git lfs install

# Tell Git LFS which file types to treat specially
git lfs track "*.h5"

# From here on, this behaves like a normal Git workflow
git add model.h5
git commit -m "Add trained model"
git push

Where Git LFS Falls Short for ML

Git LFS solves the file-size problem reasonably well, but it was built with general large files in mind — design assets, videos, audio — not specifically for the way machine learning teams actually work. It does not understand datasets as a distinct concept, has no built-in idea of an ML pipeline (the connected steps from raw data to a finished model), and offers no experiment-tracking features at all.

This gap — a tool that handles big files but understands nothing about datasets, pipelines, or experiments — is exactly the opening that DVC was built to fill, and it is where the rest of this guide is headed.

DVC stands for Data Version Control. It is a free, open-source tool, written in Python, that was created in 2017 by Dmitry Petrov and the team at a company called Iterative.ai, specifically to close the gap that Git and Git LFS left open. Since then it has grown into a project with well over a hundred contributors and thousands of users, and it has been adopted by data teams at organizations ranging from small startups to large, well-known technology companies.

The core idea behind DVC is refreshingly simple: it does not try to replace Git — it works alongside it. Git keeps doing exactly what it already does best (tracking code). DVC takes on the job Git was never good at: tracking large datasets and model files, while storing only a tiny pointer inside the actual Git repository.

What DVC Can Do

DVC’s commands are deliberately designed to feel almost identical to Git’s, so that anyone who already knows Git can learn DVC in an afternoon. The two tools are always used as a pair — never one without the other.

Step 1 — Initialize Both Tools

git init     # Starts tracking code and lightweight metadata
dvc init     # Creates a .dvc folder to manage data versioning

git add .dvc/
git commit -m "Initialize DVC"

Running dvc init creates a hidden .dvc folder containing a configuration file and a local cache directory — comparable to how git init creates the hidden .git folder. From this point on, the project understands both code history and data history.

Step 2 — Track a Dataset or Model

dvc add data/raw/training_images/

# This creates: data/raw/training_images.dvc
# A tiny text file containing a hash (a unique fingerprint)

git add data/raw/training_images.dvc .gitignore
git commit -m "Track training dataset v1"

Behind the scenes, dvc add copies the real data into a local cache, calculates a unique hash for it, and automatically updates .gitignore so Git never tries to track the heavy folder directly. The small .dvc file it generates is the only thing that ever travels through Git.

Step 3 — Connect to Remote Storage

dvc remote add -d myremote s3://my-bucket/dvcstore
git add .dvc/config
git commit -m "Configure DVC remote storage"

dvc push   # Uploads the actual data to remote storage

Step 4 — Restoring a Project on a New Machine

This is where the system really proves its worth. A teammate — or even the same person, six months later on a different laptop — can rebuild the exact project state with just four commands:

Switching Between Versions — The Two-Checkout Rule

Because Git tracks the pointer and DVC tracks the actual data, switching to an older version of a dataset always takes two commands working together — one for each tool:

git checkout <older-commit-hash>   # Changes the .dvc pointer back in time
dvc checkout                       # Pulls the matching data version into your workspace

Beyond simply versioning files, DVC can describe an entire ML workflow as a connected series of steps — sometimes summarized as “Makefiles for machine learning.” This feature is called a pipeline, and it is defined in a file named dvc.yaml.

A pipeline breaks a project into named stages. Each stage declares what command to run, what files it depends on, and what files it produces. DVC then automatically figures out which stages need to rerun whenever an upstream dependency changes — and, just as importantly, which stages can be safely skipped because nothing they depend on has changed.

stages:
  preprocess:
    cmd: python scripts/preprocess.py data/raw data/processed
    deps:
      - scripts/preprocess.py
      - data/raw
    outs:
      - data/processed

  train:
    cmd: python scripts/train.py data/processed models/model.pkl
    deps:
      - scripts/train.py
      - data/processed
    outs:
      - models/model.pkl

Running the entire pipeline is then a single command:

dvc repro   # Reruns only the stages whose dependencies changed

Why This Matters: The Reproducibility Test

A project earns the label “reproducible” only if this sequence reliably rebuilds the exact same model, every time, on any machine:

git checkout <commit-hash>
dvc pull
pip install -r requirements.txt
dvc repro

If running those four lines does not produce the same model artifact, the project is not actually reproducible yet — no matter how good the code looks. This four-line test is one of the simplest, most useful sanity checks any ML team can run on their own workflow.

A real ML project rarely produces just one model. It produces dozens, sometimes hundreds, of attempts — different hyperparameters, different feature sets, different random seeds. Without a system for tracking them, a team ends up relying on memory and Slack messages to answer the question, “wait, which run actually got us 94% accuracy?”

Tracking Experiments With MLflow

MLflow is an open-source platform built specifically to manage this side of the ML lifecycle. It lets a script log its hyperparameters, its performance metrics, and even the resulting model file — all tagged to one identifiable “run” that can later be compared against every other run.

import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier

mlflow.set_experiment("model_versioning_example")

with mlflow.start_run():
    clf = RandomForestClassifier(n_estimators=100)
    clf.fit(X_train, y_train)

    mlflow.log_param("n_estimators", 100)
    mlflow.log_metric("accuracy", clf.score(X_test, y_test))
    mlflow.sklearn.log_model(clf, "model")

Every run logged this way becomes searchable and comparable later — which hyperparameters were used, what accuracy resulted, and where the saved model file lives. Combined with Git tracking the code and DVC tracking the data, this closes the loop: every experiment is now traceable back to the exact code, data, and settings that produced it.

Tracking Experiments Natively With DVC

DVC also has a lighter-weight, built-in alternative for teams who want to stay inside a single tool: the dvc exp command family. This lets you run many quick variations of a pipeline and compare their results side by side, without creating a separate Git commit for every single attempt.

dvc exp run --set-param train.n_estimators=200
dvc exp run --set-param train.n_estimators=300
dvc exp show   # Compare every run's params and metrics in one table

Putting It All Together: A Three-Layer Tracking Stack

If DVC stores the real data outside of Git, an obvious question follows: outside of Git, but where exactly? The answer is a remote — any storage location DVC has been configured to push data to and pull data from.

The same way GitHub provides a shared home for Git repositories, a DVC remote provides a shared home for the actual datasets and model files. DVC is deliberately flexible about what counts as a remote — it can be a folder on another hard drive, a private Google Drive folder, or proper cloud object storage like Amazon S3, Google Cloud Storage, or Azure Blob Storage.

Configuring a Remote

# Give the remote a name and a default flag (-d)
dvc remote add -d storage s3://my-bucket/dvcstore

# Save this configuration so teammates inherit it too
git add .dvc/config
git commit -m "Configure DVC remote storage"

# Send the actual data up to that bucket
dvc push

Once configured and committed, every teammate who clones the Git repository automatically inherits the same remote configuration. They simply run dvc pull and the correct data flows down to their machine — no manual file transfers, no shared network drives passed around by hand, no “can you re-send me that folder?” messages.

Automating Cloud Logins for CI/CD

For a human working locally, authenticating to a remote like Google Drive is as simple as a one-time browser login. But an automated pipeline running on a server has no browser and no human to click “allow.” For this situation, cloud providers offer Service Accounts — a kind of restricted, key-based virtual user that can be granted access to just the specific storage folder it needs, with no login screen required. This is precisely what allows DVC to push and pull data automatically inside a CI/CD pipeline, which the next-but-one section covers in detail.

Manually running dvc add and dvc push after every training run works fine for a solo project, but it does not scale to a team that retrains models regularly. The fix is the same one traditional software adopted years ago: CI/CD — Continuous Integration and Continuous Delivery — automated pipelines that run on a server every time something meaningful changes.

Tools such as GitHub Actions and CircleCI can be configured to automatically retrain a model, version the new weights with DVC, and push the updated files to remote storage and to Git — all without a human typing a single command.

A Minimal GitHub Actions Workflow

name: Train and Version Model

on:
  push:
    branches: [main]

jobs:
  train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Install dependencies
        run: pip install -r requirements.txt
      - name: Train model and track version
        run: |
          python train_model.py
          dvc add models/model.h5
          git add models/model.h5.dvc
          git commit -m "Add new model version"
          dvc push

This workflow automatically trains a fresh model the moment new code lands on the main branch, versions that new model with DVC, and pushes both the model itself and its tracking metadata to their respective remotes — fully automated, fully traceable.

The Same Idea, Applied to Large Language Models

This pattern scales up naturally to far heavier workloads, including fine-tuning large language models. A common technique called LoRA (Low-Rank Adaptation) makes this practical: instead of updating every one of a model’s billions of parameters, LoRA adds and trains only a small set of lightweight “adapter” weights, dramatically cutting the compute and storage needed for each experiment — which makes those adapter files an excellent candidate for frequent, automated DVC versioning inside a CI/CD pipeline.

Authentication Without a Human in the Loop

Because CI servers cannot click through a browser login, the credentials for cloud storage are usually supplied as encrypted environment variables configured once in the CI platform’s settings — values such as a Service Account key, a bucket folder ID, and Git credentials for committing back the results. The pipeline decodes and uses these secrets only for the duration of the run, then discards them.

DVC is a popular and well-tested answer to ML’s version control problem, but it is honestly worth weighing its trade-offs and knowing what else exists before committing a team to any single tool.

How DVC Compares to the Alternatives

These tools are not mutually exclusive — in fact, the most common real-world setup combines several of them: Git for code, DVC for data and model artifacts, and MLflow or a similar platform layered on top for rich experiment comparison. Each tool focuses on the piece it does best.

Best Practices Worth Adopting From Day One

• Never commit raw data blobs directly into Git — always route them through DVC or an equivalent tool
• Always commit the small .dvc metafiles to Git — they are the source of truth linking data to code
• Add basic data validation early, to catch schema or format changes before they reach training
• Keep a tiny sample dataset inside the repository for quick experimentation, while the full dataset lives remotely
• Store cloud credentials as environment variables or CI secrets — never hardcode them into a script
• Use dvc gc (garbage collection) carefully — it can delete cached versions other teammates still rely on

Version control for AI has matured enormously since DVC first appeared in 2017 — but the underlying challenge keeps growing right alongside the field itself, as datasets and models both keep getting bigger.

Reasons for Optimism

Challenges That Remain

• Ever-Larger Models: As models grow from millions to billions of parameters, even pointer-based systems must work harder to track checkpoints efficiently.
• Multi-Team Scale: Tools built for a single Git-centered project can strain when dozens of teams share the same enormous, constantly evolving datasets.
• The Learning Curve: Asking data scientists to learn a second tool alongside Git remains real friction, even when the commands are intentionally familiar.
• Cost of Storage at Scale: Versioning every dataset and checkpoint means keeping many large copies somewhere, and that storage is not free.
• Governance Catching Up: As regulations increasingly require organizations to prove exactly which data trained which model, version control is shifting from a nice-to-have into a compliance necessity.

Sources & References