Data Versioning with DVC: A Hands-On Guide

A model is the product of code, data and settings. Git handles the code beautifully and the settings well enough. Data is the problem: large files bloat a repository, and changing a dataset in place destroys the version that trained last month's model. Without the old data, that model cannot be reproduced, explained or fairly compared with a new one.

DVC, Data Version Control, solves this by splitting the job. Git keeps a small pointer to each version of a dataset, and DVC keeps the data itself.

How DVC versions data

When you run dvc add on a file, DVC moves the data into its cache and writes a small, human-readable .dvc file that stands in for the data in Git (DVC: get started). You commit the pointer, not the data. Checking out an old commit brings back the old pointer, and dvc checkout brings back the matching data.

For a team, the cache is pushed to remote storage such as an object store, and teammates pull the versions they need.

Hands on

The script builds a real project from scratch with the UCI Bike Sharing data, daily rental counts for 2011 and 2012 licensed under CC BY 4.0 (UCI: Bike Sharing). It drives the same git and dvc commands you would type, so each step maps to a command line.

import json
import os
import shutil
import stat
import subprocess
import sys
from pathlib import Path

import pandas as pd

os.environ["DVC_NO_ANALYTICS"] = "1"            # no usage reporting from DVC
DVC = str(Path(sys.executable).with_name("dvc"))
day = pd.read_csv("data/day.csv")               # UCI Bike Sharing daily counts (CC BY 4.0)

project = Path("bike-project")


def make_writable(func, path, _):
    os.chmod(path, stat.S_IWRITE)               # Git marks its object files read-only on Windows
    func(path)


if project.exists():
    shutil.rmtree(project, onerror=make_writable)
project.mkdir()


def run(*cmd):
    out = subprocess.run(cmd, cwd=project, capture_output=True, text=True)
    if out.returncode:
        raise RuntimeError(out.stderr)
    return out.stdout


def git(*args):
    return run("git", *args)


git("init", "-q")
git("config", "user.email", "demo@example.com")
git("config", "user.name", "Demo")
run(DVC, "init", "-q")
(project / "data").mkdir()

# 1. Version 1 of the data: 2011 only. DVC stores the file; Git stores a small pointer to it.
day[day.dteday < "2012-01-01"].to_csv(project / "data/rides.csv", index=False)
run(DVC, "add", "-q", "data/rides.csv")
git("add", "data/rides.csv.dvc", "data/.gitignore", ".dvc", ".dvcignore")
git("commit", "-q", "-m", "Data: 2011 rides")
print("pointer file committed to Git:")
print((project / "data/rides.csv.dvc").read_text().strip())

# 2. Version 2: the full two years
day.to_csv(project / "data/rides.csv", index=False)
run(DVC, "add", "-q", "data/rides.csv")
git("commit", "-q", "-am", "Data: 2011-2012 rides")
print("\nGit history:", [line for line in git("log", "--format=%s").splitlines()])
print("rows now:", len(pd.read_csv(project / "data/rides.csv")))

# 3. Go back to version 1: Git restores the pointer, DVC restores the data it points to
git("checkout", "-q", "HEAD~1", "--", "data/rides.csv.dvc")
run(DVC, "checkout", "-q", "data/rides.csv.dvc")
print("rows after checking out the first version:", len(pd.read_csv(project / "data/rides.csv")))
git("checkout", "-q", "HEAD", "--", "data/rides.csv.dvc")
run(DVC, "checkout", "-q", "data/rides.csv.dvc")

# 4. A pipeline: DVC reruns a stage only when its code, data or parameters change
(project / "params.yaml").write_text("trees: 50\n")
(project / "train.py").write_text("""import json, pandas as pd, yaml
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error
p = yaml.safe_load(open("params.yaml"))
d = pd.read_csv("data/rides.csv")
cols = ["season", "yr", "mnth", "holiday", "weekday", "workingday", "weathersit", "temp", "atemp", "hum", "windspeed"]
train, test = d[d.dteday < "2012-07-01"], d[d.dteday >= "2012-07-01"]
m = RandomForestRegressor(n_estimators=p["trees"], random_state=42).fit(train[cols], train.cnt)
json.dump({"mae": round(mean_absolute_error(test.cnt, m.predict(test[cols])), 1)}, open("metrics.json", "w"))
""")
run(DVC, "stage", "add", "-q", "-n", "train", "-d", "train.py", "-d", "data/rides.csv", "-p", "trees",
    "-M", "metrics.json", f"\"{sys.executable}\" train.py")

for label in ["first run", "nothing changed", "trees changed to 200"]:
    if label.startswith("trees"):
        (project / "params.yaml").write_text("trees: 200\n")
    log = run(DVC, "repro")
    ran = "Running stage 'train'" in log
    mae = json.loads((project / "metrics.json").read_text())["mae"]
    print(f"{label:<22} stage ran: {'yes' if ran else 'no, skipped':<12} test MAE {mae}")

pointer file committed to Git:
outs:
- md5: f85615788cc71fc3c2c5fb26678e5551
  size: 28659
  hash: md5
  path: rides.csv

Git history: ['Data: 2011-2012 rides', 'Data: 2011 rides']
rows now: 731
rows after checking out the first version: 365
first run              stage ran: yes          test MAE 873.5
nothing changed        stage ran: no, skipped  test MAE 873.5
trees changed to 200   stage ran: yes          test MAE 885.0

What the output shows

• Git holds a fingerprint, not the data. The pointer file contains only an MD5 hash, the size, 28,659 bytes, and the file name. A 10 GB dataset would have a pointer just as small.
• Two versions, two commits. The history reads like code history: "Data: 2011 rides", then "Data: 2011-2012 rides", with 731 rows in the current file.
• Rolling back is two commands. Restoring the old pointer with Git and running dvc checkout brought back the 365-row version exactly.
• Pipelines rerun only when needed. The training stage ran the first time, was skipped when nothing had changed, and ran again when the number of trees changed in params.yaml. DVC decides by comparing hashes of the stage's dependencies and its parameter values with those recorded in dvc.lock (DVC: data pipelines).
• The metric moved with the parameter. Test error was 873.5 rentals a day with 50 trees and 885.0 with 200, a reminder that more trees is not automatically better.

A day with DVC

Once a project is set up, the routine is short:

• After changing data: dvc add the file, commit the updated pointer with Git, then dvc push the data to the remote.
• After pulling a colleague's work: git pull, then dvc pull to fetch the data the new pointers refer to.
• After changing code or parameters: dvc repro to rebuild whatever the change affects, and dvc metrics diff to compare the results with the last commit.

Each step pairs a Git action with a DVC action. Forgetting the DVC half is the most common way teams end up with pointers to data nobody can download.

When something else fits better

DVC suits files: CSVs, Parquet extracts, images, model weights. Some situations call for a different tool. Plain Git is enough for small files, though GitHub warns at 50 MiB and blocks files larger than 100 MiB (GitHub: large files). Data that lives in a lake or warehouse and changes continuously is better versioned there, for example with table formats that keep history. See Delta Lake explained.

What to version, and how

Working in a team

1. Configure a remote in shared storage and push after adding data.
2. Commit pointer files with the code that uses them, so one commit describes one state.
3. Run the pipeline in CI with dvc repro, so results are rebuilt from tracked inputs, not copied from someone's laptop.
4. Record data versions with experiments. A tracked run should name the commit or data hash it trained on. See the MLflow tutorial.

For how data versions combine with model versions and the registry, see model versioning with MLflow and DVC. For the broader habits that make results repeatable, see reproducibility in ML.

Common mistakes

• Committing large data files to Git before setting up DVC.
• Editing data in place without adding a new version.
• Forgetting to push the cache, so teammates have pointers but no data.
• Pipelines with undeclared dependencies, so DVC skips a stage that should rerun.
• Keeping parameters inside scripts instead of in a parameters file DVC can watch.

Data pipelines, versioning and workflow automation are covered in the ML Pipelines and Experiment Tracking modules of the MLOps course.

Sources

• Get Started with DVC (DVC documentation) — DVC
• Get Started: Data Pipelines (DVC documentation) — DVC
• About large files on GitHub (GitHub Docs) — GitHub
• Bike Sharing dataset (UCI Machine Learning Repository) — UCI Machine Learning Repository