Reproducibility in ML: Seeds, Configs and Pipelines

Reproducibility sounds like an academic virtue until a team argues for a week over whether a new model is better, and then discovers that rerunning the old one gives a different number anyway. A result that cannot be repeated cannot be compared, debugged or audited.

Most non-determinism in machine learning comes from a few places: random seeds, the order of data, the configuration, and the software environment. This guide measures each on a real model and shows what to record.

The experiment

The model predicts daily bike rentals from the UCI Bike Sharing data, released under CC BY 4.0 (UCI: Bike Sharing). It is a random forest, which uses randomness to sample rows and features, so it shows the effects clearly.

import hashlib
import json
import platform
from importlib.metadata import version

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error

day = pd.read_csv("data/day.csv")                               # UCI Bike Sharing (CC BY 4.0)
FEATURES = ["season", "yr", "mnth", "holiday", "weekday", "workingday", "weathersit", "temp", "atemp", "hum", "windspeed"]
train, test = day[day.dteday < "2012-07-01"], day[day.dteday >= "2012-07-01"]


def fit_and_score(data, seed):
    model = RandomForestRegressor(n_estimators=100, random_state=seed).fit(data[FEATURES], data.cnt)
    return model.predict(test[FEATURES])


# 1. No seed: two identical-looking runs disagree
a, b = fit_and_score(train, None), fit_and_score(train, None)
print("no seed, two runs give identical predictions:", np.array_equal(a, b))

# 2. A fixed seed: the runs agree exactly
a, b = fit_and_score(train, 42), fit_and_score(train, 42)
print("seed 42, two runs give identical predictions:", np.array_equal(a, b),
      f"| test MAE {mean_absolute_error(test.cnt, a):.1f}")

# 3. Same seed, same rows, different order: still a different model
shuffled = train.sample(frac=1, random_state=0)
c = fit_and_score(shuffled, 42)
print("seed 42 on the same rows shuffled, identical predictions:", np.array_equal(a, c),
      f"| test MAE {mean_absolute_error(test.cnt, c):.1f}")

# 4. How much do results move between seeds? Report a spread, not one lucky number
maes = [mean_absolute_error(test.cnt, fit_and_score(train, s)) for s in range(10)]
print(f"ten seeds: test MAE from {min(maes):.1f} to {max(maes):.1f}, mean {np.mean(maes):.1f}")

# 5. A run manifest: what someone needs to rerun this exactly
config = {"model": "RandomForestRegressor", "n_estimators": 100, "random_state": 42,
          "train_until": "2012-07-01", "features": FEATURES}
manifest = {
    "data_sha256": hashlib.sha256(open("data/day.csv", "rb").read()).hexdigest()[:16],
    "config_sha256": hashlib.sha256(json.dumps(config, sort_keys=True).encode()).hexdigest()[:16],
    "python": platform.python_version(),
    "packages": {p: version(p) for p in ["scikit-learn", "numpy", "pandas"]},
}
print(json.dumps(manifest, indent=2))

no seed, two runs give identical predictions: False
seed 42, two runs give identical predictions: True | test MAE 885.1
seed 42 on the same rows shuffled, identical predictions: False | test MAE 896.4
ten seeds: test MAE from 877.5 to 915.9, mean 898.8
{
  "data_sha256": "a6bcf826782d3c0f",
  "config_sha256": "3b04aa799df83714",
  "python": "3.11.9",
  "packages": {
    "scikit-learn": "1.9.1",
    "numpy": "2.4.6",
    "pandas": "3.0.6"
  }
}

What the output shows

• No seed, no reproducibility. Two runs of identical code on identical data gave different predictions.
• A seed fixes it, within one environment. With random_state=42, two runs matched exactly. scikit-learn's own guidance is to pass an integer and remove any use of random_state=None when results must be reproducible (scikit-learn: common pitfalls).
• Row order is part of the data. The same seed on the same rows in a different order gave a different model, with test error moving from 885.1 to 896.4. A query without a stable sort can change a model.
• Seeds alone move results noticeably. Across ten seeds, test error ranged from 877.5 to 915.9 rentals a day. Two configurations whose scores differ by less than that spread have not been shown to differ at all. Compare them across several seeds.
• A manifest makes a run repeatable. Short fingerprints of the data and configuration, plus the Python and package versions, identify exactly what produced the result.

The sources of variation, and the fixes

Deep learning and GPUs

Classical models on a CPU, like the forest above, can be reproduced bit for bit. Deep learning is harder. PyTorch's documentation states plainly that completely reproducible results are not guaranteed across releases, individual commits or platforms, and offers torch.use_deterministic_algorithms to choose deterministic algorithms where they exist (PyTorch: reproducibility). Deterministic settings can also run more slowly.

For such models, aim for statistical reproducibility instead: the same evaluation result within a stated tolerance, demonstrated across several seeds.

Comparing models fairly

The seed spread above has a practical consequence. In the MLflow tutorial, two configurations scored 885 and 906 with a single seed. That gap is inside the range one configuration covered across ten seeds here. Before calling one model better, train each with several seeds and compare the distributions, or use a held-out set large enough that the difference is clearly larger than the noise.

One master seed, many streams

Pipelines draw random numbers in several places: the data split, sampling, the model, augmentation. Reusing one seed everywhere can correlate them, and hard-coding many seeds is error-prone. A cleaner pattern is one master seed in the configuration from which every component gets its own stream. NumPy supports this directly: a SeedSequence can spawn independent child generators, with very high probability of independence (NumPy: parallel random number generation). Record the master seed, and every stream can be recreated.

When a result will not reproduce

Work through the causes in order, cheapest first:

1. Compare data fingerprints. A different hash ends the search.
2. Compare configuration fingerprints and the code commit.
3. Compare package versions and the Python version.
4. Check the seeds, including in data splitting and sampling.
5. Check hardware and parallelism, GPUs and thread counts last.

A manifest like the one above turns each step into a comparison of two lines.

Making it routine

• Put seeds and parameters in a configuration file, never scattered through code.
• Log a manifest with every run: data hash, configuration hash, commit and package versions. Experiment trackers can store it with the run.
• Version the data. See data versioning with DVC.
• Rebuild something old now and then to prove you can. See model versioning with MLflow and DVC.
• Pin the environment with lock files or container images.

Common mistakes

• Seeding only the model, not the data split or sampling.
• Unsorted queries feeding training data.
• Comparing single runs and declaring a winner inside the noise.
• Unpinned dependencies, so a rebuild months later uses different libraries.
• Expecting bit-for-bit results on GPUs without deterministic settings.

Reproducible pipelines, versioning and experiment tracking are central to the MLOps course.

Sources

• Common pitfalls and recommended practices (scikit-learn documentation) — scikit-learn
• Reproducibility (PyTorch documentation) — PyTorch
• Parallel random number generation (NumPy documentation) — NumPy
• Bike Sharing dataset (UCI Machine Learning Repository) — UCI Machine Learning Repository