Experiment Tracking: MLflow vs Weights & Biases

Experiment tracking stops results from living in notebook cells and memory. Two tools dominate the choice: MLflow, the open-source platform started by Databricks, and Weights & Biases, usually called W&B, a hosted platform with an open-source client. Both record parameters, metrics and artefacts. The real differences are in where the data lives, who runs the server, and what comes around the core.

The same experiment, logged twice

The script trains two models on the UCI Bike Sharing data, released under CC BY 4.0, and logs both runs to MLflow and to W&B. It uses no account and no network: MLflow writes to a local database, and W&B runs in offline mode, which saves runs on disk to be uploaded later with wandb sync (W&B: run offline).

import os
import shutil
from pathlib import Path

import mlflow
import pandas as pd
import wandb
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error

os.environ["WANDB_MODE"] = "offline"        # no account, no network: W&B keeps the run on disk for a later `wandb sync`
os.environ["WANDB_SILENT"] = "true"
Path("compare.db").unlink(missing_ok=True)       # start every run from a clean slate
shutil.rmtree("wandb", ignore_errors=True)

day = pd.read_csv("data/day.csv")            # UCI Bike Sharing (CC BY 4.0)
FEATURES = ["season", "yr", "mnth", "holiday", "weekday", "workingday", "weathersit", "temp", "atemp", "hum", "windspeed"]
train, test = day[day.dteday < "2012-07-01"], day[day.dteday >= "2012-07-01"]

mlflow.set_tracking_uri("sqlite:///compare.db")
mlflow.set_experiment("bike-compare")

# The same two experiments, logged to both tools with almost the same calls
for trees in (50, 200):
    params = {"n_estimators": trees, "max_depth": 8}
    model = RandomForestRegressor(**params, random_state=42).fit(train[FEATURES], train.cnt)
    mae = mean_absolute_error(test.cnt, model.predict(test[FEATURES]))

    with mlflow.start_run(run_name=f"forest-{trees}"):
        mlflow.log_params(params)
        mlflow.log_metric("test_mae", mae)

    with wandb.init(project="bike-compare", name=f"forest-{trees}", config=params, dir=".") as run:
        run.log({"test_mae": mae})

# MLflow: the runs live in a local database you can query and serve yourself
runs = mlflow.search_runs(order_by=["metrics.test_mae ASC"])
print("MLflow, read back from the local database:")
for _, r in runs.iterrows():
    print(f"   {r['tags.mlflow.runName']:<12} trees {r['params.n_estimators']:>4}  test MAE {r['metrics.test_mae']:.1f}")

# W&B: offline runs are files waiting to be synced to a W&B server, hosted or self-managed
offline = sorted(Path("wandb").glob("offline-run-*"))
print(f"\nW&B, offline: {len(offline)} runs saved on disk")
for d in offline:
    kinds = sorted({p.suffix or p.name for p in d.rglob("*") if p.is_file()})
    print(f"   one run folder holds: {', '.join(kinds)}")
    break
print("   to view them in the W&B app: wandb sync <run folder>")

MLflow, read back from the local database:
   forest-200   trees  200  test MAE 905.8
   forest-50    trees   50  test MAE 909.4

W&B, offline: 2 runs saved on disk
   one run folder holds: .log, .syncstate, .txt, .wandb
   to view them in the W&B app: wandb sync <run folder>

What the output shows

• The logging code is nearly identical. Both take a run name, a dictionary of parameters and a metric. Switching later is a small change to training code.
• MLflow's data was immediately usable. The runs were read back from the local database and sorted by error, with nothing running except the script. MLflow records runs locally or to a shared tracking server backed by a database and artefact storage (MLflow: tracking).
• W&B's offline runs were files waiting for a server. Each run folder held a log, a sync state and the run's data in W&B's own format. To compare them in W&B's dashboards, they have to be synced to a W&B server, hosted or self-managed.

That difference is the heart of the comparison. MLflow's open-source project includes the server and the UI. W&B's open-source part is the client. The dashboard is a service.

Side by side

Sources: MLflow's repository is licensed under Apache 2.0 (GitHub: MLflow), and Databricks offers a fully managed, hosted MLflow (Databricks: MLflow). The W&B client is MIT-licensed, and W&B runs as a cloud service, as Dedicated Cloud or as a self-managed server (GitHub: wandb). W&B tracks experiments with a few lines of code and shows them in an interactive dashboard or through a public API (W&B: experiments), and W&B Sweeps automates Bayesian, grid and random hyperparameter search (W&B: sweeps). W&B's documentation is now published on CoreWeave's site.

Choosing

Either choice is reversible if you keep the habits that matter: log every run, record the data version and the code commit, and keep the logging calls in one helper function rather than scattered through notebooks. For the hands-on MLflow workflow, see the MLflow tutorial. For linking runs to data versions, see model versioning with MLflow and DVC.

Questions to ask before choosing

1. Who will run the server, keep it patched and back it up?
2. Where may experiment data be stored, given your contracts and regulations?
3. How many people need to see, compare and discuss results?
4. Do you need built-in hyperparameter search, or will a library do?
5. What happens to the history if the team changes tools in two years?

The answers usually decide the matter faster than a feature list.

Lock-in and exports

The tracked history becomes valuable over time, so check how it leaves. MLflow's data sits in your database and storage, in documented formats. W&B offers export through its public API. Before committing, try exporting a project's runs, metrics and artefacts, and make sure the result is something your team could use without the original tool.

Common mistakes

• Choosing by dashboard screenshots instead of by where the data lives.
• Logging different things in different projects, which makes runs impossible to compare.
• No data version or commit in the runs.
• Assuming offline runs are backed up before they are synced.
• Scattering tracking calls through code, which makes switching tools painful.

Experiment tracking with MLflow, the registry and model versioning are covered in the Experiment Tracking and Model Management module of the MLOps course.

Sources

• MLflow (GitHub) — MLflow project
• ML Experiment Tracking (MLflow documentation) — MLflow
• MLflow on Databricks (Databricks documentation) — Databricks
• Experiments overview (Weights & Biases documentation, CoreWeave) — CoreWeave
• How do I run Weights & Biases offline? (CoreWeave documentation) — CoreWeave
• wandb (GitHub) — Weights & Biases
• Sweeps overview (Weights & Biases documentation, CoreWeave) — CoreWeave