Data Contracts: Reliable Pipelines Between Teams

Most pipeline failures are not caused by the pipeline. A column is renamed in the source application, a type changes, a value that was never null starts arriving empty, and a dashboard three systems downstream is wrong by morning. The team that made the change did not know anyone depended on it. A data contract is the fix for that ignorance: an explicit, versioned agreement between the team that produces a dataset and the teams that consume it, checked automatically.

What a contract contains

A contract is more than a schema. The Open Data Contract Standard, maintained by Bitol under the Linux Foundation, defines a YAML format whose sections show the scope (ODCS):

• Schema: columns, types, keys and meanings.
• Data quality: rules the data must meet, with thresholds.
• Service levels: how fresh the data will be, and how available.
• Team, roles and support: who owns the dataset and how to reach them.
• Infrastructure: where the data lives.

The essential ingredients are the first three plus an owner: what the data looks like, how good it will be, when it will arrive, and who is accountable.

A contract meeting real data

Here is a small contract for New York's yellow taxi trip data (NYC TLC: trip record data), covering the seven columns a consumer relies on and three quality rules, checked against the January files of three different years. The same check would run in a producer's release pipeline:

import urllib.request
from pathlib import Path
import duckdb
import pyarrow.parquet as pq

# the contract: what the producer promises, written down and versioned with the code
CONTRACT = {
    "dataset": "yellow_taxi_trips", "version": "1.0.0", "owner": "trips-platform team",
    "schema": {                                  # column: type the consumer relies on
        "tpep_pickup_datetime": "timestamp", "tpep_dropoff_datetime": "timestamp",
        "PULocationID": "int32", "payment_type": "int64", "passenger_count": "int64",
        "fare_amount": "double", "Airport_fee": "double",
    },
    "quality": [                                 # (rule, SQL counting violations, share of rows allowed to break it)
        ("pickup inside the file's month", "tpep_pickup_datetime < TIMESTAMP '{start}' OR tpep_pickup_datetime >= TIMESTAMP '{start}' + INTERVAL 1 MONTH", 0.001),
        ("fare is not negative", "fare_amount < 0", 0.02),
        ("payment type is a known code", "payment_type NOT IN (0, 1, 2, 3, 4, 5, 6)", 0.0),
    ],
}

def check(path, start):
    actual = {f.name: str(f.type).split("[")[0] for f in pq.read_schema(path)}
    problems = [f"missing column {c}" for c in CONTRACT["schema"] if c not in actual]
    problems += [f"{c} is {actual[c]}, contract says {t}" for c, t in CONTRACT["schema"].items() if c in actual and actual[c] != t]
    extra = len([c for c in actual if c not in CONTRACT["schema"]])     # columns the contract does not cover
    total = duckdb.sql(f"SELECT COUNT(*) FROM '{path}'").fetchone()[0]
    for rule, violation, allowed in CONTRACT["quality"]:
        bad = duckdb.sql(f"SELECT COUNT(*) FROM '{path}' WHERE {violation.format(start=start)}").fetchone()[0]
        if bad / total > allowed:
            problems.append(f"{rule}: {bad / total:.2%} of rows, allowed {allowed:.1%}")
    return problems, extra

for year in (2022, 2024, 2025):
    path = f"yellow_tripdata_{year}-01.parquet"
    if not Path(path).exists():
        urllib.request.urlretrieve(f"https://d37ci6vzurychx.cloudfront.net/trip-data/{path}", path)
    problems, extra = check(path, f"{year}-01-01")
    print(f"{year}: {'BREAKS the contract' if problems else 'meets the contract'} ({extra} columns outside it, ignored)")
    for line in problems:
        print("   breaking:", line)

2022: BREAKS the contract (13 columns outside it, ignored)
   breaking: missing column Airport_fee
   breaking: PULocationID is int64, contract says int32
   breaking: passenger_count is double, contract says int64
2024: meets the contract (12 columns outside it, ignored)
2025: BREAKS the contract (13 columns outside it, ignored)
   breaking: fare is not negative: 4.15% of rows, allowed 2.0%

What the check caught

2024 meets the contract, which was written against that year's file. The columns it does not mention are ignored: a contract covers what consumers depend on, not everything the producer publishes.

2022 breaks it on shape. The fee column has a different name, airport_fee in lower case, so the contracted Airport_fee is missing, and two columns have different types. Any consumer code written for the 2024 shape would fail or silently mis-read this file. Schema evolution looks at these changes in detail.

2025 breaks it on quality, not shape. Every contracted column is present with the right type, and the year's new congestion fee column is simply outside the contract, an additive change. But 4.15% of rows have a negative fare, against an allowed 2%. A schema check alone would have passed this file. The contract's quality rule is what notices that something about the data changed, and gives the consumer grounds to ask the producer what.

That is the argument for contracts in one result: shape and content both matter, and both can be checked before data is published instead of after a report is wrong.

Where contracts are enforced

A contract nobody checks is documentation. Enforcement points, from earliest to latest:

• In the producer's CI: a change that would break the contract fails the build, before it is merged. This is the most valuable point, because the producer finds out first.
• At the boundary: a schema registry rejects incompatible message schemas, backward compatibility being the default rule (Confluent: schema evolution).
• In the transformation layer: dbt model contracts define a model's columns, types and constraints, and with the contract enforced the model fails to build if it does not match; removing a column or changing a type is a breaking change (dbt docs: model contracts). The dbt tutorial introduces the tool.
• At ingestion, by the consumer: the last line of defence, as in the script above, quarantining data that fails.

dbt's documentation draws a useful line: contracts fix the shape of data up front, while data tests check content after it is built and allow thresholds and severities. Use both; data quality testing covers the test side.

Changing a contract

Contracts must be able to change, or teams will route around them.

• Additive changes, such as a new optional column, are non-breaking: publish them freely, as a minor version.
• Breaking changes, such as removing or renaming a column, changing a type or loosening a guarantee, need a new major version, notice to consumers, and a period in which old and new are both available.
• Version the contract with the code that produces the data, so every release is checked against the version it claims to meet.

Starting without a programme

Contracts fail when introduced as bureaucracy. A workable start:

1. Pick the one or two datasets whose breakage hurts most.
2. Write down what consumers actually use: columns, types, a freshness expectation and two or three quality rules.
3. Agree an owner on the producing side.
4. Add the check to the producer's pipeline, as a warning first, then blocking.
5. Extend to other datasets only where it has paid off.

The format matters less than the habit. A YAML file in a repository, checked on every change, with a named owner, is a data contract.

Data quality and validation, data security and compliance basics are part of the data governance module of the Data Engineering course.

Sources

• Open Data Contract Standard (Bitol, Linux Foundation) — Bitol
• Model contracts (dbt documentation) — dbt Labs
• Schema evolution and compatibility (Confluent documentation) — Confluent
• TLC Trip Record Data (NYC Taxi and Limousine Commission) — NYC Taxi and Limousine Commission