Serving ML Models with FastAPI and Docker

A trained model is useful to one notebook. Behind an API it becomes useful to every application that can send a request. The serving layer is thin, but small decisions in it decide whether the service is fast, safe and easy to operate: when the model loads, what inputs it accepts, how it reports its state, and which version answered.

This guide builds a prediction service for a bike-demand model with FastAPI and shows how to package it.

Design decisions

The service

The model predicts daily rentals from the UCI Bike Sharing data, released under CC BY 4.0 (UCI: Bike Sharing). The script first saves a model file, as a training pipeline would, then defines the service and calls it with FastAPI's test client, which starts the application exactly as a server does.

from contextlib import asynccontextmanager
from typing import Annotated

import pandas as pd
import skops.io as sio
from fastapi import FastAPI
from fastapi.testclient import TestClient
from pydantic import BaseModel, Field
from sklearn.ensemble import RandomForestRegressor

FEATURES = ["season", "yr", "mnth", "holiday", "weekday", "workingday", "weathersit", "temp", "atemp", "hum", "windspeed"]
MODEL_FILE, MODEL_VERSION = "bike-demand-v3.skops", "bike-demand:3"

# The training pipeline's output: a model file (trained here so the example is self-contained)
day = pd.read_csv("data/day.csv")                                # UCI Bike Sharing (CC BY 4.0)
train = day[day.dteday < "2012-07-01"]
sio.dump(RandomForestRegressor(n_estimators=100, random_state=42).fit(train[FEATURES], train.cnt), MODEL_FILE)


class Day(BaseModel):
    # the input contract: every feature with its allowed range
    season: int = Field(ge=1, le=4)
    yr: int = Field(ge=0, le=1)
    mnth: int = Field(ge=1, le=12)
    holiday: int = Field(ge=0, le=1)
    weekday: int = Field(ge=0, le=6)
    workingday: int = Field(ge=0, le=1)
    weathersit: int = Field(ge=1, le=4)
    temp: float = Field(ge=0, le=1)
    atemp: float = Field(ge=0, le=1)
    hum: float = Field(ge=0, le=1)
    windspeed: float = Field(ge=0, le=1)


state = {}


@asynccontextmanager
async def lifespan(app):
    # load once at start-up, not on every request
    state["model"] = sio.load(MODEL_FILE, trusted=["sklearn.tree._tree.Tree"])
    yield
    state.clear()


app = FastAPI(title="bike-demand", lifespan=lifespan)


@app.get("/health")
def health():
    return {"status": "alive"}                                  # the process is up


@app.get("/ready")
def ready():
    return {"ready": "model" in state, "model_version": MODEL_VERSION}   # it can serve predictions


def predict(days):
    frame = pd.DataFrame([d.model_dump() for d in days])[FEATURES]
    return [round(float(p)) for p in state["model"].predict(frame)]


@app.post("/predict")
def predict_one(day_in: Day):
    return {"rentals": predict([day_in])[0], "model_version": MODEL_VERSION}


@app.post("/predict/batch")
def predict_many(days: Annotated[list[Day], Field(min_length=1, max_length=500)]):
    return {"rentals": predict(days), "model_version": MODEL_VERSION}


with TestClient(app) as client:                                  # runs the start-up code, like a real server
    print("health:", client.get("/health").json(), "| ready:", client.get("/ready").json())

    july = day[day.dteday >= "2012-07-01"].head(3)
    first = july[FEATURES].iloc[0].to_dict()
    print("one day:", client.post("/predict", json=first).json(), "| actual", int(july.cnt.iloc[0]))
    batch = client.post("/predict/batch", json=july[FEATURES].to_dict("records")).json()
    print("three days:", batch["rentals"], "| actual", july.cnt.tolist())

    bad = client.post("/predict", json={**first, "season": 7})
    print("season 7:", bad.status_code, bad.json()["detail"][0]["msg"])
    missing = client.post("/predict", json={k: v for k, v in first.items() if k != "temp"})
    print("no temperature:", missing.status_code, missing.json()["detail"][0]["msg"])
    too_many = client.post("/predict/batch", json=[first] * 501)
    print("501 days in one call:", too_many.status_code, too_many.json()["detail"][0]["msg"])

health: {'status': 'alive'} | ready: {'ready': True, 'model_version': 'bike-demand:3'}
one day: {'rentals': 6097, 'model_version': 'bike-demand:3'} | actual 5531
three days: [6097, 6361, 6225] | actual [5531, 6227, 6660]
season 7: 422 Input should be less than or equal to 4
no temperature: 422 Field required
501 days in one call: 422 List should have at most 500 items after validation, not 501

What the output shows

• The service reported itself alive and ready, with the model version it would use.
• Predictions came back with their version. For the first three days of July 2012 the model predicted 6,097, 6,361 and 6,225 rentals; the actual counts were 5,531, 6,227 and 6,660.
• The contract rejected bad input before the model saw it. A season of 7, a missing temperature and a batch of 501 days each returned status 422 with a message saying exactly what was wrong.

Loading once, at start-up

The model loads inside FastAPI's lifespan function. The FastAPI documentation uses this exact case as its example: code that runs once before the application starts receiving requests, for shared resources such as a machine learning model. It also avoids loading the model during unrelated tests that merely import the module (FastAPI: lifespan events).

Health and readiness are different

/health answers "is the process alive?" /ready answers "can it serve predictions?" Orchestrators act differently on each. In Kubernetes, failing liveness probes lead to a restart, while failing readiness probes only stop traffic being sent to the container (Kubernetes: probes). A service still loading a large model should be alive but not ready. Restarting it would only start the load again.

Versioning the contract

Two things change over time, and they need separate versions. The model changes whenever it is retrained; the version in each response records that. The input contract changes only when features are added, removed or redefined, and that breaks callers. Keep the contract stable while models come and go, and when it must change, publish a new path such as /v2/predict and keep the old one running until clients have moved.

Where the time goes

A prediction request spends time on the network, on validating the input, on building the feature frame and on the model itself. For a small forest the model is often not the slowest part. Measure the 95th percentile latency, not just the average, and look at each step before optimising. Batch endpoints spread the fixed costs across many rows; a lean runtime such as ONNX can cut the model's share.

Packaging with Docker

FastAPI's documentation builds the image from an official Python image and starts the app with fastapi run in the exec form of CMD (FastAPI: Docker). The model file ships in the image alongside pinned dependencies:

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py bike-demand-v3.skops ./
CMD ["fastapi", "run", "app.py", "--port", "8080"]

Pin the same scikit-learn version that trained the model in requirements.txt; the saved file is not guaranteed to load under another version. See model packaging and Docker for machine learning.

Operating the service

Log each request's model version, latency and a summary of inputs, without personal data. Watch the distribution of inputs and predictions for drift. Roll out new model versions gradually, with the old one ready to take traffic back. See monitoring ML models in production. Whether an API is the right shape at all depends on how fresh predictions must be. See batch vs real-time inference.

Common mistakes

• Loading the model on every request.
• Accepting any JSON and letting the model fail on bad input.
• One endpoint for liveness and readiness, so a slow start triggers restarts.
• No version in responses, so predictions cannot be traced.
• Unpinned dependencies in the image.
• Unlimited batch sizes that let one caller exhaust memory.

Serving models with Flask or FastAPI, Docker and deployment strategies are covered in the Model Deployment and Containerization modules of the MLOps course.

Sources

• Lifespan Events (FastAPI documentation) — FastAPI
• Liveness, Readiness, and Startup Probes (Kubernetes documentation) — Kubernetes
• FastAPI in Containers - Docker (FastAPI documentation) — FastAPI
• Bike Sharing dataset (UCI Machine Learning Repository) — UCI Machine Learning Repository