MLOps Trending

Kubernetes for ML: Deploying Models at Scale

Kubernetes runs copies of your model service, replaces the ones that die and swaps versions without downtime, if the manifest is right. This guide deploys a concrete-strength model to a local cluster, kills a pod, rolls out a new version under continuous traffic with no failed requests, and rolls back.

GloryTecks Team

Training & Placement Faculty

25 May 2026 9 min read

Last updated: 8 October 2026

Share:

A model in a container answers requests on one machine. Kubernetes runs several copies of that container, spreads requests across them, replaces copies that die and swaps in a new version without stopping traffic. None of that is automatic, though: it depends on a few lines in the manifest that are easy to leave out. This guide deploys a small model service to a local cluster and checks each of those behaviours with real requests.

The model service

The model predicts the compressive strength of concrete from its mix, using the UCI Concrete Compressive Strength data: 1,030 mixes with ingredients such as cement, water and age, under CC BY 4.0 (UCI: concrete strength). Two versions are trained, one with 100 boosting rounds and one with 300, and each is baked into its own image. The service reports which version and which pod answered, so the experiment can see where requests go:

python
import os
import socket

import joblib
import pandas as pd
from fastapi import FastAPI

model = joblib.load("model.joblib")                     # baked into the image at build time
VERSION = os.environ.get("MODEL_VERSION", "unknown")
app = FastAPI()


@app.get("/health")                                     # the readiness probe calls this
def health():
    return {"status": "ok"}


@app.post("/predict")
def predict(rows: list[dict]):
    values = model.predict(pd.DataFrame(rows)[list(model.feature_names_in_)])
    return {"version": VERSION, "pod": socket.gethostname(), "strength_mpa": [round(float(v), 1) for v in values]}
dockerfile
FROM python:3.11-slim
WORKDIR /app
RUN pip install --no-cache-dir fastapi==0.142.2 uvicorn==0.54.0 scikit-learn==1.9.1 pandas==3.0.6 joblib==1.6.0
COPY app.py model.joblib ./
ARG MODEL_VERSION
ENV MODEL_VERSION=${MODEL_VERSION}
EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

How to build and test an image like this one is covered in serving ML models with FastAPI and Docker.

The manifest

Two objects do the work. A Deployment says how many copies (pods) of the container to run and how to replace them. A Service gives those pods one stable address and spreads requests across the ones that are ready.

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: strength-model
spec:
  replicas: 2
  selector:
    matchLabels:
      app: strength-model
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0          # never drop below the desired number of ready pods
      maxSurge: 1                # start one new pod at a time
  template:
    metadata:
      labels:
        app: strength-model
    spec:
      containers:
        - name: api
          image: strength-model:v1
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8000
          readinessProbe:          # no traffic until the model has loaded and /health answers
            httpGet:
              path: /health
              port: 8000
            periodSeconds: 2
          lifecycle:
            preStop:               # pause so the Service stops routing here before the server shuts down
              exec:
                command: ["sleep", "5"]
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              memory: 512Mi
---
apiVersion: v1
kind: Service
metadata:
  name: strength-model
spec:
  type: NodePort
  selector:
    app: strength-model
  ports:
    - port: 80
      targetPort: 8000
      nodePort: 30080

The lines that matter most for a model:

  • The readiness probe. The cluster only sends traffic to a pod once its readiness probe passes, which is exactly what a service that loads a model file at start-up needs. If the probe later fails, the pod's address is removed from the Service until it recovers (probe documentation). A liveness probe is different: it restarts a container that is stuck, and was not needed here.
  • The update strategy. maxUnavailable: 0 and maxSurge: 1 mean: never go below the requested number of ready pods, and add one new pod at a time. The defaults are 25% each, and the two cannot both be 0 (Deployment documentation).
  • The preStop pause. When a pod is being removed, the kubelet runs the preStop hook before stopping the server, while separately removing the pod from the Service's list of endpoints (pod lifecycle documentation). Those two things happen in parallel, so a short sleep gives the routing change time to spread before the server stops accepting connections.
  • Resources. The memory request tells the scheduler how much room a pod needs; the limit stops one leaking pod from taking the whole node. For a model, measure memory after loading and a burst of requests, then set the request a little above that.

The experiment

The cluster is kind, which runs cluster nodes as Docker containers on a laptop; kind load docker-image copies locally built images into it, so no registry is needed (kind: quick start). This config exposes the Service's node port on the laptop:

yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraPortMappings:           # expose the Service's node port on the laptop
      - containerPort: 30080
        hostPort: 30080

The script builds both versions, creates the cluster, and then tests five things:

python
import subprocess
import threading
import time

import joblib
import pandas as pd
import requests
from sklearn.ensemble import HistGradientBoostingRegressor


def sh(*command, check=True):
    return subprocess.run(command, capture_output=True, text=True, encoding="utf-8", errors="replace", check=check).stdout.strip()


def versions_answering(seconds_after):              # sample the Service after giving it time to drop old pods
    time.sleep(seconds_after)
    return sorted({requests.post(URL, json=rows, timeout=10).json()["version"] for _ in range(50)})


def ready_pods():
    return int(sh("kubectl", "get", "deployment", "strength-model", "-o", "jsonpath={.status.readyReplicas}") or 0)


mix = pd.read_csv("concrete.csv")                               # UCI Concrete Compressive Strength (CC BY 4.0)
features = [c for c in mix.columns if c != "strength_mpa"]
for version, rounds in (("v1", 100), ("v2", 300)):              # two model versions, two images
    model = HistGradientBoostingRegressor(max_iter=rounds, random_state=0).fit(mix[features], mix.strength_mpa)
    joblib.dump(model, "image/model.joblib")
    sh("docker", "build", "-q", "-t", f"strength-model:{version}", "--build-arg", f"MODEL_VERSION={version}", "image")

URL, rows = "http://127.0.0.1:30080/predict", mix[features].head(2).to_dict("records")
sh("kind", "create", "cluster", "--name", "ml", "--config", "kind.yaml")   # a one-node cluster inside Docker
try:
    for version in ("v1", "v2"):
        sh("kind", "load", "docker-image", f"strength-model:{version}", "--name", "ml")
    sh("kubectl", "apply", "-f", "model.yaml")
    sh("kubectl", "rollout", "status", "deployment/strength-model", "--timeout=300s")
    pods = {requests.post(URL, json=rows, timeout=10).json()["pod"] for _ in range(100)}
    print(f"1. Deployment with 2 replicas: {ready_pods()} ready; 100 requests to the Service were answered by {len(pods)} pods")

    sh("kubectl", "scale", "deployment/strength-model", "--replicas=4")
    sh("kubectl", "rollout", "status", "deployment/strength-model", "--timeout=300s")
    pods = {requests.post(URL, json=rows, timeout=10).json()["pod"] for _ in range(200)}
    print(f"2. Scaled to 4 replicas: {ready_pods()} ready; 200 requests were answered by {len(pods)} pods")

    victim = sorted(pods)[0]
    sh("kubectl", "delete", "pod", victim, "--wait=false")      # simulate a crashed pod
    start = time.perf_counter()
    time.sleep(1)
    while ready_pods() < 4 or victim in sh("kubectl", "get", "pods", "-o", "name"):
        time.sleep(1)
    print(f"3. Deleted one pod: Kubernetes replaced it; 4 ready again after {time.perf_counter() - start:.0f} s (varies)")

    answers, stop = [], threading.Event()

    def traffic():                                               # keep sending requests during the update
        while not stop.is_set():
            try:
                reply = requests.post(URL, json=rows, timeout=10)
                answers.append(reply.json()["version"] if reply.ok else f"HTTP {reply.status_code}")
            except requests.RequestException:
                answers.append("connection error")

    worker = threading.Thread(target=traffic)
    worker.start()
    sh("kubectl", "set", "image", "deployment/strength-model", "api=strength-model:v2")
    sh("kubectl", "rollout", "status", "deployment/strength-model", "--timeout=300s")
    stop.set()
    worker.join()
    failures = [a for a in answers if a not in ("v1", "v2")]
    print(f"4. Rolling update to v2 under continuous traffic: {len(failures)} of {len(answers)} requests failed (count varies); "
          f"versions seen {sorted(set(answers) - set(failures))}")
    print(f"   answering 10 s after the rollout finished: {versions_answering(10)}")

    sh("kubectl", "rollout", "undo", "deployment/strength-model")
    sh("kubectl", "rollout", "status", "deployment/strength-model", "--timeout=300s")
    print(f"5. kubectl rollout undo: answering 10 s after it finished: {versions_answering(10)}")
finally:
    sh("kind", "delete", "cluster", "--name", "ml", check=False)
text
1. Deployment with 2 replicas: 2 ready; 100 requests to the Service were answered by 2 pods
2. Scaled to 4 replicas: 4 ready; 200 requests were answered by 4 pods
3. Deleted one pod: Kubernetes replaced it; 4 ready again after 8 s (varies)
4. Rolling update to v2 under continuous traffic: 0 of 765 requests failed (count varies); versions seen ['v1', 'v2']
   answering 10 s after the rollout finished: ['v2']
5. kubectl rollout undo: answering 10 s after it finished: ['v1']

What happened

  • The Service spread the load. With two replicas, 100 requests were answered by both pods; after scaling to four, 200 requests reached all four. Scaling was one command, and the readiness probe kept new pods out of rotation until their model had loaded.
  • The Deployment healed itself. Deleting a pod simulated a crash. The Deployment noticed it had three pods instead of four and started a replacement, which was ready again within seconds. Nobody had to restart anything.
  • The rolling update dropped nothing. While a background thread sent requests as fast as it could, the Deployment replaced all four v1 pods with v2 pods one at a time, and not one request failed. For a while both versions answered, which is normal during a rolling update: if v2 returned a different response format, clients would see both formats mixed for that window.
  • Rollback was one command. kubectl rollout undo returned the Deployment to its previous template, and v1 answered again.

One detail from an earlier run of this script is worth knowing. When the requests were sent the instant kubectl rollout status reported success after the rollback, some still reached v2 pods. Those pods were in their preStop pause and still serving, while the Service's routing caught up. Nothing failed, but "rollout finished" does not mean "old version gone this second". The script now waits 10 seconds before checking which version answers.

What this does not show

A one-node kind cluster cannot show what happens when a whole machine fails, how pods spread across zones, or how a cloud load balancer behaves. It also leaves out autoscaling: a HorizontalPodAutoscaler can change the replica count based on CPU or custom metrics, but needs a metrics server and a realistic load to test. And the cluster only keeps the service up; it says nothing about whether v2's predictions are any good. That is a job for comparing versions before release and monitoring them afterwards.

Where Kubernetes fits

Kubernetes earns its complexity when you run several services, need zero-downtime releases and want the same deployment process everywhere. For a single model with modest traffic, a managed container service or a cloud ML endpoint may be simpler; deploying ML models on AWS, Azure and GCP compares those. Training workflows on a cluster are a different topic, covered in Kubeflow Pipelines. If containers are new to you, start with Docker for machine learning.

Common mistakes

  • No readiness probe, so traffic reaches pods that are still loading the model and gets errors.
  • Using the liveness probe for slow start-up, which restarts a pod that was only loading.
  • Leaving the default update strategy unchecked, so a quarter of capacity disappears during every release.
  • No memory request, so the scheduler packs too many model pods onto one node.
  • Reusing one image tag for every build, such as latest, which makes rollback ambiguous; give each model version its own image tag.

Why use Kubernetes for machine learning models?

To run several copies of a model service behind one address, replace crashed copies automatically, scale up or down with one command, and release new model versions without downtime, using the same process for every service.

What is the difference between a readiness probe and a liveness probe?

A readiness probe decides whether a pod receives traffic, which suits a model that takes time to load. A liveness probe decides whether a container should be restarted because it is stuck.

How do I update a model on Kubernetes without downtime?

Build a new image with its own tag, set the Deployment's image to it, and use a rolling update with maxUnavailable set to 0 and a readiness probe, so new pods take traffic only when ready and old ones leave only afterwards.

Can I practise Kubernetes without a cloud account?

Yes. Tools such as kind run a cluster inside Docker on a laptop, which is enough to practise Deployments, Services, probes, scaling and rollbacks.

Deploying model services to a cluster is part of the MLOps course.

Sources

Tags:#mlops#kubernetes#deploying#models#scale

GloryTecks Team

Training & Placement Faculty

The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.

Want to master MLOps?

Join GloryTecks’ expert-led MLOps training in Hyderabad with real-time projects and 100% placement support.

Explore course
Previous

CI/CD for Machine Learning: Pipelines That Work

Next

Docker for Machine Learning: A Practical Guide

Related articles

MLOps
13 May 2026 9 min

Monitoring ML Models in Production: A Complete Guide

A model that passed validation can fail quietly in production. This guide sets out what to monitor at each layer, how to cope with labels that arrive late and how to design alerts people act on, using a year of real sensor data in which the loudest alert was the least useful one.

GloryTecks Team
Read
MLOps
7 Apr 2026 7 min

Serving ML Models with FastAPI and Docker

Putting a model behind an API is where many projects stall. This guide builds a FastAPI prediction service that loads a model once at start-up, enforces an input contract, serves single and batch predictions, reports health and readiness separately, and shows the Dockerfile that packages it.

GloryTecks Team
Read
MLOps
12 Jun 2026 5 min

MLOps Roadmap 2026: Skills, Tools and Career Path

MLOps sits between data science, software engineering and operations, which makes it hard to know where to start. This roadmap orders the skills from foundations to production operations, gives each stage a project that proves the skill, and explains what the MLOps engineer role actually involves.

GloryTecks Team
Read