MLOps

Prometheus and Grafana for ML Monitoring

Prometheus and Grafana are the usual open-source pair for watching services. This guide instruments a real model API, scrapes it with Prometheus 3.15, unit-tests a drift alert, provisions a Grafana 13 dashboard from files and replays two real weeks of requests to see what the setup catches and what it misses.

GloryTecks Team

Training & Placement Faculty

6 May 2026 11 min read

Last updated: 8 October 2026

Share:

Prometheus and Grafana are the usual open-source pair for watching services: Prometheus collects and stores metrics and evaluates alert rules, and Grafana draws the dashboards. They work for model services too, as long as you choose metrics that describe the model as well as the server, and accept what metrics cannot tell you.

This guide builds a complete local setup and runs it. A demand model is served with FastAPI and instrumented in Python. Prometheus 3.15 scrapes it, a recording rule and two alerts watch it, promtool unit-tests the drift alert, and a Grafana 13 dashboard is provisioned from files. Then two real weeks of requests are replayed.

How the pieces fit

Prometheus pulls metrics over HTTP from each target at a scrape interval, which defaults to one minute, stores them as time series and evaluates rules at an evaluation interval with the same default (Prometheus: configuration). Recording rules precompute frequently needed or expensive expressions into new series (Prometheus: recording rules). Alerting rules fire once a condition has held for a set time; routing the notifications is Alertmanager's job, outside this demo.

Grafana queries Prometheus and draws panels. Its provisioning system defines data sources and dashboards in files that can live in version control (Grafana: provisioning).

What a model service should expose

Google's SRE book names four golden signals for any user-facing system: latency, traffic, errors and saturation (Google SRE: monitoring distributed systems). A model service needs those plus two of its own: what it predicts, and what it receives.

QuestionMetric in this demoType
How many requests, and how many fail?bike_requests_total, by statusCounter
How long does a prediction take?bike_prediction_secondsHistogram
What is the model predicting?bike_predicted_rentalsHistogram
What inputs arrive?bike_input_tempHistogram
Which model, and what is normal for it?bike_reference_mean_prediction, by model_versionGauge

A counter only goes up, and a histogram counts observations in buckets and also exposes a running sum and count (Prometheus: metric types). That sum and count are what make an average prediction cheap to compute.

Labels need care. Every label combination is a separate series with memory, CPU, disk and network costs, and Prometheus suggests keeping a metric's cardinality below 10 as a general guideline (Prometheus: instrumentation). A model version is a good label. A user ID, a request ID or a raw feature value is not.

The service

The model predicts hourly bike rentals from weather and calendar fields in the UCI Bike Sharing data, released under CC BY 4.0 (UCI: Bike Sharing). It is trained on 2011 only.

python
import time

import pandas as pd
from fastapi import FastAPI, Request
from prometheus_client import Counter, Gauge, Histogram, make_asgi_app
from pydantic import BaseModel, Field
from sklearn.ensemble import RandomForestRegressor

FEATURES = ["season", "mnth", "hr", "holiday", "weekday", "workingday", "weathersit", "temp", "atemp", "hum", "windspeed"]
VERSION = "rf-2011"
history = pd.read_csv("hour.csv")                                   # UCI Bike Sharing, hourly (CC BY 4.0)
train = history[history.yr == 0]                                    # trained on 2011 only
model = RandomForestRegressor(n_estimators=100, min_samples_leaf=2, random_state=0).fit(train[FEATURES], train.cnt)

REQUESTS = Counter("bike_requests_total", "Prediction requests by HTTP status", ["status"])
LATENCY = Histogram("bike_prediction_seconds", "Time spent in model.predict",
                    buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25])
PREDICTED = Histogram("bike_predicted_rentals", "Predicted rentals for the hour",
                      buckets=[25, 50, 100, 150, 200, 300, 400, 600])
TEMPERATURE = Histogram("bike_input_temp", "Normalised temperature received",
                        buckets=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9])
REFERENCE = Gauge("bike_reference_mean_prediction", "Mean prediction on the training data", ["model_version"])
REFERENCE.labels(VERSION).set(model.predict(train[FEATURES]).mean())


class Hour(BaseModel):
    season: int = Field(ge=1, le=4)
    mnth: int = Field(ge=1, le=12)
    hr: int = Field(ge=0, le=23)
    holiday: int = Field(ge=0, le=1)
    weekday: int = Field(ge=0, le=6)
    workingday: int = Field(ge=0, le=1)
    weathersit: int = Field(ge=1, le=4)
    temp: float = Field(ge=0, le=1)
    atemp: float = Field(ge=0, le=1)
    hum: float = Field(ge=0, le=1)
    windspeed: float = Field(ge=0, le=1)


app = FastAPI()
app.mount("/metrics", make_asgi_app())                              # Prometheus scrapes this path


@app.middleware("http")
async def count_requests(request: Request, call_next):
    response = await call_next(request)
    if request.url.path == "/predict":
        REQUESTS.labels(str(response.status_code)).inc()            # 422s never reach predict()
    return response


@app.post("/predict")
def predict(hour: Hour):
    start = time.perf_counter()
    rentals = float(model.predict(pd.DataFrame([hour.model_dump()]))[0])
    LATENCY.observe(time.perf_counter() - start)
    PREDICTED.observe(rentals)
    TEMPERATURE.observe(hour.temp)
    return {"rentals": round(rentals), "model_version": VERSION}

Mounting the client's ASGI app at /metrics is the pattern the Python client documents for FastAPI. With several worker processes, each keeps its own counters, and the client's multiprocess mode is needed (Prometheus client: FastAPI).

Prometheus configuration and rules

yaml
global:
  scrape_interval: 1s        # demo only; the default is 1m
  evaluation_interval: 1s

rule_files:
  - rules.yml

scrape_configs:
  - job_name: bike-model
    static_configs:
      - targets: ["localhost:8000"]
yaml
groups:
  - name: bike-model
    rules:
      - record: bike:predicted_rentals:mean1m
        expr: increase(bike_predicted_rentals_sum[1m]) / increase(bike_predicted_rentals_count[1m])

      - alert: PredictedDemandShift
        expr: abs(bike:predicted_rentals:mean1m / on(job, instance) bike_reference_mean_prediction - 1) > 0.4
        for: 10s
        labels:
          severity: ticket
        annotations:
          summary: Average prediction is more than 40% away from the training reference

      - alert: PredictionErrors
        expr: sum(rate(bike_requests_total{status!="200"}[1m])) / sum(rate(bike_requests_total[1m])) > 0.05
        for: 10s
        labels:
          severity: page

The recording rule turns the prediction histogram into a last-minute average. The drift alert compares it with the training reference and waits 10 seconds before firing, and the error alert watches the share of failed requests. The one-second intervals only compress the demo into minutes.

Testing the alert before it matters

An alert that has never fired is an untested promise. promtool runs unit tests for rules against input series you define (Prometheus: unit testing rules). This test feeds a minute of predictions averaging 150, then a minute averaging 60, and expects silence at 50 seconds and a firing alert at two minutes.

yaml
rule_files:
  - rules.yml
evaluation_interval: 1s

tests:
  - interval: 1s
    input_series:
      # 60 s of predictions averaging 150, then 60 s averaging 60
      - series: 'bike_predicted_rentals_sum{job="bike-model", instance="localhost:8000"}'
        values: '0+150x60 9060+60x59'
      - series: 'bike_predicted_rentals_count{job="bike-model", instance="localhost:8000"}'
        values: '0+1x120'
      - series: 'bike_reference_mean_prediction{job="bike-model", instance="localhost:8000", model_version="rf-2011"}'
        values: '144x120'
    alert_rule_test:
      - eval_time: 50s
        alertname: PredictedDemandShift
        exp_alerts: []
      - eval_time: 2m
        alertname: PredictedDemandShift
        exp_alerts:
          - exp_labels:
              severity: ticket
              job: bike-model
              instance: localhost:8000
            exp_annotations:
              summary: Average prediction is more than 40% away from the training reference

Grafana from files

yaml
apiVersion: 1
datasources:
  - name: Prometheus
    uid: prom
    type: prometheus
    url: http://localhost:9090
    access: proxy
    isDefault: true
yaml
apiVersion: 1
providers:
  - name: model-dashboards
    type: file
    options:
      path: dashboards
json
{
  "uid": "bike-model",
  "title": "Bike demand model",
  "refresh": "5s",
  "time": {"from": "now-15m", "to": "now"},
  "panels": [
    {"type": "timeseries", "title": "Requests per second by status",
     "gridPos": {"x": 0, "y": 0, "w": 12, "h": 8},
     "datasource": {"type": "prometheus", "uid": "prom"},
     "targets": [{"refId": "A", "expr": "sum by (status) (rate(bike_requests_total[1m]))", "legendFormat": "{{status}}"}]},
    {"type": "timeseries", "title": "Prediction latency, 95th percentile",
     "gridPos": {"x": 12, "y": 0, "w": 12, "h": 8},
     "datasource": {"type": "prometheus", "uid": "prom"},
     "targets": [{"refId": "A", "expr": "histogram_quantile(0.95, sum by (le) (rate(bike_prediction_seconds_bucket[1m])))"}]},
    {"type": "timeseries", "title": "Average prediction vs training reference",
     "gridPos": {"x": 0, "y": 8, "w": 12, "h": 8},
     "datasource": {"type": "prometheus", "uid": "prom"},
     "targets": [{"refId": "A", "expr": "bike:predicted_rentals:mean1m", "legendFormat": "last minute"},
                 {"refId": "B", "expr": "bike_reference_mean_prediction", "legendFormat": "training reference"}]},
    {"type": "timeseries", "title": "Median input temperature",
     "gridPos": {"x": 12, "y": 8, "w": 12, "h": 8},
     "datasource": {"type": "prometheus", "uid": "prom"},
     "targets": [{"refId": "A", "expr": "histogram_quantile(0.5, sum by (le) (rate(bike_input_temp_bucket[1m])))"}]}
  ]
}

Two weeks of traffic

The script checks the configuration and runs the rule test, starts all three servers with empty storage, sends three invalid requests, replays a week of April 2012 and then a week of January 2012, and reads the results from Prometheus and from Grafana.

python
import os
import shutil
import subprocess
import sys
import time
from pathlib import Path

import pandas as pd
import requests

for command in (["promtool", "check", "config", "prometheus.yml"], ["promtool", "test", "rules", "rules_test.yml"]):
    result = subprocess.run(command, capture_output=True, text=True)
    print("promtool", *command[1:3], "->", "SUCCESS" if result.returncode == 0 else result.stdout + result.stderr)

for folder in ("prom-data", "grafana-data"):
    shutil.rmtree(folder, ignore_errors=True)                     # start from empty storage
Path("grafana-data").mkdir()                                      # Grafana will not create its data folder
grafana_env = dict(os.environ, GF_SERVER_HTTP_PORT="3001",
                   GF_PATHS_DATA=str(Path("grafana-data").resolve()),
                   GF_PATHS_PROVISIONING=str(Path("grafana/provisioning").resolve()))
# 127.0.0.1 rather than localhost: on Windows, localhost tries IPv6 first and adds a delay to every call
SERVICE, PROMETHEUS, GRAFANA = "http://127.0.0.1:8000", "http://127.0.0.1:9090", "http://127.0.0.1:3001"
quiet = {"stdout": subprocess.DEVNULL, "stderr": subprocess.DEVNULL}
servers = [
    subprocess.Popen(["prometheus", "--config.file=prometheus.yml", "--storage.tsdb.path=prom-data",
                      "--web.listen-address=localhost:9090"], **quiet),
    subprocess.Popen([sys.executable, "-m", "uvicorn", "service:app", "--port", "8000", "--log-level", "warning"]),
    subprocess.Popen(["grafana", "server", "--homepath", str(Path(shutil.which("grafana")).parent.parent)],
                     env=grafana_env, **quiet),
]


def wait_for(url):
    for _ in range(120):
        try:
            if requests.get(url, timeout=1).ok:
                return
        except requests.ConnectionError:
            pass
        time.sleep(1)
    raise RuntimeError(f"{url} never became ready")


def promql(query):
    return requests.get(f"{PROMETHEUS}/api/v1/query", params={"query": query}).json()["data"]["result"]


FEATURES = ["season", "mnth", "hr", "holiday", "weekday", "workingday", "weathersit", "temp", "atemp", "hum", "windspeed"]
hours = pd.read_csv("hour.csv", parse_dates=["dteday"])


def replay(first_day):
    week = hours[(hours.dteday >= first_day) & (hours.dteday < pd.Timestamp(first_day) + pd.Timedelta(days=7))]
    for row in week[FEATURES].to_dict("records"):
        requests.post(f"{SERVICE}/predict", json=row).raise_for_status()
    return week


def report(label, week):
    mean = float(promql("bike:predicted_rentals:mean1m")[0]["value"][1])
    reference = float(promql("bike_reference_mean_prediction")[0]["value"][1])
    alerts = {a["metric"]["alertname"]: a["metric"]["alertstate"] for a in promql("ALERTS")}
    print(f"{label}: {len(week)} requests")
    print(f"  average prediction, last minute {mean:6.1f}  ({mean / reference - 1:+.0%} against the training reference {reference:.1f})")
    print(f"  actual average rentals          {week.cnt.mean():6.1f}")
    print(f"  alerts: {alerts or 'none'}")


try:
    for url in (f"{PROMETHEUS}/-/ready", f"{SERVICE}/metrics", f"{GRAFANA}/api/health"):
        wait_for(url)
    for row in hours[FEATURES].head(3).to_dict("records"):
        requests.post(f"{SERVICE}/predict", json=dict(row, hum=1.5))   # invalid: rejected with 422
    week = replay("2012-04-16")
    time.sleep(5)                                                 # give Prometheus time to scrape and evaluate
    report("week of 16 April 2012", week)
    time.sleep(65)                                                # let that week leave the one-minute window
    week = replay("2012-01-09")
    time.sleep(15)                                                # longer than the alert's 10 s "for" clause
    report("week of 9 January 2012", week)

    counts = {r["metric"]["status"]: int(float(r["value"][1])) for r in promql("bike_requests_total")}
    print("requests by status:", dict(sorted(counts.items())))
    p95 = float(promql("histogram_quantile(0.95, sum by (le) (rate(bike_prediction_seconds_bucket[1m])))")[0]["value"][1])
    print(f"95th percentile prediction time: {p95 * 1000:.0f} ms (varies by machine)")

    auth = ("admin", "admin")                                     # a fresh local Grafana's default login
    dashboard = requests.get(f"{GRAFANA}/api/dashboards/uid/bike-model", auth=auth).json()["dashboard"]
    print(f"\nGrafana dashboard '{dashboard['title']}':", [panel["title"] for panel in dashboard["panels"]])
    answer = requests.post(f"{GRAFANA}/api/ds/query", auth=auth, json={
        "from": "now-5m", "to": "now",
        "queries": [{"refId": "A", "datasource": {"uid": "prom"}, "expr": "bike:predicted_rentals:mean1m",
                     "instant": True, "range": False}]}).json()
    print(f"panel query answered by Grafana: {answer['results']['A']['frames'][0]['data']['values'][1][0]:.1f}")
finally:
    for server in servers:
        if os.name == "nt":                                       # Windows: stop child processes too (Grafana plugins)
            subprocess.run(["taskkill", "/T", "/F", "/PID", str(server.pid)], capture_output=True)
        else:
            server.terminate()
text
promtool check config -> SUCCESS
promtool test rules -> SUCCESS
week of 16 April 2012: 168 requests
  average prediction, last minute  151.0  (+5% against the training reference 144.2)
  actual average rentals           231.8
  alerts: none
week of 9 January 2012: 167 requests
  average prediction, last minute   64.2  (-55% against the training reference 144.2)
  actual average rentals           121.4
  alerts: {'PredictedDemandShift': 'firing'}
requests by status: {'200': 335, '422': 3}
95th percentile prediction time: 25 ms (varies by machine)

Grafana dashboard 'Bike demand model': ['Requests per second by status', 'Prediction latency, 95th percentile', 'Average prediction vs training reference', 'Median input temperature']
panel query answered by Grafana: 64.2

What the run shows

  • The April week looked normal, and was not. The average prediction was 151.0, 5% above the training reference of 144.2, so no alert fired. Actual rentals averaged 231.8: the model under-predicted by about a third, because demand grew between 2011 and 2012. Prometheus saw the predictions, not the outcomes, so nothing on the dashboard could show the error.
  • The January week fired the drift alert. The average prediction fell to 64.2, 55% below the reference, and the alert was firing after its 10-second hold.
  • Bad requests never reached the model. The three invalid rows were rejected with status 422 beside 335 successful requests, and the error alert stayed quiet because they were a small share of the minute's traffic.
  • Grafana agreed with Prometheus. It loaded the dashboard from files and returned the same 64.2 through its data source.

The limits of metrics for ML

  • Quality needs labels. Accuracy comes from joining predictions with outcomes, usually in a batch job that can then publish a gauge. See monitoring ML models in production.
  • Metrics aggregate. To answer what happened to one request, you need logs and traces; see logging and observability for ML systems.
  • Bucket edges are chosen in advance, so quantiles from histograms are estimates.
  • Distribution tests belong in batch jobs. PSI and similar statistics are easier to compute offline; see model drift and data drift.

For packaging the service itself, see serving ML models with FastAPI and Docker.

Common mistakes

  • Unbounded label values, such as user IDs, which multiply series.
  • Alerts without a hold period, which fire on a single noisy scrape.
  • Alert rules nobody has tested.
  • Dashboards edited by hand and lost when Grafana is rebuilt.
  • Treating normal-looking predictions as proof of a healthy model.

Can Prometheus monitor model accuracy?

Not directly, because accuracy needs the true outcomes, which arrive after the prediction. A batch job can join predictions with outcomes and publish the result as a gauge for Prometheus to scrape and alert on.

What metrics should an ML model service expose?

Request counts by status, prediction latency as a histogram, a histogram of the model's outputs, histograms of a few key inputs, and a gauge identifying the model version with a reference value for normal behaviour.

How do I test Prometheus alert rules?

Write a test file with input series and expected alerts at given times, and run promtool test rules. Run it in CI, so a broken rule fails before it is deployed.

Do I need Grafana if I already have Prometheus?

Prometheus has a basic expression browser, but Grafana adds shareable dashboards, many data sources and file-based provisioning, which is why the two are usually deployed together.

Performance monitoring, logging and alerts are covered in the Model Monitoring and Maintenance module of the MLOps course.

Sources

Tags:#mlops#prometheus#grafana#monitoring

GloryTecks Team

Training & Placement Faculty

The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.

Want to master MLOps?

Join GloryTecks’ expert-led MLOps training in Hyderabad with real-time projects and 100% placement support.

Explore course
Next

Production ML Systems: Architecture and Best Practices

Related articles

MLOps
13 May 2026 9 min

Monitoring ML Models in Production: A Complete Guide

A model that passed validation can fail quietly in production. This guide sets out what to monitor at each layer, how to cope with labels that arrive late and how to design alerts people act on, using a year of real sensor data in which the loudest alert was the least useful one.

GloryTecks Team
Read
MLOps
12 Jun 2026 5 min

MLOps Roadmap 2026: Skills, Tools and Career Path

MLOps sits between data science, software engineering and operations, which makes it hard to know where to start. This roadmap orders the skills from foundations to production operations, gives each stage a project that proves the skill, and explains what the MLOps engineer role actually involves.

GloryTecks Team
Read
MLOpsTrending
6 Jun 2026 7 min

MLflow Tutorial: Tracking Experiments and Models

MLflow records what produced each model so you never have to remember. This tutorial trains three models on real bike-sharing data, logs every run, compares them by querying the tracking store, registers the best one, points a champion alias at it and loads it back by that alias, all on a laptop.

GloryTecks Team
Read