Cost Optimization for ML Workloads in the Cloud
Most of an ML cloud bill comes from three decisions: how capacity is bought, how fully it is used and how efficiently each prediction is made. With live Azure prices and two real models, this guide shows reservations and spot cutting prices by 38 to 82%, and an always-on VM costing seven times its compute.
GloryTecks Team
Training & Placement Faculty
Last updated: 8 October 2026
A machine learning cloud bill is mostly the result of three decisions: how you buy capacity, how much of it you actually use, and how much work each unit of capacity does. Model architecture matters, but a well-built model on idle, on-demand machines still costs far more than it needs to.
This guide puts numbers on all three, using live prices from Azure's public price list and the measured throughput of two real models.
Prices you can query
Azure publishes its retail prices through an API that needs no account, covering regions, VM sizes, reservation terms and spot meters (Microsoft Learn: Retail Prices API). The script reads prices for a general-purpose 4-vCPU VM and a VM with an NVIDIA T4 GPU in Central India, then measures how many predictions two models make per second on four CPU threads: a 39 KB activity classifier on smartphone sensor data (UCI: HAR), and the all-MiniLM-L6-v2 sentence encoder (Hugging Face: all-MiniLM-L6-v2) on Banking77 customer queries (PolyAI: Banking77).
import time
import numpy as np
import onnxruntime as ort
import pandas as pd
import requests
import torch
from transformers import AutoModel, AutoTokenizer
HOURS_PER_MONTH, THREADS = 730, 4 # four threads stand in for a 4-vCPU VM
def monthly_prices(sku, region="centralindia"): # Azure's public Retail Prices API, no account needed
query = f"serviceName eq 'Virtual Machines' and armRegionName eq '{region}' and armSkuName eq '{sku}'"
items = requests.get("https://prices.azure.com/api/retail/prices", params={"$filter": query}, timeout=60).json()["Items"]
linux = [i for i in items if "Windows" not in i["productName"]]
pay_as_you_go = [i for i in linux if i["type"] == "Consumption"]
hourly = next(i["unitPrice"] for i in pay_as_you_go if not i["skuName"].endswith(("Spot", "Low Priority")))
spot = next(i["unitPrice"] for i in pay_as_you_go if i["skuName"].endswith("Spot"))
term = {i["reservationTerm"]: i["unitPrice"] for i in linux if i["type"] == "Reservation"} # price for the whole term
return {"on demand": hourly * HOURS_PER_MONTH, "1-year reservation": term["1 Year"] / 12,
"3-year reservation": term["3 Years"] / 36, "spot": spot * HOURS_PER_MONTH}
print("1. Monthly cost, Linux, Central India, USD for 730 hours (saving against on demand)")
table = {sku: monthly_prices(sku) for sku in ("Standard_D4s_v5", "Standard_NC4as_T4_v3")}
print(f" {'VM':<22}" + "".join(f"{k:>20}" for k in table["Standard_D4s_v5"]))
for sku, row in table.items():
print(f" {sku:<22}" + "".join(f"{v:>14,.2f} ({1 - v / row['on demand']:>3.0%})" for v in row.values()))
def rate(run, items, batch, seconds=3.0): # predictions per second, for about three seconds
run(items[:batch]) # warm-up: the first call pays one-off set-up costs
start, done = time.perf_counter(), 0
while time.perf_counter() - start < seconds:
chunk = items[done % (len(items) - batch): done % (len(items) - batch) + batch]
run(chunk)
done += len(chunk)
return done / (time.perf_counter() - start)
print(f"\n2. Predictions per second on {THREADS} CPU threads of this laptop (varies)")
options = ort.SessionOptions()
options.intra_op_num_threads = THREADS
activity = ort.InferenceSession("har_int8.onnx", options, providers=["CPUExecutionProvider"]) # 39 KB model
windows = np.loadtxt("har/test/X_test.txt", dtype=np.float32)
torch.set_num_threads(THREADS)
name = "sentence-transformers/all-MiniLM-L6-v2"
tokenizer, encoder = AutoTokenizer.from_pretrained(name), AutoModel.from_pretrained(name).eval()
queries = pd.read_csv("test.csv").text.tolist() # Banking77 test queries (CC BY 4.0)
@torch.no_grad()
def embed(texts):
encoder(**tokenizer(list(texts), padding=True, truncation=True, return_tensors="pt"))
speed = {}
for label, run, items, batches in (("activity model", lambda rows: activity.run(["label"], {"X": rows}), windows, (1, 512)),
("sentence encoder", embed, queries, (1, 64))):
speed[label] = {b: rate(run, items, b) for b in batches}
print(f" {label:<17}" + "".join(f" batch {b:>3}: {r:>11,.0f}" for b, r in speed[label].items()) + " (varies)")
print("\n3. One million predictions a day on a Standard_D4s_v5, at the batched speeds above (varies)")
price = table["Standard_D4s_v5"]["on demand"] / HOURS_PER_MONTH
for label, rates in speed.items():
hours = 1_000_000 / max(rates.values()) / 3600
cost = hours * 30 * price
print(f" {label:<17} {hours * 3600:8,.0f} VM-seconds a day; {cost:.2f} USD a month if billed only while running (varies)"
if cost >= 0.01 else f" {label:<17} {hours * 3600:8,.0f} VM-seconds a day; under 0.01 USD a month (varies)")
print(f" the VM left running all month: {table['Standard_D4s_v5']['on demand']:.2f} USD")1. Monthly cost, Linux, Central India, USD for 730 hours (saving against on demand)
VM on demand 1-year reservation 3-year reservation spot
Standard_D4s_v5 147.46 ( 0%) 91.00 (38%) 58.25 (60%) 27.25 (82%)
Standard_NC4as_T4_v3 422.67 ( 0%) 248.50 (41%) 160.50 (62%) 119.87 (72%)
2. Predictions per second on 4 CPU threads of this laptop (varies)
activity model batch 1: 36,994 batch 512: 393,539 (varies)
sentence encoder batch 1: 46 batch 64: 85 (varies)
3. One million predictions a day on a Standard_D4s_v5, at the batched speeds above (varies)
activity model 3 VM-seconds a day; under 0.01 USD a month (varies)
sentence encoder 11,781 VM-seconds a day; 19.83 USD a month if billed only while running (varies)
the VM left running all month: 147.46 USDPrices were read on 8 October 2026 and will change. The API lists a reservation's price for its whole term, so the script divides by 12 or 36 months. The throughput lines vary by run; they come from one run on a laptop, not from an Azure VM.
What the output shows
- How you buy capacity moves the price most. A one-year reservation cut the monthly cost by 38% for the CPU VM and 41% for the GPU VM; three years cut it by about 60%. Spot capacity was 82% and 72% cheaper.
- Spot has a catch. Azure can evict spot VMs with 30 seconds' notice when it needs the capacity, and offers no SLA, so they suit interruptible work such as batch jobs and training that saves checkpoints (Microsoft Learn: Spot VMs).
- Batching multiplied throughput. The small model handled about ten times as many predictions per second in batches of 512 as one at a time; the sentence encoder handled almost twice as many in batches of 64.
- Idle time dominated the bill. A million encoder predictions a day needed about 3.3 VM-hours of compute, around 20 dollars a month if billed only while running, against 147.46 dollars for the same VM left on all month: seven times as much. For the small model, the work was a few seconds a day and nearly the entire bill would be idle capacity.
- The GPU VM cost 2.9 times the CPU VM on demand, and 4.4 times as spot. It pays off only when it delivers more throughput than its price premium for your workload; see GPU vs CPU for ML.
Where to look for savings
| Lever | Typical move | Watch out for |
|---|---|---|
| Pricing model | Reserve steady capacity; use spot for interruptible jobs | Commitments you cannot use; evictions |
| Utilisation | Batch, scale to zero, or share instances across models | Cold starts when traffic returns |
| Efficiency | Batch requests, compress models, right-size instances | Latency limits on batching |
| Hardware | Use GPUs only where they win on cost per result | Paying for idle accelerators |
| Data | Keep compute near the data | Transfer charges between regions or clouds |
Training versus serving
Training is bursty and usually interruptible, which makes it a natural fit for spot capacity with frequent checkpoints, so an eviction loses minutes rather than hours. Choose the interval by what an eviction would cost: with only 30 seconds' notice, everything since the last checkpoint is lost, so frequent, cheap checkpoints are usually worth their storage. Serving is continuous but often light: many endpoints spend most of their time waiting. For steady, heavy traffic, reserved capacity is cheapest; for light or spiky traffic, batch scoring, serverless endpoints or scaling to zero usually win. See batch vs real-time inference.
A cost checklist
- Tag every resource with a team and a model, so the bill can be read.
- Measure utilisation of every endpoint and training cluster.
- Move interruptible work to spot, with checkpoints.
- Reserve only the steady baseline, after a few months of data.
- Batch and compress before adding hardware; see model compression.
- Review GPU use monthly, by cost per result.
For the platform services that run these workloads, see deploying ML models on AWS, Azure and GCP and Vertex AI and SageMaker compared.
Common mistakes
- Always-on endpoints for occasional traffic.
- Reserving capacity before measuring utilisation.
- Training on spot without checkpoints.
- Comparing CPU and GPU by speed instead of cost per result.
- Untagged resources that nobody owns.
How do I reduce the cost of serving ML models?
Are spot instances good for machine learning?
When is a GPU cheaper than a CPU for ML?
How much do reserved instances save?
Cloud MLOps on AWS, Azure and Google Cloud is covered in the Cloud MLOps Platforms module of the MLOps course.
Sources
- Azure Retail Prices REST API overview (Microsoft Learn) — Microsoft
- About Azure Spot Virtual Machines (Microsoft Learn) — Microsoft
- Human Activity Recognition Using Smartphones (UCI Machine Learning Repository) — UCI Machine Learning Repository
- all-MiniLM-L6-v2 (Hugging Face) — sentence-transformers
- Task-specific datasets, including Banking77 (GitHub, PolyAI) — PolyAI
GloryTecks Team
Training & Placement Faculty
The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.
Want to master MLOps?
Join GloryTecks’ expert-led MLOps training in Hyderabad with real-time projects and 100% placement support.
Airflow for ML Pipelines: A Practical Guide
NextScaling Model Training with Distributed Computing
Related articles
MLOps Roadmap 2026: Skills, Tools and Career Path
MLOps sits between data science, software engineering and operations, which makes it hard to know where to start. This roadmap orders the skills from foundations to production operations, gives each stage a project that proves the skill, and explains what the MLOps engineer role actually involves.
MLflow Tutorial: Tracking Experiments and Models
MLflow records what produced each model so you never have to remember. This tutorial trains three models on real bike-sharing data, logs every run, compares them by querying the tracking store, registers the best one, points a champion alias at it and loads it back by that alias, all on a laptop.
Docker for Machine Learning: A Practical Guide
How to containerise a machine-learning project so it runs the same everywhere: pinning the exact library versions a saved model depends on, a Dockerfile ordered for fast rebuilds, keeping data out of images, multi-stage builds, non-root users and GPU access.