GPU vs CPU for ML: When You Actually Need a GPU
GPUs win on large, parallel maths and lose on small, scattered work. Measured on a laptop's integrated GPU and its CPU, the GPU ran at half the CPU's speed on tiny matrices, up to nine times faster on large ones, and nearly four times faster embedding sentences in batches, but slower one sentence at a time.
GloryTecks Team
Training & Placement Faculty
Last updated: 8 October 2026
"Do I need a GPU?" usually gets answered with a guess, or with a benchmark from a data centre that has nothing in common with your workload. The real answer depends on the shape of the work: how big the operations are, how much data has to move, and whether you can batch.
This guide measures it. The same operations run on a laptop's CPU and on its integrated GPU, from tiny to large, including a real sentence encoder at two batch sizes.
Why GPUs are fast, and when that stops mattering
NVIDIA's performance guide describes a GPU as a highly parallel processor. A data-centre A100, for example, has 108 streaming multiprocessors, a 40 MB L2 cache and up to 2,039 GB/s of memory bandwidth from 80 GB of HBM2. Any function is limited by one of three things: memory bandwidth, math bandwidth or latency. Arithmetic intensity, the operations performed per byte moved, decides whether math or memory is the bottleneck (NVIDIA: GPU performance background).
The third limit, latency, is the one people forget. Small operations finish before a GPU's thousands of lanes get busy, and moving data to the device takes time of its own.
The hardware here
The laptop has an Intel Core Ultra 5 125H, using 14 CPU threads, and integrated Intel Arc graphics that share system memory. PyTorch supports Intel GPUs through its xpu device, including Core Ultra processors with Arc graphics on Windows 11 (PyTorch: Intel GPU). An integrated laptop GPU is far smaller than a data-centre card, so the absolute numbers will not transfer, but the shape of the results does.
The real workload embeds Banking77 customer queries (PolyAI: Banking77) with all-MiniLM-L6-v2, a compact sentence encoder (Hugging Face: all-MiniLM-L6-v2).
import statistics
import time
import pandas as pd
import torch
from transformers import AutoModel, AutoTokenizer
CPU, GPU = torch.device("cpu"), torch.device("xpu") # xpu: an Intel GPU, here the laptop's integrated Arc
print(f"CPU: {torch.get_num_threads()} threads | GPU: {torch.xpu.get_device_name(0)}")
def sync(device):
if device.type == "xpu":
torch.xpu.synchronize() # GPU work is asynchronous: wait before reading the clock
def timed(work, device, repeats=5): # median of several runs, after one warm-up
work()
sync(device)
runs = []
for _ in range(repeats):
start = time.perf_counter()
work()
sync(device)
runs.append(time.perf_counter() - start)
return statistics.median(runs)
print("\n1. Matrix multiplication, float32, billions of operations per second (varies)")
for n in (64, 256, 1024, 4096):
speed = {}
for device in (CPU, GPU):
a, b = torch.randn(n, n, device=device), torch.randn(n, n, device=device)
speed[device.type] = 2 * n**3 / timed(lambda: a @ b, device) / 1e9
print(f" {n:>4} x {n:<4} CPU {speed['cpu']:>7.1f} GPU {speed['xpu']:>7.1f} GPU/CPU {speed['xpu'] / speed['cpu']:5.2f}x (varies)")
a, b = torch.randn(1024, 1024), torch.randn(1024, 1024)
print(f" CPU and GPU products agree within float32 tolerance: {torch.allclose(a @ b, (a.to(GPU) @ b.to(GPU)).cpu(), rtol=1e-3, atol=1e-2)}")
print("\n2. Moving data is not free")
batch, layer = torch.randn(64, 384), torch.nn.Linear(384, 77)
print(f" copy a 64 x 384 batch to the GPU: {timed(lambda: batch.to(GPU), GPU, repeats=50) * 1e6:.0f} microseconds (varies)")
print(f" run a 384-to-77 linear layer on it on the CPU: {timed(lambda: layer(batch), CPU, repeats=50) * 1e6:.0f} microseconds (varies)")
print("\n3. Embedding customer queries with all-MiniLM-L6-v2")
NAME = "sentence-transformers/all-MiniLM-L6-v2"
tokenizer = AutoTokenizer.from_pretrained(NAME)
queries = pd.read_csv("test.csv").text.tolist() # Banking77 test split (CC BY 4.0), 3,080 queries
@torch.no_grad()
def embed(model, device, texts, batch_size):
out = []
for start in range(0, len(texts), batch_size):
tokens = tokenizer(texts[start:start + batch_size], padding=True, truncation=True, return_tensors="pt").to(device)
hidden, mask = model(**tokens).last_hidden_state, tokens["attention_mask"].unsqueeze(-1).float()
out.append(((hidden * mask).sum(1) / mask.sum(1)).cpu())
return torch.cat(out)
vectors = {}
for device in (CPU, GPU):
model = AutoModel.from_pretrained(NAME).eval().to(device)
embed(model, device, queries[:64], 64) # warm-up
for batch_size, texts in ((1, queries[:640]), (64, queries)):
sync(device)
start = time.perf_counter()
vectors[device.type] = embed(model, device, texts, batch_size)
sync(device)
print(f" {device.type.upper()} batch {batch_size:>2}: {len(texts) / (time.perf_counter() - start):6.0f} queries per second (varies)")
lowest = torch.nn.functional.cosine_similarity(vectors["cpu"], vectors["xpu"]).min().item()
print(f" lowest cosine similarity between CPU and GPU embeddings: {lowest:.5f} (varies)")
print(f" all {len(queries):,} CPU and GPU embeddings agree above 0.999: {lowest > 0.999}")CPU: 14 threads | GPU: Intel(R) Arc(TM) Graphics
1. Matrix multiplication, float32, billions of operations per second (varies)
64 x 64 CPU 7.2 GPU 3.7 GPU/CPU 0.52x (varies)
256 x 256 CPU 136.7 GPU 192.7 GPU/CPU 1.41x (varies)
1024 x 1024 CPU 293.0 GPU 1801.1 GPU/CPU 6.15x (varies)
4096 x 4096 CPU 434.3 GPU 3754.1 GPU/CPU 8.64x (varies)
CPU and GPU products agree within float32 tolerance: True
2. Moving data is not free
copy a 64 x 384 batch to the GPU: 177 microseconds (varies)
run a 384-to-77 linear layer on it on the CPU: 81 microseconds (varies)
3. Embedding customer queries with all-MiniLM-L6-v2
CPU batch 1: 167 queries per second (varies)
CPU batch 64: 302 queries per second (varies)
XPU batch 1: 93 queries per second (varies)
XPU batch 64: 1159 queries per second (varies)
lowest cosine similarity between CPU and GPU embeddings: 1.00000 (varies)
all 3,080 CPU and GPU embeddings agree above 0.999: TrueEvery timing is one real run and will differ on another machine, or even on another run. The agreement checks do not vary.
What the output shows
- Tiny work favoured the CPU. At 64 by 64, the GPU managed about half the CPU's rate. There was too little work to fill it.
- Big work favoured the GPU, dramatically. At 1,024 and 4,096, the GPU ran about 6 and 9 times faster than the CPU.
- Moving data cost more than the work. Copying one small batch to the GPU took more than twice as long as computing a whole linear layer on it on the CPU. For small models, transfer alone can erase the GPU's advantage.
- Batch size decided the real workload. One query at a time, the GPU embedded 93 queries a second against the CPU's 167. In batches of 64, it embedded 1,159 against the CPU's 302, nearly four times faster.
- The answers were the same. CPU and GPU products agreed within float32 tolerance, and all 3,080 embeddings agreed to above 0.999 cosine similarity.
Integrated versus dedicated GPUs
The laptop's GPU shares memory with the CPU. Dedicated GPUs carry their own memory, which is much faster, as the A100's 2,039 GB/s shows, but data has to be copied into it before any work starts. That makes the lessons above stronger, not weaker: the larger and faster the device, the more work each transfer has to feed it before it pays off. Keep data on the device across steps, and copy results back only when you need them.
When you actually need a GPU
| Workload | Usual choice | Why |
|---|---|---|
| Training neural networks on images, text or audio | GPU | Large, batched matrix maths for hours |
| Fine-tuning or serving large transformer models | GPU | Compute and memory beyond a CPU |
| High-volume batch inference with neural networks | GPU, often | Batching fills the device |
| Low-traffic inference, one request at a time | CPU | Launch and transfer overheads dominate |
| Gradient boosting and linear models on tabular data | CPU | Most such algorithms are not built for GPU arrays |
| Exploring and prototyping on modest data | CPU | Simpler, cheaper, fast enough |
For tabular work, scikit-learn does not use GPUs by default. A growing list of its estimators can run on GPUs through experimental Array API support, but its documentation explains that tree-based models are fundamentally not array-based algorithms (scikit-learn: FAQ).
Decide by measuring
- Profile the real workload, with real batch sizes and input lengths.
- Batch wherever latency allows, since batching is what unlocks a GPU.
- Count the data movement, not just the compute.
- Compare cost per result, not speed alone; see cost optimisation for ML workloads.
- Try a smaller model before bigger hardware; see model compression.
When one GPU is not enough, the next step is spreading training across several; see scaling model training with distributed computing.
Common mistakes
- Benchmarking tiny operations and concluding the GPU is useless.
- Serving one request at a time on a GPU without batching.
- Ignoring transfer time between CPU and GPU memory.
- Buying GPUs for gradient boosting on modest tabular data.
- Trusting another team's benchmark for your workload.
Do I need a GPU for machine learning?
Why is my GPU slower than my CPU?
How much faster is a GPU than a CPU for deep learning?
Can scikit-learn use a GPU?
Distributed training basics and scalable ML systems are part of the Advanced MLOps Concepts module of the MLOps course.
Sources
- GPU Performance Background User's Guide (NVIDIA) — NVIDIA
- Getting Started on Intel GPU (PyTorch documentation) — PyTorch
- Frequently Asked Questions (scikit-learn documentation) — scikit-learn
- all-MiniLM-L6-v2 (Hugging Face) — sentence-transformers
- Task-specific datasets, including Banking77 (GitHub, PolyAI) — PolyAI
GloryTecks Team
Training & Placement Faculty
The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.
Want to master MLOps?
Join GloryTecks’ expert-led MLOps training in Hyderabad with real-time projects and 100% placement support.
Building a Prediction API: A Complete Walkthrough
NextHandling Concept Drift in Live ML Systems
Related articles
MLOps Roadmap 2026: Skills, Tools and Career Path
MLOps sits between data science, software engineering and operations, which makes it hard to know where to start. This roadmap orders the skills from foundations to production operations, gives each stage a project that proves the skill, and explains what the MLOps engineer role actually involves.
MLflow Tutorial: Tracking Experiments and Models
MLflow records what produced each model so you never have to remember. This tutorial trains three models on real bike-sharing data, logs every run, compares them by querying the tracking store, registers the best one, points a champion alias at it and loads it back by that alias, all on a laptop.
Docker for Machine Learning: A Practical Guide
How to containerise a machine-learning project so it runs the same everywhere: pinning the exact library versions a saved model depends on, a Dockerfile ordered for fast rebuilds, keeping data out of images, multi-stage builds, non-root users and GPU access.