Edge ML Deployment: Running Models on Devices

Edge deployment means running a model on the device that collects the data, such as a phone, a watch, a camera or an industrial sensor, instead of on a server. The prediction happens where the data is, with no round trip and no connection required.

That changes what matters. Server models are judged on accuracy and throughput; device models also have to fit a size budget, run on one modest core and keep working for people the team will never meet. This guide takes a real smartphone sensor model through export, quantization and release checks.

Why run a model on the device

Runtimes

A few runtimes cover most devices. ONNX Runtime runs models exported to the ONNX format on many platforms, with Python APIs for quantization (ONNX Runtime: quantization). Google renamed TensorFlow Lite to LiteRT (Google Developers Blog: LiteRT). ExecuTorch is PyTorch's solution for inference on edge devices, from phones to embedded systems (ExecuTorch). Pick the runtime that matches your training framework and your target hardware.

The experiment

The UCI Human Activity Recognition data come from 30 volunteers aged 19 to 48 who performed six activities wearing a Samsung Galaxy S II on the waist, with accelerometer and gyroscope readings at 50 Hz. The volunteers were split 70% for training and 30% for testing, so the test measures people the model has never seen (UCI: HAR).

A small neural network is trained, converted to ONNX with sklearn-onnx (sklearn-onnx), pre-processed and quantized to 8-bit integers with ONNX Runtime, then run with one thread, as a stand-in for a small device.

import os
import time

import numpy as np
import onnxruntime as ort
from onnxruntime.quantization import QuantType, quantize_dynamic
from onnxruntime.quantization.shape_inference import quant_pre_process
from skl2onnx import to_onnx
from sklearn.neural_network import MLPClassifier


def load(split):                                               # UCI HAR: 561 features per 2.56-second window
    folder = f"har/{split}"
    return (np.loadtxt(f"{folder}/X_{split}.txt", dtype=np.float32), np.loadtxt(f"{folder}/y_{split}.txt", dtype=np.int64),
            np.loadtxt(f"{folder}/subject_{split}.txt", dtype=np.int64))


train_x, train_y, _ = load("train")                             # 21 volunteers
test_x, test_y, person = load("test")                           # 9 different volunteers
model = MLPClassifier(hidden_layer_sizes=(64,), max_iter=300, random_state=0).fit(train_x, train_y)

onnx_model = to_onnx(model, train_x[:1], options={id(model): {"zipmap": False}}, target_opset=17)
with open("har_fp32.onnx", "wb") as f:
    f.write(onnx_model.SerializeToString())
quant_pre_process("har_fp32.onnx", "har_prepared.onnx")         # recommended before quantization
quantize_dynamic("har_prepared.onnx", "har_int8.onnx", weight_type=QuantType.QInt8)

one_core = ort.SessionOptions()
one_core.intra_op_num_threads = 1                               # a single core stands in for a small device
results = {}
print(f"scikit-learn model accuracy: {(model.predict(test_x) == test_y).mean():.2%}")
for name in ("har_fp32.onnx", "har_int8.onnx"):
    session = ort.InferenceSession(name, one_core, providers=["CPUExecutionProvider"])
    labels = session.run(["label"], {"X": test_x})[0]
    timings = []
    for row in test_x[:1000]:                                   # one window at a time, as on a device
        start = time.perf_counter()
        session.run(["label"], {"X": row[None, :]})
        timings.append(time.perf_counter() - start)
    by_person = {p: (labels[person == p] == test_y[person == p]).mean() for p in np.unique(person)}
    worst, best = min(by_person, key=by_person.get), max(by_person, key=by_person.get)
    results[name] = labels
    print(f"{name:<15} {os.path.getsize(name) / 1024:6.1f} KB  accuracy {(labels == test_y).mean():.2%}  "
          f"volunteer {worst} {by_person[worst]:.1%} to volunteer {best} {by_person[best]:.1%}")
    print(f"{'':<15} median {np.median(timings) * 1000:.3f} ms per window on one core (varies by machine)")

agree = (results["har_fp32.onnx"] == results["har_int8.onnx"]).mean()
size_ratio = os.path.getsize("har_fp32.onnx") / os.path.getsize("har_int8.onnx")
print(f"\nint8 and float32 predictions agree on {agree:.2%} of windows; int8 file is {size_ratio:.1f} times smaller")
checks = {"file under 64 KB": os.path.getsize("har_int8.onnx") <= 64 * 1024,
          "accuracy within 0.5 points of float32": (results["har_fp32.onnx"] == test_y).mean() - (results["har_int8.onnx"] == test_y).mean() <= 0.005,
          "every test volunteer above 75%": all((results["har_int8.onnx"][person == p] == test_y[person == p]).mean() > 0.75 for p in np.unique(person))}
print("release checks for the device build:", {name: "pass" if ok else "FAIL" for name, ok in checks.items()})

scikit-learn model accuracy: 94.44%
har_fp32.onnx    143.1 KB  accuracy 94.44%  volunteer 10 81.0% to volunteer 24 99.5%
                median 0.056 ms per window on one core (varies by machine)
har_int8.onnx     39.1 KB  accuracy 94.30%  volunteer 10 81.0% to volunteer 24 99.5%
                median 0.028 ms per window on one core (varies by machine)

int8 and float32 predictions agree on 99.80% of windows; int8 file is 3.7 times smaller
release checks for the device build: {'file under 64 KB': 'pass', 'accuracy within 0.5 points of float32': 'pass', 'every test volunteer above 75%': 'pass'}

What the output shows

• Export changed nothing. The ONNX model scored exactly what the scikit-learn model scored, 94.44%. Check that parity after every conversion; converters occasionally change behaviour.
• Quantization was nearly free. The file shrank from 143.1 KB to 39.1 KB, 3.7 times smaller, while accuracy moved from 94.44% to 94.30% and predictions agreed on 99.80% of windows. ONNX Runtime's dynamic quantization stores weights as 8-bit integers and works out activation scales on the fly.
• Speed was not the problem here. Both versions took a small fraction of a millisecond per window on one laptop core. Real devices are slower and vary widely, so measure on the hardware you ship to.
• People were the problem. Accuracy ranged from 81.0% for volunteer 10 to 99.5% for volunteer 24, identically for both versions. Quantization cost 0.14 points; the gap between two people was more than 18 points. On devices, a model mostly meets users it never saw, and their labels rarely come back.
• Release checks turn budgets into rules. The int8 build passed all three: under 64 KB, within half a point of float32, and above 75% for every test volunteer.

Setting a device budget

Budgets should come from the product, not from what the model happens to need. In this dataset, readings arrive in sliding windows of 2.56 seconds with 50% overlap, so a new window is ready every 1.28 seconds. That is the real deadline, and inference at a fraction of a millisecond leaves enormous headroom; the work that matters is feature extraction, filtering the raw signals and waking the processor no more often than necessary. The size limit comes from the app download and from memory shared with everything else on the phone. The accuracy floor should be set per person, not only on average, because one user's bad experience is not offset by another's good one. Write all three down before training, and put them in the build as automated checks, as the script does.

Shipping and updating

• Version every model file with an ID and a checksum, and keep the app able to load the previous version.
• Roll out in stages, to a small share of devices first; see shadow deployment and canary releases.
• Collect aggregate telemetry with consent, such as confidence and class counts, rather than raw sensor data.
• Keep a fallback for devices where the model fails to load.
• Plan for drift on devices, through periodic updates or on-device personalisation.

For shrinking larger models, see model compression. For ONNX and other formats in general, see model packaging, and for when a server is the better place, see batch vs real-time inference.

Common mistakes

• Skipping the parity check after conversion.
• Benchmarking on a laptop and shipping to a low-end phone.
• Testing on the same people the model was trained on.
• No way to roll back a model update.
• Shipping float32 when int8 would do.

Model serialisation and deployment strategies are covered in the Model Deployment module of the MLOps course.

Sources

• Human Activity Recognition Using Smartphones (UCI Machine Learning Repository) — UCI Machine Learning Repository
• sklearn-onnx documentation — ONNX
• Quantize ONNX models (ONNX Runtime documentation) — ONNX Runtime
• TensorFlow Lite is now LiteRT (Google Developers Blog) — Google
• ExecuTorch documentation — PyTorch