Anomaly Detection: Techniques and Real-World Use Cases

An anomaly is a record that does not fit the pattern of the rest: an order of 15 rupees in a shop where orders cluster around 1,200 rupees, a sensor reading that jumps tenfold, a day with no sales at all. Sometimes it is the most important record in the dataset (fraud, a failing machine, a broken pipeline); sometimes it is a data-entry error. Either way you want to find it. The difficulty is that "normal" is rarely defined, and labelled examples of anomalies are almost never available.

Start with rules you can explain

Before any statistics, write down what is impossible or clearly suspicious: negative quantities, a delivery date before the order date, more than ten orders from one card in an hour, a temperature outside the sensor's physical range. Rules catch known problems exactly, are easy to explain to the people who act on the alerts, and cost nothing to run. Statistical and machine-learning methods are for the unknown problems the rules do not anticipate.

Z-scores, and how outliers hide themselves

The classic statistical test flags values more than three standard deviations from the mean. It has a weakness that the simulation below exposes: anomalies distort the mean and standard deviation they are measured against. The data is 990 ordinary transaction amounts around 1,200 rupees plus ten injected anomalies, eight very large and two tiny:

import numpy as np
from sklearn.ensemble import IsolationForest

rng = np.random.default_rng(42)
normal = rng.normal(1200, 150, 990)
anomalies = np.array([9000, 9500, 12000, 11000, 8800, 10500, 15, 20, 9900, 13000])
amounts = np.concatenate([normal, anomalies])
is_anomaly = np.r_[np.zeros(990, bool), np.ones(10, bool)]

z = (amounts - amounts.mean()) / amounts.std()
flag_z = np.abs(z) > 3

median = np.median(amounts)
mad = np.median(np.abs(amounts - median))
robust_z = 0.6745 * (amounts - median) / mad
flag_mad = np.abs(robust_z) > 3.5

iso = IsolationForest(contamination=0.01, random_state=0).fit(amounts.reshape(-1, 1))
flag_iso = iso.predict(amounts.reshape(-1, 1)) == -1

for name, flags in [("z-score > 3", flag_z), ("robust z (MAD) > 3.5", flag_mad), ("Isolation Forest", flag_iso)]:
    caught = (flags & is_anomaly).sum()
    false_alarms = (flags & ~is_anomaly).sum()
    print(f"{name:<22} caught {caught}/10  false alarms {false_alarms}")
print(f"std with anomalies {amounts.std():.1f}, without {normal.std():.1f}")

z-score > 3            caught 8/10  false alarms 0
robust z (MAD) > 3.5   caught 10/10  false alarms 1
Isolation Forest       caught 10/10  false alarms 0
std with anomalies 849.6, without 148.3

The last line explains the first. The eight huge values inflated the standard deviation from about 148 to about 850, so the 15- and 20-rupee transactions, which are wildly unusual, no longer looked extreme by that yardstick. This is called masking: outliers hide other outliers.

Robust scores: use the median instead

The median and the median absolute deviation (MAD) barely move when a few extreme values are added, so a z-score built on them keeps its yardstick honest. The modified z-score multiplies by 0.6745 so it is comparable with a standard z-score for normal data, and a cut-off of 3.5 is the value NIST's statistics handbook cites as recommended (NIST: detection of outliers). It caught all ten, at the cost of one false alarm: a legitimately unusual but genuine transaction. For a single numeric column, this is the method to try first.

Isolation Forest: many features at once

Real anomalies are often unusual only in combination: a normal amount, at a normal time, but from a new device in a different country. Isolation Forest handles many features together. It builds random trees that split the data at random points; anomalies, being few and different, get isolated in fewer splits than normal points (scikit-learn: IsolationForest). It needs no labels and scales to large datasets.

Its perfect score above comes with a caveat: contamination=0.01 told it to flag about 1% of the data, and exactly 1% was anomalous by construction. In real data the true rate is unknown, so either tune the threshold on the scores against a sample you have reviewed, or rank records by anomaly score and review the top of the list. scikit-learn documents several alternatives, including Local Outlier Factor for anomalies that are unusual relative to their neighbourhood (scikit-learn: outlier detection).

Anomalies in time series

Time adds context: 500 orders at 3 a.m. might be an anomaly, while 500 at noon is normal. Compare each point with what was expected for that moment, not with the overall average. A practical approach is to model the expected value, using a rolling median of the same hour on previous days or a forecasting model, and flag points whose residual, the gap between actual and expected, is extreme by a robust score. Time series forecasting covers building the expected-value model.

Evaluating a detector without labels

Accuracy is meaningless here: a detector that flags nothing is 99% accurate when 1% of records are anomalies. Useful measures are:

• Precision of the alerts: of the records flagged, how many were real problems once someone reviewed them. Alert fatigue kills detectors faster than missed anomalies do.
• Recall on known cases: replay past incidents, such as confirmed fraud or known outages, and check how many the detector would have caught.
• Precision at k: with a daily review budget of, say, 50 records, what share of the top 50 by anomaly score were real.

Evaluation metrics: accuracy, precision, recall and F1 explains these measures in detail, and the imbalance problem connects to handling imbalanced datasets when some labels do exist.

Where anomaly detection is used

• Payments and fraud: unusual amounts, locations, devices or velocities for an account.
• Industrial sensors: vibrations or temperatures drifting away from a machine's normal profile before a failure.
• IT operations: sudden changes in error rates, latency or traffic.
• Data quality: a daily file with half the usual rows, a column suddenly full of nulls, a price column that jumps a hundredfold, which is often the cheapest and most valuable use in data teams.

Anomaly detection is one of the applied specialisations built on the machine learning and statistics taught in the Data Science course, alongside its exploratory data analysis module, which includes outlier detection.

Sources

• Novelty and Outlier Detection (scikit-learn) — scikit-learn documentation
• sklearn.ensemble.IsolationForest — scikit-learn documentation
• NIST: Detection of Outliers — NIST