Outlier Detection for Analysts: Methods and Tools

Every analyst meets outliers: an order a hundred times the usual size, a delivery that took a year, a price of zero. Some are errors, some are the most important records in the dataset, and a method cannot tell which. The NIST handbook's definition is honest about this: an outlier is an observation an abnormal distance from the others, and deciding what counts as abnormal is left to the analyst, after first understanding what normal looks like (NIST: what are outliers). This guide compares the common detection methods on the same real data and then covers what to do with the results.

The data: order values

Every product order placed with a UK online gift retailer over a year, excluding cancellations, postage and fees (UCI: Online Retail), summed to one value per order:

import io
import urllib.request
import zipfile
import numpy as np
import pandas as pd

url = "https://archive.ics.uci.edu/static/public/352/online+retail.zip"
with zipfile.ZipFile(io.BytesIO(urllib.request.urlopen(url).read())) as archive:
    raw = pd.read_excel(archive.open("Online Retail.xlsx"), dtype={"InvoiceNo": str, "StockCode": str})
services = {"POST", "DOT", "M", "m", "C2", "D", "S", "B", "BANK CHARGES", "AMAZONFEE", "CRUK"}
lines = raw[~raw["InvoiceNo"].str.startswith("C") & ~raw["StockCode"].isin(services)
            & ~raw["StockCode"].str.startswith("gift_") & (raw["Quantity"] > 0) & (raw["UnitPrice"] > 0)]
orders = lines.assign(value=lines["Quantity"] * lines["UnitPrice"]).groupby("InvoiceNo")["value"].sum()
print(f"{len(orders)} orders | mean {orders.mean():.1f} | median {orders.median():.1f} | skew {orders.skew():.1f}")

19773 orders | mean 519.5 | median 302.6 | skew 52.9

The mean is far above the median and the skew is extreme: a long tail of large orders, typical of business data such as order values, incomes, durations and prices. That shape matters for choosing a method.

Four methods, very different answers

z = (orders - orders.mean()) / orders.std()
q1, q3 = orders.quantile([0.25, 0.75])
iqr = q3 - q1
mad = (orders - orders.median()).abs().median()
modified_z = 0.6745 * (orders - orders.median()) / mad
log_z = (np.log10(orders) - np.log10(orders).mean()) / np.log10(orders).std()
flags = {
    "z-score beyond 3": (z.abs() > 3).sum(),
    "Tukey fence, 1.5 x IQR": ((orders < q1 - 1.5 * iqr) | (orders > q3 + 1.5 * iqr)).sum(),
    "Tukey fence, 3 x IQR ('far out')": ((orders < q1 - 3 * iqr) | (orders > q3 + 3 * iqr)).sum(),
    "modified z-score beyond 3.5": (modified_z.abs() > 3.5).sum(),
    "z-score of log value beyond 3 (high)": (log_z > 3).sum(),
    "z-score of log value beyond 3 (low)": (log_z < -3).sum(),
}
for method, count in flags.items():
    print(f"{method:<40} {count:>5} orders")

z-score beyond 3                           118 orders
Tukey fence, 1.5 x IQR                    1797 orders
Tukey fence, 3 x IQR ('far out')          1018 orders
modified z-score beyond 3.5               1488 orders
z-score of log value beyond 3 (high)        53 orders
z-score of log value beyond 3 (low)        299 orders

• The z-score measures distance from the mean in standard deviations. On skewed data it is weakened by the very outliers it looks for: they inflate the standard deviation, so only 118 orders qualify.
• Tukey's fences flag values beyond 1.5 times the interquartile range outside the middle half of the data, or 3 times for "far out" values (Wikipedia: outlier). Because the quartiles are not affected by extremes, they flag many more: 1,797 at the usual setting.
• The modified z-score replaces the mean and standard deviation with the median and the median absolute deviation, with values beyond 3.5 labelled potential outliers (NIST: detection of outliers). It is robust in the same way, flagging 1,488.
• A z-score on the logarithm treats the data on a multiplicative scale, which suits values that vary by orders of magnitude. It flags only 53 unusually large orders, and also 299 unusually small ones that the other methods cannot see, because their lower limits fall below zero.

No method is correct in general. For skewed business data, the robust methods and the log scale are usually more informative than the plain z-score, and the counts show that the threshold is a choice, not a fact. Treat flags as a review list.

Look at what was flagged

print(orders.nlargest(3).round(1).to_dict())
print(f"top 1% of orders by value: {orders.nlargest(len(orders) // 100).sum() / orders.sum():.1%} of revenue")

{'581483': 168469.6, '541431': 77183.6, '574941': 52940.9}
top 1% of orders by value: 19.2% of revenue

The two largest orders, over 168,000 and 77,000, are not valuable customers: in the raw data both were cancelled within minutes by matching cancellation invoices, as data cleaning, step by step shows. They are real records that should not count as sales. Meanwhile, the top 1% of orders carry 19.2% of revenue; most of those are genuine large wholesale orders that the business depends on. Removing even just the top 1% "as outliers" would delete almost a fifth of revenue.

Deciding what to do with each outlier

For summaries of a typical case, report the median, or a mean calculated after capping values at a high percentile, alongside the uncapped total. For models, consider a log transformation rather than deletion. Always report how many values were changed or excluded and why.

Detecting outliers within groups

A value can be ordinary overall but extreme for its group: an order that is normal for a wholesale customer could be extraordinary for a small retail customer, or for a small country. Running the same methods per group, such as per customer segment, product category or country, finds problems that a single threshold misses. It also avoids flagging whole groups that are simply larger than others.

Tools

The methods above need nothing more than a spreadsheet or a few lines of pandas: quartiles, medians and standard deviations are built into every analytics tool. Box plots show Tukey's fences visually, and a histogram on a log scale reveals whether a long tail hides separate clusters. More advanced, multivariate techniques belong to anomaly detection; anomaly detection techniques covers them.

For checking a dataset before analysis, see how to audit a messy dataset. Outlier detection is part of the exploratory data analysis module of the Data Analytics course.

Sources

• What are outliers in the data? (NIST/SEMATECH e-Handbook of Statistical Methods) — NIST
• Outlier: Tukey's fences (Wikipedia) — Wikipedia
• Detection of outliers: modified Z-scores (NIST/SEMATECH e-Handbook) — NIST
• Online Retail (UCI Machine Learning Repository) — UCI Machine Learning Repository