Statistical Concepts Every Data Analyst Must Know

Most analysts do not need advanced statistics, but they cannot do without the basics. Every week they report averages that may be skewed, rates from samples, and differences that might be noise. The concepts below are the ones that come up again and again, each shown on the same real data: 12,330 sessions on an online shop, 1,908 of which ended in a purchase (UCI: Online Shoppers Purchasing Intention).

import io
import urllib.request
import zipfile
import numpy as np
import pandas as pd
from scipy import stats

url = "https://archive.ics.uci.edu/static/public/468/online+shoppers+purchasing+intention+dataset.zip"
sessions = pd.read_csv(zipfile.ZipFile(io.BytesIO(urllib.request.urlopen(url).read())).open("online_shoppers_intention.csv"))
bought = sessions["Revenue"].astype(int)
print(f"{len(bought)} sessions, conversion rate {bought.mean():.3f}")

12330 sessions, conversion rate 0.155

1. Distributions: the average is not the typical value

minutes = sessions["ProductRelated_Duration"] / 60
print(f"minutes on product pages: mean {minutes.mean():.1f}, median {minutes.median():.1f}, "
      f"90th percentile {minutes.quantile(0.9):.1f}")

minutes on product pages: mean 19.9, median 10.0, 90th percentile 48.0

The average session spends almost 20 minutes on product pages, but the typical session spends 10. A minority of very long sessions pulls the mean up. Durations, order values, incomes and response times are nearly always skewed this way. Report the median for "typical", the mean when totals matter, and a high percentile to show the tail. Outlier detection for analysts covers what to do with the extremes.

2. Sampling variability: a sample is one draw of many

Analysts rarely see everything; they see a week, a survey or a test group. How much would the answer change with a different sample?

rng = np.random.default_rng(42)
estimates = [bought.sample(300, random_state=rng).mean() for _ in range(1000)]
print(f"1,000 samples of 300 sessions: middle 95% of estimates "
      f"{np.percentile(estimates, 2.5):.3f} to {np.percentile(estimates, 97.5):.3f}")

1,000 samples of 300 sessions: middle 95% of estimates 0.117 to 0.193

The true rate is 15.5%, yet samples of 300 sessions give anything from about 12% to 19%. A report comparing this week's 13% with last week's 18% from samples of that size may be describing nothing but chance. The spread of an estimate across samples is its standard error, and it shrinks as samples grow (Wikipedia: standard error).

3. Confidence intervals: report the uncertainty with the number

A confidence interval turns that spread into a range reported alongside the estimate. For rates, the Wilson method described in the NIST handbook works well even for small samples and rates near zero (NIST: confidence intervals for proportions):

def wilson(successes, n, z=1.96):
    p = successes / n
    centre = (p + z**2 / (2 * n)) / (1 + z**2 / n)
    half = z * np.sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / (1 + z**2 / n)
    return centre - half, centre + half

groups = sessions.groupby("Weekend")["Revenue"].agg(["sum", "size"])
for weekend, (x, n) in groups.iterrows():
    low, high = wilson(x, n)
    print(f"{'weekend' if weekend else 'weekday'}: {x}/{n} = {x / n:.3f}, 95% CI {low:.3f} to {high:.3f}")

weekday: 1409/9462 = 0.149, 95% CI 0.142 to 0.156
weekend: 499/2868 = 0.174, 95% CI 0.161 to 0.188

Weekend sessions convert at 17.4%, with an interval of 16.1% to 18.8%; weekday sessions at 14.9%, with a narrower interval because there are more of them. Showing intervals, or at least sample sizes, stops readers treating every difference as real.

4. Hypothesis tests and p-values

Is the weekend difference more than chance? A two-proportion z-test asks how surprising the difference would be if weekend and weekday sessions really converted at the same rate:

def two_proportion_p(x1, n1, x2, n2):
    pooled = (x1 + x2) / (n1 + n2)
    z = (x1 / n1 - x2 / n2) / np.sqrt(pooled * (1 - pooled) * (1 / n1 + 1 / n2))
    return z, 2 * stats.norm.sf(abs(z))

(x1, n1), (x2, n2) = groups.loc[True], groups.loc[False]
z, p = two_proportion_p(x1, n1, x2, n2)
print(f"difference {x1 / n1 - x2 / n2:.3f} ({x1 / n1 / (x2 / n2) - 1:.0%} relative), z = {z:.2f}, p = {p:.4f}")

difference 0.025 (17% relative), z = 3.25, p = 0.0011

A p-value of 0.0011 means a difference this large would be rare if there were no real difference. It does not mean there is a 99.9% chance that weekends convert better. The American Statistical Association's statement on p-values is explicit: a p-value does not measure the probability that a hypothesis is true, nor the size or importance of an effect, and decisions should not rest only on whether it passes a threshold such as 0.05 (ASA statement on p-values).

5. Effect size: significant is not the same as important

The weekend effect is 2.5 percentage points, 17% in relative terms. Whether that matters is a business question: it might justify weekend campaigns, or it might be too small to act on. With very large samples, trivially small differences become statistically significant; with small samples, important differences can fail to reach significance. Always report the size of the difference, with its interval, next to any test result. The test also says nothing about why: weekend visitors may simply be different people with different intentions, which is the point of correlation vs causation.

6. Small groups produce extreme rates

by_browser = sessions.groupby("Browser")["Revenue"].agg(sessions="size", conversion="mean").round(3)
print(by_browser.sort_values("conversion", ascending=False).to_string())

         sessions  conversion
Browser                      
12             10       0.300
13             61       0.262
10            163       0.196
5             467       0.184
4             736       0.177
11              6       0.167
8             135       0.156
2            7961       0.154
1            2462       0.148
7              49       0.122
6             174       0.115
3             105       0.048
9               1       0.000

The best-converting browser has 10 sessions; the worst has one. The large groups sit close to the overall 15.5%. Rankings of stores, sales reps, regions or pages always show this pattern: the top and bottom of the list are crowded with small groups, because small samples vary most. Before praising or blaming a small group, check its size and interval.

7. Test many things and something will look significant

rng = np.random.default_rng(7)
false_alarms = 0
for _ in range(100):
    fake_segment = rng.random(len(bought)) < 0.5
    x1, n1 = bought[fake_segment].sum(), fake_segment.sum()
    x2, n2 = bought[~fake_segment].sum(), (~fake_segment).sum()
    false_alarms += two_proportion_p(x1, n1, x2, n2)[1] < 0.05
print(f"random splits with p < 0.05: {false_alarms} of 100")

random splits with p < 0.05: 4 of 100

Each of these 100 "segments" is pure coin-flip noise, yet four of them look significant at the 5% level, close to the one in 20 expected. Slicing a dashboard by every dimension until something stands out is the same exercise. Decide the comparisons in advance, treat surprising findings from wide searches as hypotheses to confirm on new data, and adjust thresholds when running many tests.

Choosing a method for the question

A checklist before reporting a difference

1. How big is each group, and how wide is each interval?
2. Is the metric skewed, so that a median tells the story better than a mean?
3. How large is the difference in business terms, not just in significance?
4. Was this comparison planned, or found by searching through many?
5. What else differs between the groups that could explain the result?

For how statistics fits into the wider analytics process, see business analytics explained. Summary statistics, distributions and correlation analysis are part of the exploratory data analysis module of the Data Analytics course.

Sources

• Online Shoppers Purchasing Intention Dataset (UCI Machine Learning Repository) — UCI Machine Learning Repository
• Confidence intervals for proportions (NIST/SEMATECH e-Handbook of Statistical Methods) — NIST
• The ASA Statement on p-Values: Context, Process, and Purpose — American Statistical Association, 2016
• Standard error (Wikipedia) — Wikipedia