Statistics for Data Science: The 20% You Actually Need

Statistics courses cover a great deal that data scientists rarely use, and skip habits they need every day. The core is smaller than it looks: summarise data honestly, say how sure you are, test claims without fooling yourself, and recognise when a comparison is misleading. Each idea below is shown on real data with the code that produced the numbers.

1. Summaries: when the median beats the mean

The German credit dataset records 1,000 loans, including each loan's amount and duration and whether the borrower turned out to be a good or bad credit risk (OpenML: credit-g). Loan amounts are typical of money data: most are modest, a few are large.

import numpy as np
import pandas as pd
from scipy import stats
from sklearn.datasets import fetch_openml

credit = fetch_openml(data_id=31, as_frame=True).frame
amount = credit["credit_amount"]
print(f"{len(amount)} loans | mean {amount.mean():.1f} | median {amount.median():.1f} | skew {stats.skew(amount):.2f}")
print("10th and 90th percentiles:", amount.quantile([0.1, 0.9]).round(1).tolist())

1000 loans | mean 3271.3 | median 2319.5 | skew 1.95
10th and 90th percentiles: [932.0, 7179.4]

The mean is 41% higher than the median, pulled up by a long tail of large loans. For a "typical loan", report the median; for totals, such as how much money is lent, the mean is the right basis because it multiplies back to the sum. Percentiles describe the spread better than a standard deviation for skewed data: 80% of loans fall between 932 and 7,179.4 in the dataset's currency units. Always look at the shape before choosing the summary.

2. Uncertainty: confidence intervals by bootstrapping

A sample statistic is an estimate. A confidence interval says how precise it is. The bootstrap makes intervals for almost any statistic without formulas: resample the data with replacement many times, recompute the statistic each time, and take the middle 95% of the results.

rng = np.random.default_rng(0)
boot_medians = [np.median(rng.choice(amount, len(amount))) for _ in range(5000)]
boot_means = [np.mean(rng.choice(amount, len(amount))) for _ in range(5000)]
print("95% interval for the median:", np.percentile(boot_medians, [2.5, 97.5]).round(1))
print("95% interval for the mean:  ", np.percentile(boot_means, [2.5, 97.5]).round(1))

95% interval for the median: [2171.  2443.5]
95% interval for the mean:   [3100.  3445.7]

Report estimates with intervals like these, not as bare numbers. "The median loan is about 2,320, and plausibly between 2,171 and 2,444" tells a decision-maker how much to trust the figure. The bootstrap assumes the sample represents the population; it cannot fix a biased sample, which is why how data was collected matters as much as how it is analysed.

3. Comparing groups: p-values and effect sizes

Do loans that went bad tend to be longer? Compare durations for the two groups, with two tests, and measure how big the difference is:

good = credit.loc[credit["class"] == "good", "duration"]
bad = credit.loc[credit["class"] == "bad", "duration"]
print(f"loan duration in months - good risks: median {good.median():.0f}, bad risks: median {bad.median():.0f}")
print(f"Welch t-test p = {stats.ttest_ind(bad, good, equal_var=False).pvalue:.1e}, "
      f"Mann-Whitney p = {stats.mannwhitneyu(bad, good).pvalue:.1e}")
cohens_d = (bad.mean() - good.mean()) / np.sqrt((bad.var() + good.var()) / 2)
print(f"effect size (Cohen's d): {cohens_d:.2f}")

loan duration in months - good risks: median 18, bad risks: median 24
Welch t-test p = 2.4e-10, Mann-Whitney p = 8.0e-11
effect size (Cohen's d): 0.46

Bad loans had a median duration of 24 months against 18. Both tests give tiny p-values: Welch's t-test compares means without assuming equal variances, and the Mann-Whitney test compares the groups by ranks, without assuming normal data (SciPy: mannwhitneyu). A p-value says how surprising the data would be if there were no difference. It is not the probability that the difference is real, and it says nothing about how large the difference is; the American Statistical Association's statement on p-values makes exactly these points (ASA: statement on p-values). The effect size does that job: a Cohen's d of 0.46 means the average bad loan is about half a standard deviation longer, a moderate difference. With large samples, trivial differences become "significant"; always report the size alongside the p-value.

And note what this does not show: that long loans cause defaults. Longer loans may go to different borrowers or purposes. Observational comparisons describe associations.

4. Testing many things manufactures findings

A 5% significance level means a 5% false-alarm rate per test when nothing is going on. Run many tests and false alarms become likely. Simulate studies that each compare 20 pairs of groups drawn from the same distribution, so every "finding" is false:

def any_false_alarm(tests=20):
    pvalues = [stats.ttest_ind(rng.normal(size=50), rng.normal(size=50)).pvalue for _ in range(tests)]
    return min(pvalues) < 0.05

print("studies of 20 null comparisons with at least one p < 0.05:",
      f"{np.mean([any_false_alarm() for _ in range(1000)]):.1%} (theory: {1 - 0.95 ** 20:.1%})")

studies of 20 null comparisons with at least one p < 0.05: 66.2% (theory: 64.2%)

Two thirds of these studies would report a "significant" result from pure noise. This is what happens when an analyst checks twenty customer segments, twenty metrics or twenty dashboard filters and reports the one that "worked". Decide the main comparison in advance, adjust for multiple tests (the Bonferroni correction simply divides the threshold by the number of tests), and treat unplanned findings as hypotheses for a new test. A/B testing for data scientists applies the same discipline to experiments.

5. Simpson's paradox: when combining groups reverses the story

In 1973, the University of California, Berkeley, admitted a noticeably lower share of women than men to graduate study, which raised concerns of bias. The counts for the six largest departments:

berkeley = pd.DataFrame({
    "dept": list("AABBCCDDEEFF"), "sex": ["men", "women"] * 6,
    "applied":  [825, 108, 560, 25, 325, 593, 417, 375, 191, 393, 373, 341],
    "admitted": [512, 89, 353, 17, 120, 202, 138, 131, 53, 94, 22, 24]})
overall = berkeley.groupby("sex")[["admitted", "applied"]].sum()
print("overall admission rate:", (overall["admitted"] / overall["applied"]).round(3).to_dict())
by_dept = berkeley.assign(rate=berkeley["admitted"] / berkeley["applied"]).pivot(index="dept", columns="sex", values="rate")
print(by_dept.round(3).T.to_string())
print("applications by department:", berkeley.pivot(index="sex", columns="dept", values="applied").to_dict("index"))

overall admission rate: {'men': 0.445, 'women': 0.304}
dept       A     B      C      D      E      F
sex                                           
men    0.621  0.63  0.369  0.331  0.277  0.059
women  0.824  0.68  0.341  0.349  0.239  0.070
applications by department: {'men': {'A': 825, 'B': 560, 'C': 325, 'D': 417, 'E': 191, 'F': 373}, 'women': {'A': 108, 'B': 25, 'C': 593, 'D': 375, 'E': 393, 'F': 341}}

Overall, 44.5% of men and 30.4% of women were admitted. Department by department, women were admitted at a higher rate in four of the six. The explanation is in the last line: most men applied to departments A and B, which admitted most applicants, while most women applied to C, D, E and F, which admitted far fewer. Combining departments mixed two effects, where people applied and how they were treated, and produced a misleading total. These rates match those published for the case, whose department-level analysis found, if anything, a small but statistically significant bias in favour of women (Wikipedia: Simpson's paradox).

The lesson for any dashboard or report: before comparing groups, ask what else differs between them, and break the comparison down by the variables that could explain it.

The core, in one table

Descriptive statistics, probability, distributions, hypothesis testing and correlation make up the mathematics and statistics module of the Data Science course.

Sources

• ASA Statement on Statistical Significance and P-Values — American Statistical Association, 2016
• Simpson's paradox (Wikipedia) — Wikipedia
• credit-g (OpenML dataset 31) — www.openml.org
• scipy.stats.mannwhitneyu (SciPy) — SciPy documentation