Probability for Data Science: Concepts You Must Know

Data science is reasoning under uncertainty, and probability is its language. You do not need measure theory to use it well. You need a handful of ideas that come up constantly: interpreting a test result, choosing between actions, modelling counts, understanding why averages behave, and judging whether a model's probabilities mean what they say. Each idea below is computed exactly and checked by simulation, the most reliable way to build intuition.

1. Conditional probability and Bayes' rule

A screening test catches 95% of people who have a condition (its sensitivity) and correctly clears 90% of those who do not (its specificity). The condition affects 1% of people. Someone tests positive. How likely are they to have it?

import numpy as np
from scipy import stats

prevalence, sensitivity, specificity = 0.01, 0.95, 0.90
p_positive = sensitivity * prevalence + (1 - specificity) * (1 - prevalence)
print("P(condition | positive), Bayes:", round(sensitivity * prevalence / p_positive, 4))

rng = np.random.default_rng(0)
people = 1_000_000
has_condition = rng.random(people) < prevalence
tests_positive = np.where(has_condition, rng.random(people) < sensitivity, rng.random(people) < 1 - specificity)
print("simulated:", round(has_condition[tests_positive].mean(), 4),
      f"({(has_condition & tests_positive).sum()} true positives among {tests_positive.sum()} positives)")

P(condition | positive), Bayes: 0.0876
simulated: 0.0877 (9477 true positives among 108033 positives)

Under 9%. The test is good, but the condition is rare, so the 10% false-positive rate applied to the 99% of healthy people produces far more positives than the 95% sensitivity applied to the 1% who are ill. Ignoring the base rate like this is a well-documented reasoning error, the base rate fallacy (Wikipedia: base rate fallacy). The same arithmetic governs fraud alerts, spam filters and defect detection: when positives are rare, even an accurate classifier's alarms are mostly false. Counting simulated people, as above, is often clearer than the formula.

2. Expected value: probability plus consequences

Decisions need more than probabilities; they need costs and benefits. Suppose a churn model gives each customer a probability of leaving. A retention offer costs 200, a retained customer is worth 5,000, and the offer cuts a customer's churn risk by a third. Who should get it?

customer_value, offer_cost, churn_reduction = 5000, 200, 1 / 3
for p_churn in (0.03, 0.12, 0.30):
    expected_saving = p_churn * churn_reduction * customer_value
    print(f"churn risk {p_churn:.2f}: expected saving {expected_saving:.0f} vs cost {offer_cost} ->",
          "send offer" if expected_saving > offer_cost else "skip")

churn risk 0.03: expected saving 50 vs cost 200 -> skip
churn risk 0.12: expected saving 200 vs cost 200 -> skip
churn risk 0.30: expected saving 500 vs cost 200 -> send offer

The break-even risk is 0.12: above it, the offer pays for itself on average. This is how a model's probabilities turn into a decision threshold, and why the threshold should come from costs rather than defaulting to 0.5. It also only works if the probabilities are accurate, which is the subject of section 5.

3. Distributions for counts: binomial and Poisson

Two distributions describe most counts you will meet (NIST: gallery of distributions):

• Binomial: the number of successes in a fixed number of independent trials, such as sign-ups among 500 visitors.
• Poisson: the number of events in a fixed interval when they occur independently at a steady average rate, such as support tickets per hour.

Both answer the question "is this surprising?":

print("P(30 or more sign-ups | 500 visitors, 4% rate):", round(stats.binom.sf(29, 500, 0.04), 4))
print("P(12 or more tickets in an hour | average 6):", round(stats.poisson.sf(11, 6), 4))

P(30 or more sign-ups | 500 visitors, 4% rate): 0.0196
P(12 or more tickets in an hour | average 6): 0.0201

With a usual 4% sign-up rate, 20 sign-ups are expected from 500 visitors; 30 or more would happen by chance only about 2% of the time, so it is worth investigating. The same holds for 12 tickets in an hour that normally averages 6. SciPy's sf, the survival function, gives the probability of exceeding a value directly (SciPy: scipy.stats); sf(29, ...) means more than 29, that is, 30 or more. Both models rest on independence: if one event makes others more likely, as when an outage triggers a wave of tickets, real counts spread wider than the model predicts.

4. Why averages become normal: the central limit theorem

Many quantities are skewed: order values, session lengths, incomes. Yet averages of them behave predictably. The simulation draws from an exponential distribution, which is strongly skewed with a mean of 100, and looks at averages of different sample sizes:

print("skew of the individual values:", round(stats.skew(rng.exponential(scale=100, size=1_000_000)), 2))
for n in (1, 5, 30, 200):
    means = rng.exponential(scale=100, size=(20_000, n)).mean(axis=1)
    print(f"means of {n:>3}: average {means.mean():.1f}, spread {means.std():.1f} (100/sqrt(n) = {100 / np.sqrt(n):.1f}), skew {stats.skew(means):.2f}")

skew of the individual values: 2.0
means of   1: average 100.9, spread 101.5 (100/sqrt(n) = 100.0), skew 2.01
means of   5: average 99.3, spread 44.4 (100/sqrt(n) = 44.7), skew 0.88
means of  30: average 99.8, spread 18.1 (100/sqrt(n) = 18.3), skew 0.34
means of 200: average 100.1, spread 7.0 (100/sqrt(n) = 7.1), skew 0.11

Three facts appear in the numbers. The averages centre on the true mean of 100, the law of large numbers. Their spread shrinks in proportion to one over the square root of the sample size, exactly as 100/√n predicts. And their skew falls towards zero, the shape of a normal distribution: the central limit theorem. This is why confidence intervals and A/B tests on averages work even when individual values are far from normal. Notice, too, that 30 is not a magic number: at n = 30, the means are still visibly skewed for data this lopsided.

5. Calibration: can a model's probabilities be trusted?

Classifiers output numbers between 0 and 1, but those numbers are not automatically probabilities. A model is well calibrated if, among cases it scores at 80%, about 80% are actually positive (scikit-learn: probability calibration). Two models on the MAGIC telescope data used in overfitting and underfitting, compared:

from sklearn.calibration import calibration_curve
from sklearn.datasets import fetch_openml
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

data = fetch_openml(data_id=1120, as_frame=True)
X, y = data.data, (data.target == "h").astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)
for name, model in {"naive Bayes": GaussianNB(),
                    "logistic regression": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))}.items():
    proba = model.fit(X_train, y_train).predict_proba(X_test)[:, 1]
    observed, predicted = calibration_curve(y_test, proba, n_bins=5, strategy="quantile")
    confident = proba >= 0.9
    print(f"{name}: predicted {predicted.round(2)} | observed {observed.round(2)}")
    print(f"  cases given 90% or more: {confident.sum()}, actually positive: {y_test[confident].mean():.1%}")

naive Bayes: predicted [0.   0.01 0.02 0.16 0.9 ] | observed [0.13 0.18 0.35 0.43 0.67]
  cases given 90% or more: 828, actually positive: 77.3%
logistic regression: predicted [0.08 0.14 0.23 0.46 0.84] | observed [0.08 0.12 0.23 0.52 0.81]
  cases given 90% or more: 398, actually positive: 95.5%

Naive Bayes is badly overconfident: where it says 0% to 2%, 13% to 35% are actually positive, and of the cases it scores at 90% or more, only 77% are. The scikit-learn documentation notes this tendency of Gaussian naive Bayes to push probabilities towards 0 or 1. Logistic regression's predicted and observed columns nearly match. Feed naive Bayes probabilities into the expected-value rule from section 2 and the decisions would be wrong, even if its ranking of cases were decent. Check calibration before using probabilities as probabilities, and recalibrate if needed.

6. Independence, and when it fails

Almost every formula above assumes independent events. Real data often violates it: repeat customers, clustered failures, sessions from the same user, measurements from the same machine. The consequence is usually overconfidence: results look more certain than they are. When observations are grouped, analyse at the group level or use methods that account for the grouping, and split data for validation by group, not by row.

Where each concept appears in daily work

Probability underpins the analysis of experiments; A/B testing for data scientists applies it to real product decisions. Probability, distributions and hypothesis testing are part of the mathematics and statistics module of the Data Science course.

Sources

• Base rate fallacy (Wikipedia) — Wikipedia
• Gallery of Distributions (NIST/SEMATECH e-Handbook of Statistical Methods) — NIST
• Statistical functions, scipy.stats (SciPy) — SciPy documentation
• Probability calibration (scikit-learn) — scikit-learn documentation