Exploratory Data Analysis (EDA): A Step-by-Step Walkthrough

Exploratory data analysis (EDA) is the work of understanding a dataset before you model it: what each row represents, what is missing, how values are distributed, how variables relate, and what could mislead you. The NIST handbook describes it as an attitude more than a set of techniques, aimed at uncovering structure, spotting anomalies and testing assumptions (NIST: what is EDA?). Skipping it is how models end up learning from errors, gaps and leaked answers.

This walkthrough explores a real dataset step by step in pandas: the Titanic passenger list of 1,309 people, with whether each survived (OpenML: titanic). It is well known, but it hides several traps that show why EDA matters. Each step ends with what the finding means for modelling.

Step 1: Load the data and ask what a row is

import pandas as pd
from sklearn.datasets import fetch_openml

df = fetch_openml("titanic", version=1, as_frame=True).frame
df["survived"] = df["survived"].astype(int)
print(df.shape)
print(df.columns.tolist())

(1309, 14)
['pclass', 'survived', 'name', 'sex', 'age', 'sibsp', 'parch', 'ticket', 'fare', 'cabin', 'embarked', 'boat', 'body', 'home.dest']

Each row is a passenger. Before any statistics, read the column list and write down what each means and when it was known. sibsp and parch count siblings or spouses and parents or children aboard; boat is the lifeboat and body a body identification number. That second question, when each value became known, will matter in step 8.

Step 2: Missing values, and whether missingness means something

missing = df.isna().mean().round(3)
print(missing[missing > 0].sort_values(ascending=False).to_dict())
print("survival when age is missing:", round(df.loc[df["age"].isna(), "survived"].mean(), 3),
      "| when known:", round(df.loc[df["age"].notna(), "survived"].mean(), 3))

{'body': 0.908, 'cabin': 0.775, 'boat': 0.629, 'home.dest': 0.431, 'age': 0.201, 'embarked': 0.002, 'fare': 0.001}
survival when age is missing: 0.278 | when known: 0.408

Cabin is missing for 78% of passengers, too sparse to use directly. Age is missing for 20%, and the missing group survived far less often, 27.8% against 40.8%. Missingness is not random here. Grouping by class shows age is missing for 29.3% of third-class passengers against 12.1% in first and 5.8% in second, so the gaps fall mostly on the group that survived least. Decision: when filling missing ages, also keep an indicator column recording that age was missing, because the gap itself carries information. Data preprocessing shows how to do both inside a pipeline.

Step 3: Look at the target first

print("survival rate:", round(df["survived"].mean(), 3))

survival rate: 0.382

38.2% survived. This is the baseline every later finding is compared with, and a model that predicted "did not survive" for everyone would already be right 61.8% of the time. Decision: judge models against that baseline, with metrics that account for it.

Step 4: Summarise each numeric column

print(df[["age", "fare", "sibsp", "parch"]].describe().round(2))
print("fare skew:", round(df["fare"].skew(), 2), "| zero fares:", int((df["fare"] == 0).sum()))

           age     fare    sibsp    parch
count  1046.00  1308.00  1309.00  1309.00
mean     29.88    33.30     0.50     0.39
std      14.41    51.76     1.04     0.87
min       0.17     0.00     0.00     0.00
25%      21.00     7.90     0.00     0.00
50%      28.00    14.45     0.00     0.00
75%      39.00    31.28     1.00     0.00
max      80.00   512.33     8.00     9.00
fare skew: 4.37 | zero fares: 17

Read the summary column by column. Ages run from a few months (0.17 years) to 80, plausible values. Fare is the odd one: the median is 14.45 but the mean is 33.30 and the maximum 512.33, a strong right skew, and 17 passengers paid nothing. A histogram would show a tall bar near zero and a long thin tail; plotting the logarithm of fare would spread it out. Extreme values deserve a question before a transformation: are they errors, or real?

Step 5: Investigate the oddity instead of trimming it

top = df.loc[df["fare"] == df["fare"].max()]
print(len(top), "passengers paid", top["fare"].iloc[0], "on ticket", top["ticket"].unique().tolist())

group_size = df.groupby("ticket")["ticket"].transform("size")
print("passengers sharing a ticket:", round((group_size > 1).mean(), 3))
print("tickets with one fare for all holders:", round((df.groupby("ticket")["fare"].nunique() <= 1).mean(), 3))

df["fare_per_person"] = df["fare"] / group_size
print(df.groupby("pclass")[["fare", "fare_per_person"]].median().round(2))

4 passengers paid 512.3292 on ticket ['PC 17755']
passengers sharing a ticket: 0.455
tickets with one fare for all holders: 0.999
         fare  fare_per_person
pclass                        
1       60.00            29.70
2       15.05            12.35
3        8.05             7.75

The highest fare was not paid four times; four people shared one ticket, and each row repeats the ticket's total. 45.5% of passengers share a ticket, and on 99.9% of tickets every holder shows the same fare. The fare column is a group price, not a personal one. Dividing by the number of holders halves the median first-class fare and narrows the gap between classes. Decision: use fare per person, and consider group size as a feature of its own. Trimming the "outliers" would have thrown away real passengers and missed the real problem.

Step 6: Relate variables to the target, and to each other

print(df.groupby("sex", observed=True)["survived"].agg(["mean", "size"]).round(3))
print(pd.crosstab(df["sex"], df["pclass"], values=df["survived"], aggfunc="mean").round(2))

         mean  size
sex                
female  0.727   466
male    0.191   843
pclass     1     2     3
sex                     
female  0.97  0.89  0.49
male    0.34  0.15  0.15

Women survived at 72.7%, men at 19.1%. The cross-tabulation, which computes the mean survival for every combination of sex and class (pandas: crosstab), shows more than either variable alone: 97% of first-class women survived but only 49% of third-class women, while men in second and third class fared equally badly. The effect of class depends on sex, an interaction. Decision: prefer models that capture interactions, such as tree ensembles, or add interaction features to a linear model.

Step 7: Check what a correlation misses

print("correlation of age with survival:", round(df["age"].corr(df["survived"]), 2))
bands = pd.cut(df["age"], [0, 12, 18, 40, 60, 80])
print(df.groupby(bands, observed=True)["survived"].agg(["mean", "size"]).round(3))

correlation of age with survival: -0.06
           mean  size
age                  
(0, 12]   0.574    94
(12, 18]  0.414    99
(18, 40]  0.387   626
(40, 60]  0.423   194
(60, 80]  0.242    33

A correlation of -0.06 suggests age barely matters. The bands say otherwise: children up to 12 survived at 57.4% and passengers over 60 at 24.2%, with little difference in between. Correlation measures only straight-line relationships, and this one is not a straight line. Decision: never drop a feature because of a low correlation alone; look at the relationship in groups or on a chart.

Step 8: Hunt for leakage

Some columns record what happened after the outcome. Here is how survival looks against whether boat and body were recorded:

for column in ("boat", "body"):
    print(df.groupby(df[column].notna())["survived"].agg(["mean", "size"]).round(3).rename_axis(f"{column} recorded"))

                mean  size
boat recorded             
False          0.028   823
True           0.981   486
                mean  size
body recorded             
False          0.421  1188
True           0.000   121

98.1% of passengers with a lifeboat recorded survived; no passenger with a body number did. These columns are consequences of the outcome, not causes, and would not exist at prediction time. A model given them would score superbly in testing and be useless in practice, the classic form of data leakage (scikit-learn: data leakage). Decision: drop boat and body. The same check applies to any dataset: for each column, ask whether its value would be known at the moment you need the prediction.

Step 9: Final checks, then write it down

print("duplicate rows:", int(df.duplicated().sum()))
print(df["embarked"].value_counts(dropna=False).to_dict())

duplicate rows: 0
{'S': 914, 'C': 270, 'Q': 123, nan: 2}

No duplicates, and only two passengers lack a port of embarkation, easily filled with the most common value. EDA is finished when its findings are written as decisions others can check:

Choosing charts for each question

The same steps work visually, and charts are how findings are usually shared:

• Distribution of one number: histogram or box plot, on a log scale for skewed values such as fare.
• Rates across groups: bar chart of the survival rate per group, with group sizes labelled.
• Two categories together: a heat grid of the cross-tabulation above.
• Two numbers: scatter plot, with transparency when points overlap.
• Missing values: bar chart of missing percentage per column.

Show the numbers behind any chart you present; a chart without group sizes can make a group of 33 passengers look as solid as one of 626.

The decisions above feed straight into feature engineering, the next stage of the workflow. Data understanding and summaries, visualisation with Matplotlib and Seaborn, outlier detection, feature relationships and data storytelling make up the exploratory data analysis module of the Data Science course.

Sources

• What is EDA? (NIST/SEMATECH e-Handbook of Statistical Methods) — NIST
• titanic (OpenML dataset 40945) — www.openml.org
• Common pitfalls: data leakage (scikit-learn) — scikit-learn documentation
• pandas.crosstab — pandas documentation