Data Preprocessing: Cleaning Data the Right Way

Models need numbers in a consistent form: no missing values, comparable scales, categories turned into columns. Getting data into that form is preprocessing, and it is where many models quietly go wrong, not through bad code but through steps done in the wrong order. The central rule fits in one sentence: every step that learns something from data must learn it from the training set only. This guide applies that rule to missing values, scaling and encoding, and packages the result so it cannot be broken later.

Split first

Split the data into training and test sets before computing anything from it: before imputing medians, fitting scalers or choosing categories. A median computed on all rows includes the test rows, so the test set is no longer unseen, and the evaluation becomes optimistic. scikit-learn's guide to common pitfalls names this data leakage and recommends pipelines as the fix (scikit-learn: common pitfalls). Cleaning that learns nothing, such as fixing data types, trimming whitespace or removing exact duplicate rows, can safely happen before the split.

Missing values: fill them, and record that they were missing

A SimpleImputer replaces missing values with the mean, median, most frequent value or a constant. Missingness is often informative in itself: a customer who never gave their age may behave differently from one who did. add_indicator=True adds a column marking which rows were imputed, so the model can learn from the pattern (scikit-learn: imputation). Prefer the median for skewed numbers, and consider whether a missing value really means "unknown" or actually means zero, such as no purchases.

Scaling: which scaler and when

Distance-based and gradient-based models, such as k-nearest neighbours, SVMs, k-means, linear models with regularisation and neural networks, are sensitive to scale. Tree-based models, including random forests and gradient boosting, are not, so scaling them changes nothing (scikit-learn: preprocessing).

Categories: one-hot encoding that survives production

Nominal categories such as city or plan become one column per category with OneHotEncoder. Set handle_unknown="ignore": in production, a category the model never saw in training will appear sooner or later, and without this setting the pipeline raises an error instead of producing a prediction. For categories with a natural order, such as small, medium and large, an OrdinalEncoder with an explicit order is more compact.

Everything in one pipeline

ColumnTransformer applies a different preprocessing branch to each group of columns, and Pipeline chains it with the model, so fitting the pipeline fits every step on the training data only (scikit-learn: pipelines and composite estimators). The data below is twelve made-up customer records with the usual problems: missing ages, one extreme spender and text categories.

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import OneHotEncoder, RobustScaler, StandardScaler

df = pd.DataFrame({
    "age": [34, 51, np.nan, 29, 45, 62, 38, np.nan, 27, 55, 41, 33],
    "monthly_spend": [1200, 540, 3100, 80, 990, 150000, 700, 1800, 420, 1300, 860, 2500],
    "city": ["Hyderabad", "Pune", "Chennai", "Hyderabad", "Pune", "Chennai",
             "Hyderabad", "Pune", "Hyderabad", "Chennai", "Pune", "Hyderabad"],
    "plan": ["basic", "premium", "basic", "basic", "premium", "premium",
             "basic", "premium", "basic", "premium", "basic", "premium"],
    "churned": [0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0],
})
X, y = df.drop(columns="churned"), df["churned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)

preprocess = ColumnTransformer([
    ("age", make_pipeline(SimpleImputer(strategy="median", add_indicator=True), StandardScaler()), ["age"]),
    ("spend", RobustScaler(), ["monthly_spend"]),
    ("cats", OneHotEncoder(handle_unknown="ignore"), ["city", "plan"]),
])
model = Pipeline([("prep", preprocess), ("clf", LogisticRegression())]).fit(X_train, y_train)

print(list(model.named_steps["prep"].get_feature_names_out()))
print(model.named_steps["prep"].transform(X_train).shape)

['age__age', 'age__missingindicator_age', 'spend__monthly_spend', 'cats__city_Chennai', 'cats__city_Hyderabad', 'cats__city_Pune', 'cats__plan_basic', 'cats__plan_premium']
(9, 8)

Four raw columns became eight model features: the imputed and scaled age, a missing-age flag, robust-scaled spend, and one column per city and plan. In this compact setup the missing-age flag passes through the scaler too, which is harmless for a linear model; put the indicator in its own branch if you want it left as 0 and 1. A new customer from a city the model has never seen, with no age given, still gets a prediction:

new_customer = pd.DataFrame({"age": [np.nan], "monthly_spend": [950], "city": ["Bengaluru"], "plan": ["basic"]})
print(model.named_steps["prep"].transform(new_customer).round(2).tolist())
print(model.predict_proba(new_customer).round(3).tolist())

[[-0.29, 2.83, -0.05, 0.0, 0.0, 0.0, 1.0, 0.0]]
[[0.645, 0.355]]

The unseen city became all zeros instead of an error, the missing age was filled with the training median, and its flag is set. The fitted pipeline is one object: save it with the model and every future prediction goes through exactly the same steps, which is the main protection against training and production drifting apart.

Outliers: understand before you transform

The 150,000 monthly spend above could be a data error or a real, very valuable customer. Look before deciding. RobustScaler limits its influence on scaling; a log transform compresses skewed values; capping at a high percentile is another option. Deleting rows is the last resort, and only for values you have confirmed are wrong.

A preprocessing checklist

1. Types fixed, whitespace trimmed and exact duplicates removed, all before the split.
2. Train and test sets split, stratified by the target for classification.
3. Every learned step (imputers, scalers, encoders, feature selection) inside one pipeline, fitted on training data.
4. Missing values imputed with a strategy chosen per column, with indicators where missingness may mean something.
5. Scalers chosen to match the model and the data's outliers; none needed for tree-based models.
6. Categorical encoders set to handle unseen categories.
7. The fitted pipeline saved as a single object and used for every prediction.

If a step cannot be expressed inside the pipeline, that is a warning sign that it may be learning from data it should not see.

Preprocessing follows exploration; the EDA walkthrough shows how to find the problems first, and feature engineering builds on the cleaned data. Handling missing values, encoding and feature scaling form the preprocessing and feature engineering module of the Data Science course.

Sources

• Pipelines and composite estimators — scikit-learn documentation
• Imputation of missing values — scikit-learn documentation
• Preprocessing data — scikit-learn documentation
• Common pitfalls and recommended practices — scikit-learn documentation