End-to-End Machine Learning Project: A Complete Walkthrough

Most tutorials stop when a model prints a score. Real projects start earlier, with a question and a way of measuring success, and end later, with a model someone can use and a plan for noticing when it stops working. This walkthrough runs one project through every stage on real data, the California housing dataset, so each decision and its effect are visible. The dataset is small enough to run on a laptop in seconds, which makes it ideal for practising the process rather than fighting infrastructure.

Step 1: Frame the problem before touching code

The task: predict the median house value of a California district from census data about it, such as median income, house age, average rooms and location. The dataset has 20,640 districts from the 1990 US census, eight numeric features, and a target measured in hundreds of thousands of dollars (scikit-learn: California housing).

Framing means writing down three things before modelling:

• The decision the model supports. Here, say, flagging districts whose prices look out of line with their characteristics for a closer look.
• The metric. Mean absolute error (MAE), the average size of a miss in the target's own units, because a stakeholder can understand "off by about 30,000 dollars on average" and it is not dominated by a few extreme misses.
• What good enough means. A useful model must clearly beat the simplest possible guess; without a baseline, a score has no meaning.

Skipping this step is the most common reason projects drift: without a fixed metric, every new experiment gets judged by a different number.

Step 2: Load the data and lock away a test set

import numpy as np
import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split

df = fetch_california_housing(as_frame=True).frame
print(df.shape, "| median target:", round(df["MedHouseVal"].median(), 3), "| max:", round(df["MedHouseVal"].max(), 3))

X = df.drop(columns="MedHouseVal")
y = df["MedHouseVal"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
print(len(X_train), "training rows,", len(X_test), "test rows")

(20640, 9) | median target: 1.797 | max: 5.0
16512 training rows, 4128 test rows

The test set is split off first and then left alone until the very end. Every look at it while choosing models leaks a little information and makes the final score optimistic (scikit-learn: cross-validation). Exploration, feature ideas and model comparisons all happen on the training rows. Exploratory data analysis belongs here too, on the training data: distributions, missing values, and relationships between income and price. Note the maximum value of exactly 5.0; it will matter later.

Step 3: Set a baseline, then compare candidates with cross-validation

A baseline that always predicts the median tells you what zero skill looks like. Candidates are then compared with 5-fold cross-validation on the training data only:

from sklearn.dummy import DummyRegressor
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

candidates = {
    "median baseline": DummyRegressor(strategy="median"),
    "ridge regression": make_pipeline(StandardScaler(), Ridge()),
    "gradient boosting": HistGradientBoostingRegressor(random_state=0),
}
for name, model in candidates.items():
    mae = -cross_val_score(model, X_train, y_train, cv=5, scoring="neg_mean_absolute_error").mean()
    print(f"{name}: MAE {mae:.3f}")

median baseline: MAE 0.886
ridge regression: MAE 0.529
gradient boosting: MAE 0.316

Linear regression cuts the baseline error by about 40%; gradient boosting cuts it by about 64%. Boosted trees capture non-linear effects, such as prices rising steeply with income or depending sharply on location, that a linear model cannot, and they need no feature scaling (scikit-learn: histogram-based gradient boosting). The scaler sits inside the ridge pipeline, so it is fitted only on each training fold. Cross-validation in machine learning explains why these fold averages are more trustworthy than a single split.

This is also where you would tune hyperparameters and try new features, still inside cross-validation. Resist tuning for long before the error analysis in step 5: knowing where the model fails usually suggests better changes than a wider parameter search.

Step 4: One honest evaluation on the test set

With a model chosen, fit it on all the training data and score it once on the test set:

from sklearn.metrics import mean_absolute_error

model = HistGradientBoostingRegressor(random_state=0).fit(X_train, y_train)
pred = model.predict(X_test)
print("test MAE:", round(mean_absolute_error(y_test, pred), 3))

test MAE: 0.307

The test MAE of 0.307 is close to the cross-validated 0.316, a good sign that the model was not overfitted to the comparison process. In the target's units, the typical miss is about 30,700 dollars on districts whose median value is about 179,700 dollars. That is the number to report, together with the baseline of 0.886, so readers see how much the model adds.

Step 5: Find where the model fails

An average hides where errors concentrate. Breaking the test error down by income band and by the target ceiling reveals two different stories:

errors = pd.DataFrame({"income": X_test["MedInc"], "actual": y_test, "error": (pred - y_test).abs()})
errors["income band"] = pd.qcut(errors["income"], 4, labels=["low", "mid-low", "mid-high", "high"])
print(errors.groupby("income band", observed=True)["error"].mean().round(3).to_dict())

capped = errors["actual"] >= 5.0
print("capped rows:", capped.sum(), "| MAE capped:", round(errors.loc[capped, "error"].mean(), 3),
      "| MAE rest:", round(errors.loc[~capped, "error"].mean(), 3))

{'low': 0.246, 'mid-low': 0.294, 'mid-high': 0.322, 'high': 0.366}
capped rows: 184 | MAE capped: 0.634 | MAE rest: 0.292

Errors grow with income: expensive districts are harder to predict, partly because prices vary more there. The larger finding is the ceiling. The target stops at 5.0, so every district worth more than about 500,000 dollars was recorded as 5.0. The model's error on those 184 test districts is more than twice its error on the rest, because their true values are unknown, not because the model is weak.

Each finding leads to a decision, not just a chart:

• The ceiling could be handled by removing capped rows from training and reporting the model as valid only below the cap, or by treating the top band as a separate classification problem. Either way, the limitation goes into the documentation.
• The income trend suggests reporting error as a percentage for expensive districts, or checking whether location features at a finer grain help.

Loops back from error analysis to features and data are normal; most real improvement comes from them. Retrain, re-run cross-validation, and only then look at the test set again, knowing each extra look makes it slightly less independent.

Step 6: Save the model and prove it reloads

A model that exists only in a notebook's memory cannot be used. Save the fitted model and check that the loaded copy predicts identically (scikit-learn: model persistence):

import joblib

joblib.dump(model, "house_model.joblib")
reloaded = joblib.load("house_model.joblib")
print(np.allclose(reloaded.predict(X_test), pred), round(reloaded.predict(X_test.iloc[[0]])[0], 3))

True 0.551

Two cautions from the scikit-learn documentation: load such files only from sources you trust, because loading can execute code, and load them with the same library versions used to save them. Record the versions, the training data date and the test MAE alongside the file.

Step 7: Serve it and watch it

Deployment wraps the saved model in something others can call, a batch job that scores new districts every night, or an API that returns a prediction on request. Model deployment from notebook to production covers those options.

After launch, monitoring matters more than any extra decimal of accuracy:

• Input drift. Compare the distributions of incoming features with the training data; a shift in income levels or new areas means the model is extrapolating.
• Performance once truth arrives. When real sale values become known, recompute MAE on recent predictions and compare it with the test MAE.
• A retraining rule. Decide in advance what error level or drift triggers retraining, and keep the old model so you can roll back.

A checklist for your own projects

If you are still choosing what to build, data science projects for freshers lists twelve with real datasets, and this process applies to each. Saving models with pickle and joblib, and serving them, are part of the model deployment module of the Data Science course.

Sources

• California Housing dataset (scikit-learn) — scikit-learn documentation
• Cross-validation: evaluating estimator performance (scikit-learn) — scikit-learn documentation
• Histogram-based gradient boosting (scikit-learn) — scikit-learn documentation
• Model persistence (scikit-learn) — scikit-learn documentation