Feature Engineering: A Practical Guide with Examples

Feature engineering is the work of turning raw columns into inputs that make patterns easy for a model to learn. A model can only use the information it is given, in the form it is given. The same data presented differently can halve a model's error, and on tabular problems better features often do more than a better algorithm.

This guide measures each step instead of listing techniques. The data is 17,379 hours of bike rentals from the Capital Bikeshare system in Washington, D.C., across 2011 and 2012, with the hour, the day type and the weather (UCI: bike sharing dataset). The task: predict how many bikes are rented in an hour. Every result is the mean absolute error (MAE) in rentals per hour, so lower is better.

The setup: a baseline and an honest evaluation

import numpy as np
import pandas as pd
from sklearn.datasets import fetch_openml
from sklearn.dummy import DummyRegressor
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.linear_model import Ridge
from sklearn.model_selection import TimeSeriesSplit, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

df = fetch_openml("Bike_Sharing_Demand", version=2, as_frame=True).frame
y = df["count"]
workday = (df["workingday"] == "True").astype(int)
weather = ["temp", "humidity", "windspeed"]
print(df.shape, "| mean rentals per hour:", round(y.mean(), 1))

cv = TimeSeriesSplit(n_splits=5, gap=48, max_train_size=10000, test_size=1000)
def mae(X, model=None):
    model = model or make_pipeline(StandardScaler(), Ridge())
    return round(-cross_val_score(model, X, y, cv=cv, scoring="neg_mean_absolute_error").mean(), 1)

raw = df[weather].assign(hour=df["hour"], workday=workday)
print("predicting the training mean:", mae(raw, DummyRegressor()))

(17379, 13) | mean rentals per hour: 189.5
predicting the training mean: 177.5

The rows are in time order, so the evaluation must respect time: each fold trains on earlier hours and tests on the next 1,000, with a two-day gap between them so neighbouring hours cannot leak into the test (scikit-learn: TimeSeriesSplit). A shuffled split would let the model peek at the future and flatter every feature. Guessing the training mean every hour misses by 177.5 rentals on average; that is the number to beat.

The first model is a linear regression with ridge regularisation, deliberately simple, because a linear model makes the effect of each feature easy to see. The data comes as OpenML dataset 42712 (OpenML: Bike_Sharing_Demand).

Step 1: The hour as a plain number

print("hour as a number:", mae(raw))

hour as a number: 131.3

Better than the baseline, but the model is handicapped. Given the hour as a number from 0 to 23, a linear model can only learn "later means more" or "later means fewer", a straight line through the day. Real demand has a morning peak, a midday dip and an evening peak, a shape no straight line can follow.

Step 2: Encode the hour so its shape can be learned

Two common fixes. A cyclic encoding replaces the hour with its sine and cosine, placing the hours on a circle so 23:00 sits next to midnight. A one-hot encoding gives each hour its own column, so the model learns a separate level for every hour:

cyclic = df[weather].assign(hour_sin=np.sin(2 * np.pi * df["hour"] / 24),
                            hour_cos=np.cos(2 * np.pi * df["hour"] / 24), workday=workday)
hours = pd.get_dummies(df["hour"], prefix="hour", dtype=float)
onehot = pd.concat([df[weather], hours], axis=1).assign(workday=workday)
print("sine and cosine of hour:", mae(cyclic))
print("one column per hour:", mae(onehot))

sine and cosine of hour: 117.0
one column per hour: 99.0

The cyclic features help, but one sine and cosine pair can only describe a single smooth wave per day, not two sharp peaks. One-hot encoding, with 24 separate levels, cuts the error to 99.0. The scikit-learn documentation works through the same trade-off on this dataset, including spline encodings that sit between the two (scikit-learn: time-related feature engineering).

Step 3: Add the interaction the domain suggests

Commuting suggests that working days and other days have different daily shapes. The data confirms it:

by_hour = y.groupby([workday.rename("working day"), df["hour"]]).mean().round().unstack(0)
print(by_hour.loc[[8, 13, 17]])

working day      0      1
hour
8            106.0  477.0
13           373.0  198.0
17           324.0  525.0

On working days, rentals peak at 8 am and 5 pm with a dip at midday; on weekends and holidays, the busiest time is early afternoon. The model above learns one daily shape plus a flat shift for working days, so it cannot fit both. A separate set of hourly columns that are switched on only on working days lets it learn each shape:

workday_hours = hours.mul(workday, axis=0).add_prefix("workday_")
interact = pd.concat([onehot, workday_hours], axis=1)
print("hour x working day:", mae(interact))

hour x working day: 81.5

One idea from domain knowledge, expressed as 24 extra columns, removes another 18% of the error. This is the pattern behind most valuable features: a fact about how the process works, turned into a form the model can use.

Step 4: Domain features the raw data hides

Two more pieces of information are in the data: the sky condition, from clear to heavy rain, and the year. The service grew between the two years:

print(y.groupby(df["year"]).mean().round(1).to_dict())
full = pd.concat([interact, pd.get_dummies(df["weather"], prefix="sky", dtype=float)], axis=1).assign(year=df["year"])
print("plus sky conditions and year:", mae(full))

{0: 143.8, 1: 234.7}
plus sky conditions and year: 72.7

Mean rentals rose from 143.8 per hour in the first year to 234.7 in the second, a trend the earlier features could not see. Adding the sky condition as one-hot columns and the year as a number takes the error down to 72.7. From the 131.3 it started at, the same linear model, with better inputs, now has 45% less error. A year column is a blunt way to capture growth, and a model trained on it assumes the trend continues; for real forecasting, a trend feature needs regular retraining and monitoring.

Step 5: The right features depend on the model

Gradient-boosted trees split on thresholds, so they can learn the daily shape from the plain hour without any encoding. Comparing them on different feature sets:

boosting = HistGradientBoostingRegressor(random_state=0)
print("boosting, raw columns:", mae(raw, boosting))
print("boosting, raw + year + sky:", mae(raw.assign(year=df["year"], sky=df["weather"].cat.codes), boosting))
print("boosting, all one-hot features:", mae(full, boosting))

boosting, raw columns: 74.1
boosting, raw + year + sky: 48.3
boosting, all one-hot features: 54.3

Three lessons from three lines. Trees do not need the hour encoded; on raw columns they already match the fully engineered linear model. What trees cannot do is invent information that is missing: adding the year and sky condition, new information rather than a new encoding, cuts their error by a third. And the one-hot encoding that helped the linear model hurts the trees, which now need many splits to recover what one numeric hour column gave them. Feature engineering is a conversation with a particular model, not a fixed recipe.

Other feature types worth knowing

• Ratios and differences. Price per square metre, spend per visit, days since last purchase. They encode relationships a model would otherwise need many splits or terms to approximate.
• Group aggregates. A customer's average order value or a store's typical weekly sales. Compute them from training data only, and for time-ordered data only from the past.
• Date and time parts. Hour, weekday, month, holidays, and time since an event, encoded as shown above.
• Target encoding for many categories. Replace a high-cardinality category, such as a postcode, with a smoothed average of the target for that category. Done naively it leaks the answer; scikit-learn's TargetEncoder uses internal cross fitting so a row's encoding never includes its own target (scikit-learn: TargetEncoder).
• Transformations. Logarithms for skewed amounts, and bins when a relationship changes in steps.
• Missing-value indicators. A column recording that a value was missing, when missingness carries information, as the Titanic ages did in the EDA walkthrough.

Rules every new feature must pass

1. Available at prediction time. If a value is known only after the outcome, it is leakage, however predictive it looks.
2. Computed without the test data. Any statistic, such as a mean, count or encoding, is learned from training data inside the cross-validation, ideally inside a pipeline. Data preprocessing shows how.
3. Measured, not assumed. Keep a feature only if validation error improves on the same folds; in this article every addition was kept because the MAE fell.
4. Maintainable. Every feature must be rebuilt in production from live data. A clever feature from a one-off spreadsheet is a future outage.

Feature engineering, encoding and scaling are part of the data preprocessing and feature engineering module of the Data Science course.

Sources

• Bike Sharing (UCI Machine Learning Repository) — UCI Machine Learning Repository
• Bike_Sharing_Demand (OpenML dataset 42712) — www.openml.org
• Time-related feature engineering (scikit-learn example) — scikit-learn documentation
• TimeSeriesSplit (scikit-learn cross-validation guide) — scikit-learn documentation
• sklearn.preprocessing.TargetEncoder — scikit-learn documentation