Ensemble Learning: Bagging, Boosting and Stacking Explained

An ensemble combines several models so that their errors partly cancel out. The idea is old and simple: a crowd of reasonable guessers often beats any one of them. If three classifiers are each right 80% of the time and make their mistakes independently, a majority vote is right about 90% of the time, because it fails only when at least two of the three fail together. The catch is in the word independently. Models trained on the same data tend to make the same mistakes, and then voting gains nothing.

Bagging, boosting and stacking are three ways of building models that err differently and combining them well. Each is measured below on one real dataset: 569 breast tissue samples described by 30 numeric features, labelled malignant or benign (scikit-learn: breast cancer dataset). Every score is the mean accuracy of 5-fold cross-validation repeated three times, with the spread across those 15 folds.

Bagging: many trees on resampled data

A single decision tree is unstable: small changes in the training data can produce a very different tree. Bagging, short for bootstrap aggregating, trains many copies of the model on random samples of the training data, drawn with replacement, and averages their votes (scikit-learn: bagging). A random forest goes one step further and also lets each split consider only a random subset of features, which makes the trees even less alike.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import (AdaBoostClassifier, BaggingClassifier, GradientBoostingClassifier,
                              RandomForestClassifier, StackingClassifier)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import RepeatedStratifiedKFold, StratifiedKFold, cross_val_predict, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier

X, y = load_breast_cancer(return_X_y=True)
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=0)

def score(model):
    s = cross_val_score(model, X, y, cv=cv)
    return f"{s.mean():.3f} (spread {s.std():.3f})"

print("single tree:  ", score(DecisionTreeClassifier(random_state=0)))
print("bagged trees: ", score(BaggingClassifier(DecisionTreeClassifier(), n_estimators=100, random_state=0)))
print("random forest:", score(RandomForestClassifier(n_estimators=300, random_state=0)))

single tree:   0.927 (spread 0.019)
bagged trees:  0.958 (spread 0.013)
random forest: 0.960 (spread 0.012)

Averaging 100 trees lifts accuracy by three points and narrows the spread between folds. That is bagging's job: it reduces variance, the part of the error caused by a model reacting too strongly to the particular sample it was trained on. It does little for bias; if every tree is systematically wrong in the same way, their average is too. The bias-variance trade-off explains the two kinds of error in detail.

Bagging trains its models independently, so it parallelises well, and adding more trees does not cause overfitting; the score simply levels off.

Boosting: weak models that learn from each other's mistakes

Boosting takes the opposite approach. It trains models one after another, and each new model concentrates on the examples the previous ones got wrong. AdaBoost does this by re-weighting the training examples after each round; gradient boosting fits each new tree to the remaining errors (scikit-learn: AdaBoost). The individual models are deliberately weak, often stumps: trees with a single split.

for n in (1, 10, 50, 200):
    print(f"{n:>3} stumps:", score(AdaBoostClassifier(n_estimators=n, random_state=0)))

  1 stumps: 0.890 (spread 0.017)
 10 stumps: 0.944 (spread 0.014)
 50 stumps: 0.968 (spread 0.013)
200 stumps: 0.971 (spread 0.012)

One stump, a single yes-or-no question about one feature, is right 89% of the time. Two hundred of them, each correcting the last, reach 97.1%, better than the random forest. Boosting mainly reduces bias: it builds a flexible model out of simple parts.

The price is sensitivity. Because boosting keeps chasing errors, it can end up fitting noise and mislabelled examples, and unlike bagging, adding rounds can eventually overfit. The number of rounds and the learning rate need tuning with cross-validation, and the rounds must run in sequence. Gradient-boosted tree libraries such as XGBoost, LightGBM and scikit-learn's histogram-based boosting are the most common winners on tabular data; XGBoost explained covers why.

Stacking: let a model decide how to combine models

Stacking combines models of different types. Each base model makes predictions, and a final model, often a simple logistic regression, learns how much to trust each one. To avoid the final model learning from predictions made on data the base models had already seen, it is trained on out-of-fold predictions, which scikit-learn's stacking classes produce internally with cross-validation (scikit-learn: stacked generalization).

members = [
    ("forest", RandomForestClassifier(n_estimators=300, random_state=0)),
    ("svm", make_pipeline(StandardScaler(), SVC(probability=True, random_state=0))),
    ("boosting", GradientBoostingClassifier(random_state=0)),
]
for name, model in members:
    print(f"{name}:", score(model))
stack = StackingClassifier(members, final_estimator=LogisticRegression(max_iter=1000), cv=5)
print("stacking:", score(stack))
print("logistic regression alone:", score(make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))))

forest: 0.960 (spread 0.012)
svm: 0.974 (spread 0.012)
boosting: 0.960 (spread 0.013)
stacking: 0.972 (spread 0.011)
logistic regression alone: 0.975 (spread 0.016)

The stack beats two of its three members and has the smallest spread, but it does not beat its best member, the support vector machine, and a plain logistic regression edges out everything. This is a real and common outcome. A linear model reaching 97.5% suggests the two classes are close to separable by a straight boundary once features are scaled, so a well-regularised linear model is already near the best achievable here, and combining models has little left to fix.

Why diversity decides whether an ensemble helps

Combining models helps only when they make different mistakes. Counting each model's out-of-fold errors, and the cases two models both get wrong, shows how much they overlap:

folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
models = {
    "forest": RandomForestClassifier(n_estimators=300, random_state=0),
    "boosting": GradientBoostingClassifier(random_state=0),
    "svm": make_pipeline(StandardScaler(), SVC()),
    "logistic": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
}
wrong = {name: cross_val_predict(m, X, y, cv=folds) != y for name, m in models.items()}
print({name: int(w.sum()) for name, w in wrong.items()})
for a, b in [("forest", "boosting"), ("forest", "logistic"), ("svm", "logistic")]:
    print(f"{a} and {b} both wrong: {int((wrong[a] & wrong[b]).sum())}")

{'forest': 24, 'boosting': 20, 'svm': 13, 'logistic': 12}
forest and boosting both wrong: 15
forest and logistic both wrong: 6
svm and logistic both wrong: 8

The two tree ensembles share 15 of their errors: they learn similar rules and fail on the same samples, so combining them adds little. The forest and the logistic regression share only 6, which is the kind of diversity an ensemble can exploit. The SVM and logistic regression share 8 of their 13 and 12 errors: both draw smooth boundaries on scaled features and stumble on the same hard cases. When building an ensemble, pick members from different families, check their error overlap, and always compare the ensemble with its best single member.

Bagging, boosting and stacking compared

Voting is the simplest relative: average the predictions of a few different models without training a combiner. It is a reasonable first step before stacking.

Choosing in practice

• Start with a simple model and a strong single ensemble. On tabular data, a random forest is a robust baseline that needs little tuning, and gradient boosting is usually the stronger model after tuning.
• Reach for stacking last. It adds training time, prediction latency and complexity, and as measured here, it may not beat the best member. Use it when the error overlap shows real diversity and the gain justifies the cost, as in competitions.
• Count the costs outside accuracy. An ensemble of hundreds of trees is slower to serve and harder to explain than one model. For a regulated decision, a slightly less accurate but explainable model can be the right choice.
• Validate everything with the same cross-validation. Differences of a few tenths of a percentage point are often within the spread between folds, as the table of results above shows.

For how ensembles fit among the other core algorithms, see the top 10 machine learning algorithms. Decision trees, random forests and the other supervised models they build on are covered in the machine learning core module of the Data Science course.

Sources

• Bagging meta-estimator (scikit-learn) — scikit-learn documentation
• AdaBoost (scikit-learn) — scikit-learn documentation
• Stacked generalization (scikit-learn) — scikit-learn documentation
• Breast cancer Wisconsin (diagnostic) dataset (scikit-learn) — scikit-learn documentation