XGBoost Explained: Why It Wins Kaggle Competitions

For years, XGBoost was the most common ingredient in winning machine learning competition entries on tabular data. Its authors report that among the 29 winning solutions published on Kaggle's blog in 2015, 17 used XGBoost, and that every winning team in the top ten of KDD Cup 2015 used it (Chen and Guestrin: XGBoost). It remains a standard tool in industry. This guide explains how it works, shows it on real data, and puts its reputation in perspective.

The idea: gradient boosting

XGBoost is a fast, carefully engineered implementation of gradient-boosted decision trees. Boosting builds trees one at a time:

1. Start with a simple prediction, such as the average outcome.
2. Fit a small tree to the current errors, more precisely to the gradient of the loss, which says in which direction and by how much each prediction should move.
3. Add that tree's output to the predictions, scaled down by a learning rate.
4. Repeat, each new tree correcting what the ensemble so far gets wrong.

Many shallow trees, each nudging the predictions a little, add up to a model that captures complex, non-linear patterns and interactions. Ensemble learning explained measures how boosting differs from bagging, and gradient descent explained covers the gradient idea it borrows.

What XGBoost added

Gradient boosting existed before XGBoost. Its contribution was making it robust and fast at scale (Chen and Guestrin: XGBoost):

• A regularised objective, which penalises complex trees and large leaf values, so the model is less prone to overfitting.
• Sparsity-aware split finding, which learns a default direction for missing values at every split, so missing data needs no imputation.
• Approximate split finding with a weighted quantile sketch and histogram-based methods, which make training on large data fast.
• Systems engineering: parallelism, cache-aware memory access and out-of-core computation for data larger than memory.

XGBoost on real data

The Adult census dataset, extracted from the 1994 US Census, asks whether a person's annual income exceeds 50,000 dollars (UCI: Adult). It has text categories and missing values, typical of real tabular data:

import pandas as pd
import xgboost as xgb
from sklearn.compose import make_column_selector, make_column_transformer
from sklearn.datasets import fetch_openml
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler

census = fetch_openml(data_id=1590, as_frame=True)
X, y = census.data, (census.target == ">50K").astype(int)
print(X.shape, "| share earning over 50K:", round(y.mean(), 3))
print("categorical columns:", sum(X.dtypes == "category"), "| missing values:", X.isna().sum()[lambda s: s > 0].to_dict())
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
X_fit, X_valid, y_fit, y_valid = train_test_split(X_train, y_train, test_size=0.2, stratify=y_train, random_state=0)

(48842, 14) | share earning over 50K: 0.239
categorical columns: 8 | missing values: {'workclass': 2799, 'occupation': 2809, 'native-country': 857}

With tree_method="hist" and enable_categorical=True, XGBoost uses the pandas categories directly, without one-hot encoding (XGBoost: categorical data), and the missing values need no imputation. Early stopping watches a validation set and keeps the best number of trees:

for learning_rate in (0.3, 0.05):
    model = xgb.XGBClassifier(n_estimators=2000, learning_rate=learning_rate, max_depth=6, tree_method="hist",
                              enable_categorical=True, early_stopping_rounds=50, eval_metric="logloss", random_state=0)
    model.fit(X_fit, y_fit, eval_set=[(X_valid, y_valid)], verbose=False)
    auc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])
    print(f"learning rate {learning_rate}: best after {model.best_iteration + 1} trees | test ROC AUC {auc:.4f}")

learning rate 0.3: best after 45 trees | test ROC AUC 0.9272
learning rate 0.05: best after 300 trees | test ROC AUC 0.9290

This is the central trade-off in boosting. A large learning rate reaches its best in 45 trees; a small one needs 300 but ends slightly better, because each tree's contribution is smaller and more careful. The usual recipe is a modest learning rate with early stopping deciding the number of trees, then tuning tree depth and regularisation (XGBoost: parameters). Hyperparameter tuning compared covers how to search the rest.

The parameters that matter most

• learning_rate (also called eta): how much each tree contributes. Smaller values need more trees but usually generalise better.
• n_estimators with early stopping: the number of trees, best chosen automatically on validation data.
• max_depth: how complex each tree can be; typical values are 3 to 10. Deeper trees capture more interactions and overfit more easily.
• subsample and colsample_bytree: the share of rows and columns each tree sees. Values below 1 add randomness that often reduces overfitting.
• reg_lambda, reg_alpha and gamma: regularisation on leaf values and on adding splits, the objective terms that distinguish XGBoost's formulation.

What the model relies on

gain = pd.Series(model.get_booster().get_score(importance_type="gain")).sort_values(ascending=False)
print(gain.head(5).round(1).to_dict())

{'relationship': 94.5, 'marital-status': 61.0, 'capital-gain': 28.8, 'education': 20.1, 'occupation': 13.4}

Gain importance measures how much each feature's splits improved the loss on average. Household relationship and marital status lead, followed by capital gains and education. Two cautions. Importance describes the model, not causes in the world. And this dataset includes sensitive attributes such as sex and race; a real income or credit model would need a fairness review before any such features, or their proxies, were used in decisions.

Does XGBoost really win? An honest comparison

categories = make_column_selector(dtype_include="category")
numbers = make_column_selector(dtype_exclude="category")
encode = make_pipeline(SimpleImputer(strategy="most_frequent"), OneHotEncoder(handle_unknown="ignore"))
others = {
    "logistic regression": make_pipeline(make_column_transformer((encode, categories), (StandardScaler(), numbers)),
                                         LogisticRegression(max_iter=3000)),
    "random forest": make_pipeline(make_column_transformer((encode, categories), ("passthrough", numbers)),
                                   RandomForestClassifier(n_estimators=300, random_state=0, n_jobs=-1)),
    "scikit-learn gradient boosting": HistGradientBoostingClassifier(categorical_features="from_dtype", random_state=0),
}
for name, other in others.items():
    auc = roc_auc_score(y_test, other.fit(X_train, y_train).predict_proba(X_test)[:, 1])
    print(f"{name}: test ROC AUC {auc:.4f}")

logistic regression: test ROC AUC 0.9039
random forest: test ROC AUC 0.9031
scikit-learn gradient boosting: test ROC AUC 0.9285

XGBoost's 0.929 clearly beats logistic regression and random forest, at 0.904 and 0.903. But scikit-learn's own histogram-based gradient boosting, with default settings, matches it at 0.9285. The lesson is that the winner is gradient boosting as a method, and modern implementations, XGBoost, LightGBM, CatBoost and scikit-learn's, are close in accuracy on most tabular problems. Choose among them by practical needs: speed on your data size, GPU support, handling of categories, and what your team and deployment platform already use.

Why boosting keeps winning competitions

• Tabular data suits trees. Thresholds on individual features match how business and scientific data behave, and trees need no scaling.
• Boosting reduces bias step by step while regularisation and early stopping control variance.
• It handles messy data: missing values, mixed types and irrelevant features.
• It trains fast enough to iterate, and competitions are won by many quick experiments on features.

Independent benchmarks support the pattern: on medium-sized tabular datasets, tree-based models such as XGBoost remain ahead of deep learning, as the top 10 machine learning algorithms overview discusses.

Common mistakes with XGBoost

• Too many trees without early stopping, which overfits and wastes time.
• A learning rate too high for the final model; lower it and let early stopping add trees.
• Leaking the test set into early stopping. Use a separate validation set, as above, not the test data.
• Reading importance as causation, or trusting a single importance measure; compare gain with permutation importance.

Decision trees and random forests, the foundations of boosted trees, are part of the machine learning core module of the Data Science course.

Sources

• XGBoost: A Scalable Tree Boosting System — KDD 2016, arXiv
• Categorical Data (XGBoost documentation) — XGBoost
• XGBoost Parameters (XGBoost documentation) — XGBoost
• Adult (UCI Machine Learning Repository) — UCI Machine Learning Repository