Hyperparameter Tuning: Grid Search vs Random Search vs Bayesian

Hyperparameters are the settings you choose before training, such as the learning rate of a boosted model or the regularisation strength of an SVM. Unlike the model's weights, they are not learned from the data, so they are chosen by trying candidates and keeping the best by cross-validation. The three main search strategies are grid search, random search and Bayesian optimisation. This guide runs all three on the same model with the same budget, then shows what the results do and do not mean.

The data is the spambase dataset: 4,601 emails described by 57 frequency features, labelled spam or not (OpenML: spambase). A quarter is held out as a test set that no search ever sees, and every search uses 5-fold cross-validation on the rest.

Step 0: the defaults, and how noisy the score is

import numpy as np
import optuna
from scipy.stats import loguniform, randint
from sklearn.datasets import fetch_openml
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import (GridSearchCV, RandomizedSearchCV, StratifiedKFold,
                                     cross_val_score, train_test_split)

data = fetch_openml(data_id=44, as_frame=True)
X, y = data.data, (data.target == "1").astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
model = HistGradientBoostingClassifier(early_stopping=False, random_state=0)

scores = cross_val_score(model, X_train, y_train, cv=cv)
print(f"defaults: mean {scores.mean():.4f}, fold scores {scores.round(3)}, spread {scores.std():.4f}")

defaults: mean 0.9510, fold scores [0.952 0.948 0.948 0.962 0.945], spread 0.0061

Before tuning anything, look at the spread. The five folds range from 94.5% to 96.2% accuracy with a standard deviation of 0.006. A tuned model that beats the defaults by a few thousandths may simply have been lucky with the folds. Keep that number in mind for everything that follows. Cross-validation in machine learning explains where the spread comes from.

Grid search: every combination of chosen values

Grid search tries every combination of the values you list (scikit-learn: tuning hyperparameters). Three learning rates and two values for each of three other settings make 24 candidates:

grid = {"learning_rate": [0.03, 0.1, 0.3], "max_leaf_nodes": [15, 63],
        "min_samples_leaf": [10, 40], "l2_regularization": [0.0, 1.0]}
grid_search = GridSearchCV(model, grid, cv=cv).fit(X_train, y_train)
print(f"grid search, {len(grid_search.cv_results_['params'])} candidates: best CV {grid_search.best_score_:.4f}")
print(grid_search.best_params_)

grid search, 24 candidates: best CV 0.9536
{'l2_regularization': 0.0, 'learning_rate': 0.3, 'max_leaf_nodes': 15, 'min_samples_leaf': 10}

Grid search is simple and exhaustive within its grid, but the cost multiplies with every setting added: five values for each of four settings would be 625 candidates, each trained five times.

Random search: the same budget, more variety

Random search samples each candidate from a distribution per setting, here log-uniform for rates and strengths, and stops after a fixed number:

space = {"learning_rate": loguniform(0.01, 0.5), "max_leaf_nodes": randint(8, 128),
         "min_samples_leaf": randint(5, 100), "l2_regularization": loguniform(1e-3, 10)}
random_search = RandomizedSearchCV(model, space, n_iter=24, cv=cv, random_state=0).fit(X_train, y_train)
print(f"random search, 24 candidates: best CV {random_search.best_score_:.4f}")
print({k: round(v, 3) for k, v in random_search.best_params_.items()})

random search, 24 candidates: best CV 0.9530
{'l2_regularization': 0.037, 'learning_rate': 0.263, 'max_leaf_nodes': 80, 'min_samples_leaf': 14}

Bergstra and Bengio showed why random search is usually the better default: for most problems only a few hyperparameters really matter, and different ones matter on different data sets (Bergstra and Bengio: random search). A grid spends its budget repeating the same few values of the important setting; random search tries a new value of every setting in every candidate. That shows up in a count after the next step.

Bayesian optimisation: learn from earlier trials

Bayesian methods build a model of how the score depends on the settings and use it to pick promising candidates. Optuna's TPE sampler is a popular implementation; its first 10 trials are random by default, and later ones are guided by the results so far (Optuna: TPESampler):

def objective(trial):
    params = {"learning_rate": trial.suggest_float("learning_rate", 0.01, 0.5, log=True),
              "max_leaf_nodes": trial.suggest_int("max_leaf_nodes", 8, 128),
              "min_samples_leaf": trial.suggest_int("min_samples_leaf", 5, 100),
              "l2_regularization": trial.suggest_float("l2_regularization", 1e-3, 10, log=True)}
    return cross_val_score(model.set_params(**params), X_train, y_train, cv=cv).mean()

optuna.logging.set_verbosity(optuna.logging.WARNING)
study = optuna.create_study(direction="maximize", sampler=optuna.samplers.TPESampler(seed=0))
study.optimize(objective, n_trials=24)
print(f"Bayesian (Optuna TPE), 24 trials: best CV {study.best_value:.4f}")
print({k: round(v, 3) for k, v in study.best_params.items()})

Bayesian (Optuna TPE), 24 trials: best CV 0.9525
{'learning_rate': 0.072, 'max_leaf_nodes': 117, 'min_samples_leaf': 23, 'l2_regularization': 0.001}

The results, read honestly

Now the one comparison that counts: each chosen configuration trained on all the training data and scored once on the untouched test set.

chosen = {"defaults": {}, "grid": grid_search.best_params_,
          "random": random_search.best_params_, "Bayesian": study.best_params}
for name, params in chosen.items():
    final = HistGradientBoostingClassifier(early_stopping=False, random_state=0, **params).fit(X_train, y_train)
    print(f"{name:>8}: test accuracy {final.score(X_test, y_test):.4f}")
print("distinct learning rates tried - grid:", len(set(grid_search.cv_results_["param_learning_rate"])),
      "| random:", len(set(random_search.cv_results_["param_learning_rate"])))

defaults: test accuracy 0.9592
    grid: test accuracy 0.9592
  random: test accuracy 0.9661
Bayesian: test accuracy 0.9661
distinct learning rates tried - grid: 3 | random: 24

Three observations:

• All three searches beat the defaults in cross-validation, by 0.15 to 0.26 percentage points. That is less than half the fold-to-fold spread of 0.6 points. On this data, the defaults of a modern gradient-boosting library were already close to the best available.
• On the test set, random and Bayesian search score 96.6% against 95.9%. With 1,151 test emails, that gap is 8 emails, about one standard error. Suggestive, not proof.
• Random search tried 24 different learning rates to the grid's 3. That is Bergstra and Bengio's point in one line.

With only 24 trials, Optuna had just 14 guided ones after its random start, so this comparison does not show Bayesian search at its best. Its advantage grows when each trial is expensive and the budget allows the model of the score to learn, as with deep networks that take hours per trial.

When tuning really matters

Gradient boosting has good defaults. Other models do not forgive bad settings. The same random search over an SVM's two main hyperparameters, C and gamma:

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

svm = make_pipeline(StandardScaler(), SVC())
print(f"SVM defaults: CV {cross_val_score(svm, X_train, y_train, cv=cv).mean():.4f}")
svm_search = RandomizedSearchCV(svm, {"svc__C": loguniform(1e-2, 1e3), "svc__gamma": loguniform(1e-4, 1)},
                                n_iter=24, cv=cv, random_state=0).fit(X_train, y_train)
results = svm_search.cv_results_["mean_test_score"]
print(f"SVM candidates ranged from {results.min():.4f} to {results.max():.4f}")
print({k: round(v, 4) for k, v in svm_search.best_params_.items()}, f"test accuracy {svm_search.score(X_test, y_test):.4f}")

SVM defaults: CV 0.9281
SVM candidates ranged from 0.6061 to 0.9322
{'svc__C': 10.3233, 'svc__gamma': 0.0151} test accuracy 0.9470

Across the candidates, cross-validated accuracy ran from 61% to 93%. A careless manual choice could land anywhere in that range; a search reliably finds the good region. Tuning matters most for models that are sensitive to their settings, and when you do not know good defaults.

Which method to use

Rules for honest tuning

1. Tune on training data with cross-validation only. The test set is used once, at the end.
2. Compare gains with the spread between folds. Differences smaller than it are not evidence.
3. Search rates and strengths on a log scale. The useful range of a learning rate spans orders of magnitude.
4. Fix the budget in advance. Searching until a number looks good is overfitting to the validation folds.
5. Report the tuned result with its spread, and for a rigorous estimate of the whole procedure, use nested cross-validation.

How much a tuned model is worth depends on the metric you tune for; evaluation metrics explained covers the choice. Cross-validation and hyperparameter tuning are part of the model evaluation and optimisation module of the Data Science course.

Sources

• Tuning the hyper-parameters of an estimator (scikit-learn) — scikit-learn documentation
• Random Search for Hyper-Parameter Optimization — Journal of Machine Learning Research, 2012
• optuna.samplers.TPESampler — Optuna documentation
• spambase (OpenML dataset 44) — www.openml.org