Gradient Descent Explained: The Engine Behind ML

Most machine learning models learn the same way. They start with arbitrary parameters, measure how wrong their predictions are with a loss function, and adjust the parameters to reduce that loss, over and over. Gradient descent is the rule for that adjustment. Linear and logistic regression can use it, neural networks depend on it, and gradient boosting borrows its idea. Understanding it explains why training sometimes crawls, sometimes explodes, and why scaling features matters.

The idea in one paragraph

Picture standing on a hillside in fog, trying to reach the valley floor. You cannot see the valley, but you can feel the slope under your feet. So you take a step downhill, feel the slope again, and repeat. In machine learning, the hillside is the loss plotted against the model's parameters, the slope is the gradient, and the size of each step is the learning rate. Each update moves every parameter a little in the direction opposite its gradient (Google: gradient descent).

Gradient descent from scratch

Here is a regression line, y = w·x + b, fitted by gradient descent to 200 noisy points generated around a true line with slope 2.5 and intercept 4. The loss is the mean squared error, and the two gradient lines come from differentiating it with respect to w and b:

import numpy as np

rng = np.random.default_rng(0)
x = rng.uniform(0, 10, 200)
y = 4 + 2.5 * x + rng.normal(0, 2, 200)        # true intercept 4, slope 2.5, plus noise

def gradient_descent(x, y, lr, steps):
    w, b = 0.0, 0.0
    losses = []
    for _ in range(steps):
        error = w * x + b - y
        losses.append(np.mean(error ** 2))     # mean squared error
        w -= lr * 2 * np.mean(error * x)       # gradient of the loss with respect to w
        b -= lr * 2 * np.mean(error)           # gradient of the loss with respect to b
    return w, b, losses

w, b, losses = gradient_descent(x, y, lr=0.01, steps=5000)
exact = np.linalg.lstsq(np.column_stack([x, np.ones_like(x)]), y, rcond=None)[0]
print("gradient descent:", round(w, 3), round(b, 3), "| exact least squares:", exact.round(3))
best = losses[-1]

gradient descent: 2.449 4.094 | exact least squares: [2.449 4.094]

Starting from zero, 5,000 small steps land on the same slope and intercept as the exact least-squares solution. The fitted values differ from the true 2.5 and 4 because of the noise in the data; gradient descent finds the best line for the data it has. For linear regression an exact formula exists, but for logistic regression, neural networks and most other models it does not, and iterative methods like this are how they are trained.

The learning rate: the setting that decides everything

for lr in (0.001, 0.01, 0.03):
    _, _, l = gradient_descent(x, y, lr, 500)
    print(f"lr {lr}: loss after 10 steps {l[10]:.2f}, after 500 {l[-1]:.3g}")

lr 0.001: loss after 10 steps 76.61, after 500 6.18
lr 0.01: loss after 10 steps 7.09, after 500 4.2
lr 0.03: loss after 10 steps 118866.19, after 500 6.02e+128

Three behaviours from one setting. At 0.001 the loss falls steadily but is still well above the minimum of about 4.17 after 500 steps; training crawls. At 0.01 it is nearly at the minimum. At 0.03 each step overshoots the valley and lands higher up the opposite side, so the loss grows without limit, to around 10 to the power 128 after 500 steps: the model diverges (Google: hyperparameters). If a training loss suddenly shoots up or becomes NaN, a learning rate that is too large is the first suspect.

Why scaling features speeds training up

The feature x runs from 0 to 10 with a mean of about 5. That makes the loss surface a long, narrow valley: very steep in one direction, very flat in the other. A learning rate small enough not to overshoot the steep direction crawls along the flat one. Standardising x, subtracting its mean and dividing by its standard deviation, turns the valley into a round bowl:

def steps_needed(x, y, lr, tol=1e-3):
    _, _, losses = gradient_descent(x, y, lr, steps=2000)
    return next(i for i, loss in enumerate(losses) if loss - best < tol)

x_scaled = (x - x.mean()) / x.std()
for name, data, rates in (("raw x", x, (0.01, 0.025)), ("standardised x", x_scaled, (0.1, 0.3))):
    for lr in rates:
        print(f"{name}, lr {lr}: {steps_needed(data, y, lr)} steps to reach the minimum")

raw x, lr 0.01: 864 steps to reach the minimum
raw x, lr 0.025: 345 steps to reach the minimum
standardised x, lr 0.1: 29 steps to reach the minimum
standardised x, lr 0.3: 7 steps to reach the minimum

On the raw feature, even a learning rate of 0.025, close to the point where it diverges, needs 345 steps. On the standardised feature, 7 steps reach the same loss, about fifty times fewer. The model is equally good either way, since scaling changes the parameters' units, not the fit. With several features on very different scales, such as income in the hundreds of thousands next to a count of children, the effect is far stronger, which is why scikit-learn's documentation strongly recommends scaling for its gradient descent models (scikit-learn: stochastic gradient descent).

Batch, mini-batch and stochastic gradient descent

The version above computes the gradient from all 200 points at every step: batch gradient descent. With millions of rows that is slow, so most training uses a random subset, a mini-batch, per step. Stochastic gradient descent in its strict sense uses one example per step. Comparing them over the same five passes through the data:

def minibatch_descent(x, y, lr, batch_size, epochs, seed=0):
    order_rng = np.random.default_rng(seed)
    w, b, updates = 0.0, 0.0, 0
    for _ in range(epochs):
        order = order_rng.permutation(len(x))
        for start in range(0, len(x), batch_size):
            i = order[start:start + batch_size]
            error = w * x[i] + b - y[i]
            w -= lr * 2 * np.mean(error * x[i])
            b -= lr * 2 * np.mean(error)
            updates += 1
    return round(np.mean((w * x + b - y) ** 2), 3), updates

for batch_size, lr in ((200, 0.1), (20, 0.1), (1, 0.01)):
    loss, updates = minibatch_descent(x_scaled, y, lr, batch_size, epochs=5)
    print(f"batch size {batch_size:>3}: loss {loss} after {updates} updates")
print("best possible loss:", round(best, 3))

batch size 200: loss 42.199 after 5 updates
batch size  20: loss 4.17 after 50 updates
batch size   1: loss 4.178 after 1000 updates
best possible loss: 4.169

Each pass through the data costs about the same computation whatever the batch size, but smaller batches turn it into more updates. Five full-batch updates barely start the descent; fifty mini-batch updates of 20 reach the minimum. Single examples also get there, but each gradient is noisy, so they need a smaller learning rate and wander around the minimum instead of settling on it. Mini-batches, often of 32 to 256 examples, are the usual compromise (Ruder: overview of gradient descent optimisation algorithms).

Beyond plain gradient descent

Real training usually adds refinements on top of the basic rule:

• Momentum keeps a running average of recent gradients, so steps build speed along consistent directions and zig-zags across narrow valleys cancel out.
• Adaptive optimisers such as RMSprop and Adam give each parameter its own effective learning rate. Adam is a common default for neural networks.
• Learning rate schedules start larger and shrink the rate as training goes on, moving quickly at first and settling precisely at the end.
• Early stopping ends training when validation loss stops improving, one of the simplest defences against overfitting.

For linear and logistic regression, the loss is a single bowl, so any reasonable settings reach the same minimum. Neural network losses have many dips and flat regions, which is why initialisation, optimiser and schedule matter more there. The deep learning guide builds a small network whose backpropagation computes exactly the gradients used here.

Troubleshooting training

Gradient boosting uses the same idea in a different space: each new tree is fitted to the negative gradient of the loss with respect to the current predictions, a step downhill made of a tree instead of a weight change. XGBoost explained covers how that works in practice.

Gradient descent is the engine behind many of the top 10 machine learning algorithms. Forward and backward propagation, and the neural networks they train, are covered in the deep learning module of the Data Science course.

Sources

• Linear regression: gradient descent (Google Machine Learning Crash Course) — Google for Developers
• Linear regression: hyperparameters (Google Machine Learning Crash Course) — Google for Developers
• Stochastic gradient descent (scikit-learn) — scikit-learn documentation
• An overview of gradient descent optimization algorithms — arXiv, 2016