Deep Learning Guide: Neural Networks Explained Simply

Deep learning is machine learning with neural networks of many layers. It powers image recognition, speech recognition and large language models, and it can look like magic from the outside. Inside, a neural network is a stack of simple operations, multiply, add and bend, whose numbers are adjusted, a little at a time, to reduce a measured error. The clearest way to see that is to build one small enough to read.

The problem: XOR

The XOR function outputs 1 when exactly one of its two inputs is 1. It is a classic test because no straight line can separate its answers: the points (0, 1) and (1, 0) must be on one side and (0, 0) and (1, 1) on the other. A linear model cannot learn it. A network with one hidden layer can.

A network in NumPy, trained by hand

The network has two inputs, a hidden layer of four units with the tanh activation, and one output unit with the sigmoid activation, which squashes any number into the range 0 to 1:

import numpy as np
import torch
import torch.nn as nn

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y = np.array([[0], [1], [1], [0]], dtype=float)

rng = np.random.default_rng(1)
W1, b1 = rng.normal(0, 1, (2, 4)), np.zeros((1, 4))      # input -> hidden weights
W2, b2 = rng.normal(0, 1, (4, 1)), np.zeros((1, 1))      # hidden -> output weights

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

learning_rate = 1.0
for step in range(5001):
    # Forward pass: compute predictions
    hidden = np.tanh(X @ W1 + b1)
    output = sigmoid(hidden @ W2 + b2)
    loss = np.mean((output - y) ** 2)

    # Backward pass: gradients of the loss with respect to every weight (chain rule)
    d_output = 2 * (output - y) / len(X) * output * (1 - output)
    dW2, db2 = hidden.T @ d_output, d_output.sum(0, keepdims=True)
    d_hidden = d_output @ W2.T * (1 - hidden ** 2)
    dW1, db1 = X.T @ d_hidden, d_hidden.sum(0, keepdims=True)

    # Gradient descent: move every weight a little against its gradient
    W1 -= learning_rate * dW1; b1 -= learning_rate * db1
    W2 -= learning_rate * dW2; b2 -= learning_rate * db2

    if step in (0, 1000, 5000):
        print(f"step {step:>4}: loss {loss:.4f}")

print(output.round(2).ravel())

step    0: loss 0.2400
step 1000: loss 0.0008
step 5000: loss 0.0001
[0.   0.99 0.99 0.01]

The final predictions are 0, 0.99, 0.99 and 0.01 for the four inputs: XOR, learned from four examples. Every idea in deep learning is in those thirty lines.

What each part does

Layers. A layer multiplies its inputs by a weight matrix and adds a bias. X @ W1 + b1 turns 2 inputs into 4 hidden values. Stacking layers lets the network build features from features.

Activation functions. Without a non-linear function between layers, any stack of layers collapses into a single linear transformation, and XOR stays impossible. tanh and sigmoid bend the values; modern networks mostly use ReLU in hidden layers because it trains faster in deep stacks.

The loss. A single number measuring how wrong the predictions are, here the mean squared error. Training is nothing but making this number smaller.

Backpropagation. The chain rule of calculus, applied layer by layer from the output backwards, gives the gradient of the loss with respect to every weight: how much the loss would change if that weight moved slightly. The d_... lines compute exactly that. The free textbook Deep Learning by Goodfellow, Bengio and Courville derives it in full (Deep Learning book).

Gradient descent. Each step moves every weight a small amount in the direction that reduces the loss. The learning rate sets the step size: too small and training crawls, too large and it overshoots and diverges.

The same network in PyTorch

Frameworks compute the backward pass for you. In PyTorch, you describe the forward computation, and automatic differentiation derives every gradient (PyTorch: learn the basics):

torch.manual_seed(1)
net = nn.Sequential(nn.Linear(2, 4), nn.Tanh(), nn.Linear(4, 1), nn.Sigmoid())
optimiser = torch.optim.SGD(net.parameters(), lr=1.0)
X_t = torch.tensor(X, dtype=torch.float32)
y_t = torch.tensor(y, dtype=torch.float32)

for step in range(5001):
    optimiser.zero_grad()
    loss = nn.functional.mse_loss(net(X_t), y_t)
    loss.backward()                     # backpropagation, computed automatically
    optimiser.step()                    # gradient descent update

print(round(loss.item(), 4), net(X_t).detach().numpy().round(2).ravel())

0.0001 [0.01 0.99 0.99 0.01]

The same four ideas, layers, activations, loss and an optimiser, appear as named building blocks from torch.nn (PyTorch: torch.nn). Real models differ in scale and in the types of layers used, such as convolutional layers for images and attention layers for language, not in these principles.

Training real networks: the settings that matter

XOR fits in four examples; real datasets do not, and a few practical settings decide whether training succeeds:

• Mini-batches. Instead of computing gradients on all the data at once, each step uses a small random batch, often 32 to 256 examples. It is faster and the noise it adds even helps generalisation.
• The optimiser and learning rate. Adam is a common default that adapts the step size per weight; the learning rate is still the single most important setting to tune, and schedules that lower it during training often help.
• Controls against overfitting. Dropout randomly switches off units during training, weight decay penalises large weights, and early stopping ends training when validation loss stops improving.
• A validation set watched throughout. Plot training and validation loss per epoch. Training loss falling while validation loss rises is the signature of overfitting, and the moment to stop or regularise.
• Normalised inputs. Features on similar scales, and normalisation layers inside deep networks, keep gradients well behaved.

What "deep" adds

With more layers, a network can learn a hierarchy of features: in images, edges, then shapes, then objects; in text, character patterns, then words, then meaning in context. That hierarchy is why deep networks excel on unstructured data such as images, audio and language, where hand-designing features is impractical. Computer vision basics shows convolutional layers learning image features.

When deep learning is the right tool, and when it is not

On spreadsheets of customer or transaction data, tree ensembles such as XGBoost often match deep networks with less data, tuning and compute; XGBoost explained covers why. Deep learning earns its complexity where the input is raw and high-dimensional.

Gradient descent itself is explained in more depth in gradient descent: the engine behind machine learning. Neural network basics, activation functions, forward and backward propagation, CNNs and sequence models make up the deep learning module of the Data Science course.

Sources

• Learn the Basics (PyTorch tutorials) — PyTorch documentation
• torch.nn — PyTorch documentation
• Deep Learning (Goodfellow, Bengio and Courville) — MIT Press, free online edition