Computer Vision Basics: From Pixels to Predictions

To a computer, an image is not a cat or a stop sign; it is a grid of numbers. Computer vision is the field of turning those numbers into answers: what is in the image, where it is, which pixels belong to it. Modern vision systems do this with convolutional neural networks and their successors. The ideas underneath are surprisingly concrete, and a small working model trains in seconds on a laptop.

An image is an array

A greyscale image is a two-dimensional array of brightness values; a colour image adds a third dimension for the red, green and blue channels. scikit-learn ships 1,797 small handwritten-digit images, each 8 by 8 pixels with values from 0 to 16 (scikit-learn: load_digits):

import numpy as np
import torch
import torch.nn as nn
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

digits = load_digits()
print(digits.images.shape, digits.images[0].min(), digits.images[0].max())

(1797, 8, 8) 0.0 15.0

A phone photo is the same idea at a much larger scale: millions of pixels, each with three channel values.

Convolution: a small filter that finds patterns

The core operation in vision models is convolution: slide a small grid of weights, a kernel, across the image, and at each position multiply and sum. Different kernels respond to different patterns. This one responds to vertical edges, where brightness jumps from left to right:

img = np.zeros((6, 6))
img[:, 3:] = 9                                   # dark left half, bright right half
kernel = np.array([[-1, 0, 1],
                   [-1, 0, 1],
                   [-1, 0, 1]])
edges = np.array([[np.sum(img[i:i + 3, j:j + 3] * kernel) for j in range(4)] for i in range(4)])
print(edges.astype(int))

[[ 0 27 27  0]
 [ 0 27 27  0]
 [ 0 27 27  0]
 [ 0 27 27  0]]

The output is zero over flat regions and large exactly where the edge is. A convolutional neural network (CNN) does not use hand-designed kernels like this; it learns the kernel weights from data. Early layers typically end up detecting edges and textures, and later layers combine them into parts and whole objects. Pooling layers shrink the maps in between, so later layers see a wider area of the image. In PyTorch, a layer of learned filters is a Conv2d (PyTorch: Conv2d).

A small CNN, trained for real

The digits make a quick experiment: a baseline that treats each image as 64 unrelated numbers, and a small CNN that exploits the spatial structure. The training loop follows the pattern of PyTorch's classifier tutorial (PyTorch: training a classifier):

X = digits.images / 16.0
y = digits.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)

baseline = LogisticRegression(max_iter=2000).fit(X_train.reshape(len(X_train), -1), y_train)
print("logistic regression on raw pixels:", round(baseline.score(X_test.reshape(len(X_test), -1), y_test), 3))

torch.manual_seed(0)
cnn = nn.Sequential(
    nn.Conv2d(1, 16, kernel_size=3, padding=1), nn.ReLU(), nn.MaxPool2d(2),   # 8x8 -> 4x4
    nn.Conv2d(16, 32, kernel_size=3, padding=1), nn.ReLU(), nn.MaxPool2d(2),  # 4x4 -> 2x2
    nn.Flatten(), nn.Linear(32 * 2 * 2, 10),
)
xt = torch.tensor(X_train, dtype=torch.float32).unsqueeze(1)   # add the channel dimension
yt = torch.tensor(y_train, dtype=torch.long)
optimiser = torch.optim.Adam(cnn.parameters(), lr=0.01)
loss_fn = nn.CrossEntropyLoss()

for epoch in range(30):
    for i in range(0, len(xt), 64):                                        # mini-batches of 64
        optimiser.zero_grad()
        loss = loss_fn(cnn(xt[i:i + 64]), yt[i:i + 64])
        loss.backward()
        optimiser.step()

with torch.no_grad():
    predicted = cnn(torch.tensor(X_test, dtype=torch.float32).unsqueeze(1)).argmax(dim=1)
accuracy = (predicted == torch.tensor(y_test)).float().mean().item()
print("small CNN:", round(accuracy, 3), "| parameters:", sum(p.numel() for p in cnn.parameters()))

logistic regression on raw pixels: 0.969
small CNN: 0.982 | parameters: 6090

Both are strong, and the gap is modest, which is an honest lesson in itself: on tiny, centred 8-by-8 digits, even a linear model on raw pixels does well. The CNN's advantage grows with real images, where objects appear at different positions, sizes and lighting, and treating pixels as unrelated numbers stops working. The two convolution layers with 16 and 32 filters, plus one linear layer, add up to about 6,000 learned weights; models for real photographs have millions.

The main computer vision tasks

Each needs a different model architecture and, crucially, differently labelled data: drawing boxes or pixel masks is far more expensive than assigning one label per image, which often decides which task a project can afford.

Transfer learning: the practical default

Training a vision model from scratch needs very large labelled datasets. In practice, almost everyone starts from a model pre-trained on a large dataset and fine-tunes it on their own images. torchvision provides pre-trained models for classification, detection and segmentation for exactly this purpose (torchvision: models). With a pre-trained backbone, a few hundred well-labelled images per class can be enough for a useful classifier, because the early layers already know edges, textures and shapes.

Where computer vision is used

• Manufacturing quality control: spotting scratches, missing parts or misaligned labels on a production line, often where defects are rare and recall matters most.
• Healthcare imaging: highlighting regions of X-rays or scans for a clinician to review, as decision support rather than a final diagnosis.
• Retail and logistics: counting stock on shelves, reading barcodes and labels, and checking packages.
• Agriculture: identifying crop disease or estimating yield from drone and satellite images.
• Documents: extracting text and fields from invoices, forms and identity documents, where OCR is combined with layout understanding.

In each case the hard part is rarely the network architecture. It is collecting images that look like the ones the system will see in use, and agreeing on consistent labels, especially for borderline cases where even experts disagree.

What makes vision projects succeed

• Data quality over model choice. Consistent labels, representative images (lighting, angles, devices) and a test set from the real deployment conditions matter more than the architecture.
• Augmentation. Random crops, flips and brightness changes during training teach the model that these variations do not change the answer.
• Evaluation that matches the use. Accuracy is fine for balanced classes; for rare defects, look at precision and recall; for detection, look at how well boxes overlap the truth.

The neural network fundamentals behind this, including layers, activations and backpropagation, are explained in the deep learning guide, and the same pattern-finding ideas apply to anomaly detection in images. CNNs and computer vision basics are part of the deep learning module of the Data Science course.

Sources

• torch.nn.Conv2d — PyTorch documentation
• Training a Classifier (PyTorch tutorial) — PyTorch documentation
• sklearn.datasets.load_digits — scikit-learn documentation
• torchvision: models and pre-trained weights — PyTorch documentation