Computer Vision Interview Questions and Answers

Computer vision interview questions keep coming back to one idea: why do convolutional networks suit images, when a plain dense network can also take pixels as input? The answer is easier to give, and to remember, once you have seen it measured. The experiment below trains both kinds of network on real handwritten digits, then moves the test images sideways.

The images come from the UCI Optical Recognition of Handwritten Digits data: 1,797 digits reduced to 8x8 grids of intensity values (UCI: handwritten digits), bundled with scikit-learn as load_digits (scikit-learn: load_digits). Small images keep training to seconds on a CPU, and the principle is the same at any size.

import numpy as np
import torch
from sklearn.datasets import load_digits
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
from torch import nn

torch.set_num_threads(1)
digits = load_digits()                                          # UCI Optical Recognition of Handwritten Digits, 8x8 images
X = digits.images.astype(np.float32) / 16.0                     # pixel values 0-16 scaled to 0-1
X_tr, X_te, y_tr, y_te = train_test_split(X, digits.target, test_size=0.3, stratify=digits.target, random_state=0)
print(f"{len(X):,} images of {X.shape[1]}x{X.shape[2]} pixels, {len(set(digits.target))} classes")


def shift(images, dx):                                          # move every image dx pixels right, filling with blank
    out = np.zeros_like(images)
    out[:, :, dx:] = images[:, :, :8 - dx]
    return out


def mlp():
    return nn.Sequential(nn.Flatten(), nn.Linear(64, 128), nn.ReLU(), nn.Linear(128, 10))


def cnn():
    return nn.Sequential(nn.Unflatten(1, (1, 8)), nn.Conv2d(1, 16, 3, padding=1), nn.ReLU(),
                         nn.Conv2d(16, 32, 3, padding=1), nn.ReLU(), nn.AdaptiveMaxPool2d(1), nn.Flatten(), nn.Linear(32, 10))


def fit(make, images, labels, epochs=60, augment=False, seed=0):
    torch.manual_seed(seed)
    rng = np.random.default_rng(seed)
    net, loss_fn = make(), nn.CrossEntropyLoss()
    opt = torch.optim.Adam(net.parameters(), lr=3e-3)
    yt = torch.tensor(labels)
    for _ in range(epochs):
        batch_x = np.stack([shift(images[i:i + 1], int(rng.integers(0, 3)))[0] for i in range(len(images))]) if augment else images
        xt = torch.tensor(batch_x)
        for idx in torch.randperm(len(yt)).split(64):
            opt.zero_grad()
            loss_fn(net(xt[idx]), yt[idx]).backward()
            opt.step()
    return net.eval()


def acc(net, images, labels):
    with torch.no_grad():
        return (net(torch.tensor(images)).argmax(1).numpy() == labels).mean()


params = lambda net: sum(p.numel() for p in net.parameters())
nets = {"MLP": fit(mlp, X_tr, y_tr), "CNN": fit(cnn, X_tr, y_tr), "CNN + shift augmentation": fit(cnn, X_tr, y_tr, augment=True)}
print("\nC1-C3 test accuracy as the test images are shifted right")
for name, net in nets.items():
    row = " | ".join(f"shift {dx}px {acc(net, shift(X_te, dx), y_te):.3f}" for dx in (0, 1, 2))
    print(f"   {name:<25} {params(net):>6,} parameters | {row}")

pred = nets["CNN"](torch.tensor(X_te)).argmax(1).numpy()
cm = confusion_matrix(y_te, pred)
np.fill_diagonal(cm, 0)
pairs = sorted(((cm[i, j], i, j) for i in range(10) for j in range(10) if cm[i, j]), reverse=True)[:3]
print("\nC4 most common confusions for the CNN (true -> predicted: count):", [f"{i}->{j}: {n}" for n, i, j in pairs])
print(f"   errors in total: {int(cm.sum())} of {len(y_te)} test images")

1,797 images of 8x8 pixels, 10 classes

C1-C3 test accuracy as the test images are shifted right
   MLP                        9,610 parameters | shift 0px 0.978 | shift 1px 0.483 | shift 2px 0.157
   CNN                        5,130 parameters | shift 0px 0.987 | shift 1px 0.978 | shift 2px 0.937
   CNN + shift augmentation   5,130 parameters | shift 0px 0.985 | shift 1px 0.985 | shift 2px 0.980

C4 most common confusions for the CNN (true -> predicted: count): ['9->3: 2', '8->9: 1', '8->7: 1']
   errors in total: 7 of 540 test images

C1. Why use a CNN instead of a dense network for images?

On centred test images the two looked similar: 97.8% for the dense network and 98.7% for the CNN. The difference appeared when the digits moved. Shifted one pixel to the right, the dense network fell to 48.3%, and at two pixels to 15.7%, barely above guessing among ten classes. The CNN kept 97.8% and 93.7%.

The reason is structural. A dense layer learns a separate weight for every pixel position, so a stroke that moves one pixel lands on weights that never saw it. A convolution applies the same small filter at every position, connecting each output to a local patch and sharing its weights across the image (CS231n: convolutional networks; PyTorch: Conv2d). A pattern is detected wherever it appears, and pooling then summarises where it was found.

C2. Doesn't the CNN need more parameters?

Fewer, here. The CNN had 5,130 parameters against the dense network's 9,610, because each filter is reused across the image instead of each pixel having its own weights. On realistic image sizes the gap is far larger: a dense layer on a 224 by 224 colour image needs over 150,000 weights per neuron, while a 3 by 3 convolution filter has 27 whatever the image size.

C3. What does data augmentation do?

It teaches the invariances you want by showing them. Training the same CNN on randomly shifted copies of each digit raised its accuracy on images shifted by two pixels from 93.7% to 98.0%, and cost almost nothing on centred images, 98.5% against 98.7%. Choose augmentations that match how real inputs vary, such as shifts, small rotations and lighting changes, and avoid ones that change the label: a horizontally flipped digit is not the same digit.

C4. How do you analyse a vision model's errors?

Look at the confusion matrix, then at the images. The CNN made 7 errors on 540 test images, and the most common was a 9 read as a 3, twice. Pairs like 9 and 3, or 8 and 9, share strokes, so the next steps are to look at those specific images, check whether they are badly written or mislabelled, and decide whether more examples of those classes would help. Error analysis on a handful of images often teaches more than another round of tuning, and it is the step interviewers most often find missing from candidates' projects.

Other questions to be ready for

• What is transfer learning? Start from a network trained on a large image collection, keep its early layers, which detect general edges and textures, and retrain the final layers on your data. It is the default when you have few labelled images.
• What does a pooling layer do? It summarises a neighbourhood, for example by its maximum, which reduces size and adds tolerance to small shifts. The CNN here used global max pooling.
• How do you handle class imbalance in images? Weighted losses, resampling, targeted augmentation of rare classes, and per-class metrics instead of overall accuracy.
• What is a receptive field? The region of the input image that can influence one unit. Each 3 by 3 convolution widens it by two pixels, so the CNN's second layer already sees a 5 by 5 patch; deeper layers, pooling and strides let later units see whole objects.
• Classification, detection or segmentation? Classification gives one label per image, detection adds a box around each object, and segmentation labels every pixel. Each needs different labels, models and metrics, so ask which one the problem actually needs.
• Which metrics for object detection? Intersection over union for box overlap and mean average precision across classes and thresholds.

How to answer vision questions

• Explain inductive bias: local connectivity and weight sharing match how images work.
• Back claims with a test: shifting, rotating or darkening images shows what a model has really learned.
• Prefer transfer learning and augmentation to training from scratch on small datasets.
• Look at the images behind every error count, not just the numbers.

The basics are taught in computer vision basics and the deep learning guide. For training failures, see deep learning interview questions; for image generation, diffusion models explained; and for the full round, data science interview questions and answers.

Computer vision is part of the Data Science course.

Sources

• Optical Recognition of Handwritten Digits (UCI Machine Learning Repository) — UCI Machine Learning Repository
• load_digits (scikit-learn documentation) — scikit-learn
• Convolutional Neural Networks (Stanford CS231n course notes) — Stanford University
• Conv2d (PyTorch 2.14 documentation) — PyTorch