How Convolutions Work

Every network so far flattened its input into a feature vector. For images that's a disaster: flattening throws away the very thing that makes an image an image — pixels near each other are related. Convolutional neural networks (CNNs) fix this with one elegant operation. In this lesson you'll see exactly what a convolution computes, slide kernels over an image yourself, master the output-size arithmetic, and implement a convolution from scratch in NumPy.

Why dense layers fail on images

Two fatal problems with feeding raw pixels into nn.Linear:

1. Parameter explosion. A modest 224×224 RGB photo has 224 · 224 · 3 = 150,528 input values. One dense hidden layer of just 1,000 units already needs about 150 million weights — for a single layer, before the network has done anything useful. More parameters means more memory, more compute, and far more data needed to avoid memorizing noise.
2. No translation awareness. To a dense layer, pixel 3,201 and pixel 14,878 are unrelated inputs. If it learns to spot a cat's ear in the top-left corner, it has learned nothing about ears in the bottom-right — every position needs its own separate weights for the same concept.

The fix: learn one small filter that detects a pattern, and reuse it at every position in the image. That's a convolution.

Convolution: a small filter, slid everywhere

A convolutional filter (or kernel) is a tiny grid of weights — typically 3×3 or 5×5. It slides across the image, and at each position computes a weighted sum: multiply the kernel by the pixels underneath, element-wise, and add everything up. The result at each position becomes one pixel of the output, called a feature map — a map of where in the image the kernel's pattern appears.

Try it yourself. Slide different kernels over the image and watch what each one extracts:

Notice the classic hand-crafted kernels:

• an edge-detection kernel (positive center, negative surround) lights up where brightness changes sharply and goes quiet on flat regions;
• a sharpen kernel exaggerates those changes;
• a blur kernel (all-positive, averaging) smooths them away.

For decades, computer-vision engineers designed such kernels by hand. The CNN insight is to make the kernel weights learnable parameters: the network discovers whatever filters minimize the loss — edge detectors usually emerge in the first layer all by themselves. And because one kernel is reused at every position, a 3×3 filter costs 9 weights instead of millions, and a pattern learned anywhere is recognized everywhere: built-in translation awareness.

Channels, stride, and padding

Real images have channels (RGB = 3), and each convolution layer produces many feature maps. Vocabulary for a layer that maps 3 input channels to 8 output channels with 3×3 kernels:

• Each output channel has its own kernel of shape 3×3×3 — it spans all input channels and sums across them. So the layer holds 8 kernels, 8 · (3·3·3) = 216 weights (plus 8 biases).
• Stride is the step size of the slide. Stride 1 visits every position; stride 2 skips every other one, halving the output's width and height.
• Padding adds a border of zeros around the input so the kernel can center on edge pixels. Without padding, every 3×3 convolution shrinks the image by 2 pixels per side pair; with padding 1, the size is preserved.

All of that folds into one formula for the output size along each dimension, with input size n, kernel size k, padding p, and stride s:

out = floor((n + 2p − k) / s) + 1

Worked example — a 28×28 image, 3×3 kernel, padding 1, stride 1: out = floor((28 + 2 − 3) / 1) + 1 = 28. Same size, as promised. Now stride 2: out = floor((28 + 2 − 3) / 2) + 1 = floor(13.5) + 1 = 14. Halved. You'll use this formula constantly when designing architectures — the flatten layer at the end needs to know exactly what size arrives.

Pooling: cheap downsampling

A pooling layer shrinks feature maps without any learnable weights. Max pooling with a 2×2 window and stride 2 splits the map into 2×2 tiles and keeps only the largest value in each — width and height halve, and the strongest activations survive. Why bother?

• It cuts computation for every following layer by 4x.
• It makes the representation slightly translation-tolerant: if the edge moves one pixel, the max of its neighborhood often doesn't change.
• Growing "receptive fields": after pooling, one pixel of a feature map summarizes a larger region of the original image, letting later layers see bigger patterns.

The feature hierarchy

Stack the pattern — convolution, nonlinearity, pooling — a few times and something remarkable happens. The first layer, looking at raw pixels, learns edges and color blobs. The second layer, looking at edge maps, learns textures and corners — combinations of edges. Deeper layers combine those into object parts (eyes, wheels, feathers), and the final layers into whole objects. Nobody programs this hierarchy; it emerges from training, exactly like the learned features of the MLPs you trained — but now organized spatially.

Convolution from scratch

Ten lines of NumPy make the operation concrete. Here's a vertical-edge kernel sliding over a tiny image that's dark on the left, bright on the right:

Read the output: the feature map is zero on the flat regions and strongly negative exactly along the dark-to-bright boundary — the kernel found the vertical edge and nothing else. (Sign just encodes the edge's direction; a ReLU after the convolution would keep one polarity.) Note the size: (6 − 3)/1 + 1 = 4, matching the formula with no padding.

The PyTorch layers

In PyTorch these operations are single layers:

import torch
from torch import nn
 
conv = nn.Conv2d(in_channels=3, out_channels=8,
                 kernel_size=3, stride=1, padding=1)
pool = nn.MaxPool2d(kernel_size=2, stride=2)
 
x = torch.randn(1, 3, 64, 64)     # [batch, channels, height, width]
h = conv(x)
print(h.shape)                    # [1, 8, 64, 64]  (padding=1 preserves size)
h = pool(h)
print(h.shape)                    # [1, 8, 32, 32]  (pooling halves H and W)
 
n_weights = sum(p.numel() for p in conv.parameters())
print(n_weights)                  # 8*(3*3*3) + 8 = 224 parameters

224 parameters to process an image of any size — compare that to the 150 million a dense layer needed. Convolution + ReLU + pooling is the block we'll stack into a full CNN in the next lesson.

Next up: stacking conv-relu-pool blocks into a complete CNN and training it on a real image dataset in PyTorch.