DL GEN AI · Week 3
This page takes the dense layer apart, because on image data it is the wrong machine. §1 fixes what an image is, and §2 defines the words the rest of the page uses so that no term arrives undefined. §§3–7 say why convolution is the only thing that could come next; §§8–12 show the thing sliding and name the two constraints; §13 runs the two claims in torch; §§14–15 shape it into a real classifier; §§16–17 place it in history.
flowchart LR IMG[image as a grid] --> FAIL[three MLP failures] FAIL --> FIX[two constraints] FIX --> SLIDE[filter slides] SLIDE --> KNOB[stride and pad] KNOB --> LAYERS[pool norm dropout] LAYERS --> ARCH[classifier]
Failures motivate the constraints; the constraints leave a sliding filter; knobs and layers turn the filter into a classifier.
A digital image is not a picture to a program. It is a grid of numbers, and each cell is a pixel — the smallest unit. You see a cat; the program sees a matrix. A 32×32 colour image is a 32 × 32 × 3 array of numbers, each 0–255 (the 3 is RGB); a 480×480 JPG is 480 × 480 × 3. That array is all the computer gets. The job: turn it into class probabilities, e.g. 0.80 cat, 0.15 dog, 0.05 bird. There is no perception step. That is the entire gap between you and the model.
Three kinds of image matter here, and they differ only in how many numbers each pixel carries:
| Type | Channels | Per pixel | Range |
|---|---|---|---|
| Binary | 1 | 1 bit | 0 or 1 |
| Grayscale | 1 | 8 bits | 0 (black) to 255 (white) |
| Colour (RGB) | 3 | 3 × 8 bits | 0–255 each, one per channel |
A colour image is three stacked grids — one red, one green, one blue, each called a channel — held as a single tensor. A 224×224 RGB image is therefore a \(224 \times 224 \times 3\) array, which is
\[ 224 \cdot 224 \cdot 3 = 150{,}528 \]input dimensions. Hold that number. §4 turns it into 22 billion.
Vision modelling is not one problem. It is several, and the distinction that matters for the whole week is what the network must hand back.
Keep the last one in mind. When the output is a grid the same size as the input, "the answer moves when the image moves" is not a defect — it is exactly what you asked for. This is the distinction between equivariance (per-pixel outputs that transform with the input) and invariance (a single label that ignores the transform) — the distinction that makes a convolutional layer the right tool instead of an approximate one.
The words below carry the rest of the page. Each is defined once, and the later sections use them without re-explaining.
An image tensor is the array a vision model reads. Torch orders it
(N, C, H, W): N images in the batch, C channels, H rows and W columns. Eight
RGB photos at 224×224 are (8, 3, 224, 224). A pixel is one
(row, column) position. A channel is one grid of numbers, one per position,
and the channel depth is how many such grids are stacked: 3 for RGB, 1 for
grayscale.
A kernel — the same thing as a filter — is a small grid of learned weights. It slides over the input; at each stop it multiplies the patch under it by its weights and adds the products into one number. Its depth always equals the input's channel depth, so a 3×3 kernel on an RGB image is 3×3×3. A feature map, also called an activation map, is the grid of those numbers written by one kernel. One kernel writes one feature map; 64 kernels write 64, so the output depth is the filter count.
Stride is how far the kernel moves per step. Stride 1 moves one pixel, so neighbouring stops share most of their patch. Stride 2 moves two, so the kernel makes about half as many stops per side. Padding is a border of zeros, P pixels deep on each side of the input. It lets a kernel centre on an edge pixel, and padding 1 with a 3×3 kernel at stride 1 leaves H and W unchanged.
The receptive field of one output cell is the region of the input that can change its value. One 3×3 convolution gives 3×3. Stack a second 3×3 on top and the cell reads three cells of the first map, each of which already read 3×3 of the input, so it sees 5×5. Every extra 3×3 layer at stride 1 adds one pixel of reach on each side: n stacked 3×3 layers see \((2n+1) \times (2n+1)\).
Parameter sharing is the rule that one kernel uses the same weights at every position. The layer stores the kernel once, so its parameter count does not grow with H and W. The consequence is translation equivariance: shift the input one pixel and the feature map shifts one pixel with the same values. Translation invariance is stronger: the output does not change at all. A classifier wants invariance for its final label; pooling (§14) buys part of it by discarding position inside each window.
Pooling replaces each small window of a feature map with one number. A 2×2 max pool with stride 2 keeps the largest value in each 2×2 block, so both sides halve and the channel depth is untouched. It has no weights, so training cannot change it. Discarding the exact position inside the block is the point: a feature that moves one pixel usually survives.
Take a table: rows are samples, columns are features like "age" and "income". The features have no spatial relationship to each other — swapping their columns changes nothing. That is precisely the structure a fully connected layer assumes: an unordered bag of features, every one connected to every one. For a table, that assumption is correct and costs nothing.
An image is not a table. A 28×28 picture is a grid, and the grid's geometry is real information: pixel (10, 10) sits beside pixel (10, 11), and every visual feature lives in such neighbourhoods. Flatten the grid to 784 numbers and all notion of "neighbour" is gone — the network cannot reach the geometry. Three separate failures follow, and none of them is fixed by tuning, more epochs, or a wider hidden layer.
A fully connected layer from 150,528 inputs to one hidden unit needs 150,528 weights — per unit. Hidden layers are normally wider than their input, so a shallow network lands around
\[ 150{,}528^2 = 22{,}658{,}678{,}784 \]weights. Twenty-three billion. Training them needs matching training data, matching memory, and matching computation. No labelled image dataset contains 23 billion examples.
A smaller version makes the same point: a 100×100 RGB image is \(100 \cdot 100 \cdot 3 = 30{,}000\) inputs, and a first hidden layer of only 1,000 neurons already costs 30 million weights. A modern sensor outputs far more than 100×100, so multiply that by another factor of ten and it is unwieldy immediately.
So this failure is not "expensive". It is impossible — and it is the weakest of the three, because it is a resource problem that better hardware would eventually paper over. Hold that thought; the next two are not resource problems at all.
Nearby image pixels are statistically related. In an image of a shoe, the pixel at \((i,j)\) has a lot to say about \((i,j+1)\) and very little to say about the pixel on the opposite corner.
A fully connected layer has no notion of "nearby". It treats the relationship between every input pair as equal, because from its point of view there is no arrangement of the inputs to have a relationship with — just an index.
Draw the conclusion properly, because it is stronger than it first looks. Not "permuting hurts a little" — it does not hurt at all. The label survived the permutation, because the label was a property of the multiset of pixels, and the fully connected layer reads only a multiset. A fully connected layer is blind to image geometry, by construction, in a way that no amount of training data can fix. It has no input that carries position.
An image of a tree is still an image of a tree if you shift it left by a few pixels. But that shift changes every input to the network. So a fully connected model must learn the pattern of pixels that signify a tree separately at every position. That is worse than inefficient — it is a direct loss of accuracy, and of data efficiency, on the way.
Watch the failure happen. Train on images with the cat near the centre, and the network learns position-specific weights that look for a cat in the middle. Move the cat, and the network starts failing — not gracefully degrading, failing, because the thing it learned was tied to a location. The goal of any vision network is the opposite: identify the object regardless of where it sits in the frame.
Take the fully connected layer and forbid two things. Nothing else changes.
Locality alone buys you the parameter reduction and hands the layer a real notion of nearby pixels. Weight sharing alone buys you the parameter reduction and gives every position the same feature detector. Together they buy all three things this page needs: far fewer parameters, explicit use of spatial relationships, and no re-learning a pattern at every position.
Here is the part worth sitting with. A convolutional layer is exactly what is left after you impose these two constraints on a fully connected layer. You did not choose it from a menu. You took the machine you already understand, removed everything you did not need, and the residue has a name. §10 walks the algebra that makes this precise.
Payoff in one comparison. A fully connected output unit on a 224×224 RGB image needs 150,528 weights. An entire convolutional layer with an 11×11 kernel producing 96 output channels needs
\[ 11 \cdot 11 \cdot 3 \cdot 96 = 34{,}848 \]weights — for all 96 feature maps at once. The convolutional layer completes 96 full passes over the image using fewer weights than a single fully connected neuron would need.
Picture a flashlight beam covering a 5×5 patch in the top-left corner, then sliding across the whole image one step at a time. The flashlight is the filter (also called kernel or neuron); the patch it lights is the receptive field. The filter itself is an array of numbers — the weights — and its depth must match the input's, so here it is 5 × 5 × 3. A biological aside is worth having: Hubel and Wiesel (1962) found visual-cortex neurons that fire only for edges at one orientation, organised in columns. Specialised detectors, each with its own job — that is the whole idea behind a filter.
At each position: multiply filter values by the pixel values underneath, element by element (75 multiplications for a 5×5×3 filter), add them up, write down one number. Slide to the next position, repeat. A 5×5 filter fits in 784 positions on a 32×32 image, so the output is a 28 × 28 × 1 array — the activation map (feature map). Use two filters instead of one and you get 28 × 28 × 2. More filters, deeper output, more information kept.
Patch the filter currently covers, and a vertical-edge filter:
Elementwise multiply, then sum — keep the left column, drop the middle, subtract the right:
\[ (1-3) + (4-6) + (7-9) = -2 + -2 + -2 = -6 \]Output: −6. Left brighter than right gives positive, right brighter gives negative, uniform gives zero. This filter was written by hand to show the arithmetic. In a trained network nobody writes it: random weights plus gradient descent on the loss arrive at detectors of the same kind. A curve detector tells the same story: 6600 where the curve is, 0 where it is not.
First-layer filters detect edges, colours, curves. Feed their activation maps into a second conv layer and its filters combine those into semicircles and corners; deeper still, paws and beaks; at the end, filters that fire on handwriting or pink objects (Zeiler–Fergus visualised exactly this ladder). Each layer's input describes where the previous layer's features were, so depth builds abstraction. Receptive fields grow with depth: a deeper filter sees a wider region of the original image through the layers below it.
Then the fully connected layer caps the network: it takes the last activation maps and outputs one number per class (10 for digits). It learns which high-level features vote for which class — paw-maps vote dog, wing-maps vote bird — as weights on the previous layer. Training is four steps on repeat: forward pass, loss, backward pass (which weights contributed, via dL/dW), weight update against the gradient, scaled by the learning rate. Too large a rate overshoots; too small crawls.
This is the derivation, with the jargon translated. A dense layer on a 28×28 image flattened to 784 inputs, with 784 hidden units, needs \(784^2 = 614{,}656\) weights — and on the grid its weight needs an index for input row, input column, and hidden unit, i.e. a 4th-order tensor. Two facts about natural images then delete almost all of it:
Constraint 1 — weight sharing (translation equivariance). A vertical edge top-left and the same edge bottom-right are the same feature, so the detector should not care where it sits: shifting the input should shift the output. Therefore a weight may depend only on the relative offset between input and output, not on absolute position. In symbols, each absolute-position weight collapses onto an offset weight (\(w\) indexed by position becomes \(v\) indexed by offset). The parameter count now equals the size of \(v\) — independent of image size. Equivariance (shift in → shift out) falls out for free; nobody bolted it on. Invariance is the stronger property, and pooling is what buys it.
Constraint 2 — locality. A pixel correlates strongly with its neighbours and barely at all with the far side of the image, so set \(v\) to zero outside a small window (say 3×3). The sum shrinks to that window. That is the convolutional layer, and there is nothing further to derive.
| Layer | Weights | Depends on image size? |
|---|---|---|
| Dense, 784 inputs → 784 hidden | 614,656 | yes, quadratically |
| Conv 3×3, 1 input channel → 8 output channels | 72 | no |
| Conv 3×3, 3 input channels → 64 output channels (+ 64 biases) | 1,728 (+ 64) | no |
Channels bring back a 4th-order tensor, but a small one: one filter per output channel, each seeing all input channels, so \(C_i \times C_o \times K \times K\) weights plus \(C_o\) biases. For 224×224 RGB → 64 maps that is 1,728 weights doing the work the dense layer needed billions for.
The count is the visible symptom; the restricted family is the cause. These two constraints forbid every mapping that treats one position differently from another, so the layer can only express functions built from the same detector everywhere. That is a much smaller family of functions than the dense layer can express, and a smaller family that still fits the training data is the standard reason to expect lower test error. A dense network can represent a shifted edge; it has to learn that edge again at every position, from data at every position.
nn.Conv2d does not flip
(it cross-correlates). Harmless: the kernel is learned, so training learns whichever
orientation it needs. Say "convolution" to people, "cross-correlation" when a
hand-computed value disagrees with a textbook.
6×6 image: rows 0–2 hold 30, rows 3–5 hold 0 — one horizontal step. Filter: 3×3, top row +1, bottom row −1. Valid convolution gives 4×4. Straddling the step:
\[ (30+30+30) - (0+0+0) = 90 \]The output spikes to 90 at the transition and sits near zero elsewhere. A filter reports change, never brightness. Real filters are not designed like this one — Gaussian init, then forward–loss–backprop until convergence, and the task decides the detectors.
One filter on one input gives one 2D feature map. A layer holds many filters, so its output is a 3D volume — one slice per filter. A neuron's receptive field is the input region affecting it. Layer 1: exactly the filter size. Stack two 3×3 layers and each layer-2 neuron reads 3×3 of layer 1, each of which read 3×3 of the input:
\[ 3 + 3 - 1 = 5 \]Deeper neurons see wider regions, so the network builds edges → corners → parts → objects without being told to.
3×3 is the default. Two stacked 3×3 layers cover the same 5×5 field as one 5×5 filter with fewer multiplications (18 vs 25) and a nonlinearity in between. Large kernels (5×5, 7×7) appear in early layers on large inputs to buy context fast.
A 1×1 convolution looks at one pixel across all channels. 32 filters of size 1×1×192 on a 56×56×192 volume give 56×56×32: same spatial size, fewer channels. It aggregates channel information per pixel, followed by a nonlinearity — a fully connected layer over channels.
Stride is the step size: S=1 moves one pixel, S=2 skips one. Strided convolution downsamples while learning, and usually replaces pooling in modern nets. Padding adds zero borders: valid means none (output shrinks), same means just enough to preserve the input size (e.g. pad 1 with a 3×3 filter at stride 1). Padding also gives edge pixels the same coverage central pixels get. One formula runs every size question (\(W\) in, \(K\) filter, \(P\) pad, \(S\) stride):
\[ W_{out} = \left\lfloor\frac{W - K + 2P}{S}\right\rfloor + 1 \]Check: 32×32, 5×5, P=0, S=1 → 28 (our §8 flashlight). 32×32, 3×3, P=1, S=2, 64 filters → 16×16×64. After each convolution comes nonlinearity (ReLU, \(\max(0,x)\) — negatives to zero, trains faster than tanh/sigmoid).
Two claims carry this page: a dense layer cannot hold an image, and the shape of a feature map follows from stride and padding. Both take a few lines to run. Predict each answer before you read the output under the code.
Predict first. A dense layer takes a 32×32 RGB image to 10 outputs. A 3×3 convolution takes the same image to 16 channels. Which holds more weights, and by what factor?
import torch, torch.nn as nn
dense = nn.Linear(32 * 32 * 3, 10)
conv = nn.Conv2d(3, 16, 3)
print(32 * 32 * 3) # 3072
print(sum(p.numel() for p in dense.parameters())) # 30730
print(sum(p.numel() for p in conv.parameters())) # 448
print(conv(torch.zeros(1, 3, 64, 64)).shape) # torch.Size([1, 16, 62, 62])30,730 = 3,072 × 10 weights + 10 biases. 448 = 3×3×3×16 weights + 16 biases. Both counts include the bias, because parameters() yields it.
What happens. The dense layer holds 30,730 parameters and the convolution holds 448. The dense layer has about 69 times as many. The gap is not the interesting part. The dense count is tied to one input size: at 64×64 the flattened image has \(64 \cdot 64 \cdot 3 = 12{,}288\) inputs, so the same layer becomes \(12{,}288 \times 10 + 10 = 122{,}890\) parameters. The convolution's 448 is the same at 32×32, at 64×64, and at 512×512, because the kernel is stored once and used at every position. That is parameter sharing, and it is why the failure in §4 has no fix inside the dense layer.
Predict first. A (1, 3, 32, 32) input goes through four 3×3 convolutions: no padding at stride 1; padding 1 at stride 1; padding 1 at stride 2; no padding at stride 2. Predict each output side before you run it.
x = torch.zeros(1, 3, 32, 32)
print(nn.Conv2d(3, 16, 3)(x).shape) # torch.Size([1, 16, 30, 30])
print(nn.Conv2d(3, 16, 3, padding=1)(x).shape) # torch.Size([1, 16, 32, 32])
print(nn.Conv2d(3, 16, 3, stride=2, padding=1)(x).shape) # torch.Size([1, 16, 16, 16])
print(nn.Conv2d(3, 16, 3, stride=2)(x).shape) # torch.Size([1, 16, 15, 15])30 = 32 − 3 + 1; 32 = 32 + 2 − 3 + 1; 16 = ⌊31 ÷ 2⌋ + 1; 15 = ⌊29 ÷ 2⌋ + 1.
What happens. Padding 1 keeps 32×32 because the ring of zeros gives every edge pixel a full window. Stride 2 gives 16×16 because it visits every second position. The padding still does work there: without it the same layer returns 15×15, since the first stop must fit the whole kernel inside the border. The channel count is 16 in all four cases, set by the number of filters and never by the input size. §12's formula is these four lines in symbols.
A 2×2 max-pool with stride 2 halves each spatial dimension, has no parameters, and keeps the strongest activation per patch. Halving twice per block is what makes later convolutions cheap, widens the receptive field, and tolerates small shifts — the maximum survives wherever it sits inside its patch. Average pooling smooths instead of selecting; both are applied per channel. Global pooling averages each whole feature map to one number — e.g. 512 maps of 7×7 become 512 activations feeding the head.
Deep layers see shifting input distributions as early weights move, which slows training and over-sensitises init. BatchNorm standardises each channel over the batch and spatial positions, then restores representational power with two learned parameters per channel:
\[ \hat{x} = \frac{x - \mu_B}{\sqrt{\sigma^2_B + \epsilon}}, \qquad y = \gamma \hat{x} + \beta \]
Training uses batch statistics; inference uses running averages (momentum 0.1) because a
single image has no meaningful batch. That is the entire reason for
train() vs eval() mode. Side effects: higher learning rates,
less init sensitivity, mild regularisation.
Dropout zeroes activations, never weights, with probability \(p\) — only during training. Survivors are scaled by \(\frac{1}{1-p}\) (so \(p=0.5\) doubles them), keeping the expected value identical at test time, when every neuron is active and unscaled. Forcing random neurons out stops co-adaptation: no neuron can rely on its neighbours being present. It matters most in the fully connected head, where most of the weights live.
Stack conv blocks (convolution → ReLU → pool), flatten, classify. Spatial size shrinks while channel count grows — fewer positions, richer features per position. A Fashion-MNIST classifier makes the shapes concrete:
| Stage | Output |
|---|---|
| Input | 1×28×28 |
| Conv 32 + pool | 32×14×14 |
| Conv 64 + pool | 64×7×7 |
| Conv 128 + pool | 128×3×3 |
| Flatten → FC 128 (+dropout) → FC 10 | 1152 → 128 → 10 |
\(128 \cdot 3 \cdot 3 = 1152\). Count biases too — everyone forgets them.
CNNs memorise small datasets, and labelled images are expensive. Augmentation synthesises training data with class-preserving transforms — rotation, flips, scaling, cropping, colour jitter. The rule is one sentence: the cat must still be a cat. Each transform teaches the model one invariance for free. The setup: 60k train / 10k test, pixels to [0,1], batch 64, ~5 epochs.
The instinct to say "CNNs are the deep learning innovation" is wrong by about twenty years. The architecture is old. What is new is that anyone could afford to train one.
| Year | What happened | Why it matters for this week |
|---|---|---|
| 1988–89 | CNNs applied to handwriting recognition, phoneme recognition, signature verification (LeCun et al. 1989a, Waibel et al. 1989, Bromley et al. 1993). Max pooling dates to Zhou & Chellappa (1988), average pooling to LeCun et al. (1989a). | The building blocks all existed before the field had a name for itself. |
| 1998 | LeNet (LeCun et al.) classifies 28×28 grayscale handwritten digits. Convolution layers, then fully connected layers; sigmoid activations rather than ReLUs; average pooling rather than max pooling. It ran on cheques. | "Immediately recognisable as a precursor of modern networks" — same skeleton, three details changed. Also the origin of the 28×28 input that recurs below. |
| 1999–2011 | The long winter. CNNs get sidelined; the field chases MLPs, SVMs and hand-engineered features, because CNNs needed data and GPUs that did not exist yet. | The architecture was not rejected on the merits. It was waiting on its enablers. |
| 2009 | ImageNet collated (Deng et al.) with a classification challenge attached: 1,281,167 training images, 1,000 categories. | Data at a scale where "more data" and "more parameters" could trade off. |
| 2011 | Before deep networks, the state of the art made ~25% top-5 error on ImageNet. | The number AlexNet is measured against. |
| 2012 | AlexNet (Krizhevsky et al.). Eight hidden layers, five convolutional then fully connected, ReLU. 11×11 stride 4 → 96 channels; max pool; 5×5 → 256; three 3×3 → 13×13×256; max pool → 6×6×256; FC 4096, 4096, 1000. ~60M parameters, most of them in the fully connected layers. SGD momentum 0.9, batch 128, dropout, L2, 2048× data augmentation. 16.4% top-5, 38.1% top-1. | "This result revealed the potential of deep learning and kick-started the modern era of AI research." It became the starting point for modern deep convolutional networks. |
| 2014 | VGG — fundamentally the same design, far deeper: 19 hidden layers, 144M parameters, 6.8% top-5 / 23.7% top-1. Then network-in-network's 1×1 convolutions and GoogLeNet's inception blocks. | For years, error fell as depth rose. Depth was the lever, and AlexNet had already shown the block was not the bottleneck. |
| 2015→ | Deeper networks become hard to train without modifications — residual connections and normalization. Current ImageNet leaders (ViT, SWIN) are transformers. | The shape of everything after it. And note what survived the handoff: CNNs are still the backbone under everything generative — the diffusion models behind today's image generators are U-Nets, convolutional networks with a skip path. |
The idea was right long before the enablers were. LeNet's architecture was essentially final in 1998 and was not competitive again until 2012. Fourteen years of the design sitting idle is the strongest available evidence that a good architecture and a trainable architecture are different properties — one is mathematics, the other is data and hardware.
The 2012 leap came from a stack of things, not from a new kind of layer. AlexNet's contribution was depth, ReLU, dropout, heavy augmentation, and GPUs used in parallel — all five of which existed separately beforehand. Nobody reading the 2012 paper found a new operator in it. When a result looks like a single idea, look at the training regime around it before you go hunting for the idea.
AlexNet (2012) is the coming-out party: 5 conv layers + dropout + augmentation, 16.4% top-5 error on ImageNet, five days on two GTX 580s. ZF Net (2013) shrinks the first filter to 7×7 to keep pixel information and adds the deconv visualisation — layers 1–2 show edges/circles, 3–5 dogs' faces and flowers. VGG (2014) says keep it deep and simple: only 3×3 (two stacked ≈ one 5×5, with an extra ReLU) and double filters after each pool. GoogLeNet (2015) breaks the sequential stack with the Inception module — 1×1 bottlenecks before 3×3/5×5 plus pooling in parallel, no fully connected head, 12× fewer parameters than AlexNet. ResNet (2015) adds the skip: H(x) = F(x) + x, so each block learns the delta — 152 layers, 3.6% error. Then detection (R-CNN → Fast → Faster: share the conv computation, then learn the region proposals too), GANs (generator vs discriminator), captioning (CNN + RNN), spatial transformers.
Click an option to check your answer. Each question maps to one section above. Options are shuffled on every load and on every reset, so position tells you nothing.
nn.Conv2d does not. Why is using "convolution" for the un-flipped version
safe?torch.nn.functional.conv2d. Check −6, and check that a patch of all 5s gives 0.One-line understanding. Images are grids, a fully connected layer reads them as unordered bags, and convolution is what remains once you force every neuron to see only a small neighbourhood and force every position to share one set of weights.