DL GEN AI · Week 3
(N, C, H, W) — N images, C channels, H×W pixels. A convolution slides one small kernel across an image, multiplying and summing each patch it covers into a single number of the feature map.
Order is the point — nn.Conv2d stays a black box until you have written
the loop it wraps — so from-scratch comes before PyTorch, and the classifier before
its training loop.
flowchart LR S[scratch conv] --> P[scratch pool] P --> TC[torch conv] TC --> TP[torch pool] TP --> NET[SimpleCNN] NET --> DATA[dataloaders] DATA --> TRAIN[train loop]
Each step replaces the previous one's hand loop with the library call, keeping shapes identical. Nothing new is learned after §4 — only wrapped and assembled.
CrossEntropyLoss does softmax internally. Softmax first and you apply it
twice. Same for shapes: torch images are (N, C, H, W) everywhere — forget the batch
or channel dim and every layer errors.
Five groups of words carry the rest of the page. Read them once. Each comes back with a number you can check.
An image tensor is a 4-D array with shape (N, C, H, W):
N images in one batch, C channels per image, H rows and W columns of pixels. A
pixel is one cell of one channel. A channel is one
full grid of numbers: an RGB photo has three, a grayscale photo has one, and a feature
map has as many as the layer writes. The batch dimension is always there, even for one
image, so a single 28×28 grayscale sample is (1, 1, 28, 28), not (28, 28).
A kernel (or filter) is a small grid of weights, 3×3 here. It slides over the input; at each position it multiplies the patch under it by its weights and adds the products into one number. When the input has several channels, one kernel holds a 3×3 grid per input channel, and the products from all of them are summed into that one number. A feature map is the grid of those numbers, also called an activation map because a nonlinearity usually follows. One kernel writes one feature map, so 64 kernels write 64 channels.
Stride is how far the kernel moves per step. Stride 1 moves one pixel; stride 2 moves two, so the output is about half as wide and half as tall. Padding is a ring of zeros around the border, so the kernel can centre on an edge pixel. Kernel 3, stride 1, padding 1 is called a same convolution: it writes the input's H and W unchanged. The classifier blocks later use it. Kernels are usually odd-sized (3×3, 5×5) so the window has a centre cell. The output side follows \[ \text{out} = \left\lfloor \frac{H + 2P - K}{S} \right\rfloor + 1. \]
Weight sharing means one kernel uses the same weights at every position. A feature learned at the top-left is therefore detected at every other position, and a 3×3 kernel holds nine weights per input channel whatever the image size is. That is why a convolutional layer has far fewer parameters than a dense layer over the same image.
The receptive field is the patch of the input that one output cell can see. One 3×3 convolution sees 3×3. Two in a row see 5×5, because the second combines three outputs of the first and each of those already saw 3×3. Pooling grows the field too: a pool halves the map, so the next convolution covers twice the input distance.
Pooling shrinks a feature map by summarising each window into one number. Max pooling keeps the largest value in the window. Average pooling keeps the mean. Global pooling reduces a whole map to one number per channel. Pooling has no weights, so training cannot change it. A 2×2 window with stride 2 halves both sides: 28 → 14.
A logit is the raw score the last layer writes, before any
normalisation. It can be any real number, positive or negative. A 10-class model
writes 10 logits per image. Softmax turns those 10 numbers into 10
probabilities that sum to 1, and argmax picks the largest. CrossEntropyLoss
applies softmax inside, so it takes raw logits and returns one number.
The head is the part after the last feature map.
Flattening turns the (C, H, W) block into one vector of \(C \cdot H \cdot W\)
numbers, and the fully connected (Linear) layers read that vector. The
flatten size is fixed by the input size and the number of pools, so the head must be
built for it.
The loop below writes the operation out. It is valid convolution: stride 1, no padding, so the kernel must fit entirely inside the image. Each output cell reads one local patch, and each kernel learns one feature: edges and blobs early, abstractions deeper. Technically the loop computes cross-correlation, because it does not flip the kernel; for learned filters the flip changes nothing.
import numpy as np
def conv2d(image, kernel):
hi, wi = image.shape
hk, wk = kernel.shape
ho, wo = hi - hk + 1, wi - wk + 1 # kernel must fit inside
out = np.zeros((ho, wo)) # pre-allocate the feature map
for i in range(ho):
for j in range(wo):
patch = image[i:i+hk, j:j+wk]
out[i, j] = (patch * kernel).sum() # elementwise multiply, then sum
return outThe demo: a 12×12 image with a central 6×6 white square, three kernels (horizontal edge [1s; 0s; −1s], vertical edge, sharpening). Normalise each map (minus min, over max-minus-min) before display.
Run the horizontal-edge kernel and two dark bands appear: top edge (dark→bright, ones on dark side) and bottom edge (bright→dark). Same filter, same response, anywhere in the image — that is weight sharing made visible. The vertical kernel gives the left/right pair. The sharpening kernel raises contrast at the edges.
Three kinds. Max takes the window's maximum — keeps the strongest activation, i.e. the most important feature. Average takes the mean — preserves background, sometimes used in final layers. Global averages each whole feature map to one number: 512 maps of 7×7 become 512 activations into the head. Pooling buys dimensionality reduction plus invariance to small translations: a deeper neuron sees a large input region, and the maximum survives wherever the feature sits inside it.
The 4×4 demo with 2×2 max-pool, stride 2: each 2×2 block contributes its maximum (e.g. a block holding 1, 3, 9, 7 outputs 9). Output size follows the same formula as convolution — here \(\frac{4-2}{2}+1 = 2\) per side, no padding:
\[ H_{out} = \left\lfloor \frac{H - P_{size}}{S} \right\rfloor + 1 \]def max_pool2d(image, size=2, stride=2):
h, w = image.shape
ho, wo = (h - size) // stride + 1, (w - size) // stride + 1
out = np.zeros((ho, wo))
for i in range(ho):
for j in range(wo):
window = image[i*stride:i*stride+size, j*stride:j*stride+size]
out[i, j] = np.max(window) # avg variant: np.mean(window)
return outSame loop shape as §2, different reduction. Backward: max routes the gradient to the argmax element only; average splits it evenly across the window.
On a random 10×10 image both poolers halve to 5×5, but max looks blocky (winners kept) and average looks smoother. That visual difference is the whole semantic difference: select vs smooth.
explore_conv2d(in_ch, out_ch, K, stride, padding) builds a
random (N, C, H, W) input, instantiates the layer, runs the forward pass, prints
shapes, counts parameters, and visualises 1×1 filters:
import torch.nn as nn
conv = nn.Conv2d(in_channels=3, out_channels=64,
kernel_size=3, stride=1, padding=0)
# conv.weight.shape == (64, 3, 3, 3) # (out, in, H, W) — the small 4th-order tensor
# conv.bias.shape == (64,) # one bias per output channel
out = conv(x) # nn.Module is callable: runs forward under the hoodkernel_size/stride/padding take an int (both dims) or a tuple (per-dim). Weights arrive with standard init — no manual setup.
Output height/width follow \(W_{out} = \left\lfloor \frac{W - K + 2P}{S} \right\rfloor + 1\). Parameter count, two ways that must agree — via the module, and by hand:
total = sum(p.numel() for p in conv.parameters()) # weights + biases
weights = 64 * 3 * 3 * 3 # Co * Ci * K * K = 1728
biases = 64 # one per output channel
assert total == weights + biases # 1792parameters() yields weights plus bias; numel() counts elements per tensor. Detach (.detach()) before visualising to cut grad history.
The toy: values 1…36 as a 6×6 map, with singleton batch and
channel dims added (unsqueeze) into torch's (N, C, H, W) format:
feat = torch.arange(1, 37).reshape(1, 1, 6, 6).float()
maxp = nn.MaxPool2d(kernel_size=2, stride=2) # non-overlapping, no padding
avgp = nn.AvgPool2d(kernel_size=2, stride=2)
maxp(feat).shape # (1, 1, 3, 3) — each spatial dim halved, channels untouched
# squeeze() drops singleton dims for printing; .item() takes a 0-dim tensor to floatCeiling mode off (default) means floor for output size. Kernel 2 stride 2 is the standard non-overlapping halver.
Global pooling is adaptive pooling to 1×1 — name the output size, torch figures out the window:
nn.AdaptiveMaxPool2d(1)(feat).item() # 36.0 — max over the whole map
nn.AdaptiveAvgPool2d(1)(feat).item() # 18.5 — mean over the whole mapGlobal average is the variable-size-input trick: whatever the map size, one number per channel into the classifier.
Three same-convolution blocks (K=3, S=1, P=1 preserves size; 2×2 pool halves),
channels doubling 32 → 64 → 128, then the head. super().__init__()
registers everything; nn.ReLU is stateless; dropout 0.5 acts only in
train mode:
class SimpleCNN(nn.Module):
def __init__(self):
super().__init__()
self.c1 = nn.Conv2d(1, 32, 3, 1, 1); self.r = nn.ReLU()
self.p = nn.MaxPool2d(2, 2)
self.c2 = nn.Conv2d(32, 64, 3, 1, 1)
self.c3 = nn.Conv2d(64, 128, 3, 1, 1)
self.fc1 = nn.Linear(128*3*3, 128) # 1152 -> 128
self.drop = nn.Dropout(0.5)
self.fc2 = nn.Linear(128, 10)
def forward(self, x): # (N,1,28,28) in
x = self.p(self.r(self.c1(x))) # (N,32,14,14)
x = self.p(self.r(self.c2(x))) # (N,64,7,7)
x = self.p(self.r(self.c3(x))) # (N,128,3,3)
x = x.view(x.size(0), -1) # (N,1152)
return self.fc2(self.drop(self.r(self.fc1(x)))) # (N,10) logitsTotal 241,546 parameters (320 + 18,496 + 73,856 + 147,584 + 1,290). Print the model and read it block by block — on deep nets that listing is the architecture doc.
Two usage notes. Feed random input and you get 10 meaningless
logits — the shape check is the point. And the feature-map extractor snapshots each
block with x.detach().clone(): detach cuts autograd history (no grads,
no graph kept for logging), clone copies independently so later in-place ops cannot
mutate the saved map.
transforms.Compose chains per-image transforms in order.
ToTensor converts PIL/NumPy to (C, H, W) float32 scaled 0–255 → 0–1;
Normalize(mean, std) standardises per channel (one value each for
grayscale FashionMNIST):
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
tfm = transforms.Compose([transforms.ToTensor(), # (1,28,28), 0..1
transforms.Normalize((0.5,), (0.5,))])
train_ds = datasets.FashionMNIST(root="data", train=True,
transform=tfm) # 60k (image, label) pairs
test_ds = datasets.FashionMNIST(root="data", train=False,
transform=tfm) # 10k pairs
train_loader = DataLoader(train_ds, batch_size=64,
shuffle=True, num_workers=2) # reshuffle each epoch
test_loader = DataLoader(test_ds, batch_size=64, shuffle=False)Labels pass through unchanged (ints 0–9). Shuffle train only; test is forward-only. 938 train batches and 157 test batches per epoch (ceil(60000/64), ceil(10000/64)).
One batch is images (64, 1, 28, 28) plus labels (64,). A grid viewer
takes the first 10, squeeze()s the channel dim, converts to NumPy, and
plots 2×5 with label.item() as each title — the fastest way to confirm
the pipeline before burning GPU time.
Setup order matters: model to device before creating the optimizer, so it
tracks parameters on the right device. Loss is CrossEntropyLoss —
expects (N, 10) raw logits and (N,) long labels 0–9, softmax internal. Optimizer is
Adam at lr 0.001, the standard starting point:
model = SimpleCNN().to(device) # move params before the optimizer sees them
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)The relay never changes: zero → forward → loss → backward → step. Only the layer and the loss change here.
Epoch loop over batch loop, with tqdm wrapping the loader for a live
bar. Per batch: to device, zero_grad (grads accumulate otherwise),
forward, loss, backward, step. Accumulate running loss via
loss.item() and correct predictions via outputs.argmax(1)
compared against labels; report epoch means and plot both curves. model.train()
opens each epoch (dropout on, batch stats). The run uses five epochs: train loss
0.5977 → 0.2488 and train accuracy 78.28% → 91.08%. It then scores the 10,000 test
images with model.eval() under torch.no_grad(): loss 0.2713,
accuracy 90.16%, 9016 correct.
model.train()
for epoch in range(5):
running, correct, total = 0.0, 0, 0
for inputs, labels in train_loader:
inputs, labels = inputs.to(device), labels.to(device)
optimizer.zero_grad()
outputs = model(inputs) # (64,10) logits
loss = criterion(outputs, labels)
loss.backward(); optimizer.step() # the relay, in order
running += loss.item()
correct += (outputs.argmax(1) == labels).sum().item()
total += labels.size(0)argmax over dim 1 (classes), not dim 0 (batch). size(0) is the batch length — the denominator of accuracy.
Three experiments. Each asks for a prediction before the code. Write the prediction down; the paragraph after the code is the check. Run them in a fresh interpreter.
Predict first. A 4×4 input holds the numbers 1 to 16, row by row. A 3×3 kernel of ones slides over it with stride 1. Without padding, what is the output shape, and what is its top-left value? With padding 1, what is the shape, and what does the centre 2×2 hold?
import numpy as np
def conv2d(image, kernel, padding=0):
if padding:
image = np.pad(image, padding)
hi, wi = image.shape
hk, wk = kernel.shape
out = np.zeros((hi - hk + 1, wi - wk + 1))
for i in range(out.shape[0]):
for j in range(out.shape[1]):
out[i, j] = (image[i:i+hk, j:j+wk] * kernel).sum()
return out
image = np.arange(1, 17).reshape(4, 4)
kernel = np.ones((3, 3), dtype=int)
print(conv2d(image, kernel).astype(int))
# [[54 63]
# [90 99]]
print(conv2d(image, kernel, 1).astype(int))
# [[14 24 30 22]
# [33 54 63 45]
# [57 90 99 69]
# [46 72 78 54]]np.pad adds the zero ring; the double loop is the §2 loop with padding added.
What happens. Without padding the output is 2×2, because the kernel has only two places to start in a 4-wide input: \(4 - 3 + 1 = 2\). Padding 1 gives 4×4, because \(\frac{4 + 2 - 3}{1} + 1 = 4\). The top-left padded value is \(1 + 2 + 5 + 6 = 14\): five of its nine window cells are zeros. The centre 2×2 is 54, 63, 90, 99, exactly the unpadded output, because those windows never touch the border ring. Padding changes the border, not the interior.
Predict first. A batch of one RGB image, 32×32, goes through
nn.Conv2d(3, 8, 3, padding=1). What is the output shape? How many
parameters does the layer hold? Then a 2×2 max pool reads the output: what shape now?
import torch, torch.nn as nn
x = torch.zeros(1, 3, 32, 32)
conv = nn.Conv2d(3, 8, 3, padding=1)
print(conv(x).shape) # torch.Size([1, 8, 32, 32])
print(sum(p.numel() for p in conv.parameters())) # 224
print(nn.MaxPool2d(2)(conv(x)).shape) # torch.Size([1, 8, 16, 16])Zeros are enough: the shapes and counts do not depend on the values.
What happens. Eight kernels write eight channels, and padding 1 keeps the 32×32. The count is \(8 \times 3 \times 3 \times 3 + 8 = 224\): 216 weights plus one bias per output channel. H and W do not appear, so the same layer holds 224 parameters on a 512×512 image. The pool halves each side and leaves the channels alone: 32 → 16.
Predict first. SimpleCNN (the class from §6) starts its head with
nn.Linear(128*3*3, 128), built for a 28×28 input. Feed a batch of two
32×32 images. The three pools turn 32 into 4, so the flatten holds
\(128 \times 4 \times 4 = 2048\) numbers. What does the head expect, and what does
torch report?
model = SimpleCNN()
print(model(torch.zeros(1, 1, 28, 28)).shape) # torch.Size([1, 10])
model(torch.zeros(2, 1, 32, 32))
# RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x2048 and 1152x128)
What happens. x.view(x.size(0), -1) has no fixed size, so
it hands fc1 a 2×2048 matrix. fc1 was built for 1152 input features, and the message
pairs 2048 with 1152. The fix is to match the head to the input,
nn.Linear(128*4*4, 128) for 32×32. The rule: the flatten size follows
from the input size and the number of pools, so count it before you write the head.
Click an option to check your answer. Each question maps to one section above. Options are shuffled on every load and on every reset, so position tells you nothing.
nn.Conv2d(32, 64, 3, stride=1, padding=1) on a (N,32,H,W) input. What are the weight and bias shapes, and what is preserved?conv2d (valid, stride 1) takes a 12×12 image and a 3×3 kernel. What is the output shape?model.train(). What does it switch on, and what does the evaluation pass use?nn.Conv2d(3, 8, 3, padding=1) from its weight shape. Run it and check 224; change 8 to 16, predict 448, then run.One-line understanding. Write the double loop once so the torch call is transparent, assemble the classifier, then feed it batches and run zero → forward → backward → step until the curves flatten.