Deep Learning · Module 2 of 26 · 7 concepts · about 15 min

Neurons & Neural Networks

From one weighted sum to a layer of learned features, and the shape contract that holds a stack together.

At work: The building blocks inside every image classifier, recommender and language model a company runs.

Neurons, weights, biases and layers

What it is

A neuron computes z = x·w + b, then usually applies an activation. The weights say how strongly each input contributes; positive and negative weights can support or suppress a feature, while the bias shifts the threshold at which the neuron activates. A layer evaluates many neurons in parallel: for a batch X, Z = XWᵀ + b. The output width is the number of independently learned features that layer can expose to the next layer.

Why it matters

Shapes explain most neural-network bugs. A linear layer with in_features=64, out_features=128 maps (batch, 64) to (batch, 128); its weights are learned, while its dimensions are architectural choices that trade capacity against cost. Once that contract is clear, a matrix multiplication error is not mysterious: either the previous layer emitted the wrong feature width or the next layer declared the wrong input width.

In the real world: Inside a model that predicts whether a customer will click an ad: many small weighted sums stacked in layers.

Visual example

Interactive forward pass

Move an input slider and follow its contribution through the weighted sum and activation in real time.

Try an input

x₁w = 1.251

x₂w = -0.80-0.50

x₃w = 0.600.75

bias b-0.40

Live forward pass

x₁1

x₂-0.50

x₃0.75

w₁1.25

w₂-0.80

w₃0.60

→

z = Σ(xᵢwᵢ) + b

1.70

1 × 1.25 = 1.25

-0.50 × -0.80 = 0.40

0.75 × 0.60 = 0.45

+ bias -0.40 = 1.70

→ ReLU →

neuron output

1.70

Why this matters

A layer is just many copies of this calculation with different learned weights. Seeing the one-neuron arithmetic makes the matrix form feel like a compact notation rather than magic.

Common misconception

Wrong idea: a neuron makes a binary decision by itself.

Correct idea: it produces a continuous value. A network gets its expressive power from many such values, their nonlinearities, and their composition.

import torch.nn as nn
layer = nn.Linear(64, 128)
# input: (batch, 64); output: (batch, 128)

Thinking a bias is optional decoration. Without it every neuron's decision boundary is forced through the origin, a needless and often severe restriction.

Depth and width: what each one buys

What it is

Width is how many neurons a single layer has — how many distinct features that layer can expose at once. Depth is how many layers are stacked — how many times a representation gets re-described before the output. Widening a layer multiplies its parameter count by the widened factor; deepening adds a whole new weight matrix. In PyTorch both are just the numbers you pass to a sequence of nn.Linear calls.

Why it matters

They fail differently, so the symptom tells you which to change. A network that is too narrow has an information bottleneck: no amount of training recovers features it had nowhere to put. A network that is too shallow can hold plenty of features but cannot combine them into anything abstract, so training loss plateaus well above zero while the model still generalises smoothly. Excess width tends to overfit; excess depth tends to stop training at all (module 10).

In the real world: Choosing a bigger or deeper model decides both accuracy and how much each prediction costs to serve.

Visual example

Architecture

The same input and output, two ways to spend the capacity — and two different failure modes.

64 features in, 10 classes out

	hidden widths	parameters	fails by
narrow + deep	32 → 32 → 32	4522	vanishing gradients if deeper
wide + shallow	512	38410	overfitting the training set
bottlenecked	4	310	cannot represent 10 classes well

Parameter count is not capability: the 310-parameter row fails for a structural reason no learning rate fixes.

import torch.nn as nn

narrow_deep = nn.Sequential(
    nn.Linear(64, 32), nn.ReLU(),
    nn.Linear(32, 32), nn.ReLU(),
    nn.Linear(32, 32), nn.ReLU(),
    nn.Linear(32, 10),
)
wide_shallow = nn.Sequential(nn.Linear(64, 512), nn.ReLU(), nn.Linear(512, 10))

print(sum(p.numel() for p in narrow_deep.parameters()))    # 4522
print(sum(p.numel() for p in wide_shallow.parameters()))   # 38410

Treating a hidden width smaller than the output width as harmless. A Linear(64, 4) before a 10-class head forces all ten classes through four numbers; the ceiling that creates is invisible in the code and obvious only as a stubborn training loss.

Why stacked linear layers collapse into one

What it is

Composing two affine maps gives another affine map. Concretely, W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂), so a Linear(64, 128) followed directly by a Linear(128, 10) computes exactly what some single Linear(64, 10) computes. This holds for any number of stacked linear layers: the whole stack is one matrix and one vector.

Why it matters

This is the entire argument for activation functions, and it is worth seeing as algebra rather than as advice. A ten-layer 'deep' network with no nonlinearity has the representational power of logistic regression — it will train, produce a plausible loss curve, and never learn anything a linear model could not. Forgetting one nn.ReLU() between two layers is a silent capacity bug that no error message reports.

In the real world: Without activations, a 10-layer network is no smarter than a single one, and all that extra compute buys nothing.

Visual example

The case for activations

Two linear layers with nothing between them are not deeper than one — they are the same function, written twice.

as written

	layer	shape
1	Linear(4, 8)	(4, 8)
2	Linear(8, 2)	(8, 2)

W₂(W₁x + b₁) + b₂

as computed

	layer	shape
1	Linear(4, 2)	(4, 2)

The collapse, term by term

weights: W₂W₁ → one (2, 4) matrix

biases: W₂b₁ + b₂ → one (2,) vector

a ReLU between them: max(0, ·) does not distribute, so nothing collapses

Common misconception

Wrong idea: depth alone makes a network powerful.

Correct idea: depth plus a nonlinearity does. Without the bend between layers, the extra matrices multiply out and the model is linear regression with more parameters.

import torch, torch.nn as nn

torch.manual_seed(0)
stack = nn.Sequential(nn.Linear(4, 8, bias=False), nn.Linear(8, 2, bias=False))
collapsed = nn.Linear(4, 2, bias=False)
with torch.no_grad():
    collapsed.weight.copy_(stack[1].weight @ stack[0].weight)   # (2,8) @ (8,4) -> (2,4)

x = torch.randn(5, 4)
print(torch.allclose(stack(x), collapsed(x), atol=1e-6))         # True

Believing the collapse is only about bias=False. Biases compose too — into W₂b₁ + b₂ — so a stack of biased linear layers is still exactly one biased linear layer.

Counting parameters in a Linear layer

What it is

nn.Linear(in_features, out_features) holds a weight of shape (out_features, in_features) and a bias of shape (out_features,), so it owns in_features * out_features + out_features numbers. Note the weight shape is output-first: the layer computes x @ weight.T + bias, which is why a (batch, in) input comes out as (batch, out).

Why it matters

The count is what you actually pay, three times over: parameter memory, gradient memory (one number per parameter), and optimizer state (Adam keeps two more per parameter, so four times the weights in bytes). It also tells you where a model's size lives — in most MLPs and Transformers a couple of wide layers hold nearly all the parameters, and trimming anything else saves nothing worth measuring.

In the real world: Estimating a model's memory and serving cost before training it, to see if it fits the budget.

Visual example

Memory

One Linear(784, 256) costs its parameter count four times over during Adam training.

Per-parameter state

(every row is the same size as the layer)

nn.Linear(784, 256) — 200,960 parameters

	numbers held	float32 bytes
weights	200960	803 KB
gradients	200960	803 KB
Adam m	200960	803 KB
Adam v	200960	803 KB

≈3.2 MB for one layer before a single activation is stored — and activations usually cost more.

import torch.nn as nn

layer = nn.Linear(784, 256)
print(tuple(layer.weight.shape), tuple(layer.bias.shape))   # (256, 784) (256,)
print(784 * 256 + 256)                                      # 200960

model = nn.Sequential(layer, nn.ReLU(), nn.Linear(256, 10))
for name, p in model.named_parameters():
    print(name, tuple(p.shape), p.numel())
# 0.weight (256, 784) 200960 | 0.bias (256,) 256 | 2.weight (10, 256) 2560 | 2.bias (10,) 10

Estimating GPU memory from the parameter count alone. Training also stores gradients and optimizer state per parameter, plus every activation the backward pass needs — which for a large batch is usually the dominant term, not the weights.

The shape contract between consecutive layers

What it is

Each layer publishes a contract: it accepts a trailing dimension of in_features and emits out_features. A stack is valid exactly when each layer's out_features equals the next layer's in_features, all the way down to a final width that matches what the loss expects — the number of classes for cross-entropy, one for scalar regression. PyTorch checks this at call time, not at construction time, so an inconsistent stack builds happily and fails on the first batch.

Why it matters

This is the single most common real bug in deep learning code, and it is entirely mechanical to prevent. Writing the shape at each boundary as a comment, or printing it once with a dummy batch, converts a class of runtime failures into a thing you read. It also makes refactors safe: changing a hidden width is a one-line edit if the contract is written down and a debugging session if it is not.

In the real world: Most “model won't run” errors in a new product feature come down to one layer's output not matching the next one's input.

Visual example

Deep learning

Each layer's output width must be the next layer's input width — and the last must match the loss.

A valid three-layer contract

input batch(B, 64)
Linear(64, 128) + ReLU(B, 128)
Linear(128, 32) + ReLU(B, 32)
Linear(32, 10)(B, 10)one logit per class

CrossEntropyLoss requires the final 10 to equal the number of classes; nothing else in the stack knows that.

import torch, torch.nn as nn

model = nn.Sequential(
    nn.Linear(64, 128), nn.ReLU(),   # (B, 64)  -> (B, 128)
    nn.Linear(128, 32), nn.ReLU(),   # (B, 128) -> (B, 32)
    nn.Linear(32, 10),               # (B, 32)  -> (B, 10)
)
print(model(torch.randn(4, 64)).shape)   # torch.Size([4, 10])

# A quick contract check with a dummy batch, before touching real data:
x = torch.zeros(2, 64)
for i, layer in enumerate(model):
    x = layer(x)
    print(i, type(layer).__name__, tuple(x.shape))

Reading mat1 and mat2 shapes cannot be multiplied (4x128 and 32x10) as a bug in the failing layer. The mismatch names two layers; the wrong one is usually the earlier layer's out_features, because that is the number someone changed.

Batches: the leading dimension every layer ignores

What it is

PyTorch layers are written to operate on a whole batch at once. nn.Linear applies the same weight matrix to every row of a (batch, in) input with one matrix multiply, and passes the batch axis through untouched: the output is (batch, out). The same holds for convolutions, attention and normalisation — the leading axis is examples, and the layer's parameters are shared across all of them.

Why it matters

This is the NumPy vectorisation argument with hardware money attached. One (256, 784) @ (784, 256) matmul saturates a GPU's arithmetic units; 256 separate (784,) dot products do not, and run one or two orders of magnitude slower for identical results. It is also why a batch is the unit of everything downstream: one loss per batch, one gradient estimate per batch, one optimizer step per batch.

In the real world: Scoring 256 transactions in one batch instead of one at a time makes a fraud service far cheaper to run.

import torch, torch.nn as nn

layer = nn.Linear(4, 3)
batch = torch.randn(8, 4)
one = batch[0:1]                       # keep the batch axis: (1, 4), not (4,)

print(layer(batch).shape, layer(one).shape)          # (8, 3) (1, 3)
print(torch.allclose(layer(batch)[0], layer(one)[0], atol=1e-6))   # True

Passing a single example with its batch axis squeezed away. A (784,) tensor into a Linear(784, 10) happens to work — the trailing dimension matches — but BatchNorm, loss functions and argmax(dim=1) downstream then all operate on the wrong axis. Use x.unsqueeze(0) so one example is a batch of one.

What a bias actually does

What it is

The weights decide the direction the neuron is sensitive to; the bias decides where along that direction it switches on. In one dimension, sigmoid(wx) is pinned to output 0.5 at x = 0, whatever w is; sigmoid(wx + b) crosses 0.5 at x = −b/w, so the bias slides the whole response curve left or right. Geometrically, in higher dimensions the weights orient the decision hyperplane and the bias offsets it from the origin.

Why it matters

Without a bias, every neuron's boundary is forced through the origin, which is almost never where the data's boundary is. A model on unnormalised features — pixel values in 0–255, prices in the thousands — needs exactly that offset to be useful, and a biasless layer can only compensate by distorting its weights. It is also why bias=False is safe immediately before a BatchNorm or LayerNorm: the normalisation layer contributes its own shift, so the bias would be redundant.

In the real world: It lets a model predict a sensible baseline, like typical demand, even when the inputs are all zero.

Visual example

Bias

The bias does not change the neuron's shape — it changes where the neuron turns on.

b = 0b = −4

Why this matters

A biasless neuron must switch at x = 0. Real features rarely have their threshold at zero, so the bias is what lets the same weights land in the right place.

import torch, torch.nn as nn

with_bias = nn.Linear(1, 1)
without = nn.Linear(1, 1, bias=False)
with torch.no_grad():
    with_bias.weight.fill_(2.0); with_bias.bias.fill_(-4.0)
    without.weight.fill_(2.0)

x = torch.tensor([[0.0], [2.0]])
print(torch.sigmoid(with_bias(x)).flatten())   # tensor([0.0180, 0.5000]) — turns on at x = 2
print(torch.sigmoid(without(x)).flatten())     # tensor([0.5000, 0.9820]) — pinned at x = 0

Adding bias=True to a layer followed immediately by nn.BatchNorm1d. The normalisation subtracts the batch mean, which removes the bias's effect entirely, so the parameter is trained and then discarded every step.