Deep Learning · Module 3 of 26 · 8 concepts · about 17 min

Activation Functions

Where a stack of linear layers becomes capable of learning nonlinear structure — and which bend to choose.

At work: The small choices inside a network that decide whether it trains at all, and what kind of answer it gives.

ReLU, sigmoid, tanh and GELU

What it is

Activations transform a layer's pre-activation values before they enter the next layer. ReLU is max(0, x) and is a strong default for many hidden layers because it is simple and has a healthy positive-side gradient; sigmoid maps to (0, 1), tanh to (-1, 1), and GELU is common in Transformers because it smoothly gates values. The final activation is chosen to match the output distribution: sigmoid for one binary probability, softmax for mutually exclusive classes, and often no activation at all when a stable loss function expects logits.

Why it matters

Without an activation, any number of linear layers collapses algebraically to one linear transformation. Activations add the bends that let networks carve nonlinear decision regions and represent complex functions. Their gradient behaviour also matters: saturated sigmoid or tanh units receive little gradient, while a ReLU that stays negative can stop learning entirely for that input region.

In the real world: The small functions that let networks learn complex patterns, like telling a genuine review from a fake one.

Visual example

Activation shapes

The shape determines which signals pass and which are suppressed.

ReLUsigmoidtanh

ReLU is zero for negative inputs; sigmoid and tanh saturate at their extremes.

import torch, torch.nn as nn

z = torch.tensor([-2.0, -0.5, 0.0, 0.5, 2.0])
print(nn.ReLU()(z))      # tensor([0.0000, 0.0000, 0.0000, 0.5000, 2.0000])
print(nn.Sigmoid()(z))   # tensor([0.1192, 0.3775, 0.5000, 0.6225, 0.8808])
print(nn.Tanh()(z))      # tensor([-0.9640, -0.4621, 0.0000, 0.4621, 0.9640])
print(nn.GELU()(z))      # tensor([-0.0455, -0.1543, 0.0000, 0.3457, 1.9545])

Putting sigmoid before BCEWithLogitsLoss. That loss already applies the numerically stable sigmoid internally; applying it twice weakens gradients and gives the wrong objective.

Practise it: ReLU, Forward and Backward

Saturation: why sigmoid and tanh choke the gradient

What it is

A saturated unit is one whose input has pushed it into a flat region of its activation — sigmoid near 0 or 1, tanh near ±1. Flat means the derivative is near zero, and backpropagation multiplies by that derivative (module 7), so the gradient arriving at everything upstream is scaled down. Sigmoid's derivative is at most 0.25, even at its steepest point; tanh's peaks at 1.0 but falls below 0.1 by |x| = 2.

Why it matters

Multiply enough numbers below one together and you get nothing. A ten-layer sigmoid network attenuates the gradient by at most 0.25¹⁰ ≈ 1e-6 before it reaches layer one, which is why deep sigmoid stacks were effectively untrainable and why ReLU — derivative exactly 1 on its whole positive side — was the change that made depth practical. This is the mechanism behind 'vanishing gradients', covered with its fixes in module 10.

In the real world: Why older deep networks trained slowly and often got stuck, wasting GPU time.

Visual example

Gradient flow

Backpropagation multiplies by these curves. Sigmoid never gives more than 0.25, even at its best.

ReLU′tanh′sigmoid′

Ten layers, compounded

sigmoid, best case: 0.25¹⁰ ≈ 9.5e-7

tanh, best case: 1.0¹⁰ = 1 — but only exactly at x = 0

ReLU, active units: 1¹⁰ = 1 — the gradient arrives intact

Why this matters

The forward shapes of these three functions look similarly reasonable. Their derivatives are what decide whether a deep stack trains at all.

import torch

z = torch.tensor([-6.0, -2.0, 0.0, 2.0, 6.0], requires_grad=True)
torch.sigmoid(z).sum().backward()
print(z.grad)   # tensor([0.0025, 0.1050, 0.2500, 0.1050, 0.0025])

z2 = torch.tensor([-6.0, -2.0, 0.0, 2.0, 6.0], requires_grad=True)
torch.relu(z2).sum().backward()
print(z2.grad)  # tensor([0., 0., 0., 1., 1.]) — full gradient where active

Blaming saturation on the activation alone. Unnormalised inputs and badly scaled initial weights are what push pre-activations to ±6 in the first place; swapping the activation treats the symptom, while normalising the input treats the cause.

Dying ReLU, LeakyReLU and the negative side

What it is

A ReLU unit whose pre-activation is negative for every example in the dataset outputs zero forever and receives exactly zero gradient, so its weights never change again — the unit is dead, not merely quiet. It usually happens after a large update drives the bias strongly negative, most often because the learning rate was too high. LeakyReLU keeps a small slope (typically 0.01) on the negative side so a negative unit still receives some gradient and can recover; ELU and GELU achieve the same with a smooth negative tail.

Why it matters

Dead units are capacity you paid for and cannot use, and they are invisible in the loss curve — training continues, just with a smaller effective network. A layer with 40% dead units behaves like a layer 40% narrower, which looks like an unexplained accuracy ceiling. Checking the fraction of zero activations per layer is a two-line diagnostic that turns that mystery into a number.

In the real world: A chunk of a model can quietly stop learning, leaving a recommendation model weaker than it should be.

Visual example

ReLU variants

ReLU's negative side is exactly flat — zero output and zero gradient. The variants keep a path open.

ReLULeakyReLU(0.1)GELU

Common misconception

Wrong idea: a dead ReLU will recover once the data changes.

Correct idea: it cannot. Its gradient is identically zero for every negative input, so nothing updates its weights and the input never moves back.

import torch, torch.nn as nn

acts = {}
def record(name):
    return lambda _module, _inp, out: acts.__setitem__(name, out.detach())

model = nn.Sequential(nn.Linear(64, 128), nn.ReLU(), nn.Linear(128, 10))
model[1].register_forward_hook(record("relu1"))
model(torch.randn(256, 64))

dead = (acts["relu1"] == 0).all(dim=0).float().mean()
print(f"units dead across the whole batch: {dead:.1%}")

leaky = nn.LeakyReLU(negative_slope=0.01)   # negative inputs still pass a gradient

Fixing dying ReLUs by switching to LeakyReLU and leaving the learning rate alone. The leak lets units recover, but the oversized updates that killed them are still destabilising everything else; lower the learning rate first and check whether the leak is still needed.

Practise it: ReLU, Forward and Backward

GELU and SiLU: what modern architectures actually use

What it is

GELU multiplies the input by the probability that a standard normal variable is below it — x · Φ(x) — giving a smooth curve that is close to ReLU for large positive x, close to zero for large negative x, and slightly negative just below zero. SiLU (also called Swish) is the same idea with a sigmoid gate: x · σ(x). Both are the default hidden activation in current Transformer language models and in several modern vision backbones.

Why it matters

The practical gain is smoothness. ReLU's derivative jumps from 0 to 1 at the origin, which makes the loss surface piecewise-linear and slightly harder for adaptive optimisers to traverse near that kink; GELU and SiLU are differentiable everywhere and have no dead region, which shows up as marginally better final loss at the same parameter count in large-scale training. The gain is real but small — a percent or two — which is exactly why it only became the default once runs got expensive enough for a percent to matter.

In the real world: The activations inside today's language and vision models, like those behind chat assistants.

import torch, torch.nn as nn

z = torch.tensor([-2.0, -0.5, 0.5, 2.0])
print(nn.GELU()(z))   # tensor([-0.0455, -0.1543, 0.3457, 1.9545])
print(nn.SiLU()(z))   # tensor([-0.2384, -0.1888, 0.3112, 1.7616])

# A Transformer-style feed-forward block:
block = nn.Sequential(nn.Linear(512, 2048), nn.GELU(), nn.Linear(2048, 512))

Swapping ReLU for GELU as a performance fix on a small model. It costs noticeably more compute per element, cannot be fused as cheaply, and its advantage is a fraction of a percent — worth it in a 100-GPU run, invisible next to your data problems in a small one.

Softmax: the output activation, not a hidden one

What it is

Softmax turns a vector of logits into a probability distribution: exponentiate each entry and divide by the sum, so the outputs are positive and sum to one. It is an *output* activation for mutually exclusive multi-class problems, applied once at the end, across the class dimension — never between hidden layers, where it would force a layer's features to compete for a fixed budget of one. Its numerics need the shift-before-exponentiate trick, which is the same log-sum-exp stabilisation derived in the NumPy syllabus's numerical-stability module; PyTorch's softmax and log_softmax already apply it.

Why it matters

Softmax is where a model's scores become numbers you can report, threshold, or calibrate. Getting the axis wrong is a silent bug: softmax(dim=0) on a (batch, classes) tensor normalises down the batch, so each *class* sums to one across examples and every probability changes when the batch composition changes. And applying it before nn.CrossEntropyLoss double-counts the normalisation, which weakens gradients without raising anything.

In the real world: Turning a model's scores into “70% shoes, 20% bags, 10% other” for product categorisation.

Visual example

Softmax

Shift by the max, exponentiate, divide by the sum. The shift changes nothing but prevents overflow.

Raw scores

(what the model emits)

logits [2.0, 1.0, 0.1] → probabilities

	logit	logit − max	exp	probability
cat	2	0	1	0.659
dog	1	-1	0.368	0.242
bird	0.1	-1.9	0.15	0.099

Report these; feed nn.CrossEntropyLoss the raw logit column instead.

The arithmetic

sum of exps: 1.0 + 0.368 + 0.15 = 1.518

cat: 1.0 / 1.518 = 0.659

probabilities sum to 1.000 — across classes, not across the batch

import torch

logits = torch.tensor([[2.0, 1.0, 0.1]])
print(logits.softmax(dim=1))        # tensor([[0.6590, 0.2424, 0.0986]])

# The shift is invisible but essential — same answer, no overflow:
big = torch.tensor([[1000.0, 999.0, 998.1]])
print(big.softmax(dim=1))           # tensor([[0.6590, 0.2424, 0.0986]])
print((big.exp() / big.exp().sum()))  # tensor([[nan, nan, nan]]) — done by hand

Calling softmax with no dim, or with dim=0 on a batched tensor. The default is deprecated and the wrong axis normalises across examples instead of across classes — every probability then depends on which other rows happened to share the batch.

Practise it: Scaled Dot-Product Attention

Sigmoid or softmax: multi-label vs multi-class outputs

What it is

Softmax couples the outputs — raising one probability necessarily lowers the others — which is correct when exactly one class applies. Sigmoid treats each output independently, so several can be near 1 at once, which is correct when labels are not mutually exclusive: an image that is both 'outdoors' and 'contains a dog', a document with three topic tags. The two cases pair with different losses: CrossEntropyLoss for the single-label case, BCEWithLogitsLoss for the multi-label case.

Why it matters

Choosing the wrong one makes the model structurally unable to say the right thing. A softmax head on a multi-label problem cannot assign 0.9 to two tags, so training forces it to split the probability and every tag looks uncertain. A sigmoid head on a single-label problem allows contradictory confident outputs and throws away the 'exactly one' constraint that would have made the task easier.

In the real world: A photo can be “outdoor” and “beach” and “people” at once (multi-label), but has one main product category (multi-class).

import torch, torch.nn as nn

logits = torch.tensor([[2.0, 1.8, -1.0]])

# Mutually exclusive: one label wins, probabilities sum to 1.
print(logits.softmax(dim=1))        # tensor([[0.5352, 0.4382, 0.0267]])
single_loss = nn.CrossEntropyLoss()(logits, torch.tensor([0]))

# Independent tags: two can both be high, and they do not sum to 1.
print(torch.sigmoid(logits))        # tensor([[0.8808, 0.8581, 0.2689]])
targets = torch.tensor([[1.0, 1.0, 0.0]])          # float, one per label
multi_loss = nn.BCEWithLogitsLoss()(logits, targets)

Using CrossEntropyLoss with multi-hot targets. It expects integer class indices (or a full distribution), so a multi-hot float tensor either raises on dtype or silently treats your label vector as a distribution over classes.

Where the activation goes in a block

What it is

In a modern block the conventional order is linear (or convolution) → normalisation → activation, with the activation applied after the normalisation so the nonlinearity sees a well-scaled input. The final layer of a network usually has *no* activation at all: PyTorch's classification and regression losses expect raw logits or raw values, because they fold the sigmoid or softmax into a numerically stable formulation internally.

Why it matters

Both halves cause real bugs. An activation before the normalisation wastes the normalisation — the ReLU has already clipped half the distribution the normaliser was meant to centre. An activation after the last layer double-applies it against the loss, producing a wrong objective and small gradients. 'Activation everywhere except the output' is the rule to have automatic.

In the real world: A detail that affects how stably large models train, and so how much a training run costs.

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(64, 128), nn.BatchNorm1d(128), nn.ReLU(),   # linear -> norm -> act
    nn.Linear(128, 64), nn.BatchNorm1d(64), nn.ReLU(),
    nn.Linear(64, 10),                                    # no activation here
)
loss_fn = nn.CrossEntropyLoss()   # applies log_softmax itself, stably

Ending a Sequential with nn.Softmax(dim=1) because the model 'should output probabilities'. The loss then sees probabilities it will normalise again; train on logits and call .softmax(dim=1) at inference time when you actually want to display a probability.

Choosing an activation in practice

What it is

Hidden layers: ReLU is the safe default and GELU or SiLU the current default in Transformers; use LeakyReLU if you have measured dead units. Outputs: nothing at all for regression and for any loss that takes logits, sigmoid for an independent binary probability you need to display, softmax for a displayed multi-class distribution. Special cases are driven by the range you need — tanh for an output bounded in (−1, 1), such as a normalised image from a generator.

Why it matters

Activation choice is one of the lowest-leverage hyperparameters in deep learning, and it attracts disproportionate attention because there are always new ones. The measured differences between reasonable modern activations are small and often within run-to-run noise; data quality, learning rate, and normalisation each move the needle by orders of magnitude more. Knowing that lets you spend the default choice in five seconds and get on with the parts that matter.

In the real world: Using the proven default instead of experimenting saves days on a production model.

Visual example

Decision guide

Almost every activation decision is one of five rows — and the output row is where the bugs live.

which activation, and why

	choose	because
hidden (default)	ReLU	cheap, gradient 1 where active
hidden (Transformer)	GELU / SiLU	smooth, no dead region, marginal gain at scale
hidden (dead units)	LeakyReLU	keeps a gradient path on the negative side
output + logit loss	none	the loss applies sigmoid/softmax stably itself
output for display	sigmoid / softmax	only at inference, to show a probability

Common misconception

Wrong idea: the newest activation in the literature is the best choice.

Correct idea: the differences are a fraction of a percent and often inside seed noise. The highlighted row is the one that actually breaks models — an activation applied twice, once by you and once by the loss.

import torch.nn as nn

hidden = nn.GELU()                  # or nn.ReLU() — either is defensible

regression_head = nn.Linear(64, 1)                  # no activation; MSELoss
binary_head = nn.Linear(64, 1)                      # no activation; BCEWithLogitsLoss
multiclass_head = nn.Linear(64, 10)                 # no activation; CrossEntropyLoss
bounded_head = nn.Sequential(nn.Linear(64, 3), nn.Tanh())   # output must be in (-1, 1)

Benchmarking activations on a single seed. The differences between ReLU, GELU and SiLU on a small model are usually smaller than the variance across random seeds, so a one-run comparison will confidently report whichever you ran second.