Loss Functions

A loss function converts a prediction and its target into a differentiable scalar that training can minimise. Cross-entropy penalises low probability assigned to the correct class and is the usual multiclass objective; binary cross-entropy handles one or more independent yes/no labels. MSE and MAE compare numeric targets, but differ in how strongly they punish a large residual: MSE squares it, while MAE treats every additional unit of error equally. Batch loss is usually the mean over examples, so its scale remains comparable as batch size changes.

A model does exactly what its loss rewards, not what its name suggests. The loss must match both the target encoding and the decision you care about; an excellent loss can still pair with an unsuitable business threshold or evaluation metric. Before training, state what one target represents, what one model output represents, and why reducing this scalar should improve the real outcome.

A loss is one scalar per batch, computed from predictions and targets, that is differentiable with respect to the model's parameters almost everywhere. 'One scalar' is a hard requirement of backward(): gradients are defined with respect to a single number, so a per-example loss vector must be reduced before you can call backward on it. 'Differentiable' rules out the things you actually care about — accuracy, F1, ranking position — because they are step functions whose gradient is zero wherever it exists.

This is why the loss and the metric are two different objects. You minimise cross-entropy and report accuracy; you minimise a smooth surrogate and evaluate the discrete decision. Forgetting that leads to the recurring question 'why is my loss falling while accuracy is flat?' — the answer is that the loss is being paid to make correct predictions more confident, which moves it without crossing any decision threshold.

Mean squared error averages the squared residuals: mean((y_hat − y)²). Squaring makes every error positive and makes large errors dominate — a residual of 2 contributes sixteen times as much as a residual of 0.5. MAE (mean(|y_hat − y|)) treats each unit of error equally, and Huber/SmoothL1 behaves like MSE for small residuals and like MAE beyond a threshold, which is the usual compromise when outliers are real but shouldn't own the gradient.

The choice decides what the model optimises toward. MSE's optimum is the conditional mean, so it chases outliers; MAE's optimum is the conditional median, so it ignores them. On a dataset with a handful of mislabelled extreme targets, MSE will visibly distort the fit to reduce those few residuals, and the fix is the loss, not the model. MSE's gradient is also proportional to the residual, which is convenient: big errors push harder automatically.

nn.CrossEntropyLoss is log_softmax followed by negative log-likelihood, fused into one operation. It takes the model's raw logits of shape (batch, classes) and integer class indices of shape (batch,). The fusion exists for numerics: computing softmax and then taking its log means exponentiating potentially large numbers and then taking the log of potentially tiny ones, while log_softmax does it in one stable step by subtracting the max first.

Applying softmax yourself first is the most common silent bug in PyTorch classification code. Nothing raises: the probabilities are a valid (batch, classes) tensor, so the loss dutifully treats them as logits, exponentiates values already squashed into (0, 1), and produces a flattened objective whose gradients are much smaller than they should be. The model trains, slowly and worse, and the loss curve looks plausible. The corresponding pair on the binary side is BCEWithLogitsLoss (raw scores) versus BCELoss (probabilities) — use the first.

Binary cross-entropy scores each output independently: for one output and target t ∈ {0, 1}, it is −[t·log p + (1−t)·log(1−p)], and for K independent tags it is the mean of K such terms. Categorical cross-entropy assumes exactly one class is correct and scores only the true class's probability: −log p_true. In PyTorch the shapes differ accordingly — BCEWithLogitsLoss wants float targets shaped like the logits, CrossEntropyLoss wants int64 indices with no class dimension at all.

The choice encodes a factual claim about the labels, and the wrong claim caps performance. A multi-label problem run through softmax forces the tags to compete, so a document that genuinely belongs to three topics can never score high on all three. A single-label problem run through independent sigmoids discards the 'exactly one' constraint that would have sharpened the model, and permits outputs that say 'definitely a cat and definitely a dog'. Two classes are a special case that works either way: one sigmoid output, or a softmax over two logits.

Every PyTorch loss takes reduction: "mean" (the default) averages over the batch, "sum" adds, and "none" returns one value per example. The default is what keeps the loss — and therefore the gradient scale, and therefore the usable learning rate — comparable when batch size changes. reduction="none" is the hook for anything per-example: masking padded positions, weighting hard examples, or inspecting which rows a model finds hardest. CrossEntropyLoss(weight=...) scales each class's contribution, and ignore_index drops a label entirely.

reduction="sum" couples the learning rate to the batch size: double the batch and every gradient doubles, so a rate that was stable becomes divergent. Class weights are the standard deep-learning answer to imbalance — cheaper than resampling and it keeps every example in the epoch — but they change what the loss means, so the reported number is no longer comparable with an unweighted run.

Label smoothing replaces the hard target — probability 1 on the true class, 0 elsewhere — with a slightly softened one: 1 − ε on the true class and ε/(K−1) spread over the rest, for a small ε such as 0.1. In PyTorch it is one argument: nn.CrossEntropyLoss(label_smoothing=0.1). The model is then rewarded for being mostly right rather than infinitely confident.

Hard targets ask for a logit gap that grows without bound, because −log p keeps rewarding more confidence forever. That drives overconfident, badly calibrated models whose 0.99 predictions are right far less than 99% of the time, and it makes the model brittle on noisy labels. Smoothing caps the reward for confidence, which usually improves calibration and slightly improves accuracy on large classification tasks. It is a regulariser applied to the *loss*, not to the weights or the activations — which makes it a useful contrast with weight decay and dropout in module 11.

Work backwards from the decision. Name the decision the output feeds, the encoding of one target, and the cost of each kind of mistake; the loss follows almost mechanically. Scalar target → MSE, or SmoothL1 if outliers are real. Exactly one of K classes → cross-entropy on logits. Independent tags → BCE-with-logits. Skewed costs → class weights, or a threshold tuned after training rather than a different loss. Counts → Poisson; ordered categories → an ordinal formulation rather than either regression or plain classification.

The ML syllabus's problem-framing module makes the general argument: the loss is where your definition of a good answer gets written down, and a model will exploit any mismatch. What deep learning adds is that the mismatch is harder to notice, because a big network will happily achieve a low loss on the wrong objective and produce numbers that look like success. The deep-learning-specific discipline is to check the loss against the target dtype and shape before the first run, and to keep the reported metric separate from the optimised loss.