Transfer Learning

Transfer learning reuses pretrained weights instead of asking a small target dataset to discover every useful feature from random noise. In feature extraction, the backbone remains frozen and only a new classification or regression head learns; in fine-tuning, some or all backbone layers are unfrozen with a smaller learning rate to adapt representations to the target domain. Which layers to unfreeze depends on data volume and domain distance: generic early visual features often transfer widely, while later task-specific features may need more adaptation. The validation set decides whether additional unfreezing helps rather than merely overfits.

Pretraining supplies useful visual or linguistic features before your small dataset can. It is often faster, cheaper, and more accurate than training from random initialization, particularly when labels are scarce. It also changes the experiment plan: compare a frozen baseline first, then progressively unfreeze while measuring whether the added trainable capacity improves held-out performance enough to justify its cost.

Random initialization (module 10) is chosen for one property only: it keeps activations and gradients at a workable scale. It encodes nothing about the data. Pretrained weights start from a point where the network already computes edges, textures, shapes and object parts — features your 800-image dataset would have to rediscover from its 800 examples, using gradient signal that is both noisy and scarce. The target task then only needs to learn how to combine features that already exist, which is a far smaller learning problem than constructing them.

It explains the shape of the curves: transfer learning is not a small constant improvement but a different regime. On a few hundred labelled images, a frozen backbone plus a linear head routinely beats a from-scratch network by tens of accuracy points, and reaches its best score in a handful of epochs rather than hundreds. The smaller the dataset, the larger the gap — which is exactly the case most real projects are in.

A CNN's layers form a hierarchy of specificity. The first convolutions learn oriented edges, colour blobs and simple gradients — properties of photographs in general, and near-identical whatever the source dataset was. Middle layers compose those into textures and parts: fur, wheels, corners. The last layers encode combinations specific to the source label set, and the final classifier is entirely about it: a 1000-way ImageNet head's weights are meaningless for a 4-way X-ray task. So the transfer decision is a boundary on that hierarchy — keep everything below it, retrain everything above.

It converts 'which layers should I unfreeze' from guesswork into a principle: unfreeze from the top down, and go further down the more your target domain differs from the source. It also explains the one layer you must always replace — the classifier, whose output width is the source task's class count — and why replacing it is not optional even when the class counts happen to match.

A pretrained classifier ends in a layer whose output width is the source task's class count, and that layer must be swapped for one matching your task. The mechanics differ slightly per architecture family — model.fc on ResNets, model.classifier on VGG, DenseNet and EfficientNet, model.heads.head on torchvision's ViT — so read in_features off the existing layer rather than hard-coding it. The new layer is randomly initialized and trainable by default, which is what you want: it is the only part with no useful prior. Note the interaction with freezing: set requires_grad = False on the backbone *before* attaching the new head, or the loop will freeze the head too.

This is the one mandatory edit, and the one whose failure mode is a silent wrong answer rather than an error — a model that trains, converges, and predicts among the wrong set of classes. Reading in_features from the module also means the same code works across backbones, which matters because comparing two backbones is the first experiment worth running.

A fine-tuning learning rate is typically 10-100× smaller than a from-scratch one — 1e-4 or 1e-5 where you would have used 1e-3. The reason is that the backbone is already near a good solution, so the job is a small adjustment, and a step sized for a random initialization is large enough to leave that region entirely. Discriminative learning rates go further and give each layer group its own rate, rising toward the top: the new head gets the largest rate because it is genuinely random, the deepest backbone stage a smaller one, and the earliest layers the smallest or zero. Warmup (module 8) matters here too, because the first few steps are driven by the untrained head's large gradients.

It is the difference between adaptation and destruction, and it is invisible in the first few steps — the loss falls either way, because a fresh head learning on top of scrambled features still beats chance. The damage shows up as a validation curve that plateaus below the frozen-backbone baseline, which is a confusing result unless you know to suspect the learning rate.

Fine-tuning too aggressively — too large a learning rate, too many epochs on too little data, or every layer unfrozen at once — overwrites the pretrained representations with ones fitted to your small target set. The model has not simply overfitted in the usual sense; it has discarded the general features that were the reason to start from this checkpoint. The symptom is distinctive: a fine-tuned model that scores *below* the frozen-backbone baseline on validation, despite having strictly more trainable capacity. The remedies are all forms of restraint — smaller rates, fewer unfrozen layers, early stopping, and stronger weight decay anchoring the weights near where they started.

It makes the frozen baseline non-optional. Without a frozen number to compare against, a fine-tuned run that quietly destroyed its backbone looks like a normal mediocre result, and the natural next move — train longer, unfreeze more — makes it worse. The baseline is what turns the failure into a visible, diagnosable regression.

The benefit depends on the source and target domains sharing low-level structure, and there are real cases where they do not. ImageNet features transfer weakly to grayscale medical scans with different statistics and textures, to satellite imagery at an unfamiliar scale, to audio spectrograms that are only nominally images, and to non-photographic data such as scientific plots. It also helps less when the target dataset is large — with a million labelled examples, from-scratch training catches up, and the initialization matters mostly for how fast you get there. And a mismatch in input *shape* or channel count needs explicit handling, not just a new head.

Naming the limits keeps transfer learning as a technique rather than a reflex. The decision has a cheap empirical test — train the frozen head, compare it against a small from-scratch model — and on a distant domain the answer is sometimes 'unfreeze everything and treat the checkpoint as a merely better-than-random initialization', which is a legitimate use and a different plan.

Because the choices interact, the useful order is fixed: (1) train a linear head on a frozen backbone — cheap, fast, and the baseline everything else is measured against; (2) unfreeze the top stage at a small learning rate; (3) unfreeze progressively further down, one stage at a time; (4) only then consider a larger backbone or a different pretraining source. Each step is scored on the same validation split, and each step's cost in trainable parameters is recorded beside its score. Augmentation (module 11) is held fixed across the comparison, because it interacts strongly with how much capacity is unfrozen.

It is the cheapest-first ordering, and it produces a defensible answer to 'why is the model configured this way'. It also protects against the two failure modes this module described: without step 1 there is no baseline to detect forgetting, and without one-stage-at-a-time changes a regression cannot be attributed to the unfreezing rather than to the learning rate that came with it.