RNNs, LSTMs & GRUs

An RNN processes a sequence one step at a time and passes a hidden state forward, so a token's representation can depend on preceding tokens. A plain RNN repeatedly applies the same transition, which makes gradients through many time steps prone to vanish or explode. LSTMs and GRUs add gates that decide what to retain, write, and forget; their nearly additive memory paths make long-range information easier, though not effortless, to preserve. For batched variable-length sequences, padding, masks, and hidden-state resets are part of the correctness contract.

They explain sequence modelling before attention and remain useful for small streaming or time-series tasks with strict latency budgets. They maintain only a compact state rather than an all-pairs attention matrix, but their sequential computation prevents the broad parallelism that made Transformers dominant for many offline language tasks.

A vanilla RNN holds a single hidden vector h and updates it once per timestep: h_t = tanh(W_x x_t + W_h h_{t−1} + b). The same W_x, W_h and b are applied at step 1 and at step 500 — there is no per-position weight anywhere in the model. Two consequences follow. The parameter count is independent of sequence length, so the same trained model runs on a 10-token or a 10,000-token input. And h_t is the model's entire memory of everything before step t, compressed into a fixed-size vector, which is both the source of its efficiency and the source of its bottleneck.

Parameter sharing across time is the sequence analogue of a CNN's sharing across space, and it makes the same two promises: far fewer parameters, and a pattern learned at one position that is recognisable at every other. Writing the loop by hand once is the fastest way to see that nn.RNN is not a new kind of object — it is this three-line update, fused and run in a fast kernel.

Backpropagation through time unrolls the network into one layer per timestep and applies module 7's chain rule along it. A 200-step sequence is therefore a 200-layer-deep computation whose every layer shares the same W_h — so the gradient reaching step 1 is multiplied by roughly the same factor 200 times. Module 10's arithmetic applies exactly: if that factor is slightly below 1, the gradient vanishes and the model cannot learn any dependency spanning more than a few dozen steps; slightly above 1, and it explodes, which is why gradient clipping is close to mandatory for recurrent models rather than merely prudent.

It is the concrete reason a plain RNN cannot learn 'the verb agrees with a subject 80 words earlier'. The model is not short of capacity — it is receiving no usable gradient for that dependency. And because the repeated factor is one shared matrix rather than 200 different ones, the failure is much sharper than in a feedforward stack of the same depth: there is no averaging across layers to soften it.

An LSTM carries two vectors: a hidden state h and a cell state c. Three gates, each a small learned layer with a sigmoid output between 0 and 1, control the cell state elementwise. The forget gate multiplies the old cell state, deciding what to discard. The input gate scales the candidate update, deciding what to write. The output gate decides how much of the cell state is exposed as h_t for the rest of the network to read. The crucial structural detail is the cell update c_t = f_t ⊙ c_{t−1} + i_t ⊙ g_t: information moves along it by *addition*, so when the forget gate is near 1 the cell state passes through nearly unchanged — and so does its gradient, exactly as with module 10's residual connections.

The gates are what make long-range dependencies learnable, and the mechanism is the same 'additions do not compound' insight that residual connections rely on. It is also why LSTMs are described by what each gate is *for* rather than by their equations: the equations are four parallel linear layers and some elementwise multiplications, and the design content is entirely in which quantity each gate multiplies.

A GRU merges the LSTM's forget and input gates into one update gate z_t, drops the separate cell state, and adds a reset gate controlling how much of the previous hidden state enters the candidate. The result is two gates rather than three, one state vector rather than two, and roughly three quarters of an LSTM's parameters for the same hidden size. The additive path survives — h_t = (1 − z_t) ⊙ h_{t−1} + z_t ⊙ h̃_t is an interpolation, so z_t ≈ 0 passes the old state through untouched — which is the property that matters.

It is the practical default when a recurrent model is the right choice: comparable accuracy on most tasks, faster per step, fewer parameters, and less to tune. The honest summary is that the LSTM-versus-GRU question rarely decides a project, which is itself worth knowing — it is a cheap A/B rather than an architecture debate, and the time is better spent on the data or on whether the model should be recurrent at all.

A bidirectional RNN runs two independent recurrences over the same sequence — one left to right, one right to left — and concatenates their hidden states at each position, so each output vector has seen the whole sequence on both sides. It doubles the parameters and the compute, and it typically helps substantially on tasks where the full input is available up front: classification, tagging, named-entity recognition, encoding a source sentence for translation. nn.LSTM(..., bidirectional=True) gives a hidden size of 2 × hidden per position.

The important limit is what it cannot do. The backward pass over the sequence requires the future tokens, so a bidirectional model cannot be used for autoregressive generation or for any online, streaming prediction — at inference you simply do not have the right-hand context yet. This is the same information boundary that makes causal masking necessary in module 17, arriving here first and for the same reason.

Real batches contain sequences of different lengths, so they are padded to a common length with a dummy token. Nothing about the recurrence knows those positions are fake: the RNN happily consumes them and the final hidden state ends up summarising a run of padding rather than the sentence. Two fixes exist. pack_padded_sequence tells cuDNN the true lengths so each sequence's recurrence stops where it should, and the returned h_n is each sequence's real final state. Or keep the padding and mask explicitly — exclude padded positions from the loss with ignore_index, and never pool over them.

This is the most common silent correctness bug in sequence code. Padding contributes to the loss, to any mean-pooled representation, and to the final state, so the model is trained partly to predict a token that does not exist and partly on a representation of whitespace. The metrics look plausible, and results improve noticeably when it is fixed — which is the signature of a bug rather than a tuning opportunity.

Two arguments, and the second is the decisive one. First, path length: relating token 1 to token 200 in an RNN requires information to survive 199 state updates, while self-attention (module 17) connects them in a single operation regardless of distance. Second, parallelism: h_t depends on h_{t−1}, so a 500-token sequence requires 500 sequential steps that no amount of hardware can overlap. A Transformer computes all positions' representations in one batched matrix multiplication, so training time scales with available compute rather than with sequence length — and that is what made training on internet-scale corpora feasible.

It explains a historical shift as an engineering property rather than a fashion. It also explains where recurrence still wins: its per-step cost is constant and its state is fixed-size, so for streaming inference, long-running time series, or tight latency and memory budgets on a device, an RNN remains a reasonable and sometimes better choice. Attention's advantage is in *training* throughput, and it pays for it with cost quadratic in sequence length.