Recurrent Neural Networks

Every network so far took a fixed-size input and processed it all at once. But much of the world's data arrives as a sequence — sentences, sensor readings, daily temperatures, audio — where the order carries the meaning. In this lesson you'll build the recurrence that lets a network remember, run a tiny RNN cell by hand in NumPy, meet the vanishing-gradient problem that haunts it, and train nn.RNN on a next-step prediction task.

Order is information

Compare "the movie was good, not bad" with "the movie was bad, not good". Same words, opposite meanings — only the order differs. The same goes for time series: a temperature of 15°C means something different in a falling trend than in a rising one. Context lives in the sequence.

A plain feedforward network struggles here for three reasons:

• Fixed input size. An MLP expects exactly n inputs. Sentences and series come in every length.
• No memory. Feed it one time step at a time and each prediction starts from scratch — step 10 knows nothing about steps 1–9.
• No parameter sharing across time. Concatenate a window of steps into one big input and the network must relearn the same pattern separately at every position.

The fix: give the network a hidden state that persists between steps — a working memory.

The recurrence

An RNN processes a sequence one element at a time. At step t it combines the current input x_t with its own previous hidden state h_(t-1):

h_t = tanh(Wx·x_t + Wh·h_(t-1) + b)

That's it — a linear combination of "what I see now" and "what I remember", squashed by a tanh. The same weights Wx and Wh are used at every step, so the network can handle any sequence length with a fixed number of parameters. Run the cell yourself:

Two things to notice. The hidden state changes at every step — it accumulates a summary of everything seen so far. And the two sequences contain identical values yet end in different final states: unlike an averaging model, the RNN is genuinely order-sensitive.

Unrolling through time

The loop above can be drawn as a chain: copy the cell once per time step and pass the hidden state along. Unrolled, an RNN over a 100-step sequence looks like a 100-layer feedforward network — except every "layer" shares the same weights.

Training uses backpropagation through time (BPTT): run the forward pass over the whole sequence, compute the loss, and backpropagate through the unrolled chain, summing each weight's gradient contributions across all time steps. For long sequences this gets expensive (the graph for a 10,000-step series is enormous), so in practice we use truncated BPTT: chop the sequence into chunks of, say, 50 steps, carry the hidden state forward from chunk to chunk, but detach it between chunks (hidden.detach() in PyTorch) so gradients only flow within a chunk. It's a biased approximation — the model can't learn dependencies longer than the truncation window through gradients — but it keeps memory and compute bounded, and it's what makes training on long sequences feasible at all.

The trouble with deep time

An unrolled RNN is a very deep network, and gradients flowing back through it get multiplied by (roughly) the same recurrent weights at every step. What happens when you multiply by the same number many times?

• If the effective factor is below 1, gradients vanish — by step 100 they are numerically zero, so the network cannot learn long-range dependencies. What happened 80 steps ago simply never reaches the weights.
• If it's above 1, gradients explode — the loss becomes NaN and training blows up.

Exploding gradients have a blunt but effective fix, gradient clipping (next lesson). Vanishing gradients are the deep problem: they're why vanilla RNNs in practice remember only 10–20 steps, and why the LSTM was invented. Hold that thought.

nn.RNN in PyTorch

PyTorch packages the recurrence (with all the batching and multi-layer machinery) as nn.RNN. With batch_first=True, it expects input of shape (batch, seq_len, input_size) and returns two things:

• output — shape (batch, seq_len, hidden_size): the hidden state at every time step of the top layer,
• h_n — shape (num_layers, batch, hidden_size): the final hidden state of each layer.

For "predict the next value" we take the last time step of output and map it through a linear head. Here's the full task — predicting the next point of a sine wave — to run in Colab:

import torch
from torch import nn, optim
 
torch.manual_seed(0)
 
# --- data: a sine wave, windowed into (input sequence, next value) pairs ---
t = torch.linspace(0, 60, 600)
series = torch.sin(t)
 
seq_len = 20
X = torch.stack([series[i:i + seq_len] for i in range(len(series) - seq_len)])
y = series[seq_len:]
X = X.unsqueeze(-1)          # (580, 20, 1)  — batch, seq_len, input_size
y = y.unsqueeze(-1)          # (580, 1)
 
# --- model ---
class NextStepRNN(nn.Module):
    def __init__(self, hidden_size=32):
        super().__init__()
        self.rnn = nn.RNN(input_size=1, hidden_size=hidden_size, num_layers=1,
                          batch_first=True)
        self.fc = nn.Linear(hidden_size, 1)
 
    def forward(self, x):
        out, h_n = self.rnn(x)        # out: (batch, seq_len, hidden)
        return self.fc(out[:, -1, :]) # last time step -> prediction
 
model = NextStepRNN()
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3)
 
for epoch in range(300):
    pred = model(X)
    loss = criterion(pred, y)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    if epoch % 50 == 0:
        print(f"epoch {epoch:3d}: MSE = {loss.item():.5f}")
 
with torch.inference_mode():
    print("next value after the last window:", model(X[-1:]).item())

The loss should fall to nearly zero — a sine wave is about the friendliest sequence there is. The point isn't the task; it's the plumbing: window the series, get the shapes right, take the last step's hidden state, regress.

Sequence shapes cheat sheet

Shape bugs are the number-one RNN frustration. With batch_first=True:

Without batch_first=True the default is (seq_len, batch, input_size) — a classic source of silently wrong results, since a transposed tensor often still runs.

Next: the vanishing gradient gets its cure — gated memory cells. LSTM and GRU give the network a conveyor belt for long-term memory.