Sequence Models in Practice

You can build an LSTM — now let's make it earn its keep on a realistic forecasting workflow. This lesson covers the architecture options you haven't met yet (bidirectional and stacked RNNs), the data plumbing that makes or breaks sequence models (windowing, scaling, multivariate inputs), a complete training pipeline with early stopping, and honest guidance on tuning — plus a warning about stock prices.

Bidirectional RNNs

A standard RNN reads left to right, so its state at step t only knows the past. A bidirectional RNN runs a second, independent RNN right to left and concatenates the two hidden states at each step — every position then sees both past and future context.

self.rnn = nn.LSTM(input_size, hidden_size, num_layers,
                   batch_first=True, bidirectional=True)
self.fc = nn.Linear(2 * hidden_size, output_size)   # note the 2x!

The output feature dimension doubles to 2 * hidden_size (forward and backward states concatenated), so the head must widen to match — forgetting the 2 * is the classic bidirectional bug.

When is it appropriate? Only when the whole sequence is available before you predict:

• Yes: part-of-speech tagging, named-entity recognition, classifying a complete sentence or a recorded audio clip.
• No: forecasting. To predict tomorrow you'd need the backward RNN to read the future — which is exactly what you don't have at prediction time. A bidirectional forecaster scores brilliantly in offline evaluation and is useless (or subtly leaky) in deployment.

Stacked RNNs

num_layers=2 stacks a second recurrent layer on top of the first: layer 1's hidden-state sequence becomes layer 2's input sequence. Like extra layers in an MLP, this buys hierarchical features — at the price of more parameters and slower training. Two layers is a sweet spot; beyond three rarely pays off for typical forecasting problems. Note that the dropout argument of nn.LSTM/nn.GRU applies between stacked layers, so it does nothing with num_layers=1.

Windowing: from series to samples

Recurrent layers want input of shape (samples, seq_len, features) — but a time series arrives as one long array. Sliding windows convert one into the other: each sample is a window of seq_len consecutive steps, and its target is the value right after the window. Build it yourself:

A 20-step series with seq_len=5 yields 15 samples of shape (5, 2) — that third dimension is features, which is exactly input_size for the RNN. One long recording becomes a proper supervised dataset.

Scaling — and the golden rule

RNNs are sensitive to input scale: tanh and sigmoid saturate quickly, so a series living around 3,000 (a stock index) or spanning 0–40 (temperatures) trains far better after standardization. Two rules:

1. Fit the scaler on the training split only, then apply it to both splits. Fitting on the full series leaks the test set's mean and variance into training — a subtle form of looking at the future.
2. Scale per feature. StandardScaler already works column-wise, so a multivariate series gets one mean/std per feature. Keep the scaler around: you'll need inverse_transform to report predictions in real units.

Multivariate inputs are now free: stack extra columns — other measured series, or calendar features like one-hot quarter or day-of-week — alongside the target, window everything together, and set input_size to the number of features. The model still predicts one target; it just gets more context per time step.

The full pipeline

Everything assembled on a damped sine wave — a series whose amplitude decays over time, so the model must genuinely track where it is in the decay rather than repeat one fixed cycle. Run in Colab:

import numpy as np
import matplotlib.pyplot as plt
import torch
from torch import nn, optim
from sklearn.preprocessing import StandardScaler
 
torch.manual_seed(0)
rng = np.random.default_rng(0)
 
# --- 1. the series: damped sine + noise ---
n = 800
t = np.arange(n)
series = np.exp(-t / 400) * np.sin(2 * np.pi * t / 50) + rng.normal(0, 0.03, n)
 
# --- 2. chronological split, then scale (fit on train ONLY) ---
split = int(n * 0.8)
train_raw, test_raw = series[:split], series[split:]
scaler = StandardScaler().fit(train_raw.reshape(-1, 1))
train_s = scaler.transform(train_raw.reshape(-1, 1))
test_s = scaler.transform(test_raw.reshape(-1, 1))
 
# --- 3. windowing ---
def make_windows(data, seq_len):
    X = np.stack([data[i:i + seq_len] for i in range(len(data) - seq_len)])
    y = data[seq_len:]
    return torch.tensor(X, dtype=torch.float32), torch.tensor(y, dtype=torch.float32)
 
seq_len = 30
X_train, y_train = make_windows(train_s, seq_len)   # (samples, 30, 1)
X_test, y_test = make_windows(test_s, seq_len)
 
# hold out the last 15% of training windows for validation (chronological!)
val_from = int(len(X_train) * 0.85)
X_val, y_val = X_train[val_from:], y_train[val_from:]
X_train, y_train = X_train[:val_from], y_train[:val_from]
 
# --- 4. model ---
class Forecaster(nn.Module):
    def __init__(self, input_size=1, hidden_size=64, num_layers=2, dropout=0.2):
        super().__init__()
        self.rnn = nn.LSTM(input_size, hidden_size, num_layers,
                           dropout=dropout, batch_first=True)
        self.fc = nn.Linear(hidden_size, 1)
 
    def forward(self, x):
        out, _ = self.rnn(x)
        return self.fc(out[:, -1, :])
 
model = Forecaster()
criterion = nn.MSELoss()
optimizer = optim.AdamW(model.parameters(), lr=1e-3)
 
# --- 5. training with early stopping ---
best_val, patience, bad = float("inf"), 20, 0
for epoch in range(500):
    model.train()
    loss = criterion(model(X_train), y_train)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step()
    optimizer.zero_grad()
 
    model.eval()
    with torch.inference_mode():
        val_loss = criterion(model(X_val), y_val).item()
    if val_loss < best_val:
        best_val, bad = val_loss, 0
        torch.save(model.state_dict(), "forecaster.pth")
    else:
        bad += 1
        if bad >= patience:
            print(f"early stop at epoch {epoch}, best val MSE {best_val:.5f}")
            break
 
# --- 6. evaluate on test, back in original units ---
model.load_state_dict(torch.load("forecaster.pth"))
model.eval()
with torch.inference_mode():
    preds_s = model(X_test).numpy()
 
preds = scaler.inverse_transform(preds_s).ravel()
truth = scaler.inverse_transform(y_test.numpy()).ravel()
 
mse_model = np.mean((preds - truth) ** 2)
naive = truth[:-1]                       # baseline: predict yesterday's value
mse_naive = np.mean((naive - truth[1:]) ** 2)
print(f"LSTM MSE:  {mse_model:.5f}")
print(f"naive MSE: {mse_naive:.5f}")
 
plt.figure(figsize=(12, 4))
plt.plot(truth, label="actual")
plt.plot(preds, label="LSTM prediction")
plt.legend()
plt.title("One-step-ahead forecast (test set)")
plt.show()

Note the last few lines: every forecast should be benchmarked against the naive baseline — "tomorrow equals today". On the damped sine the LSTM beats it comfortably. Keep that baseline handy; it's about to matter.

Tuning guidance

When results disappoint, turn these knobs — one at a time:

A word about stock prices

The bootcamp classic: point the pipeline at a stock index and admire a prediction curve hugging the actual prices. Don't be fooled. Daily prices are close to a random walk — the best statistical predictor of tomorrow is approximately today. A trained RNN discovers this too, and learns to output (nearly) the last value of its window. The plot looks fantastic because the prediction is the price shifted one day; the MSE barely beats the naive baseline, and any trading edge is illusory. As a learning exercise — plumbing, scaling, evaluation discipline — stock data is fine. As a money-maker, it is a lesson in why baselines exist.

Next module: sequences of words. We turn text into vectors with word embeddings — the foundation of every NLP model you'll build.