{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-0000",
   "metadata": {},
   "source": [
    "# Sequence Models in Practice\n",
    "\n",
    "Bidirectional and stacked RNNs, windowing and scaling time series properly, multivariate inputs, and a full LSTM forecasting pipeline with early stopping.\n",
    "\n",
    "*Part of the free [Deep Learning with PyTorch](https://ramadnsyh.dev/courses/deep-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/deep-learning/sequence-models-practice).*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0001",
   "metadata": {},
   "source": [
    "You can build an LSTM — now let's make it earn its keep on a realistic\n",
    "forecasting workflow. This lesson covers the architecture options you haven't\n",
    "met yet (bidirectional and stacked RNNs), the data plumbing that makes or\n",
    "breaks sequence models (windowing, scaling, multivariate inputs), a complete\n",
    "training pipeline with early stopping, and honest guidance on tuning — plus a\n",
    "warning about stock prices."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0002",
   "metadata": {},
   "source": [
    "## Bidirectional RNNs\n",
    "\n",
    "A standard RNN reads left to right, so its state at step `t` only knows the\n",
    "past. A **bidirectional** RNN runs a second, independent RNN right to left and\n",
    "concatenates the two hidden states at each step — every position then sees\n",
    "both past *and* future context."
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0003",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "self.rnn = nn.LSTM(input_size, hidden_size, num_layers,\n",
    "                   batch_first=True, bidirectional=True)\n",
    "self.fc = nn.Linear(2 * hidden_size, output_size)   # note the 2x!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0004",
   "metadata": {},
   "source": [
    "The output feature dimension doubles to `2 * hidden_size` (forward and\n",
    "backward states concatenated), so the head must widen to match — forgetting\n",
    "the `2 *` is the classic bidirectional bug.\n",
    "\n",
    "When is it appropriate? Only when the **whole sequence is available before\n",
    "you predict**:\n",
    "\n",
    "- **Yes:** part-of-speech tagging, named-entity recognition, classifying a\n",
    "  complete sentence or a recorded audio clip.\n",
    "- **No: forecasting.** To predict tomorrow you'd need the backward RNN to\n",
    "  read the future — which is exactly what you don't have at prediction time.\n",
    "  A bidirectional forecaster scores brilliantly in offline evaluation and is\n",
    "  useless (or subtly leaky) in deployment."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0005",
   "metadata": {},
   "source": [
    "## Stacked RNNs\n",
    "\n",
    "`num_layers=2` stacks a second recurrent layer on top of the first: layer 1's\n",
    "hidden-state sequence becomes layer 2's input sequence. Like extra layers in\n",
    "an MLP, this buys hierarchical features — at the price of more parameters and\n",
    "slower training. Two layers is a sweet spot; beyond three rarely pays off for\n",
    "typical forecasting problems. Note that the `dropout` argument of\n",
    "`nn.LSTM`/`nn.GRU` applies *between* stacked layers, so it does nothing with\n",
    "`num_layers=1`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0006",
   "metadata": {},
   "source": [
    "## Windowing: from series to samples\n",
    "\n",
    "Recurrent layers want input of shape `(samples, seq_len, features)` — but a\n",
    "time series arrives as one long array. **Sliding windows** convert one into\n",
    "the other: each sample is a window of `seq_len` consecutive steps, and its\n",
    "target is the value right after the window. Build it yourself:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0007",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def make_windows(data, seq_len, target_col=0):\n",
    "    X, y = [], []\n",
    "    for i in range(len(data) - seq_len):\n",
    "        X.append(data[i : i + seq_len])       # seq_len consecutive rows\n",
    "        y.append(data[i + seq_len, target_col])  # the value right after\n",
    "    return np.array(X), np.array(y)\n",
    "\n",
    "# toy multivariate series: 20 time steps, 2 features per step\n",
    "steps = np.arange(20, dtype=float)\n",
    "data = np.stack([steps, steps * 10], axis=1)   # shape (20, 2)\n",
    "\n",
    "X, y = make_windows(data, seq_len=5)\n",
    "print(f\"data: {data.shape}  ->  X: {X.shape}, y: {y.shape}\")\n",
    "print()\n",
    "print(\"first sample (rows 0-4, both features):\")\n",
    "print(X[0])\n",
    "print(f\"its target (feature 0 at row 5): {y[0]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0008",
   "metadata": {},
   "source": [
    "A 20-step series with `seq_len=5` yields 15 samples of shape `(5, 2)` — that\n",
    "third dimension is `features`, which is exactly `input_size` for the RNN. One\n",
    "long recording becomes a proper supervised dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0009",
   "metadata": {},
   "source": [
    "## Scaling — and the golden rule\n",
    "\n",
    "RNNs are sensitive to input scale: tanh and sigmoid saturate quickly, so a\n",
    "series living around 3,000 (a stock index) or spanning 0–40 (temperatures)\n",
    "trains far better after standardization. Two rules:\n",
    "\n",
    "1. **Fit the scaler on the training split only**, then apply it to both\n",
    "   splits. Fitting on the full series leaks the test set's mean and variance\n",
    "   into training — a subtle form of looking at the future.\n",
    "2. **Scale per feature.** `StandardScaler` already works column-wise, so a\n",
    "   multivariate series gets one mean/std per feature. Keep the scaler around:\n",
    "   you'll need `inverse_transform` to report predictions in real units.\n",
    "\n",
    "**Multivariate inputs** are now free: stack extra columns — other measured\n",
    "series, or calendar features like one-hot quarter or day-of-week — alongside\n",
    "the target, window everything together, and set `input_size` to the number of\n",
    "features. The model still predicts one target; it just gets more context per\n",
    "time step."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0010",
   "metadata": {},
   "source": [
    "## The full pipeline\n",
    "\n",
    "Everything assembled on a damped sine wave — a series whose amplitude decays\n",
    "over time, so the model must genuinely track where it is in the decay rather\n",
    "than repeat one fixed cycle. Run in Colab:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0011",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "from torch import nn, optim\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "torch.manual_seed(0)\n",
    "rng = np.random.default_rng(0)\n",
    "\n",
    "# --- 1. the series: damped sine + noise ---\n",
    "n = 800\n",
    "t = np.arange(n)\n",
    "series = np.exp(-t / 400) * np.sin(2 * np.pi * t / 50) + rng.normal(0, 0.03, n)\n",
    "\n",
    "# --- 2. chronological split, then scale (fit on train ONLY) ---\n",
    "split = int(n * 0.8)\n",
    "train_raw, test_raw = series[:split], series[split:]\n",
    "scaler = StandardScaler().fit(train_raw.reshape(-1, 1))\n",
    "train_s = scaler.transform(train_raw.reshape(-1, 1))\n",
    "test_s = scaler.transform(test_raw.reshape(-1, 1))\n",
    "\n",
    "# --- 3. windowing ---\n",
    "def make_windows(data, seq_len):\n",
    "    X = np.stack([data[i:i + seq_len] for i in range(len(data) - seq_len)])\n",
    "    y = data[seq_len:]\n",
    "    return torch.tensor(X, dtype=torch.float32), torch.tensor(y, dtype=torch.float32)\n",
    "\n",
    "seq_len = 30\n",
    "X_train, y_train = make_windows(train_s, seq_len)   # (samples, 30, 1)\n",
    "X_test, y_test = make_windows(test_s, seq_len)\n",
    "\n",
    "# hold out the last 15% of training windows for validation (chronological!)\n",
    "val_from = int(len(X_train) * 0.85)\n",
    "X_val, y_val = X_train[val_from:], y_train[val_from:]\n",
    "X_train, y_train = X_train[:val_from], y_train[:val_from]\n",
    "\n",
    "# --- 4. model ---\n",
    "class Forecaster(nn.Module):\n",
    "    def __init__(self, input_size=1, hidden_size=64, num_layers=2, dropout=0.2):\n",
    "        super().__init__()\n",
    "        self.rnn = nn.LSTM(input_size, hidden_size, num_layers,\n",
    "                           dropout=dropout, batch_first=True)\n",
    "        self.fc = nn.Linear(hidden_size, 1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        out, _ = self.rnn(x)\n",
    "        return self.fc(out[:, -1, :])\n",
    "\n",
    "model = Forecaster()\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = optim.AdamW(model.parameters(), lr=1e-3)\n",
    "\n",
    "# --- 5. training with early stopping ---\n",
    "best_val, patience, bad = float(\"inf\"), 20, 0\n",
    "for epoch in range(500):\n",
    "    model.train()\n",
    "    loss = criterion(model(X_train), y_train)\n",
    "    loss.backward()\n",
    "    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n",
    "    optimizer.step()\n",
    "    optimizer.zero_grad()\n",
    "\n",
    "    model.eval()\n",
    "    with torch.inference_mode():\n",
    "        val_loss = criterion(model(X_val), y_val).item()\n",
    "    if val_loss < best_val:\n",
    "        best_val, bad = val_loss, 0\n",
    "        torch.save(model.state_dict(), \"forecaster.pth\")\n",
    "    else:\n",
    "        bad += 1\n",
    "        if bad >= patience:\n",
    "            print(f\"early stop at epoch {epoch}, best val MSE {best_val:.5f}\")\n",
    "            break\n",
    "\n",
    "# --- 6. evaluate on test, back in original units ---\n",
    "model.load_state_dict(torch.load(\"forecaster.pth\"))\n",
    "model.eval()\n",
    "with torch.inference_mode():\n",
    "    preds_s = model(X_test).numpy()\n",
    "\n",
    "preds = scaler.inverse_transform(preds_s).ravel()\n",
    "truth = scaler.inverse_transform(y_test.numpy()).ravel()\n",
    "\n",
    "mse_model = np.mean((preds - truth) ** 2)\n",
    "naive = truth[:-1]                       # baseline: predict yesterday's value\n",
    "mse_naive = np.mean((naive - truth[1:]) ** 2)\n",
    "print(f\"LSTM MSE:  {mse_model:.5f}\")\n",
    "print(f\"naive MSE: {mse_naive:.5f}\")\n",
    "\n",
    "plt.figure(figsize=(12, 4))\n",
    "plt.plot(truth, label=\"actual\")\n",
    "plt.plot(preds, label=\"LSTM prediction\")\n",
    "plt.legend()\n",
    "plt.title(\"One-step-ahead forecast (test set)\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0012",
   "metadata": {},
   "source": [
    "Note the last few lines: every forecast should be benchmarked against the\n",
    "**naive baseline** — \"tomorrow equals today\". On the damped sine the LSTM\n",
    "beats it comfortably. Keep that baseline handy; it's about to matter."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0013",
   "metadata": {},
   "source": [
    "## Tuning guidance\n",
    "\n",
    "When results disappoint, turn these knobs — one at a time:\n",
    "\n",
    "| Knob | Guidance |\n",
    "|---|---|\n",
    "| `hidden_size` | 32–256. Bigger = more capacity but slower and quicker to overfit; watch the train/val gap. |\n",
    "| `seq_len` | Must cover the pattern you want captured — at least one seasonal period (a 50-step cycle needs `seq_len` ≥ 50-ish). Longer windows cost compute and can dilute recent signal. |\n",
    "| learning rate | Start at `1e-3` with Adam/AdamW; if the loss oscillates or spikes, drop to `5e-4` or `1e-4`. |\n",
    "| `num_layers` / `dropout` | 2 layers with dropout 0.2 is a solid default; dropout only acts between stacked layers. |\n",
    "| `bidirectional` | Only when the full sequence exists at prediction time — never for forecasting. |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0014",
   "metadata": {},
   "source": [
    "## A word about stock prices\n",
    "\n",
    "The bootcamp classic: point the pipeline at a stock index and admire a\n",
    "prediction curve hugging the actual prices. Don't be fooled. Daily prices are\n",
    "close to a **random walk** — the best statistical predictor of tomorrow is\n",
    "approximately today. A trained RNN discovers this too, and learns to output\n",
    "(nearly) the last value of its window. The plot looks fantastic because the\n",
    "prediction is the price shifted one day; the MSE barely beats the naive\n",
    "baseline, and any trading edge is illusory. As a *learning exercise* —\n",
    "plumbing, scaling, evaluation discipline — stock data is fine. As a\n",
    "money-maker, it is a lesson in why baselines exist."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0015",
   "metadata": {},
   "source": [
    "### 🏋️ Exercise — Go multivariate\n",
    "\n",
    "Extend the damped-sine pipeline to **three input features**: the noisy series\n",
    "itself plus two phase features, `sin(2πt/50)` and `cos(2πt/50)`, that tell the\n",
    "model where it is in the cycle. Stack them into a `(n, 3)` array, scale, and\n",
    "window with the noisy series as the target. Change `input_size` to 3, retrain,\n",
    "and compare test MSE against the univariate version — do the phase features\n",
    "help?"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0016",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0017",
   "metadata": {},
   "source": [
    "<details><summary>✅ Show solution</summary>\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "import torch\n",
    "from torch import nn, optim\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "torch.manual_seed(0)\n",
    "rng = np.random.default_rng(0)\n",
    "\n",
    "n = 800\n",
    "t = np.arange(n)\n",
    "target = np.exp(-t / 400) * np.sin(2 * np.pi * t / 50) + rng.normal(0, 0.03, n)\n",
    "phase_sin = np.sin(2 * np.pi * t / 50)\n",
    "phase_cos = np.cos(2 * np.pi * t / 50)\n",
    "data = np.stack([target, phase_sin, phase_cos], axis=1)   # (800, 3)\n",
    "\n",
    "split = int(n * 0.8)\n",
    "scaler = StandardScaler().fit(data[:split])\n",
    "train_s, test_s = scaler.transform(data[:split]), scaler.transform(data[split:])\n",
    "\n",
    "def make_windows(d, seq_len, target_col=0):\n",
    "    X = np.stack([d[i:i + seq_len] for i in range(len(d) - seq_len)])\n",
    "    y = d[seq_len:, target_col:target_col + 1]\n",
    "    return (torch.tensor(X, dtype=torch.float32),\n",
    "            torch.tensor(y, dtype=torch.float32))\n",
    "\n",
    "seq_len = 30\n",
    "X_train, y_train = make_windows(train_s, seq_len)\n",
    "X_test, y_test = make_windows(test_s, seq_len)\n",
    "print(\"X_train:\", X_train.shape)   # (samples, 30, 3)\n",
    "\n",
    "class Forecaster(nn.Module):\n",
    "    def __init__(self, input_size=3, hidden_size=64):\n",
    "        super().__init__()\n",
    "        self.rnn = nn.LSTM(input_size, hidden_size, 2, dropout=0.2,\n",
    "                           batch_first=True)\n",
    "        self.fc = nn.Linear(hidden_size, 1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        out, _ = self.rnn(x)\n",
    "        return self.fc(out[:, -1, :])\n",
    "\n",
    "model = Forecaster()\n",
    "criterion = nn.MSELoss()\n",
    "optimizer = optim.AdamW(model.parameters(), lr=1e-3)\n",
    "for epoch in range(300):\n",
    "    loss = criterion(model(X_train), y_train)\n",
    "    loss.backward()\n",
    "    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n",
    "    optimizer.step()\n",
    "    optimizer.zero_grad()\n",
    "\n",
    "model.eval()\n",
    "with torch.inference_mode():\n",
    "    test_mse = criterion(model(X_test), y_test).item()\n",
    "print(f\"multivariate test MSE (scaled units): {test_mse:.5f}\")\n",
    "```\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0018",
   "metadata": {},
   "source": [
    "Next module: sequences of *words*. We turn text into vectors with word\n",
    "embeddings — the foundation of every NLP model you'll build."
   ]
  }
 ]
}