{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-0000",
   "metadata": {},
   "source": [
    "# PyTorch & Tensors\n",
    "\n",
    "Create and reshape tensors, bridge to NumPy, move to the GPU, and let autograd compute derivatives for you.\n",
    "\n",
    "*Part of the free [Deep Learning with PyTorch](https://ramadnsyh.dev/courses/deep-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/deep-learning/pytorch-tensors).*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0001",
   "metadata": {},
   "source": [
    "PyTorch is the workhorse of modern deep learning research and much of industry.\n",
    "At its core it gives you three things: **tensors** (NumPy-like arrays), **GPU\n",
    "acceleration** (the same code runs on a graphics card), and **autograd**\n",
    "(automatic gradients — the machinery that makes training possible). This lesson\n",
    "covers all three."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0002",
   "metadata": {},
   "source": [
    "> **Run this lesson in Colab**\n",
    "> \n",
    "> Download this lesson as a notebook and run it in Google Colab\n",
    "> (free at colab.research.google.com) — PyTorch is preinstalled there. The small\n",
    "> browser demos below still run right here."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0003",
   "metadata": {},
   "source": [
    "## Creating tensors\n",
    "\n",
    "A tensor is an n-dimensional array: a scalar is 0-D, a vector 1-D, a matrix\n",
    "2-D, and a color image batch is 4-D. Everything in PyTorch is a tensor."
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0004",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "\n",
    "# From Python data\n",
    "a = torch.tensor([[1.0, 2.0, 3.0],\n",
    "                  [4.0, 5.0, 6.0]])\n",
    "\n",
    "# Factory functions\n",
    "zeros = torch.zeros(2, 3)          # all zeros\n",
    "ones  = torch.ones(5)              # all ones\n",
    "noise = torch.randn(3, 4)          # standard normal random values\n",
    "\n",
    "print(a)\n",
    "print(a.shape)                     # torch.Size([2, 3])\n",
    "print(a.dtype)                     # torch.float32"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0005",
   "metadata": {},
   "source": [
    "Two attributes you'll check constantly:\n",
    "\n",
    "- **`.shape`** — the size along each dimension. Most PyTorch bugs are shape bugs.\n",
    "- **`.dtype`** — the element type. `float32` is the default for model inputs and\n",
    "  weights; class labels must be integers (`int64`, a.k.a. `torch.long`)."
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0006",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "labels = torch.tensor([0, 2, 1])       # int64 automatically\n",
    "floats = torch.tensor([0, 2, 1], dtype=torch.float32)\n",
    "print(labels.dtype, floats.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0007",
   "metadata": {},
   "source": [
    "## Reshaping, matmul, and friends\n",
    "\n",
    "Networks constantly reshape data — flattening images into vectors, adding batch\n",
    "dimensions, reordering channels:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0008",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "x = torch.arange(12, dtype=torch.float32)   # shape [12]\n",
    "\n",
    "m = x.view(3, 4)          # reshape to 3x4 (same memory)\n",
    "m2 = x.reshape(3, -1)     # -1 means \"infer this dimension\"\n",
    "print(m.shape, m2.shape)\n",
    "\n",
    "img = x.view(1, 3, 2, 2)              # [batch, channels, height, width]\n",
    "hwc = img.permute(0, 2, 3, 1)         # reorder dims -> [1, 2, 2, 3]\n",
    "print(hwc.shape)\n",
    "\n",
    "col = x.unsqueeze(1)      # add a dimension: [12] -> [12, 1]\n",
    "back = col.squeeze()      # drop size-1 dimensions: [12, 1] -> [12]\n",
    "print(col.shape, back.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0009",
   "metadata": {},
   "source": [
    "Matrix multiplication uses the `@` operator — this is the single most important\n",
    "operation in deep learning (a linear layer is one matmul plus a bias):"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0010",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "W = torch.randn(4, 3)     # weights: 3 inputs -> 4 outputs\n",
    "x = torch.randn(3)        # one sample with 3 features\n",
    "b = torch.randn(4)\n",
    "\n",
    "y = W @ x + b             # a linear layer, by hand\n",
    "print(y.shape)            # torch.Size([4])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0011",
   "metadata": {},
   "source": [
    "Also handy: `max` vs `argmax`. Given a batch of class scores, `max(1)` gives the\n",
    "highest score per row, while `argmax(1)` gives *which class* had it — that's how\n",
    "you turn network outputs into predictions:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0012",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "scores = torch.tensor([[0.1, 0.7, 0.2],\n",
    "                       [0.8, 0.1, 0.1]])\n",
    "print(scores.argmax(1))       # tensor([1, 0]) -> predicted classes\n",
    "values, indices = scores.max(1)\n",
    "print(values, indices)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0013",
   "metadata": {},
   "source": [
    "## The NumPy bridge\n",
    "\n",
    "Tensors and NumPy arrays convert both ways cheaply (they can even share memory\n",
    "on CPU):"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0014",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "arr = np.array([[1.0, 2.0], [3.0, 4.0]])\n",
    "\n",
    "t = torch.from_numpy(arr)     # numpy -> tensor\n",
    "back = t.numpy()              # tensor -> numpy\n",
    "\n",
    "single = torch.tensor(3.14)\n",
    "print(single.item())          # one-element tensor -> Python float"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0015",
   "metadata": {},
   "source": [
    "## Moving to the GPU\n",
    "\n",
    "The magic of PyTorch: the same code runs on CPU or GPU. The modern idiom is to\n",
    "define a `device` once and move both model and data to it:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0016",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "print(device)\n",
    "\n",
    "x = torch.randn(1000, 1000).to(device)\n",
    "y = x @ x                     # runs on the GPU if one is available"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0017",
   "metadata": {},
   "source": [
    "In Colab, enable the GPU via **Runtime → Change runtime type → T4 GPU**. One\n",
    "rule to remember: **everything in one operation must live on the same device** —\n",
    "mixing a CPU tensor with a GPU tensor raises an error. And to plot or convert\n",
    "to NumPy, bring data back with `.cpu()` first."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0018",
   "metadata": {},
   "source": [
    "## Autograd: derivatives for free\n",
    "\n",
    "Here's the feature that separates PyTorch from NumPy. Mark a tensor with\n",
    "`requires_grad=True` and PyTorch records every operation done to it. Call\n",
    "`.backward()` on a scalar result, and it walks the recorded graph in reverse\n",
    "(backpropagation) to fill in `.grad` — the derivative of the result with\n",
    "respect to each input.\n",
    "\n",
    "Take **y = x² + 3x**. Calculus says dy/dx = 2x + 3, so at x = 2 the derivative\n",
    "is 7. Autograd agrees:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0019",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "x = torch.tensor(2.0, requires_grad=True)\n",
    "\n",
    "y = x**2 + 3*x        # forward pass: PyTorch records the graph\n",
    "y.backward()          # backward pass: compute dy/dx\n",
    "\n",
    "print(x.grad)         # tensor(7.) -> matches 2*2 + 3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0020",
   "metadata": {},
   "source": [
    "No symbolic math, no hand-derived formulas — and it scales to functions with\n",
    "millions of inputs, which is exactly what a neural network's loss is.\n",
    "\n",
    "For intuition, here's the same derivative computed **numerically** the way you\n",
    "might in a first calculus course — nudge x a tiny bit and measure the change.\n",
    "Run it in your browser:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0021",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def f(x):\n",
    "    return x**2 + 3*x\n",
    "\n",
    "x, h = 2.0, 1e-5\n",
    "numeric = (f(x + h) - f(x - h)) / (2 * h)   # central difference\n",
    "\n",
    "print(f\"numerical dy/dx at x=2 : {numeric:.6f}\")\n",
    "print(f\"analytic  2x + 3 at x=2: {2*x + 3:.6f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0022",
   "metadata": {},
   "source": [
    "Numerical differentiation needs one extra function evaluation *per parameter* —\n",
    "hopeless for a million-parameter model. Autograd gets every gradient in a single\n",
    "backward pass. That efficiency is why deep learning is feasible at all.\n",
    "\n",
    "One more autograd detail: **gradients accumulate**. Each `.backward()` call\n",
    "*adds* to `.grad` rather than replacing it. That's why training loops zero the\n",
    "gradients every step — you'll meet `optimizer.zero_grad()` in the next lesson."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0023",
   "metadata": {},
   "source": [
    "## Turning autograd off\n",
    "\n",
    "Tracking operations costs memory and time. When you're only *using* a model\n",
    "(predicting, evaluating), wrap the code so no graph is recorded:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0024",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "x = torch.tensor(2.0, requires_grad=True)\n",
    "\n",
    "with torch.no_grad():                 # classic way\n",
    "    y = x**2 + 3*x\n",
    "print(y.requires_grad)                # False\n",
    "\n",
    "with torch.inference_mode():          # modern, slightly faster\n",
    "    y = x**2 + 3*x\n",
    "print(y.requires_grad)                # False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0025",
   "metadata": {},
   "source": [
    "Prefer `torch.inference_mode()` for evaluation and deployment code — it's the\n",
    "stricter, faster successor to `no_grad`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0026",
   "metadata": {},
   "source": [
    "### 🏋️ Exercise — Autograd vs. calculus\n",
    "\n",
    "For the function **y = 2x³ − 5x² + 4x**, compute dy/dx at **x = 3** three ways:\n",
    "(1) with autograd, (2) analytically by hand (write the derivative formula in\n",
    "code), and confirm they match. Then demonstrate gradient **accumulation**: call\n",
    "`.backward()` a second time (rebuild `y` first) and print `x.grad` — explain\n",
    "what you see, then fix it with `x.grad.zero_()`."
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0027",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0028",
   "metadata": {},
   "source": [
    "<details><summary>✅ Show solution</summary>\n",
    "\n",
    "```python\n",
    "import torch\n",
    "\n",
    "# 1. Autograd\n",
    "x = torch.tensor(3.0, requires_grad=True)\n",
    "y = 2*x**3 - 5*x**2 + 4*x\n",
    "y.backward()\n",
    "print(\"autograd :\", x.grad)          # 6*9 - 10*3 + 4 = 28\n",
    "\n",
    "# 2. By hand: dy/dx = 6x^2 - 10x + 4\n",
    "print(\"analytic :\", 6*3.0**2 - 10*3.0 + 4)\n",
    "\n",
    "# 3. Gradients accumulate!\n",
    "y2 = 2*x**3 - 5*x**2 + 4*x\n",
    "y2.backward()\n",
    "print(\"after 2nd backward (accumulated):\", x.grad)   # 56, not 28\n",
    "\n",
    "x.grad.zero_()                        # reset, like optimizer.zero_grad()\n",
    "y3 = 2*x**3 - 5*x**2 + 4*x\n",
    "y3.backward()\n",
    "print(\"after zeroing then backward   :\", x.grad)     # 28 again\n",
    "```\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0029",
   "metadata": {},
   "source": [
    "Next up: the training loop — model, criterion, optimizer, and the\n",
    "forward-backward-step dance that turns gradients into learning."
   ]
  }
 ]
}