{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-0000",
   "metadata": {},
   "source": [
    "# Logistic Regression\n",
    "\n",
    "Turn regression into classification with the sigmoid, understand why log-loss beats MSE, and read probabilities and coefficients from a real model.\n",
    "\n",
    "*Part of the free [Machine Learning](https://ramadnsyh.dev/courses/machine-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/machine-learning/logistic-regression).*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0001",
   "metadata": {},
   "source": [
    "Despite the name, logistic regression is a *classifier* — and probably the most\n",
    "widely deployed one in industry. The trick is beautifully simple: keep the\n",
    "linear model you already know, then squash its output into a probability. In\n",
    "this lesson you'll see why a plain line fails on 0/1 labels, why the loss\n",
    "function has to change too, and how to use `LogisticRegression` properly."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0002",
   "metadata": {},
   "source": [
    "## Why not just fit a line?\n",
    "\n",
    "Labels in binary classification are 0 or 1. What if we fit ordinary linear\n",
    "regression to them and predict \"1\" whenever the line is above 0.5? Play with\n",
    "the threshold and the data below and watch what the sigmoid does that a line\n",
    "can't:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0003",
   "metadata": {},
   "source": [
    "> 🎛️ **Interactive demo** — this section has a hands-on visualization in the web version of this lesson: [open it here](https://ramadnsyh.dev/courses/machine-learning/logistic-regression)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0004",
   "metadata": {},
   "source": [
    "A straight line has two fatal problems here:\n",
    "\n",
    "1. **Unbounded outputs.** The line happily predicts 3.7 or −1.2 — meaningless\n",
    "   as a probability of a class.\n",
    "2. **Outliers drag the boundary.** Add one very obvious positive far to the\n",
    "   right and MSE forces the line to tilt toward it, *moving the decision\n",
    "   boundary* and misclassifying points near the middle. The most confident\n",
    "   examples shouldn't be the ones wrecking the fit."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0005",
   "metadata": {},
   "source": [
    "## The sigmoid: scores in, probabilities out\n",
    "\n",
    "The fix: pass the linear score `z = w·x + b` through the **sigmoid** function:\n",
    "\n",
    "**σ(z) = 1 / (1 + e^(−z))**\n",
    "\n",
    "- Large positive z → σ(z) → 1 (confident positive)\n",
    "- Large negative z → σ(z) → 0 (confident negative)\n",
    "- z = 0 → σ(z) = 0.5 (right on the boundary)\n",
    "\n",
    "The output is always between 0 and 1, so we can read it as\n",
    "**P(y = 1 given x)**. The model is still linear at heart — the sigmoid just\n",
    "translates its score into probability language."
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0006",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "z = np.linspace(-8, 8, 200)\n",
    "sigmoid = 1 / (1 + np.exp(-z))\n",
    "\n",
    "plt.plot(z, sigmoid)\n",
    "plt.axhline(0.5, color=\"gray\", ls=\"--\", lw=1)\n",
    "plt.axvline(0, color=\"gray\", ls=\"--\", lw=1)\n",
    "plt.xlabel(\"linear score  z = w·x + b\")\n",
    "plt.ylabel(\"P(y = 1)\")\n",
    "plt.title(\"The sigmoid squashes any score into (0, 1)\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0007",
   "metadata": {},
   "source": [
    "## Why the loss must change too\n",
    "\n",
    "You might be tempted to keep MSE and just wrap the prediction in a sigmoid.\n",
    "Don't — this is where the **weaknesses of gradient descent** bite. Gradient\n",
    "descent only works well when the loss surface cooperates; it can get stuck in\n",
    "**local minima**, stall on **plateaus** (flat regions where gradients vanish),\n",
    "and crawl through **saddle points**. MSE-through-a-sigmoid creates exactly\n",
    "these pathologies: the loss surface becomes **non-convex**, with plateaus\n",
    "wherever the sigmoid saturates. A confidently-wrong prediction (σ ≈ 0 when the\n",
    "truth is 1) sits on flat ground — the gradient is nearly zero and learning\n",
    "stalls precisely when the model most needs correcting.\n",
    "\n",
    "The right loss is **log-loss** (binary cross-entropy):\n",
    "\n",
    "**L = −(1/n) Σ [ yᵢ·log(pᵢ) + (1 − yᵢ)·log(1 − pᵢ) ]**\n",
    "\n",
    "Being wrong with high confidence costs almost infinitely much — log(p) blows\n",
    "up as p → 0 — so the gradient stays large exactly where MSE goes flat. Bonus:\n",
    "with log-loss the optimization problem is **convex**, one bowl-shaped valley\n",
    "with a single minimum. Gradient descent can't get trapped."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0008",
   "metadata": {},
   "source": [
    "> **One idea, two names**\n",
    "> \n",
    "> Log-loss, binary cross-entropy, and negative log-likelihood are the same thing.\n",
    "> Maximizing the probability of the observed labels is equivalent to minimizing\n",
    "> this loss."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0009",
   "metadata": {},
   "source": [
    "## Decision boundary and threshold\n",
    "\n",
    "The model's *probability* is continuous; the *decision* needs a cutoff. By\n",
    "default `predict` uses 0.5, which corresponds to the line (or hyperplane)\n",
    "where `w·x + b = 0`. But as you saw in the metrics lesson, the threshold is a\n",
    "business decision, not a math constant — lower it to catch more positives,\n",
    "raise it to reduce false alarms. Keep `predict_proba` around and apply the\n",
    "threshold yourself when the costs are asymmetric."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0010",
   "metadata": {},
   "source": [
    "## Logistic regression on real data\n",
    "\n",
    "Let's train on the breast-cancer dataset. Scaling matters: logistic regression\n",
    "is trained by an iterative optimizer, and wildly different feature scales make\n",
    "the loss valley long and narrow — slow, unstable convergence. A pipeline keeps\n",
    "it honest:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0011",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "X, y = load_breast_cancer(return_X_y=True)\n",
    "X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=42)\n",
    "\n",
    "model = make_pipeline(StandardScaler(), LogisticRegression())\n",
    "model.fit(X_tr, y_tr)\n",
    "print(f\"train accuracy: {model.score(X_tr, y_tr):.3f}\")\n",
    "print(f\"test  accuracy: {model.score(X_te, y_te):.3f}\")\n",
    "\n",
    "# probabilities for the first 5 test tumors\n",
    "proba = model.predict_proba(X_te[:5])\n",
    "print(\"\\\\nP(malignant)  P(benign)   prediction\")\n",
    "for p, pred in zip(proba, model.predict(X_te[:5])):\n",
    "    print(f\"   {p[0]:.3f}       {p[1]:.3f}      {pred}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0012",
   "metadata": {},
   "source": [
    "`predict_proba` returns one column per class, ordered by `model.classes_`\n",
    "(here 0 = malignant, 1 = benign); each row sums to 1. Notice how some\n",
    "predictions are near-certain (0.99+) while others hover near 0.5 — that\n",
    "uncertainty information is free, and thresholding is how you use it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0013",
   "metadata": {},
   "source": [
    "## Coefficients are log-odds\n",
    "\n",
    "Because the model is linear inside the sigmoid, its coefficients are\n",
    "interpretable: increasing feature j by one (scaled) unit adds `w_j` to the\n",
    "**log-odds** `log(p / (1 − p))` of the positive class. You don't need to think\n",
    "in log-odds day to day — the sign and size are what matter:\n",
    "\n",
    "- **positive coefficient** → larger feature value pushes toward class 1\n",
    "- **negative coefficient** → pushes toward class 0\n",
    "- **larger magnitude** → stronger influence (comparable across features only\n",
    "  *because* we standardized them)"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0014",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "coefs = pd.Series(model[-1].coef_[0],\n",
    "                  index=load_breast_cancer().feature_names)\n",
    "print(coefs.sort_values())     # most malignant-leaning features first"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0015",
   "metadata": {},
   "source": [
    "## Regularization: the C parameter\n",
    "\n",
    "`LogisticRegression` applies L2 regularization by default, controlled by `C` —\n",
    "the **inverse** regularization strength (the same convention SVMs use):\n",
    "\n",
    "- **small C** (e.g. 0.01) → strong regularization → smaller coefficients,\n",
    "  simpler model, may underfit\n",
    "- **large C** (e.g. 100) → weak regularization → coefficients roam free, may\n",
    "  overfit"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0016",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split, cross_val_score\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "X, y = load_breast_cancer(return_X_y=True)\n",
    "X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=42)\n",
    "\n",
    "for C in [0.01, 0.1, 1, 10, 100]:\n",
    "    model = make_pipeline(StandardScaler(), LogisticRegression(C=C))\n",
    "    cv = cross_val_score(model, X_tr, y_tr, cv=5).mean()\n",
    "    coef_size = np.abs(model.fit(X_tr, y_tr)[-1].coef_).mean()\n",
    "    print(f\"C={C:>6}   CV accuracy={cv:.3f}   mean |coef|={coef_size:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0017",
   "metadata": {},
   "source": [
    "Watch the coefficients grow with C while cross-validation accuracy peaks\n",
    "somewhere in the middle — tune C with cross-validation rather than guessing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0018",
   "metadata": {},
   "source": [
    "### 🏋️ Exercise — Threshold surgery on logistic regression\n",
    "\n",
    "= t, 0, 1).\",\n",
    "    \"Use confusion_matrix(y_te, pred) and compare the false-negative count (malignant predicted benign) across thresholds.\",\n",
    "  ]}\n",
    "  solution={`\n",
    "import numpy as np\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "X, y = load_breast_cancer(return_X_y=True)\n",
    "X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=42)\n",
    "model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_tr, y_tr)\n",
    "\n",
    "p_malignant = model.predict_proba(X_te)[:, 0]\n",
    "\n",
    "for t in [0.5, 0.3, 0.1]:\n",
    "    pred = np.where(p_malignant >= t, 0, 1)   # 0 = malignant\n",
    "    cm = confusion_matrix(y_te, pred)\n",
    "    # rows: truth (0=malignant, 1=benign); cols: prediction\n",
    "    missed = cm[0, 1]     # malignant predicted as benign\n",
    "    alarms = cm[1, 0]     # benign predicted as malignant\n",
    "    print(f\"threshold {t:.1f}: missed cancers = {missed}, false alarms = {alarms}\")\n",
    "`}\n",
    ">\n",
    "Using the breast-cancer pipeline, compare three malignancy thresholds — 0.5,\n",
    "0.3, and 0.1 — and count for each: (a) how many malignant tumors get missed\n",
    "(predicted benign) and (b) how many benign tumors get falsely flagged. What is\n",
    "the cost of driving missed cancers toward zero?"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0019",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0020",
   "metadata": {},
   "source": [
    "<details><summary>✅ Show solution</summary>\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "X, y = load_breast_cancer(return_X_y=True)\n",
    "X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=42)\n",
    "model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_tr, y_tr)\n",
    "\n",
    "p_malignant = model.predict_proba(X_te)[:, 0]\n",
    "\n",
    "for t in [0.5, 0.3, 0.1]:\n",
    "    pred = np.where(p_malignant >= t, 0, 1)   # 0 = malignant\n",
    "    cm = confusion_matrix(y_te, pred)\n",
    "    # rows: truth (0=malignant, 1=benign); cols: prediction\n",
    "    missed = cm[0, 1]     # malignant predicted as benign\n",
    "    alarms = cm[1, 0]     # benign predicted as malignant\n",
    "    print(f\"threshold {t:.1f}: missed cancers = {missed}, false alarms = {alarms}\")\n",
    "```\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0021",
   "metadata": {},
   "source": [
    "Next: what happens when there are more than two classes — and when a single\n",
    "sample can carry several labels at once."
   ]
  }
 ]
}