{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-0000",
   "metadata": {},
   "source": [
    "# Multiclass & Multilabel Classification\n",
    "\n",
    "Go beyond yes/no questions — One-vs-Rest, One-vs-One, softmax regression, metric averaging for many classes, and models that assign several labels at once.\n",
    "\n",
    "*Part of the free [Machine Learning](https://ramadnsyh.dev/courses/machine-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/machine-learning/multiclass-multilabel).*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0001",
   "metadata": {},
   "source": [
    "So far every classifier we've built answered a yes/no question: malignant or\n",
    "benign, fraud or legit. Real problems are rarely that tidy — an iris flower is\n",
    "one of *three* species, a handwritten digit is one of *ten*, and a movie can be\n",
    "an action comedy *and* a romance all at once. This lesson extends everything\n",
    "you know about binary classification to many classes, and then to many labels\n",
    "per sample."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0002",
   "metadata": {},
   "source": [
    "## One question, how many answers?\n",
    "\n",
    "Three different problem shapes hide behind the word \"classification\":\n",
    "\n",
    "- **Binary** — exactly one of 2 classes. *Is this tumor malignant?*\n",
    "- **Multiclass** — exactly one of K classes. *Which iris species is this:\n",
    "  setosa, versicolor, or virginica?* The classes are mutually exclusive — a\n",
    "  flower can't be two species.\n",
    "- **Multilabel** — any *subset* of K labels. *Which genres describe this\n",
    "  movie?* `[\"action\", \"comedy\"]` is a perfectly valid answer, and so is an\n",
    "  empty set or all of them.\n",
    "\n",
    "The distinction matters because it changes the model's output, the loss, and\n",
    "the metrics. Confuse multiclass with multilabel and you'll force a movie to\n",
    "have exactly one genre — or let a flower be two species at once."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0003",
   "metadata": {},
   "source": [
    "## Turning binary classifiers into multiclass ones\n",
    "\n",
    "Logistic regression natively separates two classes. Two classic strategies\n",
    "recycle it for K classes:\n",
    "\n",
    "**One-vs-Rest (OvR)** trains K binary classifiers, each answering \"is it class\n",
    "k, or anything else?\" — *setosa vs rest*, *versicolor vs rest*, *virginica vs\n",
    "rest*. At prediction time, all K models score the sample and the most\n",
    "confident one wins. K models total, each trained on the full dataset.\n",
    "\n",
    "**One-vs-One (OvO)** trains one binary classifier per *pair* of classes —\n",
    "K·(K−1)/2 models — and lets them vote. That's more models, but each trains on\n",
    "only two classes' worth of data, which is a win for algorithms whose training\n",
    "cost grows steeply with dataset size (scikit-learn's `SVC` uses OvO internally\n",
    "for exactly this reason)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0004",
   "metadata": {},
   "source": [
    "## Softmax: the multiclass sigmoid\n",
    "\n",
    "There's a more elegant third option. Instead of gluing binary models together,\n",
    "generalize the sigmoid itself. **Softmax** takes K raw scores and turns them\n",
    "into K probabilities that sum to 1: exponentiate every score, then divide each\n",
    "by the total."
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0005",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "classes = [\"setosa\", \"versicolor\", \"virginica\"]\n",
    "scores = np.array([2.0, 1.0, 0.1])     # raw linear scores, one per class\n",
    "\n",
    "exp_scores = np.exp(scores)\n",
    "softmax = exp_scores / exp_scores.sum()\n",
    "\n",
    "for c, s, p in zip(classes, scores, softmax):\n",
    "    print(f\"{c:11s}  score={s:4.1f}  ->  P={p:.3f}\")\n",
    "print(f\"\\\\nprobabilities sum to {softmax.sum():.1f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0006",
   "metadata": {},
   "source": [
    "A logistic regression trained this way — one weight vector per class, softmax\n",
    "on top, cross-entropy loss — is called **multinomial (softmax) logistic\n",
    "regression**. It's one coherent model rather than K independent ones, so the\n",
    "probabilities are directly comparable. Modern scikit-learn does this\n",
    "automatically: hand `LogisticRegression` a target with three classes and it\n",
    "fits the multinomial model, no extra arguments needed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0007",
   "metadata": {},
   "source": [
    "## Multiclass in practice: iris\n",
    "\n",
    "Everything else about the workflow is unchanged — same pipeline, same `fit`,\n",
    "same `predict_proba` (now with three columns). The confusion matrix just grows\n",
    "to K×K:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0008",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import ConfusionMatrixDisplay\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "iris = load_iris()\n",
    "X_tr, X_te, y_tr, y_te = train_test_split(iris.data, iris.target,\n",
    "                                          stratify=iris.target, random_state=42)\n",
    "\n",
    "model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_tr, y_tr)\n",
    "print(f\"test accuracy: {model.score(X_te, y_te):.3f}\")\n",
    "\n",
    "ConfusionMatrixDisplay.from_estimator(model, X_te, y_te,\n",
    "                                      display_labels=iris.target_names)\n",
    "plt.title(\"Iris test set\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0009",
   "metadata": {},
   "source": [
    "The matrix tells a story accuracy can't: setosa is never confused with\n",
    "anything (its petals are unmistakably small), while the mistakes live entirely\n",
    "in the versicolor/virginica corner — those two species genuinely overlap."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0010",
   "metadata": {},
   "source": [
    "## Macro, micro, weighted: averaging metrics across classes\n",
    "\n",
    "Precision, recall, and F1 are defined *per class* — with K classes you get K\n",
    "of each. To report a single number you must average them, and *how* you\n",
    "average changes the answer:\n",
    "\n",
    "- **macro** — average the K per-class scores equally. Every class counts the\n",
    "  same, so a tiny minority class can drag the score down. Use it when small\n",
    "  classes matter.\n",
    "- **micro** — pool all predictions first, then compute the metric once. Big\n",
    "  classes dominate. (For single-label multiclass, micro-F1 equals plain\n",
    "  accuracy.)\n",
    "- **weighted** — macro, but each class weighted by its number of samples. A\n",
    "  compromise that still mostly reflects the majority.\n",
    "\n",
    "On balanced iris these barely differ. On imbalanced data they diverge wildly:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0011",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_classification\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import f1_score, recall_score, accuracy_score\n",
    "\n",
    "X, y = make_classification(n_samples=1000, n_classes=3, n_informative=6,\n",
    "                           weights=[0.7, 0.2, 0.1], random_state=42)\n",
    "X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=42)\n",
    "model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_tr, y_tr)\n",
    "pred = model.predict(X_te)\n",
    "\n",
    "print(\"per-class recall:\", recall_score(y_te, pred, average=None).round(3))\n",
    "print()\n",
    "for avg in [\"macro\", \"micro\", \"weighted\"]:\n",
    "    print(f\"F1 ({avg:8s}) = {f1_score(y_te, pred, average=avg):.3f}\")\n",
    "print(f\"accuracy      = {accuracy_score(y_te, pred):.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0012",
   "metadata": {},
   "source": [
    "The model nails the 70% majority class and largely misses the 10% minority.\n",
    "Micro-F1 (= accuracy here) looks respectable; macro-F1 exposes the failure.\n",
    "When someone reports \"the F1\", always ask *which averaging*."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0013",
   "metadata": {},
   "source": [
    "## Multilabel: many tags per sample\n",
    "\n",
    "Now drop the \"exactly one\" constraint. A multilabel target is a **binary\n",
    "matrix**: one column per label, one row per sample, with 1s wherever a label\n",
    "applies. `MultiLabelBinarizer` converts lists of tags into that matrix, and\n",
    "`OneVsRestClassifier` fits one independent binary classifier per column —\n",
    "each label gets its own yes/no decision, so a sample can light up several:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0014",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.preprocessing import MultiLabelBinarizer\n",
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import hamming_loss, f1_score\n",
    "\n",
    "# synthetic \"movies\": 3 features drive which genre tags apply\n",
    "rng = np.random.default_rng(42)\n",
    "X = rng.normal(0, 1, (300, 3))\n",
    "tags = []\n",
    "for a, r, c in X:\n",
    "    t = []\n",
    "    if a > 0.2:            t.append(\"action\")\n",
    "    if r > 0.4:            t.append(\"romance\")\n",
    "    if c > 0.1 or a > 1.2: t.append(\"comedy\")\n",
    "    if not t:              t.append(\"drama\")\n",
    "    tags.append(t)\n",
    "\n",
    "mlb = MultiLabelBinarizer()\n",
    "Y = mlb.fit_transform(tags)\n",
    "print(\"labels:\", list(mlb.classes_))\n",
    "print(\"first 3 movies:\", tags[:3])\n",
    "print(\"as a binary matrix:\\\\n\", Y[:3])\n",
    "\n",
    "X_tr, X_te, Y_tr, Y_te = train_test_split(X, Y, random_state=42)\n",
    "clf = OneVsRestClassifier(LogisticRegression()).fit(X_tr, Y_tr)\n",
    "P = clf.predict(X_te)\n",
    "\n",
    "print(f\"\\\\nhamming loss    : {hamming_loss(Y_te, P):.3f}\")\n",
    "print(f\"micro-F1        : {f1_score(Y_te, P, average='micro'):.3f}\")\n",
    "print(f\"exact-match rate: {(P == Y_te).all(axis=1).mean():.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0015",
   "metadata": {},
   "source": [
    "Two multilabel-specific metrics appear here. **Hamming loss** is the fraction\n",
    "of individual label decisions that were wrong — every cell of the matrix\n",
    "counts, so 0.09 means 9% of all yes/no calls missed. The **exact-match rate**\n",
    "(subset accuracy) is far stricter: a sample only counts if *every* label is\n",
    "right. It's normal for hamming loss to look great while exact match looks\n",
    "mediocre — one wrong tag out of four ruins the exact match but barely moves\n",
    "the hamming loss."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0016",
   "metadata": {},
   "source": [
    "> **Which problem do you actually have?**\n",
    "> \n",
    "> Ask: can the true answers overlap? Species, digit, sentiment — no overlap →\n",
    "> multiclass, use softmax. Genres, article topics, symptoms — overlaps are\n",
    "> meaningful → multilabel, use one binary decision per label."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0017",
   "metadata": {},
   "source": [
    "### 🏋️ Exercise — Find the hardest genre\n",
    "\n",
    "0.2:            t.append(\"action\")\n",
    "    if r > 0.4:            t.append(\"romance\")\n",
    "    if c > 0.1 or a > 1.2: t.append(\"comedy\")\n",
    "    if not t:              t.append(\"drama\")\n",
    "    tags.append(t)\n",
    "\n",
    "mlb = MultiLabelBinarizer()\n",
    "Y = mlb.fit_transform(tags)\n",
    "X_tr, X_te, Y_tr, Y_te = train_test_split(X, Y, random_state=42)\n",
    "clf = OneVsRestClassifier(LogisticRegression()).fit(X_tr, Y_tr)\n",
    "P = clf.predict(X_te)\n",
    "\n",
    "per_label_f1 = f1_score(Y_te, P, average=None)\n",
    "freq = Y.mean(axis=0)\n",
    "for name, f1, fr in zip(mlb.classes_, per_label_f1, freq):\n",
    "    print(f\"{name:8s}  F1={f1:.2f}  frequency={fr:.2f}\")\n",
    "\n",
    "# drama is hardest: it is the rarest label AND it is defined negatively\n",
    "# (\"none of the other rules fired\"), so no single feature pushes toward it.\n",
    "`}\n",
    ">\n",
    "Using the synthetic movie-tag dataset from this lesson, compute the **per-label\n",
    "F1 score** (one number per genre) and each label's frequency in the data.\n",
    "Which genre is hardest to predict — and looking at how the tags were\n",
    "generated, can you explain *why* that label is fundamentally harder than the\n",
    "others?"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0018",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0019",
   "metadata": {},
   "source": [
    "<details><summary>✅ Show solution</summary>\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "from sklearn.preprocessing import MultiLabelBinarizer\n",
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import f1_score\n",
    "\n",
    "rng = np.random.default_rng(42)\n",
    "X = rng.normal(0, 1, (300, 3))\n",
    "tags = []\n",
    "for a, r, c in X:\n",
    "    t = []\n",
    "    if a > 0.2:            t.append(\"action\")\n",
    "    if r > 0.4:            t.append(\"romance\")\n",
    "    if c > 0.1 or a > 1.2: t.append(\"comedy\")\n",
    "    if not t:              t.append(\"drama\")\n",
    "    tags.append(t)\n",
    "\n",
    "mlb = MultiLabelBinarizer()\n",
    "Y = mlb.fit_transform(tags)\n",
    "X_tr, X_te, Y_tr, Y_te = train_test_split(X, Y, random_state=42)\n",
    "clf = OneVsRestClassifier(LogisticRegression()).fit(X_tr, Y_tr)\n",
    "P = clf.predict(X_te)\n",
    "\n",
    "per_label_f1 = f1_score(Y_te, P, average=None)\n",
    "freq = Y.mean(axis=0)\n",
    "for name, f1, fr in zip(mlb.classes_, per_label_f1, freq):\n",
    "    print(f\"{name:8s}  F1={f1:.2f}  frequency={fr:.2f}\")\n",
    "\n",
    "# drama is hardest: it is the rarest label AND it is defined negatively\n",
    "# (\"none of the other rules fired\"), so no single feature pushes toward it.\n",
    "```\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0020",
   "metadata": {},
   "source": [
    "Next module: a classifier with a completely different philosophy — support\n",
    "vector machines, which ignore most of the data and let a handful of boundary\n",
    "points decide everything."
   ]
  }
 ]
}