{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "id": "cell-0000", "metadata": {}, "source": [ "# Classification Metrics Beyond Accuracy\n", "\n", "Why accuracy lies on imbalanced data, and how to read confusion matrices, precision, recall, F1, ROC curves, and precision-recall curves instead.\n", "\n", "*Part of the free [Machine Learning](https://ramadnsyh.dev/courses/machine-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/machine-learning/classification-metrics).*" ] }, { "cell_type": "markdown", "id": "cell-0001", "metadata": {}, "source": [ "Accuracy is the first metric everyone learns — and the first one that betrays\n", "them. On real-world classification problems (fraud, disease, churn) the classes\n", "are rarely balanced, and a model can score 95%+ accuracy while being completely\n", "useless. This lesson gives you the full evaluation toolkit: the confusion\n", "matrix, precision and recall, F1, ROC/AUC, and the precision-recall curve." ] }, { "cell_type": "markdown", "id": "cell-0002", "metadata": {}, "source": [ "## The accuracy trap\n", "\n", "Suppose only 5% of transactions are fraudulent. Here's a \"model\" that never\n", "predicts fraud at all — it just answers \"legit\" every time:" ] }, { "cell_type": "code", "id": "cell-0003", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "from sklearn.datasets import make_classification\n", "from sklearn.metrics import accuracy_score\n", "\n", "X, y = make_classification(n_samples=1000, weights=[0.95], flip_y=0,\n", " random_state=42)\n", "print(f\"Positives (fraud): {y.sum()} / {len(y)}\")\n", "\n", "y_dummy = np.zeros_like(y) # always predict the majority class\n", "print(f\"Accuracy of doing nothing: {accuracy_score(y, y_dummy):.1%}\")" ] }, { "cell_type": "markdown", "id": "cell-0004", "metadata": {}, "source": [ "95% accuracy, zero frauds caught. Any metric that rewards this behavior can't\n", "be trusted on imbalanced data. To do better we need to look at *which kinds*\n", "of mistakes the model makes — not just how many." ] }, { "cell_type": "markdown", "id": "cell-0005", "metadata": {}, "source": [ "## The confusion matrix: four kinds of outcome\n", "\n", "For a binary problem, every prediction lands in one of four cells:\n", "\n", "- **True Positive (TP)** — predicted positive, actually positive\n", "- **False Positive (FP)** — predicted positive, actually negative (a false alarm)\n", "- **False Negative (FN)** — predicted negative, actually positive (a miss)\n", "- **True Negative (TN)** — predicted negative, actually negative\n", "\n", "Accuracy is `(TP + TN) / total` — it lumps both error types together. The\n", "confusion matrix keeps them separate:" ] }, { "cell_type": "code", "id": "cell-0006", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from sklearn.datasets import load_breast_cancer\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.pipeline import make_pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.metrics import ConfusionMatrixDisplay\n", "import matplotlib.pyplot as plt\n", "\n", "X, y = load_breast_cancer(return_X_y=True) # 1 = benign, 0 = malignant\n", "X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=42)\n", "\n", "model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_tr, y_tr)\n", "\n", "ConfusionMatrixDisplay.from_estimator(model, X_te, y_te,\n", " display_labels=[\"malignant\", \"benign\"])\n", "plt.title(\"Breast cancer test set\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "cell-0007", "metadata": {}, "source": [ "Rows are the truth, columns are the predictions. The diagonal is what the\n", "model got right; the off-diagonal cells are the two flavors of mistake." ] }, { "cell_type": "markdown", "id": "cell-0008", "metadata": {}, "source": [ "## Precision vs recall: which mistake hurts more?\n", "\n", "Two metrics zoom in on the positive class, each punishing a different error:\n", "\n", "**Precision = TP / (TP + FP)** — of everything I *flagged* as positive, how\n", "much really was? High precision = few false alarms.\n", "\n", "**Recall = TP / (TP + FN)** — of everything that *is* positive, how much did\n", "I catch? High recall = few misses. (Also called *sensitivity*.)\n", "\n", "Which one matters depends entirely on the cost of each mistake:\n", "\n", "- **Spam filtering** — a false positive means a real email (maybe a job offer)\n", " lands in spam. That's the expensive mistake, so optimize **precision**.\n", "- **Cancer screening** — a false negative means telling a sick patient they're\n", " healthy. That's catastrophic, so optimize **recall**; a few false alarms\n", " that trigger a follow-up test are an acceptable price." ] }, { "cell_type": "markdown", "id": "cell-0009", "metadata": {}, "source": [ "> **A memory hook**\n", "> \n", "> Precision asks \"when I speak, am I right?\" Recall asks \"did I find them all?\"\n", "> You can always get perfect recall by flagging everything — at the cost of\n", "> terrible precision. The two pull against each other." ] }, { "cell_type": "markdown", "id": "cell-0010", "metadata": {}, "source": [ "## F1 and the threshold tradeoff\n", "\n", "The **F1-score** is the harmonic mean of precision and recall:\n", "\n", "**F1 = 2 · (precision · recall) / (precision + recall)**\n", "\n", "The harmonic mean is deliberately harsh: if either precision or recall is near\n", "zero, F1 is near zero too. Use it when you want a single number that balances\n", "both — the \"do nothing\" fraud model above has an F1 of exactly 0.\n", "\n", "Most classifiers actually output a *probability*, and the default 0.5 cutoff\n", "is just a choice. Moving the threshold trades precision for recall:" ] }, { "cell_type": "code", "id": "cell-0011", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from sklearn.datasets import load_breast_cancer\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.pipeline import make_pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.metrics import precision_score, recall_score, f1_score\n", "\n", "X, y = load_breast_cancer(return_X_y=True)\n", "X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=42)\n", "model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_tr, y_tr)\n", "\n", "proba = model.predict_proba(X_te)[:, 1]\n", "print(\"threshold precision recall F1\")\n", "for t in [0.1, 0.3, 0.5, 0.7, 0.9]:\n", " pred = (proba >= t).astype(int)\n", " print(f\" {t:.1f} {precision_score(y_te, pred):.3f} \"\n", " f\"{recall_score(y_te, pred):.3f} {f1_score(y_te, pred):.3f}\")" ] }, { "cell_type": "markdown", "id": "cell-0012", "metadata": {}, "source": [ "Lower the threshold and recall climbs (you flag more) while precision drops.\n", "For a per-class summary of everything at once, `classification_report` prints\n", "precision, recall, F1, and support for each class:" ] }, { "cell_type": "code", "id": "cell-0013", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from sklearn.metrics import classification_report\n", "print(classification_report(y_te, model.predict(X_te),\n", " target_names=[\"malignant\", \"benign\"]))" ] }, { "cell_type": "markdown", "id": "cell-0014", "metadata": {}, "source": [ "## ROC curve and AUC\n", "\n", "Instead of picking one threshold, the **ROC curve** sweeps through *all* of\n", "them, plotting the true-positive rate (recall) against the false-positive rate\n", "at every cutoff. The **AUC** (area under the curve) summarizes it: 1.0 is a\n", "perfect ranker, 0.5 is coin-flipping." ] }, { "cell_type": "code", "id": "cell-0015", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from sklearn.datasets import load_breast_cancer\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.pipeline import make_pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.metrics import roc_curve, roc_auc_score\n", "import matplotlib.pyplot as plt\n", "\n", "X, y = load_breast_cancer(return_X_y=True)\n", "X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=42)\n", "model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_tr, y_tr)\n", "\n", "proba = model.predict_proba(X_te)[:, 1]\n", "fpr, tpr, _ = roc_curve(y_te, proba)\n", "\n", "plt.plot(fpr, tpr, label=f\"AUC = {roc_auc_score(y_te, proba):.3f}\")\n", "plt.plot([0, 1], [0, 1], \"k--\", label=\"random guessing\")\n", "plt.xlabel(\"False positive rate\")\n", "plt.ylabel(\"True positive rate (recall)\")\n", "plt.legend()\n", "plt.title(\"ROC curve\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "cell-0016", "metadata": {}, "source": [ "A nice property of AUC: it doesn't depend on any particular threshold, so it\n", "measures how well the model *ranks* positives above negatives." ] }, { "cell_type": "markdown", "id": "cell-0017", "metadata": {}, "source": [ "## The PR curve: when imbalance is heavy\n", "\n", "ROC has a blind spot. The false-positive rate divides by the number of\n", "*negatives* — and when negatives are 95%+ of the data, even thousands of false\n", "alarms barely move the curve. ROC-AUC then looks flattering while precision is\n", "actually awful.\n", "\n", "The **precision-recall curve** plots precision against recall across\n", "thresholds, and both quantities focus on the positive class — so it stays\n", "honest under heavy imbalance. Rules of thumb from practice:\n", "\n", "- **Roughly balanced data (20–80% positive)** — ROC-AUC and PR-AUC both work.\n", "- **Rare positives (under ~5%)** — prefer the PR curve and average precision;\n", " ROC-AUC tends to overestimate how good the model is.\n", "- **Very little data** — every metric becomes unstable; treat all scores with\n", " suspicion." ] }, { "cell_type": "code", "id": "cell-0018", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from sklearn.metrics import PrecisionRecallDisplay\n", "PrecisionRecallDisplay.from_estimator(model, X_te, y_te)" ] }, { "cell_type": "markdown", "id": "cell-0019", "metadata": {}, "source": [ "### 🏋️ Exercise — Pick a threshold for a screening model\n", "\n", "= t, 0, 1) # 0 = malignant\n", " rec = recall_score(y_te, pred, pos_label=0)\n", " prec = precision_score(y_te, pred, pos_label=0)\n", " if rec >= 0.95:\n", " best = (t, rec, prec)\n", "\n", "t, rec, prec = best\n", "print(f\"threshold={t:.2f} recall={rec:.3f} precision={prec:.3f}\")\n", "`}\n", ">\n", "Using the breast-cancer model from this lesson, treat **malignant** (label 0)\n", "as the positive class of a screening tool. Find the *highest* probability\n", "threshold that still achieves **recall ≥ 0.95** for malignant cases, and\n", "report the precision you get at that threshold. Why would a hospital prefer\n", "this over the default 0.5 cutoff?" ] }, { "cell_type": "code", "id": "cell-0020", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Your solution here" ] }, { "cell_type": "markdown", "id": "cell-0021", "metadata": {}, "source": [ "
✅ Show solution\n", "\n", "```python\n", "import numpy as np\n", "from sklearn.datasets import load_breast_cancer\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.pipeline import make_pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.metrics import precision_score, recall_score\n", "\n", "X, y = load_breast_cancer(return_X_y=True)\n", "X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=42)\n", "model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_tr, y_tr)\n", "\n", "proba_malignant = model.predict_proba(X_te)[:, 0] # class 0 = malignant\n", "\n", "best = None\n", "for t in np.linspace(0.05, 0.95, 19):\n", " pred = np.where(proba_malignant >= t, 0, 1) # 0 = malignant\n", " rec = recall_score(y_te, pred, pos_label=0)\n", " prec = precision_score(y_te, pred, pos_label=0)\n", " if rec >= 0.95:\n", " best = (t, rec, prec)\n", "\n", "t, rec, prec = best\n", "print(f\"threshold={t:.2f} recall={rec:.3f} precision={prec:.3f}\")\n", "```\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "cell-0022", "metadata": {}, "source": [ "Next: the workhorse classifier behind those probability scores — logistic\n", "regression, the sigmoid, and why it needs a different loss than MSE." ] } ] }