{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "id": "cell-0000", "metadata": {}, "source": [ "# Baselines & Benchmarks\n", "\n", "Start every project with a deliberately dumb model — dummy baselines, the class-imbalance accuracy trap, and fair benchmarking with cross-validation.\n", "\n", "*Part of the free [Machine Learning](https://ramadnsyh.dev/courses/machine-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/machine-learning/baselines-benchmarks).*" ] }, { "cell_type": "markdown", "id": "cell-0001", "metadata": {}, "source": [ "\"My model gets 92% accuracy.\" Is that good? You genuinely cannot know without\n", "something to compare against. If guessing the majority class already scores\n", "91%, that model has learned almost nothing. This lesson gives you the habit\n", "that separates rigorous practitioners from hopeful ones: **establish a dumb\n", "baseline first, benchmark everything against it, and only trust improvements\n", "you can measure fairly.**" ] }, { "cell_type": "markdown", "id": "cell-0002", "metadata": {}, "source": [ "## Why every project starts with a dumb model\n", "\n", "A **baseline** is the score of the simplest strategy imaginable — random\n", "guessing, always predicting the most common class, or one hand-written rule\n", "(\"predict survived for every female passenger\"). A **benchmark** is a\n", "stronger reference to beat: a simple standard model, a previous production\n", "system, or a public leaderboard.\n", "\n", "Starting simple pays off repeatedly:\n", "\n", "- It **calibrates every number that follows.** 92% only means something\n", " relative to the baseline's 91% — or 55%.\n", "- It **catches bugs and leaks.** If your first fancy model scores *below* a\n", " dummy, something is broken. If it scores suspiciously near 100%, suspect\n", " leakage.\n", "- It **delivers value early.** A working end-to-end pipeline with a simple\n", " model beats a half-finished sophisticated one, and effort is often not\n", " proportional to payoff — a huge grid search frequently buys a fraction of\n", " a percent." ] }, { "cell_type": "markdown", "id": "cell-0003", "metadata": {}, "source": [ "## DummyClassifier: the honest zero point\n", "\n", "scikit-learn ships baseline models that deliberately ignore the features:\n", "`DummyClassifier` and `DummyRegressor`. Let's pit one against a real model\n", "on the breast cancer dataset:" ] }, { "cell_type": "code", "id": "cell-0004", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from sklearn.datasets import load_breast_cancer\n", "from sklearn.dummy import DummyClassifier\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.neighbors import KNeighborsClassifier\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "\n", "X, y = load_breast_cancer(return_X_y=True)\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " X, y, test_size=0.25, stratify=y, random_state=42\n", ")\n", "\n", "dummy = DummyClassifier(strategy=\"most_frequent\").fit(X_train, y_train)\n", "print(f\"dummy (majority class): {dummy.score(X_test, y_test):.3f}\")\n", "\n", "real = Pipeline([\n", " (\"scaler\", StandardScaler()),\n", " (\"knn\", KNeighborsClassifier(n_neighbors=7)),\n", "]).fit(X_train, y_train)\n", "print(f\"scaled KNN : {real.score(X_test, y_test):.3f}\")" ] }, { "cell_type": "markdown", "id": "cell-0005", "metadata": {}, "source": [ "The dummy scores about 63% *without ever looking at a single feature* —\n", "because 63% of patients in this dataset have benign tumors. So the honest\n", "reading of the KNN result is not \"97% accurate\" but \"34 points above chance.\"\n", "(`DummyRegressor` plays the same role for regression, predicting the training\n", "mean or median; its R² is essentially zero by construction.)" ] }, { "cell_type": "markdown", "id": "cell-0006", "metadata": {}, "source": [ "## The accuracy trap: imbalanced classes\n", "\n", "Now the trap this protects you from. When one class dominates, accuracy\n", "becomes nearly meaningless:" ] }, { "cell_type": "code", "id": "cell-0007", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from sklearn.datasets import make_classification\n", "from sklearn.dummy import DummyClassifier\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.metrics import recall_score\n", "from sklearn.model_selection import train_test_split\n", "\n", "# Fraud-style data: only ~3% positives\n", "X, y = make_classification(n_samples=2000, n_features=8, n_informative=4,\n", " weights=[0.97, 0.03], flip_y=0, random_state=42)\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " X, y, test_size=0.25, stratify=y, random_state=42\n", ")\n", "\n", "dummy = DummyClassifier(strategy=\"most_frequent\").fit(X_train, y_train)\n", "model = LogisticRegression(max_iter=1000).fit(X_train, y_train)\n", "\n", "for name, clf in [(\"dummy\", dummy), (\"logistic\", model)]:\n", " acc = clf.score(X_test, y_test)\n", " rec = recall_score(y_test, clf.predict(X_test))\n", " print(f\"{name:8s} accuracy={acc:.3f} recall={rec:.3f}\")" ] }, { "cell_type": "markdown", "id": "cell-0008", "metadata": {}, "source": [ "The dummy hits about 97% accuracy while catching **zero** fraud cases — its\n", "recall (the fraction of true positives it finds) is 0. Anyone who reports\n", "\"97% accurate\" on this problem is reporting the class ratio, not model skill.\n", "On imbalanced problems, lead with metrics like recall, precision, F1, or\n", "balanced accuracy — the classification module covers them in depth — and\n", "*always* publish the dummy's score next to yours." ] }, { "cell_type": "markdown", "id": "cell-0009", "metadata": {}, "source": [ "## Benchmarking models fairly\n", "\n", "Once the baseline is planted, compare candidate models under identical\n", "conditions: same data, same preprocessing, same cross-validation splits. A\n", "loop over a dictionary of pipelines does it cleanly:" ] }, { "cell_type": "code", "id": "cell-0010", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from sklearn.datasets import load_breast_cancer\n", "from sklearn.dummy import DummyClassifier\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.model_selection import cross_val_score\n", "from sklearn.neighbors import KNeighborsClassifier\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "from sklearn.tree import DecisionTreeClassifier\n", "\n", "X, y = load_breast_cancer(return_X_y=True)\n", "\n", "def scaled(model):\n", " return Pipeline([(\"scaler\", StandardScaler()), (\"model\", model)])\n", "\n", "candidates = {\n", " \"dummy\": DummyClassifier(strategy=\"most_frequent\"),\n", " \"knn\": scaled(KNeighborsClassifier(n_neighbors=7)),\n", " \"logistic\": scaled(LogisticRegression(max_iter=1000)),\n", " \"tree (d=4)\": DecisionTreeClassifier(max_depth=4, random_state=42),\n", "}\n", "\n", "for name, model in candidates.items():\n", " scores = cross_val_score(model, X, y, cv=5)\n", " print(f\"{name:12s} {scores.mean():.3f} ± {scores.std():.3f}\")" ] }, { "cell_type": "markdown", "id": "cell-0011", "metadata": {}, "source": [ "Reading the table properly:\n", "\n", "- **Cross-validation, not a single split**, so no model wins by luck of the\n", " draw. Report the mean *and* the standard deviation.\n", "- **Overlapping error bars mean \"roughly tied.\"** If two models are within a\n", " standard deviation of each other, prefer the simpler, faster one.\n", "- **Each model gets the preprocessing it needs** (scaling for KNN and\n", " logistic regression; trees don't care), bundled in a pipeline so nothing\n", " leaks." ] }, { "cell_type": "markdown", "id": "cell-0012", "metadata": {}, "source": [ "> **Beating the benchmark is a loop, not a step**\n", "> \n", "> When your best model must improve, there are only two levers: better data\n", "> (feature engineering, cleaning, more samples) and better modeling (tuning,\n", "> different algorithms). Change one thing at a time, re-run the same benchmark,\n", "> and keep what helps. Public competitions like Kaggle work exactly this way —\n", "> a shared leaderboard is just a benchmark thousands of people iterate against." ] }, { "cell_type": "markdown", "id": "cell-0013", "metadata": {}, "source": [ "## Keep an experiments log\n", "\n", "Within a day of iterating you will forget which combination produced which\n", "score. Professionals keep a log — a spreadsheet or a plain text file is\n", "enough. For every run record:\n", "\n", "- the **data version and features** used (e.g., \"added is_alone, binned age\"),\n", "- the **model and hyperparameters** (or the grid searched),\n", "- the **validation scheme** (5-fold CV, seed 42) — it must stay constant, or\n", " scores aren't comparable,\n", "- the **score with its spread**, and one line of notes (\"helped, keep\" /\n", " \"no change, revert\").\n", "\n", "Two related habits multiply your speed: build yourself **reusable templates**\n", "for the boilerplate (imports, split, preprocessor, grid search) so each new\n", "experiment costs minutes, and resist tinkering past the point of diminishing\n", "returns — when three experiments in a row move the score by less than its\n", "standard deviation, the remaining gains probably live in the data, not the\n", "model." ] }, { "cell_type": "markdown", "id": "cell-0014", "metadata": {}, "source": [ "### 🏋️ Exercise — Benchmark on an imbalanced problem with the right metric\n", "\n", "18s} {'f1':>18s}\")\n", "for name, model in candidates.items():\n", " acc = cross_val_score(model, X, y, cv=5)\n", " f1 = cross_val_score(model, X, y, cv=5, scoring=\"f1\")\n", " print(f\"{name:10s} {acc.mean():>8.3f} ± {acc.std():.3f} \"\n", " f\"{f1.mean():>8.3f} ± {f1.std():.3f}\")\n", "`}\n", ">\n", "Rebuild the fraud-style imbalanced dataset from this lesson and benchmark\n", "three candidates — a majority-class dummy, scaled KNN, and scaled logistic\n", "regression — with 5-fold cross-validation, reporting **both** accuracy and\n", "F1 (`scoring=\"f1\"`) for each. Which metric actually distinguishes the real\n", "models from the dummy, and which model would you pick?" ] }, { "cell_type": "code", "id": "cell-0015", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Your solution here" ] }, { "cell_type": "markdown", "id": "cell-0016", "metadata": {}, "source": [ "
✅ Show solution\n", "\n", "```python\n", "from sklearn.datasets import make_classification\n", "from sklearn.dummy import DummyClassifier\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.model_selection import cross_val_score\n", "from sklearn.neighbors import KNeighborsClassifier\n", "from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "\n", "X, y = make_classification(n_samples=2000, n_features=8, n_informative=4,\n", " weights=[0.97, 0.03], flip_y=0, random_state=42)\n", "\n", "def scaled(model):\n", " return Pipeline([(\"scaler\", StandardScaler()), (\"model\", model)])\n", "\n", "candidates = {\n", " \"dummy\": DummyClassifier(strategy=\"most_frequent\"),\n", " \"knn\": scaled(KNeighborsClassifier(n_neighbors=7)),\n", " \"logistic\": scaled(LogisticRegression(max_iter=1000)),\n", "}\n", "\n", "print(f\"{'model':10s} {'accuracy':>18s} {'f1':>18s}\")\n", "for name, model in candidates.items():\n", " acc = cross_val_score(model, X, y, cv=5)\n", " f1 = cross_val_score(model, X, y, cv=5, scoring=\"f1\")\n", " print(f\"{name:10s} {acc.mean():>8.3f} ± {acc.std():.3f} \"\n", " f\"{f1.mean():>8.3f} ± {f1.std():.3f}\")\n", "```\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "cell-0017", "metadata": {}, "source": [ "That wraps the foundations — next module: regression, where you'll fit your\n", "first line with gradient descent and meet the loss functions behind almost\n", "every model." ] } ] }