{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "id": "cell-0000", "metadata": {}, "source": [ "# SVMs in Practice: Classification & Regression\n", "\n", "Tune C, gamma, and the kernel with GridSearchCV, see why scaling is non-negotiable, meet SVR's epsilon tube, and learn when to reach for SVMs at all.\n", "\n", "*Part of the free [Machine Learning](https://ramadnsyh.dev/courses/machine-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/machine-learning/svm-in-practice).*" ] }, { "cell_type": "markdown", "id": "cell-0001", "metadata": {}, "source": [ "You know what an SVM *is* — a maximum-margin classifier with a kernel-shaped\n", "boundary. This lesson is about using one well: which hyperparameters actually\n", "matter, how to tune them without fooling yourself, why an unscaled SVM is a\n", "broken SVM, and how the same margin idea turns into a regressor. We'll close\n", "with the honest question every practitioner faces: when is an SVM the right\n", "tool at all?" ] }, { "cell_type": "markdown", "id": "cell-0002", "metadata": {}, "source": [ "## The three knobs that matter\n", "\n", "For `SVC`, three hyperparameters do almost all the work:\n", "\n", "- **kernel** — the shape vocabulary. `\"linear\"` for straight boundaries,\n", " `\"rbf\"` (the default) for smooth curves. Polynomial and sigmoid kernels\n", " exist but are rarely the winner.\n", "- **C** — the price of margin violations. Small C = wide, tolerant margin\n", " (underfit risk); large C = strict fitting of every training point (overfit\n", " risk).\n", "- **gamma** — RBF only: the reach of each support vector. Small gamma = big\n", " picture; large gamma = detail-obsessed islands (overfit risk).\n", "\n", "C and gamma interact — a large-C, large-gamma model is doubly prone to\n", "memorizing noise — so they should be tuned *together*, typically over a\n", "logarithmic grid like 0.01, 0.1, 1, 10, 100." ] }, { "cell_type": "markdown", "id": "cell-0003", "metadata": {}, "source": [ "## Scaling is non-optional\n", "\n", "Before any tuning: the RBF kernel is built on **Euclidean distance**. If one\n", "feature ranges over thousands (a tumor's area) and another over fractions\n", "(its smoothness), distance is effectively computed on the big feature alone —\n", "the rest become invisible. Trees don't care about this; SVMs and KNN care\n", "enormously. Watch the same model with and without a scaler:" ] }, { "cell_type": "code", "id": "cell-0004", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from sklearn.datasets import load_breast_cancer\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.svm import SVC\n", "from sklearn.pipeline import make_pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "\n", "X, y = load_breast_cancer(return_X_y=True)\n", "X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=42)\n", "\n", "raw = SVC().fit(X_tr, y_tr)\n", "scaled = make_pipeline(StandardScaler(), SVC()).fit(X_tr, y_tr)\n", "\n", "print(f\"SVC without scaling: {raw.score(X_te, y_te):.3f}\")\n", "print(f\"SVC with scaling : {scaled.score(X_te, y_te):.3f}\")" ] }, { "cell_type": "markdown", "id": "cell-0005", "metadata": {}, "source": [ "From 0.92 to 0.98 — the error rate drops almost fourfold, from the same\n", "algorithm, just by standardizing the features first. (And this is with\n", "scikit-learn's `gamma=\"scale\"` default already partially compensating; with a\n", "fixed gamma the unscaled model collapses much harder.) Rule: **an SVM\n", "pipeline starts with a scaler.** Always." ] }, { "cell_type": "markdown", "id": "cell-0006", "metadata": {}, "source": [ "## Tuning C and gamma with GridSearchCV\n", "\n", "Because scaling is part of the model, it must live *inside* the\n", "cross-validation — otherwise the scaler peeks at validation data and your\n", "scores are quietly optimistic. A `Pipeline` inside `GridSearchCV` gets this\n", "right automatically. Parameters are addressed as `stepname__param`:" ] }, { "cell_type": "code", "id": "cell-0007", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from sklearn.datasets import load_breast_cancer\n", "from sklearn.model_selection import train_test_split, GridSearchCV\n", "from sklearn.svm import SVC\n", "from sklearn.pipeline import make_pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "\n", "X, y = load_breast_cancer(return_X_y=True)\n", "X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=42)\n", "\n", "pipe = make_pipeline(StandardScaler(), SVC())\n", "grid = {\n", " \"svc__C\": [0.1, 1, 10],\n", " \"svc__gamma\": [0.01, 0.1, 1],\n", "}\n", "search = GridSearchCV(pipe, grid, cv=3).fit(X_tr, y_tr)\n", "\n", "print(\"best params:\", search.best_params_)\n", "print(f\"best CV score : {search.best_score_:.3f}\")\n", "print(f\"test score : {search.score(X_te, y_te):.3f}\")" ] }, { "cell_type": "markdown", "id": "cell-0008", "metadata": {}, "source": [ "Nine combinations, three folds each — 27 quick fits. In a real project you'd\n", "use a wider grid (`np.logspace(-3, 3, 7)` for both C and gamma is the classic\n", "choice) and often include `\"kernel\": [\"linear\", \"rbf\"]` as a third axis.\n", "Notice the pattern in the winner: moderate C, small gamma — smooth boundaries\n", "generalize." ] }, { "cell_type": "markdown", "id": "cell-0009", "metadata": {}, "source": [ "> **Imbalanced classes? Two extra moves**\n", "> \n", "> A 99%-negative fraud dataset will hand you a 99%-accurate SVM that catches\n", "> nothing — the same accuracy trap from the metrics lesson. Pass\n", "> `scoring=\"f1\"` to GridSearchCV so tuning optimizes something honest, and try\n", "> `class_weight=\"balanced\"` in SVC, which raises the misclassification price\n", "> for the rare class." ] }, { "cell_type": "markdown", "id": "cell-0010", "metadata": {}, "source": [ "## SVR: regression with an epsilon tube\n", "\n", "The margin idea flips neatly into regression. **Support Vector Regression**\n", "fits a curve surrounded by a tube of half-width **epsilon (ε)**, and the loss\n", "is deliberately indifferent: any point *inside* the tube costs nothing, no\n", "matter where exactly it sits. Only points on or outside the tube — the\n", "support vectors — pull on the fit.\n", "\n", "So epsilon controls how much detail the model bothers to chase:" ] }, { "cell_type": "code", "id": "cell-0011", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn.svm import SVR\n", "\n", "rng = np.random.default_rng(42)\n", "x = np.sort(rng.uniform(0, 6, 80))\n", "y = np.sin(x) + rng.normal(0, 0.15, 80)\n", "X = x.reshape(-1, 1)\n", "x_plot = np.linspace(0, 6, 300).reshape(-1, 1)\n", "\n", "plt.scatter(x, y, s=15, color=\"gray\", alpha=0.6, label=\"data\")\n", "for eps, color in zip([0.05, 0.2, 0.8], [\"tab:blue\", \"tab:green\", \"tab:red\"]):\n", " svr = SVR(kernel=\"rbf\", C=10, epsilon=eps).fit(X, y)\n", " plt.plot(x_plot, svr.predict(x_plot), color=color,\n", " label=f\"eps={eps} ({len(svr.support_)} SVs)\")\n", "plt.legend()\n", "plt.title(\"SVR: bigger epsilon = wider tube = fewer support vectors\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "cell-0012", "metadata": {}, "source": [ "With ε = 0.05 the tube is skinny, 61 of 80 points stick out, and the curve\n", "wiggles after every one of them. At ε = 0.2 the tube swallows the noise and\n", "the fit hugs the true sine wave with only 15 support vectors. At ε = 0.8 the\n", "tube is so wide that just 2 points constrain it — the model flattens out and\n", "underfits. C plays the same role as in classification (how hard to punish the\n", "points outside the tube), and gamma still shapes the RBF curve." ] }, { "cell_type": "markdown", "id": "cell-0013", "metadata": {}, "source": [ "## When to reach for an SVM — and when not to\n", "\n", "SVMs shine when:\n", "\n", "- the dataset is **small to medium** (hundreds to tens of thousands of rows)\n", " and features are numeric,\n", "- the data is **high-dimensional** relative to its size (text vectors, gene\n", " expression) — margins behave well there, and a linear kernel is often\n", " enough,\n", "- you want a **smooth, flexible boundary** without designing features for it.\n", "\n", "Prefer trees and ensembles (next module) when features are a mix of\n", "categorical and numeric, when you'd rather skip scaling, or when you need\n", "feature importances out of the box.\n", "\n", "The hard limit is **scale**. Kernel SVM training grows roughly quadratically\n", "with the number of samples — at hundreds of thousands of rows, fitting (and\n", "grid-searching!) becomes painful. When that happens, drop the kernel: use\n", "`LinearSVC`, or `SGDClassifier(loss=\"hinge\")`, which train a linear SVM in\n", "time proportional to the data size and handle millions of rows. You lose\n", "curved boundaries but keep the margin philosophy." ] }, { "cell_type": "markdown", "id": "cell-0014", "metadata": {}, "source": [ "### 🏋️ Exercise — Add the kernel to the search\n", "\n", "Extend the lesson's grid search to also try the **linear kernel**. Use a list\n", "of two parameter grids so the linear kernel is searched over C only, while\n", "the RBF kernel is searched over C and gamma. Which kernel wins on the\n", "breast-cancer data, and by how much? What does the small gap tell you about\n", "this dataset's geometry?" ] }, { "cell_type": "code", "id": "cell-0015", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Your solution here" ] }, { "cell_type": "markdown", "id": "cell-0016", "metadata": {}, "source": [ "
✅ Show solution\n", "\n", "```python\n", "from sklearn.datasets import load_breast_cancer\n", "from sklearn.model_selection import train_test_split, GridSearchCV\n", "from sklearn.svm import SVC\n", "from sklearn.pipeline import make_pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "\n", "X, y = load_breast_cancer(return_X_y=True)\n", "X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=42)\n", "\n", "pipe = make_pipeline(StandardScaler(), SVC())\n", "grid = [\n", " {\"svc__kernel\": [\"linear\"], \"svc__C\": [0.1, 1, 10]},\n", " {\"svc__kernel\": [\"rbf\"], \"svc__C\": [0.1, 1, 10],\n", " \"svc__gamma\": [0.01, 0.1, 1]},\n", "]\n", "search = GridSearchCV(pipe, grid, cv=3).fit(X_tr, y_tr)\n", "\n", "print(\"best params:\", search.best_params_)\n", "print(f\"best CV score : {search.best_score_:.3f}\")\n", "print(f\"test score : {search.score(X_te, y_te):.3f}\")\n", "\n", "# On this dataset linear and rbf land very close — a reminder that\n", "# scaled breast-cancer data is nearly linearly separable, and that the\n", "# simplest kernel that matches the data is the one to prefer.\n", "```\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "cell-0017", "metadata": {}, "source": [ "Next module: decision trees and ensembles — models that ask a sequence of\n", "simple questions, need no scaling at all, and combine into some of the\n", "strongest tabular-data learners around." ] } ] }