{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "id": "cell-0000", "metadata": {}, "source": [ "# Support Vector Machines: Maximum Margin\n", "\n", "Why the widest street between classes beats any other separating line, what support vectors are, and how the kernel trick bends straight lines around curved data.\n", "\n", "*Part of the free [Machine Learning](https://ramadnsyh.dev/courses/machine-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/machine-learning/support-vector-machines).*" ] }, { "cell_type": "markdown", "id": "cell-0001", "metadata": {}, "source": [ "Logistic regression draws a boundary by balancing probabilities across *all*\n", "the training points. Support vector machines take the opposite stance: most\n", "points don't matter at all — only the few sitting closest to the enemy class\n", "do. From that one idea come maximum margins, the C parameter, and (with the\n", "kernel trick) some of the most flexible decision boundaries in classical ML." ] }, { "cell_type": "markdown", "id": "cell-0002", "metadata": {}, "source": [ "## Many lines separate the data — which one is best?\n", "\n", "Take two well-separated clouds of points. Infinitely many straight lines\n", "classify the training set perfectly: one hugging the red class, one hugging\n", "the blue, and everything in between. They're all equally \"correct\" on the\n", "training data, but they will not generalize equally — a line that grazes the\n", "red cluster will misclassify the very next red point that lands slightly\n", "farther out.\n", "\n", "The SVM answer: pick the line with the **widest margin** — the biggest\n", "possible buffer zone between the boundary and the nearest point of each class.\n", "Think of it as fitting the widest possible street between the classes and\n", "drawing the boundary down the middle. Try it yourself — drag the C slider and\n", "watch the street change:" ] }, { "cell_type": "markdown", "id": "cell-0003", "metadata": {}, "source": [ "> 🎛️ **Interactive demo** — this section has a hands-on visualization in the web version of this lesson: [open it here](https://ramadnsyh.dev/courses/machine-learning/support-vector-machines)." ] }, { "cell_type": "markdown", "id": "cell-0004", "metadata": {}, "source": [ "Before it got its modern name, this algorithm was literally called the\n", "**maximum margin classifier**. A wide margin is a safety buffer: new samples\n", "from each class have room to scatter without crossing the line." ] }, { "cell_type": "markdown", "id": "cell-0005", "metadata": {}, "source": [ "## Support vectors: the only points that matter\n", "\n", "Look at the highlighted points in the visualization — the ones sitting exactly\n", "on the edges of the street. Those are the **support vectors**, and they alone\n", "define the boundary. Every other training point could be deleted and the SVM\n", "would draw *exactly the same line*.\n", "\n", "That's the machine in \"support vector machine\": an algorithm whose whole job\n", "is to find the handful of critical boundary points and let them \"support\" the\n", "margin. It has two nice consequences:\n", "\n", "- **Sparsity** — the fitted model only stores the support vectors, not the\n", " whole training set (unlike KNN).\n", "- **Robustness to easy points** — adding a thousand more obviously-red points\n", " deep inside red territory changes nothing. Compare that to linear regression\n", " on labels, where every point tugs on the fit." ] }, { "cell_type": "markdown", "id": "cell-0006", "metadata": {}, "source": [ "## Soft margins and the C parameter\n", "\n", "Real classes overlap. If the margin had to be perfectly clean, one mislabeled\n", "point could force a terrible boundary — or make separation impossible. So\n", "practical SVMs use a **soft margin**: points are allowed inside the street, or\n", "even on the wrong side, but each violation costs a penalty. The hyperparameter\n", "**C** prices that penalty:\n", "\n", "- **Small C** — violations are cheap → the SVM keeps the street *wide* and\n", " tolerant, accepting some training mistakes for a smoother boundary. Too\n", " small and it underfits.\n", "- **Large C** — violations are expensive → the SVM gets *strict*, narrowing\n", " the street to classify every training point correctly. Too large and it\n", " contorts around noise — overfitting.\n", "\n", "Go back to the slider above and verify both regimes: low C recruits many\n", "support vectors into a wide street; high C shrinks the street until only a few\n", "points touch it. You can see the same effect numerically:" ] }, { "cell_type": "code", "id": "cell-0007", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from sklearn.datasets import make_blobs\n", "from sklearn.svm import SVC\n", "\n", "X, y = make_blobs(n_samples=120, centers=2, cluster_std=1.6, random_state=7)\n", "\n", "print(\" C support vectors train accuracy\")\n", "for C in [0.01, 1, 100]:\n", " svm = SVC(kernel=\"linear\", C=C).fit(X, y)\n", " n_sv = svm.n_support_.sum()\n", " print(f\"{C:>6} {n_sv:>3} {svm.score(X, y):.3f}\")" ] }, { "cell_type": "markdown", "id": "cell-0008", "metadata": {}, "source": [ "With C = 0.01 a quarter of the dataset ends up inside the margin (all support\n", "vectors); with C = 100 just three points pin down a razor-thin street." ] }, { "cell_type": "markdown", "id": "cell-0009", "metadata": {}, "source": [ "## When no line works: the kernel trick\n", "\n", "Tolerance can't save you when the data simply isn't linearly separable — think\n", "of one class forming a ring around the other, or two interleaving crescents.\n", "No straight line, at any C, will do.\n", "\n", "The SVM's escape is a change of perspective: **map the data into a higher\n", "dimension where a line (well, a plane) *does* separate it**. A ring around a\n", "cluster is inseparable in 2-D — but add a third axis measuring \"distance from\n", "the center\" and the inner cluster floats above the ring, trivially split by a\n", "flat plane. Projected back down to 2-D, that flat plane looks like a circle.\n", "\n", "The **kernel trick** is what makes this affordable: the SVM never actually\n", "computes the high-dimensional coordinates. It only ever needs *similarities*\n", "between pairs of points, and a kernel function computes those similarities as\n", "if the mapping had been done. The most popular choice is the **RBF (radial\n", "basis function) kernel**, a Gaussian bump: two points are highly similar when\n", "close, and their similarity decays smoothly to zero with distance. An RBF-SVM\n", "boundary is, in effect, built from soft spheres of influence around each\n", "support vector — which is why it can trace almost any smooth shape." ] }, { "cell_type": "markdown", "id": "cell-0010", "metadata": {}, "source": [ "## Gamma: big picture vs detail-oriented\n", "\n", "The RBF kernel adds a second knob, **gamma (γ)**, which sets how far each\n", "support vector's influence reaches:\n", "\n", "- **Small γ** — wide influence → the model sees the *big picture*: smooth,\n", " gently curved boundaries. Too small ≈ almost linear (underfits).\n", "- **Large γ** — tiny influence → the model becomes *detail-oriented*: the\n", " boundary wraps tightly around individual points, forming islands around\n", " every training sample. Classic overfitting.\n", "\n", "Avoid very large gamma values — and note that gamma is a *distance* scale, so\n", "feature scaling directly affects it. Scaling is known to help SVMs a lot; the\n", "next lesson makes that concrete." ] }, { "cell_type": "markdown", "id": "cell-0011", "metadata": {}, "source": [ "## Linear vs RBF on two moons\n", "\n", "Let's see both kernels on `make_moons` — two interleaved crescents that no\n", "straight line can separate:" ] }, { "cell_type": "code", "id": "cell-0012", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn.datasets import make_moons\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.svm import SVC\n", "\n", "X, y = make_moons(n_samples=200, noise=0.25, random_state=42)\n", "X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)\n", "\n", "xx, yy = np.meshgrid(np.linspace(-1.8, 2.8, 200),\n", " np.linspace(-1.3, 1.8, 200))\n", "grid = np.c_[xx.ravel(), yy.ravel()]\n", "\n", "fig, axes = plt.subplots(1, 2, figsize=(9, 3.6), sharey=True)\n", "for ax, kernel in zip(axes, [\"linear\", \"rbf\"]):\n", " svm = SVC(kernel=kernel).fit(X_tr, y_tr)\n", " Z = svm.predict(grid).reshape(xx.shape)\n", " ax.contourf(xx, yy, Z, alpha=0.25, cmap=\"coolwarm\")\n", " ax.scatter(X_te[:, 0], X_te[:, 1], c=y_te, cmap=\"coolwarm\",\n", " edgecolors=\"k\", s=25)\n", " ax.set_title(f\"{kernel}: test acc = {svm.score(X_te, y_te):.2f}\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "cell-0013", "metadata": {}, "source": [ "The linear kernel does its honest best — a straight cut through the middle,\n", "around 82% — while the RBF kernel bends its boundary along the crescents and\n", "climbs past 90%. Same algorithm, different similarity function." ] }, { "cell_type": "markdown", "id": "cell-0014", "metadata": {}, "source": [ "### 🏋️ Exercise — Watch gamma overfit in real time\n", "\n", "5} {svm.score(X_tr, y_tr):.3f} {svm.score(X_te, y_te):.3f}\")\n", "\n", "# test accuracy peaks around gamma 1-10, then at gamma=100 the train\n", "# accuracy hits 1.0 while test accuracy collapses to 0.80 -> each support\n", "# vector's influence has shrunk to a tiny island and the model memorizes noise.\n", "`}\n", ">\n", "Using the same `make_moons` train/test split as in the lesson, fit RBF SVMs\n", "with gamma set to 0.1, 1, 10, and 100. Print train and test accuracy for each.\n", "At which gamma does the model generalize best, and where does memorization\n", "start? Explain the pattern in terms of each support vector's radius of\n", "influence." ] }, { "cell_type": "code", "id": "cell-0015", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Your solution here" ] }, { "cell_type": "markdown", "id": "cell-0016", "metadata": {}, "source": [ "
✅ Show solution\n", "\n", "```python\n", "from sklearn.datasets import make_moons\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.svm import SVC\n", "\n", "X, y = make_moons(n_samples=200, noise=0.25, random_state=42)\n", "X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42)\n", "\n", "print(\"gamma train acc test acc\")\n", "for g in [0.1, 1, 10, 100]:\n", " svm = SVC(kernel=\"rbf\", gamma=g).fit(X_tr, y_tr)\n", " print(f\"{g:>5} {svm.score(X_tr, y_tr):.3f} {svm.score(X_te, y_te):.3f}\")\n", "\n", "# test accuracy peaks around gamma 1-10, then at gamma=100 the train\n", "# accuracy hits 1.0 while test accuracy collapses to 0.80 -> each support\n", "# vector's influence has shrunk to a tiny island and the model memorizes noise.\n", "```\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "cell-0017", "metadata": {}, "source": [ "Next: putting SVMs to work — tuning C and gamma with grid search, why scaling\n", "is non-negotiable, and the regression cousin SVR with its epsilon tube." ] } ] }