Support Vector Machines: Maximum Margin

Logistic regression draws a boundary by balancing probabilities across all the training points. Support vector machines take the opposite stance: most points don't matter at all — only the few sitting closest to the enemy class do. From that one idea come maximum margins, the C parameter, and (with the kernel trick) some of the most flexible decision boundaries in classical ML.

Many lines separate the data — which one is best?

Take two well-separated clouds of points. Infinitely many straight lines classify the training set perfectly: one hugging the red class, one hugging the blue, and everything in between. They're all equally "correct" on the training data, but they will not generalize equally — a line that grazes the red cluster will misclassify the very next red point that lands slightly farther out.

The SVM answer: pick the line with the widest margin — the biggest possible buffer zone between the boundary and the nearest point of each class. Think of it as fitting the widest possible street between the classes and drawing the boundary down the middle. Try it yourself — drag the C slider and watch the street change:

C1support vectors6margin 2/‖w‖1.24train acc96%

Penalty C1

Before it got its modern name, this algorithm was literally called the maximum margin classifier. A wide margin is a safety buffer: new samples from each class have room to scatter without crossing the line.

Support vectors: the only points that matter

Look at the highlighted points in the visualization — the ones sitting exactly on the edges of the street. Those are the support vectors, and they alone define the boundary. Every other training point could be deleted and the SVM would draw exactly the same line.

That's the machine in "support vector machine": an algorithm whose whole job is to find the handful of critical boundary points and let them "support" the margin. It has two nice consequences:

• Sparsity — the fitted model only stores the support vectors, not the whole training set (unlike KNN).
• Robustness to easy points — adding a thousand more obviously-red points deep inside red territory changes nothing. Compare that to linear regression on labels, where every point tugs on the fit.

Soft margins and the C parameter

Real classes overlap. If the margin had to be perfectly clean, one mislabeled point could force a terrible boundary — or make separation impossible. So practical SVMs use a soft margin: points are allowed inside the street, or even on the wrong side, but each violation costs a penalty. The hyperparameter C prices that penalty:

• Small C — violations are cheap → the SVM keeps the street wide and tolerant, accepting some training mistakes for a smoother boundary. Too small and it underfits.
• Large C — violations are expensive → the SVM gets strict, narrowing the street to classify every training point correctly. Too large and it contorts around noise — overfitting.

Go back to the slider above and verify both regimes: low C recruits many support vectors into a wide street; high C shrinks the street until only a few points touch it. You can see the same effect numerically:

Python — runs in your browser

Run

from sklearn.datasets import make_blobs from sklearn.svm import SVC X, y = make_blobs(n_samples=120, centers=2, cluster_std=1.6, random_state=7) print(" C support vectors train accuracy") for C in [0.01, 1, 100]: svm = SVC(kernel="linear", C=C).fit(X, y) n_sv = svm.n_support_.sum() print(f"{C:>6} {n_sv:>3} {svm.score(X, y):.3f}")

With C = 0.01 a quarter of the dataset ends up inside the margin (all support vectors); with C = 100 just three points pin down a razor-thin street.

When no line works: the kernel trick

Tolerance can't save you when the data simply isn't linearly separable — think of one class forming a ring around the other, or two interleaving crescents. No straight line, at any C, will do.

The SVM's escape is a change of perspective: map the data into a higher dimension where a line (well, a plane) does separate it. A ring around a cluster is inseparable in 2-D — but add a third axis measuring "distance from the center" and the inner cluster floats above the ring, trivially split by a flat plane. Projected back down to 2-D, that flat plane looks like a circle.

The kernel trick is what makes this affordable: the SVM never actually computes the high-dimensional coordinates. It only ever needs similarities between pairs of points, and a kernel function computes those similarities as if the mapping had been done. The most popular choice is the RBF (radial basis function) kernel, a Gaussian bump: two points are highly similar when close, and their similarity decays smoothly to zero with distance. An RBF-SVM boundary is, in effect, built from soft spheres of influence around each support vector — which is why it can trace almost any smooth shape.

Gamma: big picture vs detail-oriented

The RBF kernel adds a second knob, gamma (γ), which sets how far each support vector's influence reaches:

• Small γ — wide influence → the model sees the big picture: smooth, gently curved boundaries. Too small ≈ almost linear (underfits).
• Large γ — tiny influence → the model becomes detail-oriented: the boundary wraps tightly around individual points, forming islands around every training sample. Classic overfitting.

Avoid very large gamma values — and note that gamma is a distance scale, so feature scaling directly affects it. Scaling is known to help SVMs a lot; the next lesson makes that concrete.

Linear vs RBF on two moons

Let's see both kernels on make_moons — two interleaved crescents that no straight line can separate:

Python — runs in your browser

Run

import numpy as np import matplotlib.pyplot as plt from sklearn.datasets import make_moons from sklearn.model_selection import train_test_split from sklearn.svm import SVC X, y = make_moons(n_samples=200, noise=0.25, random_state=42) X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=42) xx, yy = np.meshgrid(np.linspace(-1.8, 2.8, 200), np.linspace(-1.3, 1.8, 200)) grid = np.c_[xx.ravel(), yy.ravel()] fig, axes = plt.subplots(1, 2, figsize=(9, 3.6), sharey=True) for ax, kernel in zip(axes, ["linear", "rbf"]): svm = SVC(kernel=kernel).fit(X_tr, y_tr) Z = svm.predict(grid).reshape(xx.shape) ax.contourf(xx, yy, Z, alpha=0.25, cmap="coolwarm") ax.scatter(X_te[:, 0], X_te[:, 1], c=y_te, cmap="coolwarm", edgecolors="k", s=25) ax.set_title(f"{kernel}: test acc = {svm.score(X_te, y_te):.2f}") plt.show()

The linear kernel does its honest best — a straight cut through the middle, around 82% — while the RBF kernel bends its boundary along the crescents and climbs past 90%. Same algorithm, different similarity function.

Check your understanding

5 questions · free

1. Q1.Among all lines that perfectly separate the training data, why does the SVM prefer the one with the widest margin?

   AIt is the fastest one to computeBThe buffer zone gives new, slightly-scattered points the best chance of landing on the correct sideCIt always passes through the originDIt minimizes the number of features used

2. Q2.You delete a training point that is NOT a support vector and refit the SVM. What happens to the decision boundary?

   AIt shifts toward the deleted point's classBIt becomes widerCNothing — only support vectors define the boundaryDThe model can no longer be fit

3. Q3.In an SVM, what does a SMALL value of C do?

   AMakes margin violations expensive, forcing a narrow, strict marginBMakes violations cheap, allowing a wide, tolerant margin that may accept some training errorsCIncreases the influence radius of each support vectorDSwitches the kernel from linear to RBF

4. Q4.What is the kernel trick?

   ARandomly re-initializing the SVM to escape local minimaBComputing high-dimensional similarities directly, so data can be separated as if it were mapped to a higher dimension — without ever computing the mappingCReducing the data to two dimensions before fittingDAveraging several linear SVMs trained on bootstrapped data

5. Q5.An RBF-SVM's boundary forms tiny islands around individual training points and gets 100% training accuracy but poor test accuracy. Which change is most likely to help?

   AIncrease gammaBDecrease gammaCRemove the support vectorsDSwitch to a linear kernel with larger C

Exercise: Watch gamma overfit in real time

Using the same make_moons train/test split as in the lesson, fit RBF SVMs with gamma set to 0.1, 1, 10, and 100. Print train and test accuracy for each. At which gamma does the model generalize best, and where does memorization start? Explain the pattern in terms of each support vector's radius of influence.

Next: putting SVMs to work — tuning C and gamma with grid search, why scaling is non-negotiable, and the regression cousin SVR with its epsilon tube.