SVMs in Practice: Classification & Regression

You know what an SVM is — a maximum-margin classifier with a kernel-shaped boundary. This lesson is about using one well: which hyperparameters actually matter, how to tune them without fooling yourself, why an unscaled SVM is a broken SVM, and how the same margin idea turns into a regressor. We'll close with the honest question every practitioner faces: when is an SVM the right tool at all?

The three knobs that matter

For SVC, three hyperparameters do almost all the work:

• kernel — the shape vocabulary. "linear" for straight boundaries, "rbf" (the default) for smooth curves. Polynomial and sigmoid kernels exist but are rarely the winner.
• C — the price of margin violations. Small C = wide, tolerant margin (underfit risk); large C = strict fitting of every training point (overfit risk).
• gamma — RBF only: the reach of each support vector. Small gamma = big picture; large gamma = detail-obsessed islands (overfit risk).

C and gamma interact — a large-C, large-gamma model is doubly prone to memorizing noise — so they should be tuned together, typically over a logarithmic grid like 0.01, 0.1, 1, 10, 100.

Scaling is non-optional

Before any tuning: the RBF kernel is built on Euclidean distance. If one feature ranges over thousands (a tumor's area) and another over fractions (its smoothness), distance is effectively computed on the big feature alone — the rest become invisible. Trees don't care about this; SVMs and KNN care enormously. Watch the same model with and without a scaler:

Python — runs in your browser

Run

from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.svm import SVC from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler X, y = load_breast_cancer(return_X_y=True) X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=42) raw = SVC().fit(X_tr, y_tr) scaled = make_pipeline(StandardScaler(), SVC()).fit(X_tr, y_tr) print(f"SVC without scaling: {raw.score(X_te, y_te):.3f}") print(f"SVC with scaling : {scaled.score(X_te, y_te):.3f}")

From 0.92 to 0.98 — the error rate drops almost fourfold, from the same algorithm, just by standardizing the features first. (And this is with scikit-learn's gamma="scale" default already partially compensating; with a fixed gamma the unscaled model collapses much harder.) Rule: an SVM pipeline starts with a scaler. Always.

Tuning C and gamma with GridSearchCV

Because scaling is part of the model, it must live inside the cross-validation — otherwise the scaler peeks at validation data and your scores are quietly optimistic. A Pipeline inside GridSearchCV gets this right automatically. Parameters are addressed as stepname__param:

Python — runs in your browser

Run

from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split, GridSearchCV from sklearn.svm import SVC from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler X, y = load_breast_cancer(return_X_y=True) X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=42) pipe = make_pipeline(StandardScaler(), SVC()) grid = { "svc__C": [0.1, 1, 10], "svc__gamma": [0.01, 0.1, 1], } search = GridSearchCV(pipe, grid, cv=3).fit(X_tr, y_tr) print("best params:", search.best_params_) print(f"best CV score : {search.best_score_:.3f}") print(f"test score : {search.score(X_te, y_te):.3f}")

Nine combinations, three folds each — 27 quick fits. In a real project you'd use a wider grid (np.logspace(-3, 3, 7) for both C and gamma is the classic choice) and often include "kernel": ["linear", "rbf"] as a third axis. Notice the pattern in the winner: moderate C, small gamma — smooth boundaries generalize.

SVR: regression with an epsilon tube

The margin idea flips neatly into regression. Support Vector Regression fits a curve surrounded by a tube of half-width epsilon (ε), and the loss is deliberately indifferent: any point inside the tube costs nothing, no matter where exactly it sits. Only points on or outside the tube — the support vectors — pull on the fit.

So epsilon controls how much detail the model bothers to chase:

Python — runs in your browser

Run

import numpy as np import matplotlib.pyplot as plt from sklearn.svm import SVR rng = np.random.default_rng(42) x = np.sort(rng.uniform(0, 6, 80)) y = np.sin(x) + rng.normal(0, 0.15, 80) X = x.reshape(-1, 1) x_plot = np.linspace(0, 6, 300).reshape(-1, 1) plt.scatter(x, y, s=15, color="gray", alpha=0.6, label="data") for eps, color in zip([0.05, 0.2, 0.8], ["tab:blue", "tab:green", "tab:red"]): svr = SVR(kernel="rbf", C=10, epsilon=eps).fit(X, y) plt.plot(x_plot, svr.predict(x_plot), color=color, label=f"eps={eps} ({len(svr.support_)} SVs)") plt.legend() plt.title("SVR: bigger epsilon = wider tube = fewer support vectors") plt.show()

With ε = 0.05 the tube is skinny, 61 of 80 points stick out, and the curve wiggles after every one of them. At ε = 0.2 the tube swallows the noise and the fit hugs the true sine wave with only 15 support vectors. At ε = 0.8 the tube is so wide that just 2 points constrain it — the model flattens out and underfits. C plays the same role as in classification (how hard to punish the points outside the tube), and gamma still shapes the RBF curve.

When to reach for an SVM — and when not to

SVMs shine when:

• the dataset is small to medium (hundreds to tens of thousands of rows) and features are numeric,
• the data is high-dimensional relative to its size (text vectors, gene expression) — margins behave well there, and a linear kernel is often enough,
• you want a smooth, flexible boundary without designing features for it.

Prefer trees and ensembles (next module) when features are a mix of categorical and numeric, when you'd rather skip scaling, or when you need feature importances out of the box.

The hard limit is scale. Kernel SVM training grows roughly quadratically with the number of samples — at hundreds of thousands of rows, fitting (and grid-searching!) becomes painful. When that happens, drop the kernel: use LinearSVC, or SGDClassifier(loss="hinge"), which train a linear SVM in time proportional to the data size and handle millions of rows. You lose curved boundaries but keep the margin philosophy.

Check your understanding

5 questions · free

1. Q1.Why must feature scaling happen for an RBF-kernel SVM?

   AScaling makes training deterministicBThe RBF kernel is distance-based, so a large-range feature drowns out all the othersCSVC raises an error on unscaled inputDScaling removes outliers from the data

2. Q2.Why should the StandardScaler live inside the Pipeline passed to GridSearchCV, rather than being applied to the whole dataset first?

   AIt runs faster inside the pipelineBGridSearchCV cannot handle pre-scaled dataCScaling first leaks validation-fold statistics into training, making CV scores optimisticDThe scaler needs access to the parameter grid

3. Q3.In SVR, what happens to training points that fall strictly inside the epsilon tube?

   AThey contribute zero loss and do not affect the fitted curveBThey are the support vectorsCThey are removed from the datasetDThey get quadratic loss like in ridge regression

4. Q4.You increase epsilon in SVR from 0.1 to 1.0. What do you expect?

   AMore support vectors and a wigglier curveBFewer support vectors and a smoother (possibly underfit) curveCNo change — epsilon only affects prediction speedDThe kernel switches to linear

5. Q5.You have 2 million rows of numeric data and want an SVM-style classifier. What is the practical choice?

   ASVC with an RBF kernel and a large grid searchBSVC with gamma set very high to speed up trainingCLinearSVC or SGDClassifier with hinge loss — linear SVMs that scale to millions of rowsDSVR with a wide epsilon

Exercise: Add the kernel to the search

Extend the lesson's grid search to also try the linear kernel. Use a list of two parameter grids so the linear kernel is searched over C only, while the RBF kernel is searched over C and gamma. Which kernel wins on the breast-cancer data, and by how much? What does the small gap tell you about this dataset's geometry?

Next module: decision trees and ensembles — models that ask a sequence of simple questions, need no scaling at all, and combine into some of the strongest tabular-data learners around.