K-Nearest Neighbors

K-Nearest Neighbors (KNN) is machine learning at its most human: to classify something new, look at the most similar things you've seen before and go with the majority. You already do this — guessing a stranger's home town from their accent, or a movie's genre from its poster. In this lesson you'll build that intuition into a working classifier and meet two ideas that follow you through all of ML: the complexity knob and feature scaling.

Play with it first

KNN has no training phase to speak of — it just stores the data. All the action happens at prediction time. Drag the test point around, change k (the number of neighbors consulted), and toggle the decision regions to see the boundary the model implies:

predictedclass BvotesA:1 · B:2 · C:2

Neighbors k5

Weighting

UniformDistance (1/d)

Show decision regions

Two things to notice while exploring: with k = 1 the decision regions are jagged and every stray training point gets its own little island, while with a large k the boundary becomes smooth and stubborn. Keep that trade-off in mind — the rest of the lesson explains it.

How KNN decides

Given a new point, KNN does three things:

1. Measure the distance from the new point to every training point.
2. Find the k closest training points.
3. Take a majority vote of their labels (for regression: average their values).

The standard distance is Euclidean distance — the straight-line distance you'd measure with a ruler:

d(a, b) = sqrt( Σ (aᵢ − bᵢ)² )

scikit-learn's KNeighborsClassifier exposes this via the Minkowski parameter p: p=2 is Euclidean, p=1 is Manhattan distance (sum of absolute differences — city-block travel). Both are worth trying when tuning.

Uniform vs distance-weighted voting

By default every one of the k neighbors gets an equal vote (weights="uniform"). With weights="distance", closer neighbors count more — a neighbor right next to the test point can outvote several far-away ones. Distance weighting often helps when classes overlap, and it has a side effect worth knowing: on the training data itself, the nearest neighbor of a training point is itself at distance zero, so training accuracy becomes perfect. That's not a bug, but it means training accuracy tells you nothing — evaluate on held-out data, as always.

The k knob: jagged vs smooth

k is a complexity dial, just like tree depth in the last lesson:

• Small k (1–3): the model reacts to every individual point, including noise and mislabeled examples. Jagged boundary, high training accuracy, risk of overfitting.
• Large k: predictions average over a wide neighborhood, smoothing away detail. Push k toward the size of the dataset and every prediction becomes the majority class — underfitting.

The right k lives in between and depends on the data, so we search for it. Here is KNN on the classic iris dataset (150 flowers, 4 measurements, 3 species), scanning several values of k:

Python — runs in your browser

Run

from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from sklearn.neighbors import KNeighborsClassifier X, y = load_iris(return_X_y=True) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.3, stratify=y, random_state=42 ) print(" k train test") for k in [1, 3, 5, 9, 15, 25, 51]: knn = KNeighborsClassifier(n_neighbors=k).fit(X_train, y_train) print(f"{k:2d} {knn.score(X_train, y_train):.3f} " f"{knn.score(X_test, y_test):.3f}")

The pattern: training accuracy is highest at k = 1 and decays as k grows, while test accuracy peaks somewhere in the middle. (Tip: prefer odd values of k for binary problems to avoid tied votes.)

Distance models need scaled features

Here's KNN's biggest gotcha. Distance treats all features as if they were in the same units. If one feature ranges 0–1 and another ranges 0–10,000, the big-range feature completely dominates the distance — even if it's pure noise. Watch a useless feature destroy the model, and scaling rescue it:

Python — runs in your browser

Run

import numpy as np from sklearn.model_selection import train_test_split from sklearn.neighbors import KNeighborsClassifier from sklearn.preprocessing import StandardScaler rng = np.random.default_rng(0) n = 300 # Feature 1: informative (class 0 near 0, class 1 near 3) x1 = np.r_[rng.normal(0, 1, n // 2), rng.normal(3, 1, n // 2)] y = np.r_[np.zeros(n // 2, int), np.ones(n // 2, int)] # Feature 2: pure noise, but on a huge scale x2 = rng.normal(0, 1, n) * 1000 X = np.c_[x1, x2] X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.3, stratify=y, random_state=42 ) knn = KNeighborsClassifier(n_neighbors=5).fit(X_train, y_train) print(f"raw features : test accuracy = {knn.score(X_test, y_test):.3f}") scaler = StandardScaler().fit(X_train) # fit on TRAIN only X_train_s = scaler.transform(X_train) X_test_s = scaler.transform(X_test) knn = KNeighborsClassifier(n_neighbors=5).fit(X_train_s, y_train) print(f"scaled features : test accuracy = {knn.score(X_test_s, y_test):.3f}")

Unscaled, the noisy feature's huge magnitude drowns out the informative one and accuracy hovers near coin-flipping. After StandardScaler (subtract the mean, divide by the standard deviation — both computed from training data only), every feature speaks at the same volume. Alternatives include MinMaxScaler (squash to a 0–1 range) and RobustScaler (uses median and quartiles, resistant to outliers).

Where KNN struggles

KNN is a lazy learner: fit just stores the data, and all the work happens at predict time. That inverts the usual cost profile and brings some real weaknesses:

• Slow predictions. Each prediction compares against (potentially) every training point. With millions of rows, that hurts exactly where speed matters — in production.
• Memory hungry. The model is the training set; nothing gets compressed into a small set of learned parameters.
• The curse of dimensionality. With many features, all points become nearly equidistant from each other, and "nearest" stops meaning much. KNN shines with few, well-scaled, informative features.
• Sensitive to irrelevant features and scale, as you just saw.

Still, KNN is a superb first tool: no assumptions about the data's shape, naturally multiclass, and a strong sanity-check baseline before reaching for heavier machinery.

Check your understanding

5 questions · free

1. Q1.How does a KNN classifier with k=7 label a new point?

   AIt fits a line separating the classesBIt finds the 7 closest training points and takes a majority vote of their labelsCIt picks the label of the single closest point 7 timesDIt averages the coordinates of 7 random points

2. Q2.Decreasing k from 25 to 1 usually makes the decision boundary…

   Asmoother, reducing overfittingBmore jagged, increasing the risk of overfittingCidentical — k only affects speedDlinear

3. Q3.Why does feature scaling matter so much for KNN?

   AIt makes the algorithm run fasterBDistances are dominated by features with large numeric ranges, regardless of how informative they areCscikit-learn refuses to fit unscaled dataDScaling changes the value of k

4. Q4.With weights="distance", training accuracy becomes 100%. Why?

   ADistance weighting removes all noise from the dataBEach training point's nearest neighbor is itself at distance zero, giving it infinite voting weightCIt sets k=1 automaticallyDIt is a bug in scikit-learn

5. Q5.Which is a genuine weakness of KNN?

   ATraining is extremely slowBIt can only handle two classesCPrediction requires comparing against the stored training data, which is slow for large datasetsDIt cannot do regression

Exercise: Tune k, weights, and distance metric on iris

Extend the iris example into a small manual search: loop over odd k from 1 to 25, both weights options ("uniform", "distance"), and both distance metrics (p=1 Manhattan, p=2 Euclidean). Print the best test accuracy and the combination that achieved it. (In the next lesson you'll learn GridSearchCV, which does this search properly with cross-validation.)

Next up: scikit-learn pipelines — how to chain scaling, encoding, and models into one leak-proof object you can tune end to end.