Clustering with K-Means

Everything so far had a supervisor: every row came with the right answer, and the model's job was to match it. Clustering removes the answer key. You hand the algorithm raw points and ask, "which of these belong together?" In this lesson you'll run the K-Means loop by hand, learn how to pick the number of clusters, and see exactly where K-Means breaks — and what to use when it does.

Learning without labels

In supervised learning, "good" means "close to the labels". Without labels, we need a different definition, and clustering uses geometry: a good grouping puts similar points in the same cluster and dissimilar points in different ones. That makes clustering genuinely useful when labels don't exist yet — segmenting customers, grouping documents by theme, compressing colors in an image — but it also means you have to judge whether the clusters mean anything. The algorithm will always give you groups; it can't tell you if they matter.

The K-Means loop

K-Means is the workhorse. You choose k, the number of clusters, and the algorithm alternates two moves until nothing changes:

1. Assign — give each point to its nearest centroid
2. Update — move each centroid to the mean of its assigned points

Try it yourself. Step through the iterations below and watch the two moves alternate — and keep an eye on the inertia readout as the centroids settle:

iteration0next stepassigninertia—

Clusters k3

Step (assign)▶ PlayReset centroidsAdd points: click the chart

A few things you probably noticed: the assignments and centroids stabilize after just a handful of iterations, the inertia only ever goes down, and if you re-initialize, the final clusters can differ. K-Means is only guaranteed to find a local optimum, which is why scikit-learn's KMeans runs several random restarts (n_init) and keeps the best one by default.

Inertia, and why it always drops with k

Inertia is the quantity K-Means minimizes: the sum of squared distances from each point to its own centroid. Lower inertia means tighter clusters — but be careful using it to choose k. Adding a cluster can only ever reduce inertia (in the extreme, k = n gives inertia 0, with every point as its own "cluster"). So you can't just pick the k with the lowest inertia; you look for the point of diminishing returns.

Choosing k: elbow and silhouette

The elbow method plots inertia against k and looks for the bend — the k after which extra clusters stop paying for themselves. The silhouette score measures, for each point, how much closer it is to its own cluster than to the nearest other cluster (from −1 to +1, higher is better) — and unlike inertia, it peaks at a good k instead of always decreasing:

Python — runs in your browser

Run

import matplotlib.pyplot as plt from sklearn.datasets import make_blobs from sklearn.cluster import KMeans from sklearn.metrics import silhouette_score X, _ = make_blobs(n_samples=300, centers=4, cluster_std=0.9, random_state=42) ks = range(2, 10) inertias, silhouettes = [], [] for k in ks: km = KMeans(n_clusters=k, n_init=5, random_state=42).fit(X) inertias.append(km.inertia_) silhouettes.append(silhouette_score(X, km.labels_)) fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9, 3.5)) ax1.plot(ks, inertias, "o-") ax1.set_xlabel("k"); ax1.set_ylabel("Inertia"); ax1.set_title("Elbow") ax2.plot(ks, silhouettes, "o-", color="tab:orange") ax2.set_xlabel("k"); ax2.set_ylabel("Silhouette"); ax2.set_title("Silhouette") plt.tight_layout() plt.show() best_k = list(ks)[silhouettes.index(max(silhouettes))] print(f"Best k by silhouette: {best_k}")

The data was generated with 4 blobs, and both diagnostics agree: the elbow bends at 4 and the silhouette peaks there. On real data the signals are rarely this clean — treat them as evidence, not verdicts, and sanity-check the clusters against domain knowledge.

Where K-Means fails

The assign-to-nearest-centroid rule carves space into convex regions, so K-Means silently assumes clusters are roughly spherical, similar in size, and separable by straight boundaries. Give it two interleaved crescents and it fails confidently:

Python — runs in your browser

Run

import matplotlib.pyplot as plt from sklearn.datasets import make_moons from sklearn.cluster import KMeans X, y_true = make_moons(n_samples=300, noise=0.06, random_state=42) km = KMeans(n_clusters=2, n_init=5, random_state=42) labels = km.fit_predict(X) fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9, 3.8)) ax1.scatter(X[:, 0], X[:, 1], c=y_true, s=12, cmap="coolwarm") ax1.set_title("True structure: two moons") ax2.scatter(X[:, 0], X[:, 1], c=labels, s=12, cmap="coolwarm") c = km.cluster_centers_ ax2.scatter(c[:, 0], c[:, 1], c="white", edgecolors="k", s=120, linewidths=2) ax2.set_title("K-Means: splits them wrong") for ax in (ax1, ax2): ax.set_aspect("equal") plt.tight_layout() plt.show()

K-Means slices the moons with a straight cut because that's all it can do. Other classic failure modes: clusters with very different densities or sizes (the big cluster "steals" points from the small one), outliers dragging centroids around, and non-numeric data you can't take a mean of (variants like K-Modes and K-Prototypes exist for that).

Beyond K-Means: DBSCAN and agglomerative clustering

Two alternatives cover most of what K-Means can't:

• DBSCAN grows clusters from dense regions: points with enough neighbors within radius eps seed a cluster, and it expands through connected dense areas. It finds arbitrarily shaped clusters, doesn't need k, and labels sparse points as noise (-1).
• Agglomerative clustering starts with every point as its own cluster and repeatedly merges the closest pair, building a hierarchy (visualized as a dendrogram). With linkage="single" ("closest points" distance) it chains along curved shapes nicely.

Python — runs in your browser

Run

import matplotlib.pyplot as plt from sklearn.datasets import make_moons from sklearn.cluster import KMeans, DBSCAN, AgglomerativeClustering X, _ = make_moons(n_samples=300, noise=0.06, random_state=42) models = [ ("K-Means", KMeans(n_clusters=2, n_init=5, random_state=42)), ("DBSCAN (eps=0.2)", DBSCAN(eps=0.2)), ("Agglomerative (single)", AgglomerativeClustering(n_clusters=2, linkage="single")), ] fig, axes = plt.subplots(1, 3, figsize=(11, 3.6)) for ax, (name, model) in zip(axes, models): labels = model.fit_predict(X) ax.scatter(X[:, 0], X[:, 1], c=labels, s=10, cmap="viridis") ax.set_title(name) ax.set_aspect("equal") plt.tight_layout() plt.show()

Both alternatives recover the moons perfectly. So why is K-Means still the default? It's fast, it scales to millions of points, its clusters come with centroids you can interpret and reuse (as you'll see next lesson), and many real datasets — especially after scaling — really are blob-shaped. DBSCAN's weakness is choosing eps and handling clusters of varying density; agglomerative clustering is O(n²) and struggles past tens of thousands of points.

Check your understanding

4 questions · free

1. Q1.In the K-Means loop, what happens during the 'update' step?

   AEach point is reassigned to its nearest centroidBEach centroid moves to the mean of the points assigned to itCThe number of clusters k is increased by oneDPoints far from all centroids are removed as outliers

2. Q2.Why can't you choose k by simply picking the value with the lowest inertia?

   AInertia is random and changes on every runBInertia always decreases as k grows, reaching 0 when every point is its own clusterCInertia only works for two-dimensional dataDInertia increases with k, so you'd always pick k = 2

3. Q3.You cluster customers on income (in dollars) and number of visits (0–50) without scaling. What happens?

   ANothing — K-Means is scale-invariantBThe clusters are determined almost entirely by income, since it dominates the distanceCK-Means refuses to run on features with different unitsDThe visits feature dominates because it has fewer unique values

4. Q4.Which dataset shape would make DBSCAN a better choice than K-Means?

   AFour well-separated round blobs of similar sizeBTwo interleaved crescent-shaped clusters with some noise pointsCA single Gaussian cloudDData with only categorical features

Exercise: Break K-Means, then fix the diagnosis

Generate blobs with very different spreads: 3 centers with cluster_std=[0.5, 2.5, 0.5] (400 points, random_state=7). Run K-Means with the correct k = 3, then compare its labels to the true ones with a side-by-side scatter plot and compute the silhouette score of both labelings. Which failure mode from the lesson are you seeing?

Next up: putting clusters to work — segmenting customers into named personas and compressing an image down to a handful of colors.