PCA: Dimensionality Reduction

Clustering grouped rows; this lesson compresses columns. Real datasets routinely have dozens or hundreds of features, and that width causes real problems: you can't plot it, distances get less meaningful, and models overfit. Principal component analysis (PCA) is the classic fix — it finds a small set of new axes that keep as much of the data's variation as possible. You'll see how it works geometrically, how to choose the number of components, and how to drop it into a scikit-learn pipeline.

The curse of dimensionality

Every extra feature adds a dimension to the space your data lives in, and high-dimensional space behaves badly. The volume grows exponentially, so a fixed number of rows spreads thinner and thinner — with 30 features, 569 patients (the breast-cancer dataset we'll use) barely sketch the space. Distances lose contrast: in very high dimensions, the nearest and farthest neighbors of a point end up almost equally far away, which quietly degrades anything built on distance — KNN, K-Means, SVMs with RBF kernels. And more features means more parameters, which means more ways to memorize noise.

Dimensionality reduction attacks this from three angles at once:

• Visualization — compress to 2 or 3 dimensions so you can actually look at the data
• Compression — store or transmit the same information in fewer numbers
• Feature extraction — build a few dense, informative features out of many redundant ones, and feed those to a model

The good news: real features are rarely independent. A tumor's radius, perimeter, and area are three columns telling one story. When features are correlated, the data doesn't fill the 30-dimensional space — it hugs a much lower-dimensional sheet inside it, and PCA finds that sheet.

Directions of maximal variance

PCA's rule is simple: find the direction along which the data varies the most. That's principal component 1 (PC1). Then find the direction at a right angle to it with the most remaining variance — PC2 — and so on. Projecting the data onto the first few components keeps the spread (the information) and throws away the flat, noisy directions.

Try it below. Rotate the candidate axis through the cloud and watch the projected variance change, then project onto PC1 and see how much of the 2-D structure survives in 1-D:

PC1 var92.2%PC2 var7.8%rotation30°

Rotate data30°Project onto PC1

Notice that the best axis isn't either original feature — it's a diagonal combination of them. That's the general pattern: each principal component is a weighted mix of all the original features, and the components are computed in one shot via singular value decomposition (SVD), the same matrix factorization trick that powers half of classical ML. You'll also notice what gets lost: the small wiggles perpendicular to PC1. PCA bets that low-variance directions are noise. Usually a good bet — not always.

Scale first, always

PCA chases variance, and variance has units. In the breast-cancer data, mean area lives in the hundreds while mean smoothness lives around 0.1 — unscaled, the "biggest" direction is just whichever feature has the biggest numbers. Watch how badly this skews things:

Python — runs in your browser

Run

from sklearn.datasets import load_breast_cancer from sklearn.preprocessing import StandardScaler from sklearn.decomposition import PCA X, y = load_breast_cancer(return_X_y=True) print(f"Data: {X.shape[0]} patients x {X.shape[1]} features") pca_raw = PCA().fit(X) pca_scaled = PCA().fit(StandardScaler().fit_transform(X)) print(f"\nPC1 share of variance, unscaled: {pca_raw.explained_variance_ratio_[0]:.1%}") print(f"PC1 share of variance, scaled : {pca_scaled.explained_variance_ratio_[0]:.1%}")

Unscaled, PC1 "explains" over 98% of the variance — but it's just pointing at the large-unit features, not at structure. After standardizing, PC1 carries a believable 44%, and the rest is spread across many components.

Choosing the number of components

Each fitted PCA exposes explained_variance_ratio_ — the fraction of total variance each component captures. Its cumulative sum is the standard tool for choosing n_components: plot it and read off how many components you need to keep, say, 95% of the variance.

Python — runs in your browser

Run

import numpy as np import matplotlib.pyplot as plt from sklearn.datasets import load_breast_cancer from sklearn.preprocessing import StandardScaler from sklearn.decomposition import PCA X, y = load_breast_cancer(return_X_y=True) X_scaled = StandardScaler().fit_transform(X) pca = PCA().fit(X_scaled) cumvar = np.cumsum(pca.explained_variance_ratio_) plt.figure(figsize=(7, 4)) plt.plot(range(1, 31), cumvar, "o-") plt.axhline(0.95, color="gray", ls="--") plt.xlabel("Number of components") plt.ylabel("Cumulative explained variance") plt.title("30 features, but how much real information?") plt.show() n95 = int(np.argmax(cumvar >= 0.95)) + 1 print(f"Components for 95% of the variance: {n95}") print(f"First 2 components already hold {cumvar[1]:.1%}")

Ten components carry 95% of the variance of thirty features — a third of the width for almost all of the information. That redundancy is exactly the radius/perimeter/area correlation showing up in the math. A convenient shortcut: PCA(n_components=0.95) picks the count for you automatically.

Thirty dimensions on one screen

The most immediate payoff is visualization. Project the standardized data onto PC1 and PC2 and color each patient by diagnosis — a plot that would be impossible with the raw 30 columns:

Python — runs in your browser

Run

import matplotlib.pyplot as plt from sklearn.datasets import load_breast_cancer from sklearn.preprocessing import StandardScaler from sklearn.decomposition import PCA data = load_breast_cancer() X_scaled = StandardScaler().fit_transform(data.data) pca = PCA(n_components=2) X2 = pca.fit_transform(X_scaled) plt.figure(figsize=(7, 5)) for label, color in [(0, "tab:red"), (1, "tab:blue")]: mask = data.target == label plt.scatter(X2[mask, 0], X2[mask, 1], s=14, c=color, alpha=0.6, label=data.target_names[label]) plt.xlabel(f"PC1 ({pca.explained_variance_ratio_[0]:.0%} of variance)") plt.ylabel(f"PC2 ({pca.explained_variance_ratio_[1]:.0%} of variance)") plt.legend() plt.title("Breast-cancer data in PCA space") plt.show()

Remember: PCA never saw the diagnosis labels. It compressed 30 measurements into 2 numbers per patient using variance alone — and the malignant and benign groups separate almost cleanly anyway. That's strong evidence the labels are learnable, spotted before training a single classifier.

PCA as a preprocessing step

Because PCA is a transformer, it slots straight into a Pipeline between the scaler and the model. Fewer, decorrelated inputs can mean faster training and sometimes less overfitting:

Python — runs in your browser

Run

from sklearn.datasets import load_breast_cancer from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler from sklearn.decomposition import PCA from sklearn.svm import SVC from sklearn.model_selection import cross_val_score X, y = load_breast_cancer(return_X_y=True) models = [ ("SVC on all 30 features", make_pipeline(StandardScaler(), SVC())), ("SVC on 10 PCA components", make_pipeline(StandardScaler(), PCA(n_components=10), SVC())), ("SVC on 2 PCA components", make_pipeline(StandardScaler(), PCA(n_components=2), SVC())), ] for name, model in models: scores = cross_val_score(model, X, y, cv=5) print(f"{name:26s} accuracy = {scores.mean():.3f}")

Ten components match the full model's accuracy with a third of the inputs — and even two components stay remarkably close, which the scatter plot above already predicted. In a real project you'd treat n_components as a hyperparameter and let GridSearchCV tune it along with the classifier's settings, since the whole pipeline cross-validates as one unit.

What PCA can't do

PCA has two honest limitations. First, it's linear: components are straight axes, so if your data curls along a spiral or an S-shaped surface, no rotation captures it — that's what nonlinear methods like t-SNE (next lesson) and kernel PCA are for. Second, components trade interpretability for compactness: "0.22 times mean radius minus 0.10 times smoothness plus 28 more terms" is much harder to explain to a stakeholder than any original column. And one subtle trap: PCA is unsupervised, so it keeps high-variance directions whether or not they help your prediction task. Almost always they do — but if the signal lives in a low-variance direction, PCA will happily throw it away.

Check your understanding

5 questions · free

1. Q1.What is principal component 1 (PC1)?

   AThe original feature with the highest varianceBThe direction in feature space along which the data varies the mostCThe feature most correlated with the targetDThe average of all features

2. Q2.You run PCA on unscaled data where one feature is measured in thousands and the rest in single digits. What happens?

   APCA refuses to run on mixed unitsBPC1 essentially points at the large-unit feature, regardless of the real structureCNothing — PCA is scale-invariantDThe small features are automatically upweighted

3. Q3.The cumulative explained-variance curve reaches 0.95 at component 10 (out of 30). What does that tell you?

   AThe model will be 95% accurate with 10 featuresB10 components retain 95% of the data's total variance — the other 20 directions add little spreadC10 of the original columns can be deletedDThe data has exactly 10 clusters

4. Q4.Which task is PCA fundamentally unable to handle?

   ACompressing 30 correlated features to 10BVisualizing high-dimensional data in 2-DCCapturing structure that curls along a nonlinear surfaceDSpeeding up a downstream classifier

5. Q5.Why can PCA discard information that matters for classification?

   AIt always keeps the wrong componentsBIt is unsupervised: it ranks directions by variance, not by how well they separate the classesCIt only works on two classesDIt shuffles the labels during fitting

Exercise: Compress the digits

Repeat the workflow on load_digits (64 pixel features). Scale the data, find how many components you need for 90% of the variance, then compare a logistic-regression pipeline on all 64 features against one that reduces to that number of components first (3-fold cross-validation). How much accuracy does the compression cost?

Next up: when straight axes aren't enough — t-SNE for visualizing nonlinear structure, and topic modeling for finding themes in text.