t-SNE & Topic Modeling

PCA gave us one tool for shrinking dimensions, but it can only rotate and project along straight axes. This lesson covers two unsupervised techniques that go further: t-SNE, which unfolds nonlinear structure into striking 2-D maps, and topic modeling, which discovers the hidden themes in a pile of documents. Different data, same spirit — find structure nobody labeled.

Where PCA plots fall short

The digits dataset packs each handwritten digit into 64 pixel features. There are ten obvious groups in there — one per digit — but similarity between digits isn't a straight-line affair: a curvy 3 sits "near" an 8 in ways no single linear axis captures. Project onto the top two principal components and much of that neighborhood structure smears together:

Python — runs in your browser

Run

import numpy as np import matplotlib.pyplot as plt from sklearn.datasets import load_digits from sklearn.decomposition import PCA X, y = load_digits(return_X_y=True) rng = np.random.default_rng(0) idx = rng.choice(len(X), 300, replace=False) X, y = X[idx], y[idx] X_pca = PCA(n_components=2).fit_transform(X) plt.figure(figsize=(6.5, 5)) sc = plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y, s=18, cmap="tab10") plt.colorbar(sc, label="digit") plt.title("Digits projected with PCA") plt.xlabel("PC1"); plt.ylabel("PC2") plt.show()

A few digits (0s, 6s) form loose islands, but the middle is a traffic jam. PCA did its job — it kept the two highest-variance directions — but the variance that separates a 4 from a 9 lives on a curved surface that no two straight axes can flatten.

t-SNE: preserving neighborhoods

t-distributed Stochastic Neighbor Embedding (t-SNE) takes a different goal: instead of preserving global variance, preserve local similarity. For every pair of points it computes "how likely are these two to be neighbors?" in the original high-dimensional space, then arranges points in 2-D so those neighbor probabilities match as closely as possible. Points that were close stay close; everything else is negotiable.

The knob you'll actually turn is perplexity — roughly, the number of neighbors each point tries to stay faithful to. Small perplexity focuses on very local structure (and can shatter clusters into fragments); larger values look further out and give smoother, more global layouts.

Python — runs in your browser

Run

import numpy as np import matplotlib.pyplot as plt from sklearn.datasets import load_digits from sklearn.manifold import TSNE X, y = load_digits(return_X_y=True) rng = np.random.default_rng(0) idx = rng.choice(len(X), 300, replace=False) X, y = X[idx], y[idx] fig, axes = plt.subplots(1, 2, figsize=(10, 4.2)) for ax, perp in zip(axes, [5, 30]): Z = TSNE(n_components=2, perplexity=perp, init="pca", learning_rate="auto", random_state=42).fit_transform(X) ax.scatter(Z[:, 0], Z[:, 1], c=y, s=14, cmap="tab10") ax.set_title(f"t-SNE, perplexity={perp}") ax.set_xticks([]); ax.set_yticks([]) plt.tight_layout() plt.show()

Even on this small 300-digit sample, the ten classes resolve into distinct islands — remember, t-SNE never saw the colors. Perplexity 5 produces tighter, more fragmented clumps; perplexity 30 gives the cleaner map. On the full dataset the separation is even more dramatic, which is why t-SNE became the standard for eyeballing embeddings, from digits to word vectors to single-cell genomics.

Reading a t-SNE plot honestly

t-SNE optimizes neighbor probabilities, and it will cheerfully distort everything else to get them. Keep these rules in mind:

• Distances between clusters mean little. Two islands far apart aren't necessarily more different than two nearby ones.
• Cluster sizes mean little. t-SNE expands dense blobs and shrinks sparse ones; the areas on screen don't reflect spread in the data.
• Different runs and perplexities give different pictures. Always try a couple of perplexity values before trusting a pattern.
• It's visualization-only. There's no transform for new points, so you can't use t-SNE coordinates as features in a deployed pipeline; refitting changes the whole map.

One practical tip that applies to both: for very wide data (thousands of features), run PCA down to about 50 components first, then t-SNE on that — faster and less noisy.

From pixels to words: bag-of-words

Dimensionality reduction gets even more interesting on text. First we need numbers: the bag-of-words representation counts how often each vocabulary word occurs in each document, ignoring order entirely. Each document becomes one very wide, very sparse row — one column per word.

• CountVectorizer produces raw counts.
• TfidfVectorizer reweights them by tf-idf, shrinking words that appear everywhere ("the", "and") and boosting words distinctive to a few documents.

Python — runs in your browser

Run

from sklearn.feature_extraction.text import CountVectorizer corpus = [ "the rocket launched into orbit around the earth", "astronauts aboard the space station photographed the planets", "the telescope captured images of a distant galaxy", "nasa announced a new mission to land on the moon", "the satellite orbits the earth twice every day", "engineers tested the rocket engines before the launch", "simmer the tomato sauce and season the pasta with basil", "bake the bread until the crust turns golden brown", "chop the onions and fry them in olive oil", "the chef seasoned the soup with garlic and pepper", "knead the dough and let it rise before baking", "roast the vegetables with olive oil and sea salt", "the striker scored a stunning goal in the final minute", "the goalkeeper saved a penalty during the cup match", "the team won the league after a dramatic last match", "fans cheered loudly as the midfielder scored twice", "the coach praised the defenders after the away match", "the referee showed a red card early in the derby", ] vec = CountVectorizer(stop_words="english") X_counts = vec.fit_transform(corpus) vocab = vec.get_feature_names_out() print(f"Document-term matrix: {X_counts.shape[0]} docs x {X_counts.shape[1]} words") print(f"Sample vocabulary: {list(vocab[:8])}") print(f"\nDoc 0: '{corpus[0]}'") row = X_counts[0].toarray().ravel() print("Non-zero counts:", {vocab[i]: int(row[i]) for i in row.nonzero()[0]})

Eighteen tiny documents already produce a matrix dozens of columns wide — real corpora hit tens of thousands. And you can probably see three themes hiding in there. Topic modeling is how the machine finds them.

Topic modeling: LSA and LDA

A topic model factorizes the document-term matrix into two smaller pieces: documents-to-topics ("doc 3 is 90% space") and topics-to-words ("the space topic loves rocket, orbit, earth"). Two classic approaches:

• LSA (Latent Semantic Analysis) applies truncated SVD — literally PCA's engine — to the (usually tf-idf) matrix. Fast and deterministic, but topic weights can be negative, which makes them awkward to read.
• LDA (Latent Dirichlet Allocation) is a probabilistic model: each document is a mixture of topics, each topic a distribution over words. It works on raw counts and its topics are proper probabilities — usually the more interpretable of the two.

Python — runs in your browser

Run

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer from sklearn.decomposition import TruncatedSVD, LatentDirichletAllocation corpus = [ "the rocket launched into orbit around the earth", "astronauts aboard the space station photographed the planets", "the telescope captured images of a distant galaxy", "nasa announced a new mission to land on the moon", "the satellite orbits the earth twice every day", "engineers tested the rocket engines before the launch", "simmer the tomato sauce and season the pasta with basil", "bake the bread until the crust turns golden brown", "chop the onions and fry them in olive oil", "the chef seasoned the soup with garlic and pepper", "knead the dough and let it rise before baking", "roast the vegetables with olive oil and sea salt", "the striker scored a stunning goal in the final minute", "the goalkeeper saved a penalty during the cup match", "the team won the league after a dramatic last match", "fans cheered loudly as the midfielder scored twice", "the coach praised the defenders after the away match", "the referee showed a red card early in the derby", ] def top_words(model, vocab, n=5): for k, comp in enumerate(model.components_): words = [vocab[i] for i in comp.argsort()[::-1][:n]] print(f" topic {k}: {', '.join(words)}") # LSA on tf-idf tfidf = TfidfVectorizer(stop_words="english") X_tfidf = tfidf.fit_transform(corpus) lsa = TruncatedSVD(n_components=3, random_state=42).fit(X_tfidf) print("LSA topics (TruncatedSVD):") top_words(lsa, tfidf.get_feature_names_out()) # LDA on raw counts counts = CountVectorizer(stop_words="english") X_counts = counts.fit_transform(corpus) lda = LatentDirichletAllocation(n_components=3, max_iter=20, random_state=42).fit(X_counts) print("\nLDA topics:") top_words(lda, counts.get_feature_names_out()) doc_topics = lda.transform(X_counts) print(f"\nDoc 0 topic mixture: {doc_topics[0].round(2)}") print(f"Doc 12 topic mixture: {doc_topics[12].round(2)}")

Both models recover the space / cooking / football split without ever being told those themes exist — the topics are just directions (LSA) or word distributions (LDA) in bag-of-words space. Note the last two lines: LDA also gives each document a topic mixture, which makes a great compact feature vector for downstream tasks like clustering articles or routing support tickets. In practice, choosing the number of topics works like choosing k in K-Means: try several values and judge whether the top words tell coherent stories.

The same "compress, then look" recipe extends beyond text — run PCA on face images and the components become ghostly "eigenfaces"; feed t-SNE the topic mixtures of news articles and related stories cluster together. Unsupervised learning keeps paying rent as a lens on data.

Check your understanding

5 questions · free

1. Q1.What does t-SNE try to preserve when mapping data to 2-D?

   AThe global variance of the data, like PCABThe pairwise neighbor relationships — points close in high dimensions stay close in 2-DCThe exact distances between all pairs of pointsDThe order of the rows in the dataset

2. Q2.In a t-SNE plot, two clusters appear far apart and one looks much bigger than the other. What can you safely conclude?

   AThe far-apart clusters are the most dissimilar groups in the dataBThe bigger cluster has more variance in the original spaceCOnly that the points within each cluster are similar to each other — inter-cluster distances and sizes are not meaningfulDThe data has exactly two classes

3. Q3.Why can't you use fitted t-SNE coordinates as features for scoring new data in production?

   At-SNE has no transform for unseen points — embedding new data means refitting, which changes the whole mapBt-SNE coordinates are always negativeCt-SNE only works on imagesDThe coordinates are integers, not floats

4. Q4.What is the key difference between LSA and LDA topics?

   ALSA works only on images, LDA only on textBLSA topics come from SVD and can have negative weights; LDA topics are probability distributions over wordsCLDA is a supervised algorithmDLSA requires deep learning

5. Q5.In the bag-of-words representation, what does each column of the matrix correspond to?

   AA documentBA sentenceCA vocabulary word, with cells counting its occurrences per documentDA topic

Exercise: Add a fourth topic

Extend the lesson's corpus with 5–6 short sentences about a fourth theme of your choosing (weather, music, finance...). Refit LDA with n_components=4 and print the top 5 words per topic — does your new theme get its own topic? Then refit with n_components=3 on the same expanded corpus and observe what goes wrong.

Next up: a new module — time series, where the order of the rows finally matters and yesterday is your best predictor of today.