{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "id": "cell-0000", "metadata": {}, "source": [ "# t-SNE & Topic Modeling\n", "\n", "Visualize nonlinear structure with t-SNE, learn its caveats, then discover hidden themes in text with LSA and LDA.\n", "\n", "*Part of the free [Machine Learning](https://ramadnsyh.dev/courses/machine-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/machine-learning/tsne-topic-modeling).*" ] }, { "cell_type": "markdown", "id": "cell-0001", "metadata": {}, "source": [ "PCA gave us one tool for shrinking dimensions, but it can only rotate and\n", "project along straight axes. This lesson covers two unsupervised techniques\n", "that go further: **t-SNE**, which unfolds nonlinear structure into striking\n", "2-D maps, and **topic modeling**, which discovers the hidden themes in a pile\n", "of documents. Different data, same spirit — find structure nobody labeled." ] }, { "cell_type": "markdown", "id": "cell-0002", "metadata": {}, "source": [ "## Where PCA plots fall short\n", "\n", "The digits dataset packs each handwritten digit into 64 pixel features.\n", "There are ten obvious groups in there — one per digit — but similarity\n", "between digits isn't a straight-line affair: a curvy 3 sits \"near\" an 8 in\n", "ways no single linear axis captures. Project onto the top two principal\n", "components and much of that neighborhood structure smears together:" ] }, { "cell_type": "code", "id": "cell-0003", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn.datasets import load_digits\n", "from sklearn.decomposition import PCA\n", "\n", "X, y = load_digits(return_X_y=True)\n", "rng = np.random.default_rng(0)\n", "idx = rng.choice(len(X), 300, replace=False)\n", "X, y = X[idx], y[idx]\n", "\n", "X_pca = PCA(n_components=2).fit_transform(X)\n", "\n", "plt.figure(figsize=(6.5, 5))\n", "sc = plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y, s=18, cmap=\"tab10\")\n", "plt.colorbar(sc, label=\"digit\")\n", "plt.title(\"Digits projected with PCA\")\n", "plt.xlabel(\"PC1\"); plt.ylabel(\"PC2\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "cell-0004", "metadata": {}, "source": [ "A few digits (0s, 6s) form loose islands, but the middle is a traffic jam.\n", "PCA did its job — it kept the two highest-variance directions — but the\n", "variance that separates a 4 from a 9 lives on a curved surface that no two\n", "straight axes can flatten." ] }, { "cell_type": "markdown", "id": "cell-0005", "metadata": {}, "source": [ "## t-SNE: preserving neighborhoods\n", "\n", "**t-distributed Stochastic Neighbor Embedding (t-SNE)** takes a different\n", "goal: instead of preserving global variance, preserve **local similarity**.\n", "For every pair of points it computes \"how likely are these two to be\n", "neighbors?\" in the original high-dimensional space, then arranges points in\n", "2-D so those neighbor probabilities match as closely as possible. Points\n", "that were close stay close; everything else is negotiable.\n", "\n", "The knob you'll actually turn is **perplexity** — roughly, the number of\n", "neighbors each point tries to stay faithful to. Small perplexity focuses on\n", "very local structure (and can shatter clusters into fragments); larger\n", "values look further out and give smoother, more global layouts." ] }, { "cell_type": "code", "id": "cell-0006", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn.datasets import load_digits\n", "from sklearn.manifold import TSNE\n", "\n", "X, y = load_digits(return_X_y=True)\n", "rng = np.random.default_rng(0)\n", "idx = rng.choice(len(X), 300, replace=False)\n", "X, y = X[idx], y[idx]\n", "\n", "fig, axes = plt.subplots(1, 2, figsize=(10, 4.2))\n", "for ax, perp in zip(axes, [5, 30]):\n", " Z = TSNE(n_components=2, perplexity=perp, init=\"pca\",\n", " learning_rate=\"auto\", random_state=42).fit_transform(X)\n", " ax.scatter(Z[:, 0], Z[:, 1], c=y, s=14, cmap=\"tab10\")\n", " ax.set_title(f\"t-SNE, perplexity={perp}\")\n", " ax.set_xticks([]); ax.set_yticks([])\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "cell-0007", "metadata": {}, "source": [ "Even on this small 300-digit sample, the ten classes resolve into distinct\n", "islands — remember, t-SNE never saw the colors. Perplexity 5 produces\n", "tighter, more fragmented clumps; perplexity 30 gives the cleaner map. On\n", "the full dataset the separation is even more dramatic, which is why t-SNE\n", "became *the* standard for eyeballing embeddings, from digits to word vectors\n", "to single-cell genomics." ] }, { "cell_type": "markdown", "id": "cell-0008", "metadata": {}, "source": [ "## Reading a t-SNE plot honestly\n", "\n", "t-SNE optimizes neighbor probabilities, and it will cheerfully distort\n", "everything else to get them. Keep these rules in mind:\n", "\n", "- **Distances between clusters mean little.** Two islands far apart aren't\n", " necessarily more different than two nearby ones.\n", "- **Cluster sizes mean little.** t-SNE expands dense blobs and shrinks\n", " sparse ones; the areas on screen don't reflect spread in the data.\n", "- **Different runs and perplexities give different pictures.** Always try a\n", " couple of perplexity values before trusting a pattern.\n", "- **It's visualization-only.** There's no `transform` for new points, so you\n", " can't use t-SNE coordinates as features in a deployed pipeline; refitting\n", " changes the whole map." ] }, { "cell_type": "markdown", "id": "cell-0009", "metadata": {}, "source": [ "> **The modern alternative: UMAP**\n", "> \n", "> UMAP solves a similar neighbor-preservation problem but runs much faster,\n", "> scales to millions of points, preserves global structure somewhat better,\n", "> and *can* transform new data. It's not bundled with scikit-learn (package\n", "> `umap-learn`), but in practice it has largely replaced t-SNE for big\n", "> datasets. The reading skills above apply to UMAP plots too." ] }, { "cell_type": "markdown", "id": "cell-0010", "metadata": {}, "source": [ "One practical tip that applies to both: for very wide data (thousands of\n", "features), run PCA down to about 50 components first, then t-SNE on that —\n", "faster and less noisy." ] }, { "cell_type": "markdown", "id": "cell-0011", "metadata": {}, "source": [ "## From pixels to words: bag-of-words\n", "\n", "Dimensionality reduction gets even more interesting on text. First we need\n", "numbers: the **bag-of-words** representation counts how often each\n", "vocabulary word occurs in each document, ignoring order entirely. Each\n", "document becomes one very wide, very sparse row — one column per word.\n", "\n", "- `CountVectorizer` produces raw counts.\n", "- `TfidfVectorizer` reweights them by **tf-idf**, shrinking words that\n", " appear everywhere (\"the\", \"and\") and boosting words distinctive to a few\n", " documents." ] }, { "cell_type": "code", "id": "cell-0012", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from sklearn.feature_extraction.text import CountVectorizer\n", "\n", "corpus = [\n", " \"the rocket launched into orbit around the earth\",\n", " \"astronauts aboard the space station photographed the planets\",\n", " \"the telescope captured images of a distant galaxy\",\n", " \"nasa announced a new mission to land on the moon\",\n", " \"the satellite orbits the earth twice every day\",\n", " \"engineers tested the rocket engines before the launch\",\n", " \"simmer the tomato sauce and season the pasta with basil\",\n", " \"bake the bread until the crust turns golden brown\",\n", " \"chop the onions and fry them in olive oil\",\n", " \"the chef seasoned the soup with garlic and pepper\",\n", " \"knead the dough and let it rise before baking\",\n", " \"roast the vegetables with olive oil and sea salt\",\n", " \"the striker scored a stunning goal in the final minute\",\n", " \"the goalkeeper saved a penalty during the cup match\",\n", " \"the team won the league after a dramatic last match\",\n", " \"fans cheered loudly as the midfielder scored twice\",\n", " \"the coach praised the defenders after the away match\",\n", " \"the referee showed a red card early in the derby\",\n", "]\n", "\n", "vec = CountVectorizer(stop_words=\"english\")\n", "X_counts = vec.fit_transform(corpus)\n", "vocab = vec.get_feature_names_out()\n", "\n", "print(f\"Document-term matrix: {X_counts.shape[0]} docs x {X_counts.shape[1]} words\")\n", "print(f\"Sample vocabulary: {list(vocab[:8])}\")\n", "print(f\"\\\\nDoc 0: '{corpus[0]}'\")\n", "row = X_counts[0].toarray().ravel()\n", "print(\"Non-zero counts:\", {vocab[i]: int(row[i]) for i in row.nonzero()[0]})" ] }, { "cell_type": "markdown", "id": "cell-0013", "metadata": {}, "source": [ "Eighteen tiny documents already produce a matrix dozens of columns wide —\n", "real corpora hit tens of thousands. And you can probably see three themes\n", "hiding in there. Topic modeling is how the machine finds them." ] }, { "cell_type": "markdown", "id": "cell-0014", "metadata": {}, "source": [ "## Topic modeling: LSA and LDA\n", "\n", "A **topic model** factorizes the document-term matrix into two smaller\n", "pieces: documents-to-topics (\"doc 3 is 90% space\") and topics-to-words\n", "(\"the space topic loves *rocket*, *orbit*, *earth*\"). Two classic\n", "approaches:\n", "\n", "- **LSA (Latent Semantic Analysis)** applies truncated SVD — literally PCA's\n", " engine — to the (usually tf-idf) matrix. Fast and deterministic, but\n", " topic weights can be negative, which makes them awkward to read.\n", "- **LDA (Latent Dirichlet Allocation)** is a probabilistic model: each\n", " document is a mixture of topics, each topic a distribution over words.\n", " It works on raw counts and its topics are proper probabilities — usually\n", " the more interpretable of the two." ] }, { "cell_type": "code", "id": "cell-0015", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n", "from sklearn.decomposition import TruncatedSVD, LatentDirichletAllocation\n", "\n", "corpus = [\n", " \"the rocket launched into orbit around the earth\",\n", " \"astronauts aboard the space station photographed the planets\",\n", " \"the telescope captured images of a distant galaxy\",\n", " \"nasa announced a new mission to land on the moon\",\n", " \"the satellite orbits the earth twice every day\",\n", " \"engineers tested the rocket engines before the launch\",\n", " \"simmer the tomato sauce and season the pasta with basil\",\n", " \"bake the bread until the crust turns golden brown\",\n", " \"chop the onions and fry them in olive oil\",\n", " \"the chef seasoned the soup with garlic and pepper\",\n", " \"knead the dough and let it rise before baking\",\n", " \"roast the vegetables with olive oil and sea salt\",\n", " \"the striker scored a stunning goal in the final minute\",\n", " \"the goalkeeper saved a penalty during the cup match\",\n", " \"the team won the league after a dramatic last match\",\n", " \"fans cheered loudly as the midfielder scored twice\",\n", " \"the coach praised the defenders after the away match\",\n", " \"the referee showed a red card early in the derby\",\n", "]\n", "\n", "def top_words(model, vocab, n=5):\n", " for k, comp in enumerate(model.components_):\n", " words = [vocab[i] for i in comp.argsort()[::-1][:n]]\n", " print(f\" topic {k}: {', '.join(words)}\")\n", "\n", "# LSA on tf-idf\n", "tfidf = TfidfVectorizer(stop_words=\"english\")\n", "X_tfidf = tfidf.fit_transform(corpus)\n", "lsa = TruncatedSVD(n_components=3, random_state=42).fit(X_tfidf)\n", "print(\"LSA topics (TruncatedSVD):\")\n", "top_words(lsa, tfidf.get_feature_names_out())\n", "\n", "# LDA on raw counts\n", "counts = CountVectorizer(stop_words=\"english\")\n", "X_counts = counts.fit_transform(corpus)\n", "lda = LatentDirichletAllocation(n_components=3, max_iter=20,\n", " random_state=42).fit(X_counts)\n", "print(\"\\\\nLDA topics:\")\n", "top_words(lda, counts.get_feature_names_out())\n", "\n", "doc_topics = lda.transform(X_counts)\n", "print(f\"\\\\nDoc 0 topic mixture: {doc_topics[0].round(2)}\")\n", "print(f\"Doc 12 topic mixture: {doc_topics[12].round(2)}\")" ] }, { "cell_type": "markdown", "id": "cell-0016", "metadata": {}, "source": [ "Both models recover the space / cooking / football split without ever being\n", "told those themes exist — the topics are just directions (LSA) or word\n", "distributions (LDA) in bag-of-words space. Note the last two lines: LDA also\n", "gives each *document* a topic mixture, which makes a great compact feature\n", "vector for downstream tasks like clustering articles or routing support\n", "tickets. In practice, choosing the number of topics works like choosing k in\n", "K-Means: try several values and judge whether the top words tell coherent\n", "stories.\n", "\n", "The same \"compress, then look\" recipe extends beyond text — run PCA on face\n", "images and the components become ghostly \"eigenfaces\"; feed t-SNE the topic\n", "mixtures of news articles and related stories cluster together. Unsupervised\n", "learning keeps paying rent as a *lens* on data." ] }, { "cell_type": "markdown", "id": "cell-0017", "metadata": {}, "source": [ "### 🏋️ Exercise — Add a fourth topic\n", "\n", "Extend the lesson's corpus with 5–6 short sentences about a **fourth theme**\n", "of your choosing (weather, music, finance...). Refit LDA with\n", "`n_components=4` and print the top 5 words per topic — does your new theme\n", "get its own topic? Then refit with `n_components=3` on the same expanded\n", "corpus and observe what goes wrong." ] }, { "cell_type": "code", "id": "cell-0018", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Your solution here" ] }, { "cell_type": "markdown", "id": "cell-0019", "metadata": {}, "source": [ "
✅ Show solution\n", "\n", "```python\n", "from sklearn.feature_extraction.text import CountVectorizer\n", "from sklearn.decomposition import LatentDirichletAllocation\n", "\n", "corpus = [\n", " \"the rocket launched into orbit around the earth\",\n", " \"astronauts aboard the space station photographed the planets\",\n", " \"the telescope captured images of a distant galaxy\",\n", " \"nasa announced a new mission to land on the moon\",\n", " \"the satellite orbits the earth twice every day\",\n", " \"engineers tested the rocket engines before the launch\",\n", " \"simmer the tomato sauce and season the pasta with basil\",\n", " \"bake the bread until the crust turns golden brown\",\n", " \"chop the onions and fry them in olive oil\",\n", " \"the chef seasoned the soup with garlic and pepper\",\n", " \"knead the dough and let it rise before baking\",\n", " \"roast the vegetables with olive oil and sea salt\",\n", " \"the striker scored a stunning goal in the final minute\",\n", " \"the goalkeeper saved a penalty during the cup match\",\n", " \"the team won the league after a dramatic last match\",\n", " \"fans cheered loudly as the midfielder scored twice\",\n", " \"the coach praised the defenders after the away match\",\n", " \"the referee showed a red card early in the derby\",\n", " # new theme: weather\n", " \"heavy rain flooded the streets across the city\",\n", " \"a cold front will bring snow to the mountains tonight\",\n", " \"the forecast predicts sunshine and clear skies tomorrow\",\n", " \"strong winds and thunderstorms hit the coast this weekend\",\n", " \"the heatwave broke temperature records across the region\",\n", " \"morning fog reduced visibility on the highway\",\n", "]\n", "\n", "vec = CountVectorizer(stop_words=\"english\")\n", "X = vec.fit_transform(corpus)\n", "vocab = vec.get_feature_names_out()\n", "\n", "for n in (4, 3):\n", " lda = LatentDirichletAllocation(n_components=n, max_iter=25,\n", " random_state=42).fit(X)\n", " print(f\"LDA with {n} topics:\")\n", " for k, comp in enumerate(lda.components_):\n", " words = [vocab[i] for i in comp.argsort()[::-1][:5]]\n", " print(f\" topic {k}: {', '.join(words)}\")\n", " print()\n", "\n", "# With 4 topics each theme gets its own clean word list; with 3,\n", "# two themes are forced to share a topic and the top words blur.\n", "```\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "cell-0020", "metadata": {}, "source": [ "Next up: a new module — time series, where the order of the rows finally\n", "matters and yesterday is your best predictor of today." ] } ] }