{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "id": "cell-0000", "metadata": {}, "source": [ "# Word Embeddings: Word2Vec & FastText\n", "\n", "Why one-hot vectors fail, how Word2Vec learns meaning from context, and how FastText handles typos and words it has never seen.\n", "\n", "*Part of the free [Deep Learning with PyTorch](https://ramadnsyh.dev/courses/deep-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/deep-learning/word-embeddings).*" ] }, { "cell_type": "code", "id": "cell-0001", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "%pip install -q gensim" ] }, { "cell_type": "markdown", "id": "cell-0002", "metadata": {}, "source": [ "Neural networks eat numbers, not words. Before any deep NLP model can read a\n", "sentence, every word has to become a vector — and *how* you build those\n", "vectors decides whether the model starts from \"cat and kitten are related\" or\n", "from total ignorance. In this lesson you'll see why the naive encoding fails,\n", "how Word2Vec learns meaning from raw text, and how FastText extends the idea\n", "to typos and brand-new words." ] }, { "cell_type": "markdown", "id": "cell-0003", "metadata": {}, "source": [ "## Why one-hot vectors fail\n", "\n", "The simplest encoding assigns each word its own slot: with a vocabulary of\n", "50,000 words, \"cat\" becomes a 50,000-dimensional vector that is all zeros\n", "except for a single 1. Two problems kill this approach:\n", "\n", "1. **It's huge.** One vector per word, each as long as the vocabulary.\n", "2. **Every word is equally distant from every other word.** \"cat\" is exactly\n", " as far from \"kitten\" as it is from \"carburetor\".\n", "\n", "The second problem is the fatal one. A standard way to measure vector\n", "similarity is **cosine similarity** — the cosine of the angle between two\n", "vectors (1 = same direction, 0 = perpendicular, −1 = opposite). Run this and\n", "watch every one-hot pair come out perpendicular:" ] }, { "cell_type": "code", "id": "cell-0004", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "\n", "vocab = [\"cat\", \"kitten\", \"dog\", \"carburetor\"]\n", "one_hot = np.eye(len(vocab)) # each row is a word vector\n", "\n", "def cosine(a, b):\n", " return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))\n", "\n", "for i, w1 in enumerate(vocab):\n", " for j, w2 in enumerate(vocab):\n", " if i < j:\n", " print(f\"cos({w1}, {w2}) = {cosine(one_hot[i], one_hot[j]):.1f}\")" ] }, { "cell_type": "markdown", "id": "cell-0005", "metadata": {}, "source": [ "Every similarity is 0.0. One-hot vectors carry **no notion of meaning** — the\n", "model has to relearn from scratch that \"good\" and \"great\" are related, in\n", "every task, from its own limited training data." ] }, { "cell_type": "markdown", "id": "cell-0006", "metadata": {}, "source": [ "## The distributional hypothesis\n", "\n", "The fix comes from a 1957 insight by linguist J.R. Firth: *\"You shall know a\n", "word by the company it keeps.\"* Words that appear in similar contexts tend to\n", "have similar meanings. You've never needed a dictionary to guess that in\n", "\"I poured a glass of *tezgüino* and got drunk\", tezgüino is some kind of\n", "alcoholic drink — the surrounding words told you.\n", "\n", "Word embeddings operationalize this: learn a dense vector (typically 50–300\n", "dimensions) for each word such that **words appearing in similar contexts get\n", "similar vectors**. Similarity is no longer zero everywhere — it reflects\n", "usage, which is a good proxy for meaning." ] }, { "cell_type": "markdown", "id": "cell-0007", "metadata": {}, "source": [ "## Word2Vec: two training games\n", "\n", "Word2Vec (Mikolov et al., 2013) learns embeddings by playing a fill-in-the-blank\n", "game over billions of sentence windows. Slide a window across text — say\n", "\"the quick brown fox jumps\" — and train a tiny network on one of two tasks:\n", "\n", "- **CBOW (continuous bag of words)**: given the context words (\"the\", \"quick\",\n", " \"fox\", \"jumps\"), predict the center word (\"brown\"). Fast, works well on\n", " frequent words.\n", "- **Skip-gram**: given the center word (\"brown\"), predict each context word.\n", " Slower, but better for rare words and small corpora.\n", "\n", "The network itself is almost trivially simple — one hidden layer, no\n", "activation. The magic is that to get good at the prediction game, the hidden\n", "layer is *forced* to place words used in similar contexts near each other.\n", "The predictions are then thrown away; **the hidden-layer weights are the\n", "embeddings**." ] }, { "cell_type": "markdown", "id": "cell-0008", "metadata": {}, "source": [ "> **The famous analogies**\n", "> \n", "> Trained embeddings pick up directions with consistent meaning. The vector\n", "> from \"man\" to \"woman\" is roughly the same as the one from \"king\" to \"queen\",\n", "> so `king − man + woman ≈ queen`. The same trick recovers capitals\n", "> (`paris − france + italy ≈ rome`) and verb tenses. Nobody programmed this in —\n", "> it falls out of the prediction game." ] }, { "cell_type": "markdown", "id": "cell-0009", "metadata": {}, "source": [ "## Geometry of meaning, by hand\n", "\n", "Before training anything, let's build intuition with a tiny hand-crafted\n", "embedding space: two dimensions, one for \"royalty\" and one for \"femininity\".\n", "Watch cosine similarity behave sensibly and the analogy arithmetic work:" ] }, { "cell_type": "code", "id": "cell-0010", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "\n", "# hand-crafted 2-D vectors: (royalty, femininity)\n", "words = {\n", " \"king\": np.array([0.95, 0.15]),\n", " \"queen\": np.array([0.95, 0.85]),\n", " \"man\": np.array([0.15, 0.15]),\n", " \"woman\": np.array([0.15, 0.85]),\n", " \"prince\": np.array([0.80, 0.20]),\n", " \"girl\": np.array([0.05, 0.90]),\n", " \"boy\": np.array([0.05, 0.10]),\n", "}\n", "\n", "def cosine(a, b):\n", " return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))\n", "\n", "print(f\"cos(king, queen) = {cosine(words['king'], words['queen']):.2f}\")\n", "print(f\"cos(king, girl) = {cosine(words['king'], words['girl']):.2f}\")\n", "\n", "# the analogy: king - man + woman = ?\n", "target = words[\"king\"] - words[\"man\"] + words[\"woman\"]\n", "best = max((w for w in words if w != \"king\"),\n", " key=lambda w: cosine(words[w], target))\n", "print(f\"king - man + woman -> {best}\")\n", "\n", "for w, v in words.items():\n", " plt.scatter(*v)\n", " plt.annotate(w, v, textcoords=\"offset points\", xytext=(6, 4))\n", "plt.xlabel(\"royalty\"); plt.ylabel(\"femininity\")\n", "plt.title(\"A hand-made embedding space\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "cell-0011", "metadata": {}, "source": [ "Real Word2Vec does exactly this, except the dimensions are learned rather than\n", "hand-labeled, and there are 100+ of them." ] }, { "cell_type": "markdown", "id": "cell-0012", "metadata": {}, "source": [ "## Training Word2Vec with gensim\n", "\n", "Gensim can't run in the browser, so run the rest of this lesson in the\n", "downloadable notebook (Colab works great — no GPU needed for a small corpus).\n", "The modern gensim 4.x API:" ] }, { "cell_type": "code", "id": "cell-0013", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from gensim.models import Word2Vec\n", "\n", "# a corpus = list of tokenized sentences (use thousands+ in practice)\n", "corpus = [\n", " [\"the\", \"movie\", \"was\", \"great\", \"and\", \"the\", \"acting\", \"superb\"],\n", " [\"a\", \"terrible\", \"movie\", \"with\", \"awful\", \"acting\"],\n", " [\"the\", \"film\", \"was\", \"fantastic\", \"truly\", \"great\"],\n", " [\"awful\", \"plot\", \"and\", \"terrible\", \"pacing\"],\n", " # ... many more sentences\n", "]\n", "\n", "model = Word2Vec(\n", " sentences=corpus,\n", " vector_size=100, # embedding dimensions\n", " window=5, # context words on each side\n", " min_count=1, # ignore rarer words (use 3-5 on real corpora)\n", " sg=1, # 1 = skip-gram, 0 = CBOW\n", " epochs=50,\n", " workers=4,\n", ")\n", "\n", "print(model.wv[\"great\"].shape) # (100,)\n", "print(model.wv.most_similar(\"great\", topn=5)) # neighbors by cosine\n", "print(model.wv.similarity(\"great\", \"awful\")) # a single pair" ] }, { "cell_type": "markdown", "id": "cell-0014", "metadata": {}, "source": [ "Note the gensim 4.x conventions: the trained vectors live in `model.wv`, the\n", "dimension argument is `vector_size`, and the epoch count is `epochs`.\n", "\n", "Training good embeddings needs *lots* of text, so in practice you usually load\n", "vectors pretrained on billions of words:" ] }, { "cell_type": "code", "id": "cell-0015", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import gensim.downloader as api\n", "\n", "glove = api.load(\"glove-wiki-gigaword-50\") # ~66 MB download\n", "\n", "print(glove.most_similar(\"coffee\", topn=5))\n", "print(glove.most_similar(positive=[\"king\", \"woman\"], negative=[\"man\"], topn=3))\n", "# -> queen comes out on top" ] }, { "cell_type": "markdown", "id": "cell-0016", "metadata": {}, "source": [ "## FastText: words are made of pieces\n", "\n", "Word2Vec has a blind spot: it learns one vector **per whole word**. Ask it\n", "about a typo (\"fantasttic\") or a word absent from training and it simply\n", "fails — the word is out of vocabulary (OOV).\n", "\n", "FastText (from Facebook AI) fixes this by representing each word as the sum of\n", "its **character n-grams**. \"fantastic\" becomes pieces like \"fan\", \"ant\",\n", "\"tas\", ..., plus the whole word. Consequences:\n", "\n", "- **Typos** share most n-grams with the correct word, so they land nearby.\n", "- **Morphology** comes for free: \"run\", \"running\", \"runner\" share subwords.\n", "- **OOV words** get a vector by summing their n-grams — no lookup failure.\n", "\n", "The API is a drop-in replacement:" ] }, { "cell_type": "code", "id": "cell-0017", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from gensim.models import FastText\n", "\n", "model = FastText(\n", " sentences=corpus,\n", " vector_size=100,\n", " window=5,\n", " min_count=1,\n", " epochs=50,\n", ")\n", "\n", "# works even if this exact string never appeared in training:\n", "print(model.wv[\"fantasttic\"][:5]) # no KeyError\n", "print(model.wv.similarity(\"fantastic\", \"fantasttic\")) # high, thanks to shared n-grams" ] }, { "cell_type": "markdown", "id": "cell-0018", "metadata": {}, "source": [ "For noisy user-generated text — reviews, tweets, chat logs — FastText's typo\n", "tolerance is a big practical win over vanilla Word2Vec." ] }, { "cell_type": "markdown", "id": "cell-0019", "metadata": {}, "source": [ "## The bridge to deep NLP\n", "\n", "In a PyTorch model, embeddings live in an `nn.Embedding` layer: a lookup table\n", "of shape (vocab_size, embedding_dim) that maps token IDs to vectors and is\n", "trained by backprop like any other layer. You can start it from random values,\n", "or warm-start it with pretrained vectors:" ] }, { "cell_type": "code", "id": "cell-0020", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import torch\n", "from torch import nn\n", "\n", "embedding = nn.Embedding(num_embeddings=len(glove), embedding_dim=50)\n", "embedding.weight.data.copy_(torch.tensor(glove.vectors))" ] }, { "cell_type": "markdown", "id": "cell-0021", "metadata": {}, "source": [ "Either way, the embedding layer is the standard first layer of every deep NLP\n", "model — including the sentiment classifier we build next." ] }, { "cell_type": "markdown", "id": "cell-0022", "metadata": {}, "source": [ "### 🏋️ Exercise — Analogy arithmetic with pretrained GloVe\n", "\n", "In a notebook (Colab is fine, no GPU needed), load `glove-wiki-gigaword-50`\n", "via `gensim.downloader` and test three analogies with `most_similar`: the\n", "classic `king − man + woman`, the capital-city analogy\n", "`paris − france + italy`, and one analogy of your own invention. Then find at\n", "least one analogy that produces a *wrong or biased* answer and write one\n", "sentence about why that happens." ] }, { "cell_type": "code", "id": "cell-0023", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Your solution here" ] }, { "cell_type": "markdown", "id": "cell-0024", "metadata": {}, "source": [ "
✅ Show solution\n", "\n", "```python\n", "import gensim.downloader as api\n", "\n", "glove = api.load(\"glove-wiki-gigaword-50\")\n", "\n", "# 1. king - man + woman\n", "print(glove.most_similar(positive=[\"king\", \"woman\"], negative=[\"man\"], topn=3))\n", "\n", "# 2. paris - france + italy\n", "print(glove.most_similar(positive=[\"paris\", \"italy\"], negative=[\"france\"], topn=3))\n", "\n", "# 3. my own: walking - walk + swim (verb morphology)\n", "print(glove.most_similar(positive=[\"walking\", \"swim\"], negative=[\"walk\"], topn=3))\n", "\n", "# 4. where analogies break down\n", "print(glove.most_similar(positive=[\"doctor\", \"woman\"], negative=[\"man\"], topn=3))\n", "# Embeddings absorb biases present in the training text - results can\n", "# reflect stereotypes rather than logic. Never treat them as ground truth.\n", "```\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "cell-0025", "metadata": {}, "source": [ "Next up: putting embeddings to work — building a sentiment classifier from a\n", "TF-IDF baseline all the way to an LSTM." ] } ] }