Word Embeddings: Word2Vec & FastText

Neural networks eat numbers, not words. Before any deep NLP model can read a sentence, every word has to become a vector — and how you build those vectors decides whether the model starts from "cat and kitten are related" or from total ignorance. In this lesson you'll see why the naive encoding fails, how Word2Vec learns meaning from raw text, and how FastText extends the idea to typos and brand-new words.

Why one-hot vectors fail

The simplest encoding assigns each word its own slot: with a vocabulary of 50,000 words, "cat" becomes a 50,000-dimensional vector that is all zeros except for a single 1. Two problems kill this approach:

1. It's huge. One vector per word, each as long as the vocabulary.
2. Every word is equally distant from every other word. "cat" is exactly as far from "kitten" as it is from "carburetor".

The second problem is the fatal one. A standard way to measure vector similarity is cosine similarity — the cosine of the angle between two vectors (1 = same direction, 0 = perpendicular, −1 = opposite). Run this and watch every one-hot pair come out perpendicular:

Every similarity is 0.0. One-hot vectors carry no notion of meaning — the model has to relearn from scratch that "good" and "great" are related, in every task, from its own limited training data.

The distributional hypothesis

The fix comes from a 1957 insight by linguist J.R. Firth: "You shall know a word by the company it keeps." Words that appear in similar contexts tend to have similar meanings. You've never needed a dictionary to guess that in "I poured a glass of tezgüino and got drunk", tezgüino is some kind of alcoholic drink — the surrounding words told you.

Word embeddings operationalize this: learn a dense vector (typically 50–300 dimensions) for each word such that words appearing in similar contexts get similar vectors. Similarity is no longer zero everywhere — it reflects usage, which is a good proxy for meaning.

Word2Vec: two training games

Word2Vec (Mikolov et al., 2013) learns embeddings by playing a fill-in-the-blank game over billions of sentence windows. Slide a window across text — say "the quick brown fox jumps" — and train a tiny network on one of two tasks:

• CBOW (continuous bag of words): given the context words ("the", "quick", "fox", "jumps"), predict the center word ("brown"). Fast, works well on frequent words.
• Skip-gram: given the center word ("brown"), predict each context word. Slower, but better for rare words and small corpora.

The network itself is almost trivially simple — one hidden layer, no activation. The magic is that to get good at the prediction game, the hidden layer is forced to place words used in similar contexts near each other. The predictions are then thrown away; the hidden-layer weights are the embeddings.

Geometry of meaning, by hand

Before training anything, let's build intuition with a tiny hand-crafted embedding space: two dimensions, one for "royalty" and one for "femininity". Watch cosine similarity behave sensibly and the analogy arithmetic work:

Real Word2Vec does exactly this, except the dimensions are learned rather than hand-labeled, and there are 100+ of them.

Training Word2Vec with gensim

Gensim can't run in the browser, so run the rest of this lesson in the downloadable notebook (Colab works great — no GPU needed for a small corpus). The modern gensim 4.x API:

from gensim.models import Word2Vec
 
# a corpus = list of tokenized sentences (use thousands+ in practice)
corpus = [
    ["the", "movie", "was", "great", "and", "the", "acting", "superb"],
    ["a", "terrible", "movie", "with", "awful", "acting"],
    ["the", "film", "was", "fantastic", "truly", "great"],
    ["awful", "plot", "and", "terrible", "pacing"],
    # ... many more sentences
]
 
model = Word2Vec(
    sentences=corpus,
    vector_size=100,   # embedding dimensions
    window=5,          # context words on each side
    min_count=1,       # ignore rarer words (use 3-5 on real corpora)
    sg=1,              # 1 = skip-gram, 0 = CBOW
    epochs=50,
    workers=4,
)
 
print(model.wv["great"].shape)                 # (100,)
print(model.wv.most_similar("great", topn=5)) # neighbors by cosine
print(model.wv.similarity("great", "awful"))  # a single pair

Note the gensim 4.x conventions: the trained vectors live in model.wv, the dimension argument is vector_size, and the epoch count is epochs.

Training good embeddings needs lots of text, so in practice you usually load vectors pretrained on billions of words:

import gensim.downloader as api
 
glove = api.load("glove-wiki-gigaword-50")   # ~66 MB download
 
print(glove.most_similar("coffee", topn=5))
print(glove.most_similar(positive=["king", "woman"], negative=["man"], topn=3))
# -> queen comes out on top

FastText: words are made of pieces

Word2Vec has a blind spot: it learns one vector per whole word. Ask it about a typo ("fantasttic") or a word absent from training and it simply fails — the word is out of vocabulary (OOV).

FastText (from Facebook AI) fixes this by representing each word as the sum of its character n-grams. "fantastic" becomes pieces like "fan", "ant", "tas", ..., plus the whole word. Consequences:

• Typos share most n-grams with the correct word, so they land nearby.
• Morphology comes for free: "run", "running", "runner" share subwords.
• OOV words get a vector by summing their n-grams — no lookup failure.

The API is a drop-in replacement:

from gensim.models import FastText
 
model = FastText(
    sentences=corpus,
    vector_size=100,
    window=5,
    min_count=1,
    epochs=50,
)
 
# works even if this exact string never appeared in training:
print(model.wv["fantasttic"][:5])                      # no KeyError
print(model.wv.similarity("fantastic", "fantasttic"))  # high, thanks to shared n-grams

For noisy user-generated text — reviews, tweets, chat logs — FastText's typo tolerance is a big practical win over vanilla Word2Vec.

The bridge to deep NLP

In a PyTorch model, embeddings live in an nn.Embedding layer: a lookup table of shape (vocab_size, embedding_dim) that maps token IDs to vectors and is trained by backprop like any other layer. You can start it from random values, or warm-start it with pretrained vectors:

import torch
from torch import nn
 
embedding = nn.Embedding(num_embeddings=len(glove), embedding_dim=50)
embedding.weight.data.copy_(torch.tensor(glove.vectors))

Either way, the embedding layer is the standard first layer of every deep NLP model — including the sentiment classifier we build next.

Next up: putting embeddings to work — building a sentiment classifier from a TF-IDF baseline all the way to an LSTM.