Sentiment Analysis End to End

Sentiment analysis — deciding whether a piece of text is positive or negative — is the classic first "real" NLP project: the task is easy to state, the labels are cheap to get, and every technique from the last lesson gets put to work. In this lesson you'll build the same classifier three ways: a classical baseline that runs right here in your browser, a deep Embedding + LSTM model in PyTorch, and finally the modern transformer shortcut.

The task: is this review positive or negative?

We'll treat sentiment as binary classification on short reviews: each input is a piece of text, the target is 1 (positive) or 0 (negative). The canonical benchmark is IMDB: 50,000 movie reviews split evenly into train and test, labeled by star rating (1–4 stars = negative, 7–10 = positive; the ambiguous middle is dropped).

That labeling detail is worth pausing on — it's typical dataset thinking for sentiment. Labels usually come from something users already provide (star ratings, thumbs up/down), which is what makes the task cheap. But it also means the classes reflect how people rate: app-store reviews, for instance, pile up at 1 star and 5 stars, so real sentiment datasets are often imbalanced, and accuracy alone can be misleading. Keep that in mind — we return to it in the evaluation section.

Preprocessing: from raw text to model food

Raw reviews are messy — capitalization, punctuation, wildly different lengths. A standard cleanup pipeline:

1. Lowercase everything, so "Great" and "great" are one token.
2. Tokenize — split text into words (or subwords). A simple text.lower().split() gets you surprisingly far in English.
3. Stopwords — carefully. Removing ultra-common words ("the", "a", "of") shrinks the vocabulary and can reduce noise for topic tasks. But standard stopword lists include "not", "no", and "never" — and deleting those turns "not good" into "good". For sentiment, either keep stopwords or prune the list by hand.
4. Pad or truncate to a fixed length. Neural networks train on batches, and a batch is a rectangular tensor — so every sequence in it must have the same length. Short reviews get filled with a special <pad> token; long ones get cut at, say, 200 tokens.

Always start with the baseline: TF-IDF + logistic regression

Before touching a neural network, build the ten-line classical pipeline: TF-IDF turns each review into a vector of word weights (frequent in this document, rare across the corpus = high weight), and logistic regression learns which words push a review positive or negative. This baseline is fast, interpretable, and embarrassingly hard to beat on small datasets.

Here it is end to end on sixteen mini-reviews — training, predicting new sentences, and showing which n-grams the model learned to trust:

Both new sentences come out on the right side — but notice the probabilities hover near 0.5. Sixteen examples is nowhere near enough for confident predictions; on the full IMDB training set this exact pipeline reaches roughly 88–90% accuracy. That's the number any deep model has to beat.

Evaluation beyond accuracy

A single accuracy number hides where a model fails. The confusion matrix breaks predictions into four cells — true/false positives and negatives — and from it come precision (of everything predicted positive, how much really was?) and recall (of everything really positive, how much did we catch?).

To make this concrete, let's generate a corpus with a built-in trap: mixed-sentiment reviews like "the acting was great but the plot was dull", where the verdict after "but" wins. A bag-of-words model sees the same words in both orders, so it cannot tell "great but ... dull" from "dull but ... great":

Around 75% accuracy overall — but the mixed reviews are barely above coin-flip territory, and the confusion matrix shows the misses landing on both sides. The lesson: bag-of-words throws away word order, and some sentiment lives in the order. That is exactly the itch a recurrent model scratches.

The deep version: Embedding → LSTM → linear head

The deep pipeline reads a review as a sequence: token IDs go through an nn.Embedding lookup, an LSTM reads the vectors left to right while carrying a memory, and a linear head turns the final memory into one logit. PyTorch and the IMDB download don't run in the browser, so run these cells in the downloadable notebook on Colab with a GPU.

First, data and a vocabulary. We keep words seen at least twice, reserve ID 0 for padding and ID 1 for unknown words, and clamp every review to 200 tokens:

from collections import Counter
from datasets import load_dataset
 
imdb = load_dataset("imdb")
train_texts, train_labels = imdb["train"]["text"], imdb["train"]["label"]
test_texts, test_labels = imdb["test"]["text"], imdb["test"]["label"]
 
def tokenize(text):
    return text.lower().split()
 
counter = Counter(tok for text in train_texts for tok in tokenize(text))
vocab = {"<pad>": 0, "<unk>": 1}
for word, count in counter.most_common():
    if count >= 2:
        vocab[word] = len(vocab)
 
MAX_LEN = 200
def encode(text):
    ids = [vocab.get(tok, 1) for tok in tokenize(text)][:MAX_LEN]
    return ids + [0] * (MAX_LEN - len(ids))   # pad to fixed length

The model — follow the tensor shapes in the comments, they are the whole story:

import torch
from torch import nn
 
class SentimentLSTM(nn.Module):
    def __init__(self, vocab_size, embed_dim=100, hidden_dim=128):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True)
        self.head = nn.Linear(hidden_dim, 1)
 
    def forward(self, x):              # x:   (batch, seq)       token IDs
        emb = self.embedding(x)        # emb: (batch, seq, embed) vectors
        out, (h, c) = self.lstm(emb)   # h:   (1, batch, hidden)  final state
        return self.head(h[-1])        # ->   (batch, 1)          one logit

A batch of 64 reviews enters as a (64, 200) integer tensor. The embedding layer replaces each ID with its 100-dimensional vector: (64, 200, 100). The LSTM digests the 200 steps and hands back its final hidden state h, one 128-dimensional summary per review — we take h[-1], shape (64, 128), and the head maps it to (64, 1) logits. BCEWithLogitsLoss applies the sigmoid internally:

from torch.utils.data import TensorDataset, DataLoader
 
device = "cuda" if torch.cuda.is_available() else "cpu"
 
X_train = torch.tensor([encode(t) for t in train_texts])
y_train = torch.tensor(train_labels, dtype=torch.float32)
loader = DataLoader(TensorDataset(X_train, y_train), batch_size=64, shuffle=True)
 
model = SentimentLSTM(len(vocab)).to(device)
criterion = nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
 
for epoch in range(5):
    model.train()
    total = 0.0
    for xb, yb in loader:
        xb, yb = xb.to(device), yb.to(device)
        loss = criterion(model(xb).squeeze(1), yb)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        total += loss.item() * len(xb)
    print(f"epoch {epoch + 1}: train loss {total / len(loader.dataset):.4f}")

Warm-starting with pretrained embeddings

Trained from scratch, the embedding layer has to rediscover that "superb" and "wonderful" are related — from 25,000 reviews. The previous lesson's GloVe vectors already know this from billions of words, so copy them in as the starting point:

import gensim.downloader as api
 
glove = api.load("glove-wiki-gigaword-100")   # matches embed_dim=100
 
hits = 0
with torch.no_grad():
    for word, idx in vocab.items():
        if word in glove:
            model.embedding.weight[idx] = torch.tensor(glove[word])
            hits += 1
print(f"initialized {hits}/{len(vocab)} embedding rows from GloVe")

The rows stay trainable, so fine-tuning nudges them toward sentiment-specific meanings. Warm-starting typically buys a couple of accuracy points and much faster convergence — enough to push this LSTM a little past the TF-IDF baseline on IMDB.

What you'd do today: fine-tune a transformer

Honest guidance: in 2026 nobody starts a new sentiment project with an LSTM. Pretrained transformers (BERT and descendants) read the whole sequence with attention, come already trained on enormous corpora, and fine-tune to 93–95% on IMDB in minutes. With Hugging Face, inference is three lines:

from transformers import pipeline
 
clf = pipeline("sentiment-analysis",
               model="distilbert-base-uncased-finetuned-sst-2-english")
print(clf(["An absolute masterpiece.",
           "Two hours of my life I will never get back."]))

So why build the LSTM at all? Because the transformer pipeline is this same pipeline — tokenize, embed, encode the sequence, classify from a summary vector — with attention swapped in for recurrence. Having built it once by hand, nothing inside the black box will surprise you.

Next up: a new module and a new goal — instead of predicting labels, we'll train networks that generate data, starting with autoencoders.