{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "id": "cell-0000", "metadata": {}, "source": [ "# Sentiment Analysis End to End\n", "\n", "Build a sentiment classifier three ways — a TF-IDF baseline you can run in the browser, an Embedding + LSTM model in PyTorch, and a modern transformer pipeline.\n", "\n", "*Part of the free [Deep Learning with PyTorch](https://ramadnsyh.dev/courses/deep-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/deep-learning/sentiment-analysis).*" ] }, { "cell_type": "code", "id": "cell-0001", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "%pip install -q gensim datasets transformers" ] }, { "cell_type": "markdown", "id": "cell-0002", "metadata": {}, "source": [ "Sentiment analysis — deciding whether a piece of text is positive or negative —\n", "is the classic first \"real\" NLP project: the task is easy to state, the labels\n", "are cheap to get, and every technique from the last lesson gets put to work.\n", "In this lesson you'll build the same classifier three ways: a classical\n", "baseline that runs right here in your browser, a deep Embedding + LSTM model\n", "in PyTorch, and finally the modern transformer shortcut." ] }, { "cell_type": "markdown", "id": "cell-0003", "metadata": {}, "source": [ "## The task: is this review positive or negative?\n", "\n", "We'll treat sentiment as **binary classification on short reviews**: each\n", "input is a piece of text, the target is 1 (positive) or 0 (negative). The\n", "canonical benchmark is **IMDB**: 50,000 movie reviews split evenly into train\n", "and test, labeled by star rating (1–4 stars = negative, 7–10 = positive; the\n", "ambiguous middle is dropped).\n", "\n", "That labeling detail is worth pausing on — it's typical **dataset thinking**\n", "for sentiment. Labels usually come from something users already provide (star\n", "ratings, thumbs up/down), which is what makes the task cheap. But it also\n", "means the classes reflect how people rate: app-store reviews, for instance,\n", "pile up at 1 star and 5 stars, so real sentiment datasets are often\n", "**imbalanced**, and accuracy alone can be misleading. Keep that in mind — we\n", "return to it in the evaluation section." ] }, { "cell_type": "markdown", "id": "cell-0004", "metadata": {}, "source": [ "## Preprocessing: from raw text to model food\n", "\n", "Raw reviews are messy — capitalization, punctuation, wildly different lengths.\n", "A standard cleanup pipeline:\n", "\n", "1. **Lowercase** everything, so \"Great\" and \"great\" are one token.\n", "2. **Tokenize** — split text into words (or subwords). A simple\n", " `text.lower().split()` gets you surprisingly far in English.\n", "3. **Stopwords — carefully.** Removing ultra-common words (\"the\", \"a\", \"of\")\n", " shrinks the vocabulary and can reduce noise for topic tasks. But standard\n", " stopword lists include **\"not\"**, \"no\", and \"never\" — and deleting those\n", " turns \"not good\" into \"good\". For sentiment, either keep stopwords or prune\n", " the list by hand.\n", "4. **Pad or truncate to a fixed length.** Neural networks train on batches,\n", " and a batch is a rectangular tensor — so every sequence in it must have the\n", " same length. Short reviews get filled with a special `` token; long\n", " ones get cut at, say, 200 tokens." ] }, { "cell_type": "markdown", "id": "cell-0005", "metadata": {}, "source": [ "> **Stopword lists are task-dependent**\n", "> \n", "> The words that are \"meaningless\" for document topic detection can be the most\n", "> meaningful ones for sentiment. Negations flip polarity, and intensifiers like\n", "> \"very\" and \"so\" amplify it. Never apply a stopword list without reading it\n", "> first." ] }, { "cell_type": "markdown", "id": "cell-0006", "metadata": {}, "source": [ "## Always start with the baseline: TF-IDF + logistic regression\n", "\n", "Before touching a neural network, build the ten-line classical pipeline:\n", "**TF-IDF** turns each review into a vector of word weights (frequent in this\n", "document, rare across the corpus = high weight), and **logistic regression**\n", "learns which words push a review positive or negative. This baseline is fast,\n", "interpretable, and embarrassingly hard to beat on small datasets.\n", "\n", "Here it is end to end on sixteen mini-reviews — training, predicting new\n", "sentences, and showing which n-grams the model learned to trust:" ] }, { "cell_type": "code", "id": "cell-0007", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "from sklearn.feature_extraction.text import TfidfVectorizer\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.pipeline import make_pipeline\n", "\n", "reviews = [\n", " \"a brilliant film with superb acting\",\n", " \"the plot was gripping from start to finish\",\n", " \"great movie, I loved every minute\",\n", " \"wonderful cast and a beautiful story\",\n", " \"one of the best films I have ever seen\",\n", " \"sharp writing and a fantastic soundtrack\",\n", " \"funny, touching, and full of heart\",\n", " \"an instant classic, truly amazing\",\n", " \"a dull film with wooden acting\",\n", " \"the plot was boring and predictable\",\n", " \"terrible movie, I wanted my money back\",\n", " \"awful pacing and a pointless story\",\n", " \"one of the worst films I have ever seen\",\n", " \"lazy writing and a forgettable soundtrack\",\n", " \"not funny, not touching, just empty\",\n", " \"a complete mess, truly disappointing\",\n", "]\n", "labels = [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]\n", "\n", "clf = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)), LogisticRegression())\n", "clf.fit(reviews, labels)\n", "\n", "new_reviews = [\n", " \"a fantastic story with brilliant acting\",\n", " \"boring, predictable, and a total mess\",\n", "]\n", "for text, p in zip(new_reviews, clf.predict_proba(new_reviews)[:, 1]):\n", " verdict = \"positive\" if p >= 0.5 else \"negative\"\n", " print(f\"P(positive) = {p:.2f} -> {verdict:8s} '{text}'\")\n", "\n", "coefs = clf.named_steps[\"logisticregression\"].coef_[0]\n", "names = clf.named_steps[\"tfidfvectorizer\"].get_feature_names_out()\n", "order = np.argsort(coefs)\n", "print(\"most negative n-grams:\", \", \".join(names[order[:4]]))\n", "print(\"most positive n-grams:\", \", \".join(names[order[-4:]]))" ] }, { "cell_type": "markdown", "id": "cell-0008", "metadata": {}, "source": [ "Both new sentences come out on the right side — but notice the probabilities\n", "hover near 0.5. Sixteen examples is nowhere near enough for confident\n", "predictions; on the full IMDB training set this exact pipeline reaches roughly\n", "88–90% accuracy. **That's the number any deep model has to beat.**" ] }, { "cell_type": "markdown", "id": "cell-0009", "metadata": {}, "source": [ "## Evaluation beyond accuracy\n", "\n", "A single accuracy number hides *where* a model fails. The **confusion matrix**\n", "breaks predictions into four cells — true/false positives and negatives — and\n", "from it come **precision** (of everything predicted positive, how much really\n", "was?) and **recall** (of everything really positive, how much did we catch?).\n", "\n", "To make this concrete, let's generate a corpus with a built-in trap:\n", "**mixed-sentiment reviews** like \"the acting was great but the plot was dull\",\n", "where the verdict after \"but\" wins. A bag-of-words model sees the same words\n", "in both orders, so it *cannot* tell \"great but ... dull\" from\n", "\"dull but ... great\":" ] }, { "cell_type": "code", "id": "cell-0010", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn.feature_extraction.text import TfidfVectorizer\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.pipeline import make_pipeline\n", "from sklearn.metrics import classification_report, ConfusionMatrixDisplay\n", "\n", "rng = np.random.default_rng(42)\n", "pos = [\"great\", \"brilliant\", \"superb\", \"wonderful\", \"fantastic\", \"moving\"]\n", "neg = [\"dull\", \"awful\", \"boring\", \"terrible\", \"predictable\", \"lazy\"]\n", "nouns = [\"film\", \"plot\", \"acting\", \"soundtrack\", \"dialogue\", \"ending\"]\n", "\n", "texts, labels = [], []\n", "for _ in range(40): # easy reviews\n", " texts.append(f\"the {rng.choice(nouns)} was {rng.choice(pos)}\"); labels.append(1)\n", " texts.append(f\"the {rng.choice(nouns)} was {rng.choice(neg)}\"); labels.append(0)\n", "for _ in range(20): # mixed reviews - the clause after 'but' wins\n", " a, b = rng.choice(nouns, 2, replace=False)\n", " texts.append(f\"the {a} was {rng.choice(pos)} but the {b} was {rng.choice(neg)}\"); labels.append(0)\n", " texts.append(f\"the {a} was {rng.choice(neg)} but the {b} was {rng.choice(pos)}\"); labels.append(1)\n", "\n", "X_tr, X_te, y_tr, y_te = train_test_split(\n", " texts, labels, test_size=0.3, stratify=labels, random_state=0)\n", "\n", "clf = make_pipeline(TfidfVectorizer(), LogisticRegression()) # unigrams only\n", "clf.fit(X_tr, y_tr)\n", "pred = clf.predict(X_te)\n", "\n", "print(classification_report(y_te, pred, target_names=[\"negative\", \"positive\"]))\n", "mixed = [i for i, t in enumerate(X_te) if \"but\" in t]\n", "ok = sum(1 for i in mixed if pred[i] == y_te[i])\n", "print(f\"mixed-sentiment reviews correct: {ok}/{len(mixed)}\")\n", "\n", "ConfusionMatrixDisplay.from_predictions(\n", " y_te, pred, display_labels=[\"negative\", \"positive\"], cmap=\"Blues\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "cell-0011", "metadata": {}, "source": [ "Around 75% accuracy overall — but the mixed reviews are barely above coin-flip\n", "territory, and the confusion matrix shows the misses landing on both sides.\n", "The lesson: **bag-of-words throws away word order**, and some sentiment lives\n", "in the order. That is exactly the itch a recurrent model scratches." ] }, { "cell_type": "markdown", "id": "cell-0012", "metadata": {}, "source": [ "## The deep version: Embedding → LSTM → linear head\n", "\n", "The deep pipeline reads a review as a *sequence*: token IDs go through an\n", "`nn.Embedding` lookup, an LSTM reads the vectors left to right while carrying\n", "a memory, and a linear head turns the final memory into one logit. PyTorch and\n", "the IMDB download don't run in the browser, so run these cells in the\n", "downloadable notebook on Colab with a GPU.\n", "\n", "First, data and a vocabulary. We keep words seen at least twice, reserve ID 0\n", "for padding and ID 1 for unknown words, and clamp every review to 200 tokens:" ] }, { "cell_type": "code", "id": "cell-0013", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from collections import Counter\n", "from datasets import load_dataset\n", "\n", "imdb = load_dataset(\"imdb\")\n", "train_texts, train_labels = imdb[\"train\"][\"text\"], imdb[\"train\"][\"label\"]\n", "test_texts, test_labels = imdb[\"test\"][\"text\"], imdb[\"test\"][\"label\"]\n", "\n", "def tokenize(text):\n", " return text.lower().split()\n", "\n", "counter = Counter(tok for text in train_texts for tok in tokenize(text))\n", "vocab = {\"\": 0, \"\": 1}\n", "for word, count in counter.most_common():\n", " if count >= 2:\n", " vocab[word] = len(vocab)\n", "\n", "MAX_LEN = 200\n", "def encode(text):\n", " ids = [vocab.get(tok, 1) for tok in tokenize(text)][:MAX_LEN]\n", " return ids + [0] * (MAX_LEN - len(ids)) # pad to fixed length" ] }, { "cell_type": "markdown", "id": "cell-0014", "metadata": {}, "source": [ "The model — follow the tensor shapes in the comments, they are the whole\n", "story:" ] }, { "cell_type": "code", "id": "cell-0015", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import torch\n", "from torch import nn\n", "\n", "class SentimentLSTM(nn.Module):\n", " def __init__(self, vocab_size, embed_dim=100, hidden_dim=128):\n", " super().__init__()\n", " self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)\n", " self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True)\n", " self.head = nn.Linear(hidden_dim, 1)\n", "\n", " def forward(self, x): # x: (batch, seq) token IDs\n", " emb = self.embedding(x) # emb: (batch, seq, embed) vectors\n", " out, (h, c) = self.lstm(emb) # h: (1, batch, hidden) final state\n", " return self.head(h[-1]) # -> (batch, 1) one logit" ] }, { "cell_type": "markdown", "id": "cell-0016", "metadata": {}, "source": [ "A batch of 64 reviews enters as a `(64, 200)` integer tensor. The embedding\n", "layer replaces each ID with its 100-dimensional vector: `(64, 200, 100)`. The\n", "LSTM digests the 200 steps and hands back its final hidden state `h`, one\n", "128-dimensional summary per review — we take `h[-1]`, shape `(64, 128)`, and\n", "the head maps it to `(64, 1)` logits. `BCEWithLogitsLoss` applies the sigmoid\n", "internally:" ] }, { "cell_type": "code", "id": "cell-0017", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from torch.utils.data import TensorDataset, DataLoader\n", "\n", "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", "\n", "X_train = torch.tensor([encode(t) for t in train_texts])\n", "y_train = torch.tensor(train_labels, dtype=torch.float32)\n", "loader = DataLoader(TensorDataset(X_train, y_train), batch_size=64, shuffle=True)\n", "\n", "model = SentimentLSTM(len(vocab)).to(device)\n", "criterion = nn.BCEWithLogitsLoss()\n", "optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)\n", "\n", "for epoch in range(5):\n", " model.train()\n", " total = 0.0\n", " for xb, yb in loader:\n", " xb, yb = xb.to(device), yb.to(device)\n", " loss = criterion(model(xb).squeeze(1), yb)\n", " optimizer.zero_grad()\n", " loss.backward()\n", " optimizer.step()\n", " total += loss.item() * len(xb)\n", " print(f\"epoch {epoch + 1}: train loss {total / len(loader.dataset):.4f}\")" ] }, { "cell_type": "markdown", "id": "cell-0018", "metadata": {}, "source": [ "### Warm-starting with pretrained embeddings\n", "\n", "Trained from scratch, the embedding layer has to rediscover that \"superb\" and\n", "\"wonderful\" are related — from 25,000 reviews. The previous lesson's GloVe\n", "vectors already know this from billions of words, so copy them in as the\n", "starting point:" ] }, { "cell_type": "code", "id": "cell-0019", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import gensim.downloader as api\n", "\n", "glove = api.load(\"glove-wiki-gigaword-100\") # matches embed_dim=100\n", "\n", "hits = 0\n", "with torch.no_grad():\n", " for word, idx in vocab.items():\n", " if word in glove:\n", " model.embedding.weight[idx] = torch.tensor(glove[word])\n", " hits += 1\n", "print(f\"initialized {hits}/{len(vocab)} embedding rows from GloVe\")" ] }, { "cell_type": "markdown", "id": "cell-0020", "metadata": {}, "source": [ "The rows stay trainable, so fine-tuning nudges them toward sentiment-specific\n", "meanings. Warm-starting typically buys a couple of accuracy points and much\n", "faster convergence — enough to push this LSTM a little past the TF-IDF\n", "baseline on IMDB." ] }, { "cell_type": "markdown", "id": "cell-0021", "metadata": {}, "source": [ "## What you'd do today: fine-tune a transformer\n", "\n", "Honest guidance: in 2026 nobody starts a new sentiment project with an LSTM.\n", "Pretrained transformers (BERT and descendants) read the whole sequence with\n", "attention, come already trained on enormous corpora, and fine-tune to 93–95%\n", "on IMDB in minutes. With Hugging Face, inference is three lines:" ] }, { "cell_type": "code", "id": "cell-0022", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from transformers import pipeline\n", "\n", "clf = pipeline(\"sentiment-analysis\",\n", " model=\"distilbert-base-uncased-finetuned-sst-2-english\")\n", "print(clf([\"An absolute masterpiece.\",\n", " \"Two hours of my life I will never get back.\"]))" ] }, { "cell_type": "markdown", "id": "cell-0023", "metadata": {}, "source": [ "So why build the LSTM at all? Because the transformer pipeline is this same\n", "pipeline — tokenize, embed, encode the sequence, classify from a summary\n", "vector — with attention swapped in for recurrence. Having built it once by\n", "hand, nothing inside the black box will surprise you." ] }, { "cell_type": "markdown", "id": "cell-0024", "metadata": {}, "source": [ "### 🏋️ Exercise — Fix the mixed-review failure\n", "\n", "barely helps (about 5/13 mixed correct). Each bigram like\n", "# 'brilliant but' appears only once or twice - too rare to learn.\n", "\n", "# 2. Unigrams, more mixed data (range(60)):\n", "# -> still fails (about 17/37 mixed correct). More examples of a\n", "# pattern the features cannot express does not help.\n", "\n", "# 3. Bigrams AND more mixed data:\n", "clf = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)), LogisticRegression())\n", "# with: for _ in range(60): ... in the mixed-review loop\n", "# -> about 35/37 mixed correct, roughly 97% accuracy overall.\n", "# The model needs BOTH a feature that can express the pattern\n", "# (bigrams ending in 'but') and enough data to estimate it.\n", "`}\n", ">\n", "Return to the evaluation PyRunner above and try to rescue the mixed-sentiment\n", "reviews **without a neural network**. Run three experiments: (1) switch the\n", "vectorizer to bigrams with `ngram_range=(1, 2)`; (2) keep unigrams but triple\n", "the mixed examples by changing `range(20)` to `range(60)`; (3) do both. For\n", "each, note the mixed-review score. Which change matters, and why do you need\n", "both?" ] }, { "cell_type": "code", "id": "cell-0025", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Your solution here" ] }, { "cell_type": "markdown", "id": "cell-0026", "metadata": {}, "source": [ "
✅ Show solution\n", "\n", "```python\n", "# Starting from the evaluation PyRunner above, run three variants:\n", "\n", "# 1. Bigrams, same data (range(20)):\n", "# clf = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)), LogisticRegression())\n", "# -> barely helps (about 5/13 mixed correct). Each bigram like\n", "# 'brilliant but' appears only once or twice - too rare to learn.\n", "\n", "# 2. Unigrams, more mixed data (range(60)):\n", "# -> still fails (about 17/37 mixed correct). More examples of a\n", "# pattern the features cannot express does not help.\n", "\n", "# 3. Bigrams AND more mixed data:\n", "clf = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)), LogisticRegression())\n", "# with: for _ in range(60): ... in the mixed-review loop\n", "# -> about 35/37 mixed correct, roughly 97% accuracy overall.\n", "# The model needs BOTH a feature that can express the pattern\n", "# (bigrams ending in 'but') and enough data to estimate it.\n", "```\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "cell-0027", "metadata": {}, "source": [ "Next up: a new module and a new goal — instead of predicting labels, we'll\n", "train networks that *generate* data, starting with autoencoders." ] } ] }