{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11" }, "colab": { "provenance": [] } }, "cells": [ { "cell_type": "markdown", "id": "cell-0000", "metadata": {}, "source": [ "# Popularity & Weighted Ratings\n", "\n", "Build your first recommender — a \"top charts\" list that ranks movies fairly by blending average rating with vote count using the IMDB weighted-rating formula.\n", "\n", "*Part of the free [Machine Learning](https://ramadnsyh.dev/courses/machine-learning) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/machine-learning/demographic-filtering).*" ] }, { "cell_type": "markdown", "id": "cell-0001", "metadata": {}, "source": [ "Every product you use — Netflix, YouTube, Spotify, Tokopedia — spends enormous\n", "effort answering one question: *out of thousands of items, which ten should we\n", "show this person right now?* That's the recommendation problem, and this module\n", "builds the three classic answers to it from scratch. We start with the simplest\n", "one: recommend what's popular — but do it *fairly*." ] }, { "cell_type": "markdown", "id": "cell-0002", "metadata": {}, "source": [ "## The recommendation problem, and three ways to solve it\n", "\n", "A recommender system ranks items for a user. The three classic families differ\n", "in what information they use to build that ranking:\n", "\n", "- **Popularity / demographic filtering** — recommend what people in general\n", " like: \"Top 50 movies of the year\". No personalization; everyone sees the same\n", " list. It only needs item statistics (ratings, vote counts, genre, year).\n", "- **Content-based filtering** — recommend items *similar to what you already\n", " liked*: \"More like Toy Story\". It compares item attributes — genre, cast,\n", " synopsis — so it needs a good description of each item.\n", "- **Collaborative filtering** — recommend what *people with similar taste*\n", " liked: \"Users who watched this also watched…\". It ignores item attributes\n", " entirely and learns purely from behavior.\n", "\n", "They form a ladder of personalization, and real systems combine all three.\n", "This lesson climbs the first rung." ] }, { "cell_type": "markdown", "id": "cell-0003", "metadata": {}, "source": [ "## Why the raw average rating fails\n", "\n", "The obvious popularity ranking — sort by average rating — has a famous flaw.\n", "Suppose one film holds a 9.6 average from **3 votes** and another holds an 8.7\n", "from **26,000 votes**. Which is actually better? Almost certainly the second:\n", "three enthusiastic friends can produce a 9.6, but 26,000 strangers agreeing on\n", "8.7 is strong evidence. A raw sort can't tell the difference. Watch it happen:" ] }, { "cell_type": "code", "id": "cell-0004", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import pandas as pd\n", "\n", "movies = pd.DataFrame({\n", " \"title\": [\n", " \"The Shawshank Redemption\", \"The Dark Knight\", \"Inception\",\n", " \"Interstellar\", \"Parasite\", \"Whiplash\", \"Joker\",\n", " \"Avengers: Endgame\", \"La La Land\", \"Knives Out\",\n", " \"Mad Max: Fury Road\", \"Get Out\", \"The Emoji Movie\",\n", " \"Midnight Static\", \"The Lost Reel\",\n", " ],\n", " \"vote_average\": [8.7, 8.5, 8.4, 8.4, 8.5, 8.4, 8.2,\n", " 8.3, 7.9, 7.8, 7.6, 7.6, 5.4, 9.6, 9.2],\n", " \"vote_count\": [26000, 31000, 34000, 32000, 16000, 14000, 24000,\n", " 25000, 17000, 12000, 21000, 15000, 4000, 3, 12],\n", "})\n", "\n", "# The naive \"top chart\": sort by average rating\n", "naive = movies.sort_values(\"vote_average\", ascending=False)\n", "print(naive.head(5).to_string(index=False))" ] }, { "cell_type": "markdown", "id": "cell-0005", "metadata": {}, "source": [ "Two movies nobody has heard of — *Midnight Static* (3 votes) and *The Lost\n", "Reel* (12 votes) — beat *The Shawshank Redemption*. The average rating alone\n", "answers \"how much did the people who voted like it?\" but ignores \"how many\n", "people is that opinion based on?\"" ] }, { "cell_type": "markdown", "id": "cell-0006", "metadata": {}, "source": [ "## The IMDB weighted rating\n", "\n", "IMDB's classic fix blends each movie's own average with the global average,\n", "weighted by how many votes the movie has:\n", "\n", "**WR = (v / (v + m)) · R + (m / (v + m)) · C**\n", "\n", "- **v** — number of votes for the movie (`vote_count`)\n", "- **R** — the movie's own average rating (`vote_average`)\n", "- **C** — the mean rating across *all* movies\n", "- **m** — the minimum votes required to be taken seriously (a tuning knob)\n", "\n", "Read it as a tug-of-war. When `v` is huge compared to `m`, the first fraction\n", "approaches 1 and **WR ≈ R** — the movie has earned the right to its own score.\n", "When `v` is tiny, the second fraction dominates and **WR ≈ C** — the movie is\n", "pulled toward \"just average\" until it collects more evidence. Statisticians\n", "call this *shrinkage*: shrink unreliable estimates toward the global mean.\n", "\n", "How do you pick `m`? A common recipe is a quantile of the vote counts — e.g.\n", "\"you need more votes than 70% of the catalog\". Let's build the fair chart:" ] }, { "cell_type": "code", "id": "cell-0007", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import pandas as pd\n", "\n", "movies = pd.DataFrame({\n", " \"title\": [\n", " \"The Shawshank Redemption\", \"The Dark Knight\", \"Inception\",\n", " \"Interstellar\", \"Parasite\", \"Whiplash\", \"Joker\",\n", " \"Avengers: Endgame\", \"La La Land\", \"Knives Out\",\n", " \"Mad Max: Fury Road\", \"Get Out\", \"The Emoji Movie\",\n", " \"Midnight Static\", \"The Lost Reel\",\n", " ],\n", " \"vote_average\": [8.7, 8.5, 8.4, 8.4, 8.5, 8.4, 8.2,\n", " 8.3, 7.9, 7.8, 7.6, 7.6, 5.4, 9.6, 9.2],\n", " \"vote_count\": [26000, 31000, 34000, 32000, 16000, 14000, 24000,\n", " 25000, 17000, 12000, 21000, 15000, 4000, 3, 12],\n", "})\n", "\n", "C = movies[\"vote_average\"].mean() # global mean rating\n", "m = movies[\"vote_count\"].quantile(0.70) # 70th-percentile vote count\n", "\n", "v = movies[\"vote_count\"]\n", "R = movies[\"vote_average\"]\n", "movies[\"score\"] = (v / (v + m)) * R + (m / (v + m)) * C\n", "\n", "print(f\"C = {C:.2f} m = {m:.0f} votes\\\\n\")\n", "chart = movies.sort_values(\"score\", ascending=False)\n", "print(chart[[\"title\", \"vote_average\", \"vote_count\", \"score\"]]\n", " .round(2).head(8).to_string(index=False))\n", "print(\"\\\\n...and the old 'winners' now:\")\n", "print(chart[chart.vote_count < 100][[\"title\", \"vote_average\", \"vote_count\", \"score\"]]\n", " .round(2).to_string(index=False))" ] }, { "cell_type": "markdown", "id": "cell-0008", "metadata": {}, "source": [ "The 3-vote wonder collapses from 9.6 down to roughly the global mean, while\n", "heavily-voted films keep scores close to their true averages. On IMDB's real\n", "Top 250, `m` is set high enough (25,000 votes) that low-evidence movies are\n", "also *excluded outright* — with a real catalog of thousands of films you'd use\n", "`q=0.90` or higher rather than our small-sample 0.70." ] }, { "cell_type": "markdown", "id": "cell-0009", "metadata": {}, "source": [ "> **Filter → score → sort**\n", "> \n", "> Production popularity shelves usually add a filtering step before scoring:\n", "> restrict to a genre, a year range, or a runtime window (\"Top animated films of\n", "> the 2010s\"), then compute the weighted rating within that slice, then sort.\n", "> The same three-step recipe — filter, score, sort — powers every \"Top 10 in\n", "> Indonesia today\" row you've ever seen." ] }, { "cell_type": "markdown", "id": "cell-0010", "metadata": {}, "source": [ "## Where popularity lists shine — and where they stop\n", "\n", "Popularity ranking is genuinely useful:\n", "\n", "- **Cold start.** A brand-new visitor has no history, so personalized methods\n", " have nothing to work with. Showing them what's broadly loved is the best\n", " available move — this is why logged-out homepages are wall-to-wall charts.\n", "- **Trending shelves.** \"Popular this week\" computed over a recent time window\n", " is a strong, cheap signal that requires zero user data.\n", "- **A sanity baseline.** Any fancy recommender that can't beat \"recommend the\n", " most popular items\" isn't earning its complexity. Always measure against it.\n", "\n", "But the limits are built into the definition:\n", "\n", "- **No personalization.** Everyone gets the same list. If you love obscure\n", " documentaries, the chart still hands you superhero blockbusters.\n", "- **Popularity bias / feedback loops.** Popular items get recommended, which\n", " makes them more popular, which keeps them recommended. Niche gems stay\n", " buried, and the catalog's \"long tail\" never gets exposure.\n", "\n", "Fixing the first limitation is the job of the next two lessons." ] }, { "cell_type": "markdown", "id": "cell-0011", "metadata": {}, "source": [ "### 🏋️ Exercise — Tune the evidence threshold\n", "\n", "Write a function `top_chart(df, q, k=5)` that computes the IMDB weighted\n", "rating using `m = quantile(q)` of the vote counts and returns the top `k`\n", "movies. Run it with `q = 0.05`, `q = 0.50`, and `q = 0.90` on the lesson's\n", "movie table. At which setting does *Midnight Static* (9.6 average, 3 votes)\n", "sneak back into the top 5 — and why?" ] }, { "cell_type": "code", "id": "cell-0012", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "# Your solution here" ] }, { "cell_type": "markdown", "id": "cell-0013", "metadata": {}, "source": [ "
✅ Show solution\n", "\n", "```python\n", "import pandas as pd\n", "\n", "movies = pd.DataFrame({\n", " \"title\": [\"The Shawshank Redemption\", \"The Dark Knight\", \"Inception\",\n", " \"Interstellar\", \"Parasite\", \"Whiplash\", \"Joker\",\n", " \"Avengers: Endgame\", \"La La Land\", \"Knives Out\",\n", " \"Mad Max: Fury Road\", \"Get Out\", \"The Emoji Movie\",\n", " \"Midnight Static\", \"The Lost Reel\"],\n", " \"vote_average\": [8.7, 8.5, 8.4, 8.4, 8.5, 8.4, 8.2,\n", " 8.3, 7.9, 7.8, 7.6, 7.6, 5.4, 9.6, 9.2],\n", " \"vote_count\": [26000, 31000, 34000, 32000, 16000, 14000, 24000,\n", " 25000, 17000, 12000, 21000, 15000, 4000, 3, 12],\n", "})\n", "\n", "def top_chart(df, q, k=5):\n", " df = df.copy()\n", " C = df[\"vote_average\"].mean()\n", " m = df[\"vote_count\"].quantile(q)\n", " v, R = df[\"vote_count\"], df[\"vote_average\"]\n", " df[\"score\"] = (v / (v + m)) * R + (m / (v + m)) * C\n", " return df.sort_values(\"score\", ascending=False).head(k), m\n", "\n", "for q in [0.05, 0.50, 0.90]:\n", " chart, m = top_chart(movies, q)\n", " print(f\"q={q} (m={m:.0f} votes)\")\n", " print(chart[[\"title\", \"score\"]].round(2).to_string(index=False))\n", " print()\n", "```\n", "\n", "
" ] }, { "cell_type": "markdown", "id": "cell-0014", "metadata": {}, "source": [ "Next up: content-based filtering — turning \"you liked Toy Story\" into \"you\n", "might like Finding Nemo\" by measuring how similar two movies *are*." ] } ] }