{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cell-0000",
   "metadata": {},
   "source": [
    "# pandas: DataFrames & Series\n",
    "\n",
    "Learn pandas' two core structures — Series and DataFrame — and the first-look toolkit every analysis starts with, from head() and info() to derived columns and value_counts().\n",
    "\n",
    "*Part of the free [Python for Data Science](https://ramadnsyh.dev/courses/python-for-data-science) course by [Muhammad Ramadiansyah](https://ramadnsyh.dev). This notebook is generated from the interactive lesson — [read it online](https://ramadnsyh.dev/courses/python-for-data-science/pandas-basics).*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0001",
   "metadata": {},
   "source": [
    "If NumPy is the engine of data science in Python, **pandas** is the cockpit.\n",
    "It wraps NumPy arrays in labeled, table-shaped structures that feel like a\n",
    "spreadsheet you can program. In this lesson you'll build DataFrames from\n",
    "scratch, take a \"first look\" at a dataset the way analysts actually do, and\n",
    "create new columns from existing ones."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0002",
   "metadata": {},
   "source": [
    "## Series and DataFrame\n",
    "\n",
    "pandas has two core objects:\n",
    "\n",
    "- **Series** — a single labeled column of values (a NumPy array plus an index)\n",
    "- **DataFrame** — a whole table: multiple Series sharing the same row index"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0003",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# A Series: values + an index of labels\n",
    "ratings = pd.Series([4.5, 3.8, 4.9], index=[\"Inception\", \"Tenet\", \"Interstellar\"])\n",
    "print(ratings)\n",
    "print()\n",
    "print(ratings[\"Tenet\"])   # look values up by label"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0004",
   "metadata": {},
   "source": [
    "A DataFrame is what you'll work with 95% of the time. The most common way to\n",
    "build one by hand is from a **dictionary of lists** — keys become column names,\n",
    "lists become the columns:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0005",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "movies = pd.DataFrame({\n",
    "    \"title\":  [\"Inception\", \"Tenet\", \"Interstellar\", \"Dunkirk\", \"Oppenheimer\"],\n",
    "    \"year\":   [2010, 2020, 2014, 2017, 2023],\n",
    "    \"minutes\":[148, 150, 169, 106, 180],\n",
    "    \"rating\": [8.8, 7.3, 8.7, 7.8, 8.3],\n",
    "    \"genre\":  [\"Sci-Fi\", \"Sci-Fi\", \"Sci-Fi\", \"War\", \"Biography\"],\n",
    "})\n",
    "print(movies)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0006",
   "metadata": {},
   "source": [
    "Each column of a DataFrame **is** a Series — same index, one dtype per column.\n",
    "That per-column uniformity is what keeps pandas fast under the hood."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0007",
   "metadata": {},
   "source": [
    "## Loading real data\n",
    "\n",
    "In real projects you rarely type data in — you load it. The workhorse is\n",
    "`pd.read_csv`, which also has cousins for Excel, JSON, SQL, and more. Browser\n",
    "cells can't read files, so run this one in the downloadable notebook or Colab:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0008",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# local file, a URL, or a path on Google Drive all work\n",
    "df = pd.read_csv(\"data/cereal.csv\")\n",
    "\n",
    "# useful options you'll reach for constantly:\n",
    "df = pd.read_csv(\"data/cereal.csv\", index_col=\"name\")   # use a column as the index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0009",
   "metadata": {},
   "source": [
    "Everything below works identically whether your DataFrame came from a CSV or\n",
    "a dictionary — so we'll keep using our inline movies table."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0010",
   "metadata": {},
   "source": [
    "## First look: head, info, describe\n",
    "\n",
    "Whenever a dataset lands on your desk, the same four commands come first.\n",
    "Think of it as the data scientist's handshake:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0011",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "movies = pd.DataFrame({\n",
    "    \"title\":  [\"Inception\", \"Tenet\", \"Interstellar\", \"Dunkirk\", \"Oppenheimer\"],\n",
    "    \"year\":   [2010, 2020, 2014, 2017, 2023],\n",
    "    \"minutes\":[148, 150, 169, 106, 180],\n",
    "    \"rating\": [8.8, 7.3, 8.7, 7.8, 8.3],\n",
    "    \"genre\":  [\"Sci-Fi\", \"Sci-Fi\", \"Sci-Fi\", \"War\", \"Biography\"],\n",
    "})\n",
    "\n",
    "print(movies.head(3))        # first rows — sanity-check the parsing\n",
    "print()\n",
    "print(movies.tail(2))        # last rows — catch trailing junk\n",
    "print()\n",
    "print(\"shape:\", movies.shape)   # (rows, columns)\n",
    "print()\n",
    "print(movies.dtypes)         # one dtype per column"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0012",
   "metadata": {},
   "source": [
    "`info()` and `describe()` go one level deeper — structure and statistics:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0013",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "movies = pd.DataFrame({\n",
    "    \"title\":  [\"Inception\", \"Tenet\", \"Interstellar\", \"Dunkirk\", \"Oppenheimer\"],\n",
    "    \"year\":   [2010, 2020, 2014, 2017, 2023],\n",
    "    \"minutes\":[148, 150, 169, 106, 180],\n",
    "    \"rating\": [8.8, 7.3, 8.7, 7.8, 8.3],\n",
    "    \"genre\":  [\"Sci-Fi\", \"Sci-Fi\", \"Sci-Fi\", \"War\", \"Biography\"],\n",
    "})\n",
    "\n",
    "movies.info()                # column names, non-null counts, dtypes, memory\n",
    "print()\n",
    "print(movies.describe())     # count/mean/std/min/quartiles/max for numeric columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0014",
   "metadata": {},
   "source": [
    "> **Read info() like a detective**\n",
    "> \n",
    "> `info()` answers three questions at a glance: how many rows do I have, which\n",
    "> columns have missing values (non-null count below the row count), and did any\n",
    "> numeric column sneak in as `object` (usually a sign of dirty data like\n",
    "> `\"1,200\"` or `\"N/A\"` strings)?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0015",
   "metadata": {},
   "source": [
    "## Selecting columns\n",
    "\n",
    "Square brackets with a name give you one column (a Series); a **list** of\n",
    "names gives you a smaller DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0016",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "movies = pd.DataFrame({\n",
    "    \"title\":  [\"Inception\", \"Tenet\", \"Interstellar\", \"Dunkirk\", \"Oppenheimer\"],\n",
    "    \"year\":   [2010, 2020, 2014, 2017, 2023],\n",
    "    \"minutes\":[148, 150, 169, 106, 180],\n",
    "    \"rating\": [8.8, 7.3, 8.7, 7.8, 8.3],\n",
    "    \"genre\":  [\"Sci-Fi\", \"Sci-Fi\", \"Sci-Fi\", \"War\", \"Biography\"],\n",
    "})\n",
    "\n",
    "print(movies[\"rating\"])              # one column -> Series\n",
    "print()\n",
    "print(movies[[\"title\", \"rating\"]])   # list of columns -> DataFrame (note the double brackets)\n",
    "print()\n",
    "print(movies.rating.mean())          # dot access works for simple names"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0017",
   "metadata": {},
   "source": [
    "Dot access (`movies.rating`) is convenient for reading, but it breaks on column\n",
    "names with spaces and can't create new columns — for anything serious, prefer\n",
    "brackets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0018",
   "metadata": {},
   "source": [
    "## Adding derived columns\n",
    "\n",
    "New columns are created by assigning to a name that doesn't exist yet. Thanks\n",
    "to vectorization, arithmetic between columns happens row by row automatically:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0019",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "movies = pd.DataFrame({\n",
    "    \"title\":  [\"Inception\", \"Tenet\", \"Interstellar\", \"Dunkirk\", \"Oppenheimer\"],\n",
    "    \"year\":   [2010, 2020, 2014, 2017, 2023],\n",
    "    \"minutes\":[148, 150, 169, 106, 180],\n",
    "    \"rating\": [8.8, 7.3, 8.7, 7.8, 8.3],\n",
    "})\n",
    "\n",
    "movies[\"hours\"] = (movies[\"minutes\"] / 60).round(2)\n",
    "movies[\"age\"] = 2026 - movies[\"year\"]\n",
    "movies[\"is_long\"] = movies[\"minutes\"] > 150     # boolean column\n",
    "\n",
    "print(movies)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0020",
   "metadata": {},
   "source": [
    "This is the same elementwise thinking from the NumPy lesson — no loops needed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0021",
   "metadata": {},
   "source": [
    "## Counting categories with value_counts\n",
    "\n",
    "For categorical columns, `value_counts()` is your best friend — it tallies how\n",
    "often each value appears, most frequent first:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0022",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "orders = pd.DataFrame({\n",
    "    \"order_id\": range(1, 11),\n",
    "    \"payment\": [\"card\", \"cash\", \"card\", \"e-wallet\", \"card\",\n",
    "                \"cash\", \"card\", \"e-wallet\", \"card\", \"card\"],\n",
    "    \"amount\": [25.0, 12.5, 40.0, 18.0, 33.0, 9.5, 55.0, 21.0, 30.0, 14.5],\n",
    "})\n",
    "\n",
    "print(orders[\"payment\"].value_counts())\n",
    "print()\n",
    "print(orders[\"payment\"].value_counts(normalize=True))   # as proportions\n",
    "print()\n",
    "print(orders[\"payment\"].unique())      # just the distinct values\n",
    "print(orders[\"payment\"].nunique())     # how many distinct values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0023",
   "metadata": {},
   "source": [
    "`normalize=True` turns counts into shares — \"60% of orders paid by card\" is\n",
    "usually more useful than \"6 orders\"."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0024",
   "metadata": {},
   "source": [
    "## Fixing dtypes\n",
    "\n",
    "Columns sometimes arrive with the wrong type — numbers stored as strings,\n",
    "categories stored as plain objects. `astype` converts them:"
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0025",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame({\n",
    "    \"product\": [\"laptop\", \"mouse\", \"monitor\"],\n",
    "    \"price\": [\"999\", \"25\", \"310\"],       # oops: strings!\n",
    "    \"category\": [\"computers\", \"accessories\", \"computers\"],\n",
    "})\n",
    "print(df.dtypes)\n",
    "\n",
    "df[\"price\"] = df[\"price\"].astype(int)\n",
    "df[\"category\"] = df[\"category\"].astype(\"category\")   # memory-efficient for repeated labels\n",
    "\n",
    "print()\n",
    "print(df.dtypes)\n",
    "print(\"Total:\", df[\"price\"].sum())      # now math works"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0026",
   "metadata": {},
   "source": [
    "Until the conversion, `df[\"price\"].sum()` would have concatenated the strings\n",
    "into `\"99925310\"` — a classic silent bug that `dtypes` catches early."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0027",
   "metadata": {},
   "source": [
    "### 🏋️ Exercise — Build and explore a mini e-commerce table\n",
    "\n",
    "Create a DataFrame of 6 products with columns `product`, `category`, `price`,\n",
    "and `quantity` (invent the values). Then: (1) add a `revenue` column equal to\n",
    "price × quantity, (2) print `head()`, `info()`, and `describe()`, (3) compute\n",
    "the total revenue, and (4) show what **percentage** of products falls in each\n",
    "category using `value_counts`."
   ]
  },
  {
   "cell_type": "code",
   "id": "cell-0028",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Your solution here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0029",
   "metadata": {},
   "source": [
    "<details><summary>✅ Show solution</summary>\n",
    "\n",
    "```python\n",
    "import pandas as pd\n",
    "\n",
    "sales = pd.DataFrame({\n",
    "    \"product\":  [\"laptop\", \"mouse\", \"keyboard\", \"monitor\", \"webcam\", \"headset\"],\n",
    "    \"category\": [\"computers\", \"accessories\", \"accessories\", \"computers\", \"accessories\", \"audio\"],\n",
    "    \"price\":    [1200.0, 25.0, 45.0, 310.0, 60.0, 85.0],\n",
    "    \"quantity\": [3, 20, 15, 5, 8, 10],\n",
    "})\n",
    "\n",
    "sales[\"revenue\"] = sales[\"price\"] * sales[\"quantity\"]\n",
    "\n",
    "print(sales.head())\n",
    "sales.info()\n",
    "print(sales.describe())\n",
    "\n",
    "print(\"Total revenue:\", sales[\"revenue\"].sum())\n",
    "print(sales[\"category\"].value_counts(normalize=True) * 100)\n",
    "```\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cell-0030",
   "metadata": {},
   "source": [
    "Next lesson: the operations that turn pandas into a query engine — filtering\n",
    "rows with boolean masks, sorting, and grouping."
   ]
  }
 ]
}