pandas: DataFrames & Series

If NumPy is the engine of data science in Python, pandas is the cockpit. It wraps NumPy arrays in labeled, table-shaped structures that feel like a spreadsheet you can program. In this lesson you'll build DataFrames from scratch, take a "first look" at a dataset the way analysts actually do, and create new columns from existing ones.

Series and DataFrame

pandas has two core objects:

• Series — a single labeled column of values (a NumPy array plus an index)
• DataFrame — a whole table: multiple Series sharing the same row index

A DataFrame is what you'll work with 95% of the time. The most common way to build one by hand is from a dictionary of lists — keys become column names, lists become the columns:

Each column of a DataFrame is a Series — same index, one dtype per column. That per-column uniformity is what keeps pandas fast under the hood.

Loading real data

In real projects you rarely type data in — you load it. The workhorse is pd.read_csv, which also has cousins for Excel, JSON, SQL, and more. Browser cells can't read files, so run this one in the downloadable notebook or Colab:

import pandas as pd
 
# local file, a URL, or a path on Google Drive all work
df = pd.read_csv("data/cereal.csv")
 
# useful options you'll reach for constantly:
df = pd.read_csv("data/cereal.csv", index_col="name")   # use a column as the index

Everything below works identically whether your DataFrame came from a CSV or a dictionary — so we'll keep using our inline movies table.

First look: head, info, describe

Whenever a dataset lands on your desk, the same four commands come first. Think of it as the data scientist's handshake:

info() and describe() go one level deeper — structure and statistics:

Selecting columns

Square brackets with a name give you one column (a Series); a list of names gives you a smaller DataFrame:

Dot access (movies.rating) is convenient for reading, but it breaks on column names with spaces and can't create new columns — for anything serious, prefer brackets.

Adding derived columns

New columns are created by assigning to a name that doesn't exist yet. Thanks to vectorization, arithmetic between columns happens row by row automatically:

This is the same elementwise thinking from the NumPy lesson — no loops needed.

Counting categories with value_counts

For categorical columns, value_counts() is your best friend — it tallies how often each value appears, most frequent first:

normalize=True turns counts into shares — "60% of orders paid by card" is usually more useful than "6 orders".

Fixing dtypes

Columns sometimes arrive with the wrong type — numbers stored as strings, categories stored as plain objects. astype converts them:

Until the conversion, df["price"].sum() would have concatenated the strings into "99925310" — a classic silent bug that dtypes catches early.

Next lesson: the operations that turn pandas into a query engine — filtering rows with boolean masks, sorting, and grouping.