Almost all practical machine learning is done in Python, and most of the hours go into handling data, not into training models: loading it, cleaning it, reshaping it and looking at it. Three libraries carry nearly all of that work. NumPy gives you fast arrays, pandas gives you tables, and Matplotlib draws the pictures that tell you whether the data means what you think. If you come from infrastructure, this is the toolkit that replaces your shell pipelines of awk, sort and uniq once data gets serious.
- Set up an isolated Python environment and work productively in a notebook
- Use NumPy arrays, vectorised operations, broadcasting and the
axisargument instead of Python loops - Load, inspect, filter, clean and aggregate tabular data with pandas
- Combine datasets with merges and handle missing values deliberately
- Plot distributions and relationships to understand a dataset before modelling it
Environment and notebooks
Data libraries have heavy, version-sensitive dependencies, so give every project its own virtual environment and record what you installed. This is the same reproducibility principle as lockfiles in the DevOps track.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install numpy pandas matplotlib scikit-learn jupyterlab
pip freeze > requirements.txt # record exact versions
jupyter lab # opens the notebook interface in a browserA notebook mixes code, output, charts and notes in cells that you run one at a time. It is ideal for exploration, because you can load a large file once and then poke at it for an hour. It has one serious trap: cells can be run in any order, so the state in memory may not match what the notebook shows from top to bottom. Before you trust or share a notebook, use Restart Kernel and Run All to prove that it works from a clean start.
- Use notebooks to explore. Move code that has to run repeatedly or in production into
.pymodules with tests. - Set random seeds, so that results can be reproduced.
- Notebooks store their output in the file, which makes diffs noisy and can leak data. Clear the output before committing, or use a tool that does it for you.
- A useful variant of
printin notebooks: the last expression in a cell is displayed automatically, and pandas tables render as formatted HTML.
NumPy: arrays instead of loops
A Python list holds references to separate objects scattered through memory. A NumPy array is one contiguous block of values that all share a single type, called its dtype. Operations on it run in compiled code over the whole block at once, which is called vectorisation, and it is typically tens to hundreds of times faster than a Python loop. Every other library in this track, including pandas, scikit-learn and PyTorch, is built on the same idea.
import numpy as np
latency_ms = np.array([120, 95, 310, 88, 2050, 101, 97])
latency_ms.shape # (7,) -> one dimension, seven elements
latency_ms.dtype # int64
latency_ms / 1000 # element-wise: array of seconds, no loop
latency_ms.mean() # 408.71...
np.percentile(latency_ms, 99)
slow = latency_ms > 300 # boolean array: [False False True False True False False]
latency_ms[slow] # boolean mask -> array([ 310, 2050])
slow.sum() # True counts as 1 -> 2 slow requests
slow.mean() # share that are slow -> 0.2857...
np.where(latency_ms > 300, "slow", "ok") # vectorised if/elseMost real data has more than one dimension. A two-dimensional array is a matrix with shape equal to (rows, columns). The axis argument says which dimension an operation collapses, and it confuses everybody at first. axis=0 works down the rows, giving one result per column. axis=1 works across the columns, giving one result per row.
import numpy as np
# 3 servers (rows) x 4 hourly CPU readings (columns)
cpu = np.array([[0.20, 0.35, 0.80, 0.40],
[0.10, 0.15, 0.30, 0.20],
[0.55, 0.60, 0.95, 0.70]])
cpu.shape # (3, 4)
cpu.mean(axis=0) # per hour, across servers -> shape (4,)
cpu.mean(axis=1) # per server, across hours -> shape (3,)
cpu[0] # first row
cpu[:, 2] # third column: every server at hour 2
cpu[cpu[:, 2] > 0.5] # rows of the servers that were busy at hour 2
# broadcasting: subtract each server's own mean from its row
centred = cpu - cpu.mean(axis=1, keepdims=True) # (3, 4) - (3, 1)
centred.mean(axis=1) # ~0 for every serverBroadcasting is the rule that lets arrays of different shapes be combined. NumPy compares shapes from the right, and two dimensions are compatible when they are equal or one of them is 1, in which case the smaller array is stretched along that dimension without copying. That is why (3, 4) - (3, 1) works, and why keepdims=True matters: without it the means have shape (3,), which does not line up with the rows and raises an error.
Slicing a NumPy array returns a view onto the same memory, not a copy. Modifying part = cpu[:2] changes cpu too. Call .copy() when you need an independent array. Boolean masks and lists of indices, by contrast, do return copies.
pandas: tables you can query
pandas adds labels on top of NumPy. A Series is a one-dimensional array with an index. A DataFrame is a table: a set of Series sharing one index, each column with its own type. Think of it as a spreadsheet or a SQL table held in memory.
import pandas as pd
df = pd.read_csv("requests.csv", parse_dates=["timestamp"])
df.shape # (rows, columns)
df.head() # first five rows
df.info() # column types and non-null counts: always run this first
df.describe() # count, mean, std, min, quartiles, max for numeric columns
df["status"].value_counts() # frequency of each value
df["status"].value_counts(normalize=True) # the same, as proportions
df.isna().sum() # missing values per columnThose six or seven lines are how every analysis begins. info() in particular catches the two most common data problems at once: a numeric column that loaded as text because of one stray value, and columns with missing data.
import pandas as pd
df = pd.read_csv("requests.csv", parse_dates=["timestamp"])
# select columns
df["latency_ms"] # one column -> Series
df[["route", "latency_ms"]] # several columns -> DataFrame
# filter rows with a boolean mask; combine with & | ~ and PARENTHESES
errors = df[df["status"] >= 500]
slow_checkout = df[(df["route"] == "/checkout") & (df["latency_ms"] > 500)]
api_routes = df[df["route"].isin(["/orders", "/checkout"])]
# .loc selects by label and condition, .iloc by integer position
df.loc[df["status"] >= 500, ["timestamp", "route", "status"]]
df.iloc[0:10, 0:3]
# new columns are vectorised expressions
df["latency_s"] = df["latency_ms"] / 1000
df["is_error"] = df["status"] >= 500
df["hour"] = df["timestamp"].dt.hour
df.sort_values("latency_ms", ascending=False).head(10)Two syntax points cause most beginner errors. Conditions are combined with &, | and ~, never with the Python keywords and, or and not, and each condition needs its own parentheses, because & binds more tightly than ==. And to change values in place, use a single .loc[rows, columns] = value. Chained indexing such as df[mask]["col"] = value may modify a temporary copy and silently do nothing, which pandas warns about as SettingWithCopyWarning.
Group, aggregate, merge
groupby is the workhorse of analysis, and it is the same idea as SQL's GROUP BY: split the rows into groups by a key, apply an aggregation to each group, and combine the results. It replaces the shell pipeline sort | uniq -c and goes far beyond it.
import pandas as pd
df = pd.read_csv("requests.csv", parse_dates=["timestamp"])
df["is_error"] = df["status"] >= 500
# one aggregation
df.groupby("route")["latency_ms"].mean()
# several named aggregations at once
summary = df.groupby("route").agg(
requests=("latency_ms", "size"),
p50=("latency_ms", "median"),
p99=("latency_ms", lambda s: s.quantile(0.99)),
error_rate=("is_error", "mean"),
).sort_values("error_rate", ascending=False)
# time series: requests and error rate per 5 minutes
per_5m = (
df.set_index("timestamp")
.resample("5min")
.agg({"latency_ms": "median", "is_error": "mean"})
)Note the trick in error_rate: the mean of a boolean column is the proportion of True values. The same approach gives you an availability SLI from a request log in one line.
merge joins two tables on a key, with the same join types as SQL.
import pandas as pd
requests = pd.read_csv("requests.csv")
customers = pd.read_csv("customers.csv") # customer_id, plan, region
joined = requests.merge(customers, on="customer_id", how="left")
len(requests), len(joined) # these should be EQUAL for a left join
joined["plan"].isna().sum() # requests whose customer was not found
joined.groupby("plan")["latency_ms"].median()Always compare row counts before and after a merge. If the key is not unique in the right-hand table, every matching row is duplicated, and your totals inflate silently. Pass validate="many_to_one" to make pandas raise an error instead.
Missing and messy data
Real data has gaps, wrong types, duplicates and impossible values. pandas represents a missing value as NaN, which stands for "not a number". NaN is contagious in arithmetic, is skipped by default in aggregations such as mean(), and is not equal to itself, so you must test for it with isna() and never with ==.
import pandas as pd
df = pd.read_csv("requests.csv")
df.isna().mean().sort_values(ascending=False) # share missing per column
df = df.dropna(subset=["customer_id"]) # drop rows missing an essential field
df["region"] = df["region"].fillna("unknown") # a category of its own
df["latency_ms"] = df["latency_ms"].fillna(df["latency_ms"].median())
df["latency_ms"] = pd.to_numeric(df["latency_ms"], errors="coerce") # bad text -> NaN
df["route"] = df["route"].str.strip().str.lower() # tidy strings
df = df.drop_duplicates(subset=["request_id"])
df = df[df["latency_ms"].between(0, 60_000)] # remove impossible values
df["plan"] = df["plan"].astype("category") # compact, faster groupbyThere is no universally right way to handle a missing value. Dropping rows is simplest and loses data. Filling with the median keeps the row and hides the fact that the value was unknown. Often the best choice is to fill and add a column recording that the value was missing, because the missingness itself can carry information. What matters is that you decide deliberately, write the decision down, and apply exactly the same treatment later, when the model runs in production.
For data too large for memory, read only the columns you need with usecols, process the file in pieces with chunksize, and prefer the columnar Parquet format to CSV, which is smaller, much faster, and preserves types.
Look at the data
Summary statistics can describe completely different datasets with identical numbers. A plot shows you, in a second, the outliers, the second cluster, the gap where logging was broken for a day, and the column that is secretly constant. Plot before you model, every time.
import matplotlib.pyplot as plt
import pandas as pd
df = pd.read_csv("requests.csv", parse_dates=["timestamp"])
fig, axes = plt.subplots(1, 3, figsize=(15, 4))
# 1. distribution of one variable
axes[0].hist(df["latency_ms"], bins=50)
axes[0].set_yscale("log") # latency is heavy-tailed; a log axis shows the tail
axes[0].set_xlabel("latency (ms)")
axes[0].set_title("Latency distribution")
# 2. relationship between two variables
axes[1].scatter(df["payload_kb"], df["latency_ms"], s=4, alpha=0.3)
axes[1].set_xlabel("payload (KB)")
axes[1].set_ylabel("latency (ms)")
# 3. behaviour over time
per_min = df.set_index("timestamp")["latency_ms"].resample("1min").median()
axes[2].plot(per_min.index, per_min.values)
axes[2].set_title("Median latency per minute")
fig.tight_layout()
fig.savefig("latency_overview.png", dpi=150)| Question | Plot |
|---|---|
| What values does this variable take? | Histogram; box plot to compare groups |
| How do two numeric variables relate? | Scatter plot |
| How does it change over time? | Line plot |
| How do categories compare? | Bar chart |
| Which features move together? | Heatmap of df.corr(numeric_only=True) |
Use the object-oriented style shown here, fig, ax = plt.subplots() followed by methods on ax, instead of the older plt.plot() calls that act on a hidden global figure. It stays readable when a figure has several panels. For quick exploration, pandas can plot directly with df["latency_ms"].plot.hist(bins=50), which uses Matplotlib underneath.