Python for Data Science: The Essential Libraries

Python itself has no data science features. Its strength is a set of open-source libraries that fit together so well they feel like one tool: arrays underneath, tables on top, charts and statistics beside them, and machine learning built on all of it. Knowing what each library is for, and in what order to learn them, saves beginners months of installing everything at once and mastering nothing.

The stack, by job

NumPy is the foundation. Its ndarray holds values of one type in a contiguous block of memory, which is what makes whole-array arithmetic fast (NumPy documentation). You will use it directly less than you expect, because pandas and scikit-learn use it for you, but understanding arrays explains why vectorised code is fast; the NumPy guide covers that.

pandas is where most of the day goes. A DataFrame is a table with labelled columns of possibly different types, and pandas reads CSV, Excel, SQL and Parquet, cleans and reshapes data, and groups and joins it (pandas: getting started). The pandas guide walks through a realistic cleaning job.

Matplotlib draws charts with complete control over every element (Matplotlib user guide); seaborn, built on it, draws common statistical charts such as distributions, box plots and relationships between variables in one call. Most analysts use seaborn for quick exploration and Matplotlib when a chart must look exactly right.

SciPy supplies statistical tests and distributions; statsmodels adds regression and time-series models with the detailed output statisticians expect. scikit-learn covers classical machine learning: every model has the same fit and predict interface, and pipelines chain preprocessing with a model so that cross-validation stays honest (scikit-learn: getting started).

One dataset through the stack

The example below shows how the pieces hand data to each other. It uses the diabetes dataset that ships with scikit-learn: 442 patients, ten baseline measurements that the dataset provides already centred and scaled, and a measure of disease progression one year later as the target (scikit-learn: load_diabetes).

import numpy as np
import pandas as pd
from scipy import stats
from sklearn.datasets import load_diabetes
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# pandas: the data as a table
df = load_diabetes(as_frame=True).frame
print(df.shape)
print(df[["bmi", "bp", "target"]].describe().loc[["mean", "std"]].round(3))

(442, 11)
        bmi     bp   target
mean -0.000 -0.000  152.133
std   0.048  0.048   77.093

The feature columns have a mean of zero and a small spread because the dataset was published pre-scaled; the target is in its original units. Checking describe() like this is the first thing to do with any dataset, before modelling anything.

# SciPy: is progression different for patients with above-median BMI?
median_bmi = df["bmi"].median()
high = df.loc[df["bmi"] > median_bmi, "target"]
low = df.loc[df["bmi"] <= median_bmi, "target"]
t = stats.ttest_ind(high, low, equal_var=False)
print(f"mean target: high BMI {high.mean():.1f}, low BMI {low.mean():.1f}, Welch t={t.statistic:.2f}, p={t.pvalue:.1e}")

mean target: high BMI 191.5, low BMI 113.5, Welch t=12.27, p=1.2e-29

equal_var=False runs Welch's t-test, which does not assume both groups have the same variance (SciPy: ttest_ind). The difference is large and very unlikely to be chance, though a test says nothing about cause.

# NumPy + scikit-learn: from DataFrame to arrays to a cross-validated model
X = df.drop(columns="target").to_numpy()
y = df["target"].to_numpy()
print(type(X).__name__, X.dtype, X.shape)

model = make_pipeline(StandardScaler(), Ridge(alpha=1.0))
scores = cross_val_score(model, X, y, cv=5, scoring="r2")
print(scores.round(3), round(scores.mean(), 3))

ndarray float64 (442, 10)
[0.428 0.522 0.486 0.427 0.549] 0.482

The DataFrame became plain NumPy arrays for the model, and five-fold cross-validation reports an R² of about 0.48: the ten measurements explain roughly half of the variation in progression. The spread across folds, from 0.43 to 0.55, is a reminder to report more than one number; cross-validation in machine learning explains why.

The order to learn them in

1. pandas first, with just enough NumPy to understand arrays and vectorised operations. Most real work is loading, cleaning and summarising tables.
2. Plotting with seaborn and Matplotlib next, because exploratory analysis is mostly looking at data; see exploratory data analysis, step by step.
3. Statistics with SciPy alongside the statistics you are learning, not as a separate library to memorise.
4. scikit-learn once you can clean data and read a chart. Models trained on data you have not explored teach bad habits.
5. Deep learning libraries such as PyTorch or TensorFlow later, and only if your work involves images, text or other unstructured data at scale.

Setting up

Use a virtual environment or a conda environment per project, install the libraries with pinned versions, and run Jupyter from inside that environment so notebooks use the same packages as your scripts. Jupyter is ideal for exploration; move code that works into .py modules once it needs to be rerun reliably. Virtual environments with venv, pip and Poetry explains the setup.

Mistakes that slow beginners down

• Learning libraries instead of problems. Pick a dataset and a question; learn each function when the question needs it.
• Row-by-row loops in pandas. They are slow and usually unnecessary; vectorised column operations do the same work far faster.
• Modelling before looking. Skipping describe(), missing-value counts and a few charts is how data errors end up in model results.
• Mixing environments. A notebook kernel running a different environment from your terminal causes "module not found" errors and version mismatches.

These libraries sit in the wider set of Python tools covered in top Python libraries every developer should know, and the path from here is laid out in the data science roadmap. NumPy, pandas, Matplotlib, seaborn and scikit-learn are the core tools of the Data Science course, from its Python for data science module to machine learning.

Sources

• NumPy: the absolute basics for beginners — NumPy documentation
• pandas: getting started — pandas documentation
• Matplotlib user guide — Matplotlib documentation
• scipy.stats.ttest_ind — SciPy documentation
• scikit-learn: getting started — scikit-learn documentation
• load_diabetes — scikit-learn documentation