Dimensionality Reduction: PCA and t-SNE Explained

A dataset with dozens or hundreds of columns is hard to visualise and can be slow or noisy to model. Dimensionality reduction maps it into fewer dimensions while keeping as much of the important structure as possible. The two methods most people meet first, PCA and t-SNE, solve different problems: PCA compresses data for modelling and understanding, and t-SNE draws maps of it for looking. Using one for the other's job is a common mistake. Here both are measured on the same data: 1,797 handwritten digit images, each a 64-dimensional vector of pixel values.

PCA: find the directions that matter

Principal component analysis finds new axes, the principal components, along which the data varies most. The first component captures the most variance, the second the most of what is left, at right angles to the first, and so on (scikit-learn: PCA). Keeping only the first few components compresses the data with the least loss of variance.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.manifold import TSNE, trustworthiness
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_digits(return_X_y=True)
X_scaled = StandardScaler().fit_transform(X)
cumulative = np.cumsum(PCA().fit(X_scaled).explained_variance_ratio_)
print({k: round(cumulative[k - 1], 3) for k in (2, 10, 20, 30, 40)})
print("components for 95% of variance:", int(np.argmax(cumulative >= 0.95)) + 1)

{2: 0.216, 10: 0.589, 20: 0.793, 30: 0.893, 40: 0.951}
components for 95% of variance: 40

Two components keep only about a fifth of the variance: a 2D PCA plot of this data shows a lot of overlap. Forty of the 64 components keep 95%. Standardising first matters, because PCA chases variance, and unscaled features with large ranges would dominate the components.

What compression costs a model

Fewer features can mean faster training and less noise, but also less information. Putting PCA inside a pipeline and cross-validating shows the trade-off directly:

for k in (5, 10, 20, 40, None):
    steps = [StandardScaler()] + ([PCA(n_components=k)] if k else []) + [LogisticRegression(max_iter=3000)]
    accuracy = cross_val_score(make_pipeline(*steps), X, y, cv=5).mean()
    print(f"{k if k else 'all 64'} components: accuracy {accuracy:.3f}")

5 components: accuracy 0.771
10 components: accuracy 0.840
20 components: accuracy 0.899
40 components: accuracy 0.914
all 64 components: accuracy 0.920

Forty components get within a percentage point of using all 64. Twenty cost two more points. Five lose a lot. Choose the number of components from a curve like this, for your model and metric, rather than from a variance threshold alone. Keeping PCA inside the pipeline also means it is fitted only on training folds, which avoids leaking information from the test data.

t-SNE: a map of neighbourhoods

t-SNE does something different. It places points in two or three dimensions so that points that are close in the original space stay close in the map, caring much less about large distances (scikit-learn: t-SNE). The trustworthiness score measures how well each method keeps each point's nearest neighbours as neighbours, from 0 to 1 (scikit-learn: trustworthiness):

pca_2d = PCA(n_components=2).fit_transform(X_scaled)
tsne_2d = TSNE(n_components=2, perplexity=30, init="pca", random_state=0).fit_transform(X_scaled)
print("PCA:", round(trustworthiness(X_scaled, pca_2d, n_neighbors=5), 3),
      "| t-SNE:", round(trustworthiness(X_scaled, tsne_2d, n_neighbors=5), 3))

from sklearn.metrics import silhouette_score
print("digit separation (silhouette on true labels) PCA:", round(silhouette_score(pca_2d, y), 3),
      "| t-SNE:", round(silhouette_score(tsne_2d, y), 3))

PCA: 0.818 | t-SNE: 0.993
digit separation (silhouette on true labels) PCA: 0.055 | t-SNE: 0.485

In two dimensions, t-SNE keeps neighbourhoods almost perfectly. The second line measures how well the ten digits separate in each 2D map, using the silhouette score of the true labels: close to zero for PCA, where the digit groups overlap heavily, and far higher for t-SNE, where they form distinct groups. For looking at structure, t-SNE wins decisively.

Reading t-SNE plots without fooling yourself

The scikit-learn documentation lists the caveats, and they matter (scikit-learn: t-SNE):

• Distances between clusters mean little. Two islands far apart are not necessarily more different than two close ones.
• Cluster sizes mean little. t-SNE expands dense regions and contracts sparse ones.
• Results depend on perplexity and randomness. Try several perplexity values and seeds; trust only patterns that persist.
• There is no transform for new data. t-SNE maps the points it was fitted on; it is not a preprocessing step for a model.

So use t-SNE to explore: to see whether classes separate, to spot mislabelled points sitting in the wrong island, or to check whether embeddings group sensibly. Do not measure distances on it, and do not feed its output to a model.

PCA in practice: reading the components

PCA is not only a compression tool; its components can explain your data. Each component is a weighted combination of the original features, and the weights, called loadings, show which features move together. In a customer dataset, a first component with large positive loadings on visits, spend and items per order might be read as overall engagement, and a second contrasting discount use with full-price purchases as price sensitivity. Interpretations like these are hypotheses to check, not facts, but they often reveal redundant features and suggest simpler, more meaningful ones to build.

A workflow for high-dimensional data

1. Standardise numeric features.
2. Run PCA and look at the cumulative explained variance to see how much redundancy there is.
3. Visualise with t-SNE or UMAP to explore groups and outliers, with a few perplexity settings.
4. If you compress for a model, put PCA inside the pipeline and choose the number of components by cross-validated performance.
5. Report what was lost: the variance dropped and the metric change, as in the table above.

Choosing between them

UMAP is a popular alternative to t-SNE for visualisation that is often faster and preserves more of the global layout; it lives in a separate library.

Reduced data is often clustered next; clustering algorithms compared shows how. Dimensionality reduction with PCA is part of the preprocessing and feature engineering module of the Data Science course.

Sources

• Principal component analysis (scikit-learn) — scikit-learn documentation
• t-distributed Stochastic Neighbor Embedding (scikit-learn) — scikit-learn documentation
• sklearn.manifold.trustworthiness — scikit-learn documentation