PCA: Reducing Dimensions Without Losing What Matters (Not Always)

Lectures 10a and 10b. After K-means and DBSCAN (grouping with no label), PCA tackles another label-free problem: how to summarize a bunch of variables into a few, without losing what matters. Spoiler for the whole post: "what matters" is a tricky question.

What PCA actually computes

The professor goes back to the breast cancer dataset that already showed up in this playlist, 30 variables, 569 patients:

from sklearn.decomposition import PCA
pca = PCA(n_components=30)
X_pca = pca.fit_transform(X)
print(pca.explained_variance_ratio_)

Output: the first component alone explains 98.2% of the data's total variance. The remaining 29, together, explain the 1.8% left over.

An absurd concentration, and it makes sense: several of these 30 variables essentially measure the same thing in different ways (a tumor's radius, perimeter, and area are practically the same information, just in different units), so the dataset's real variation is much "narrower" than 30 numbers suggest.

Bishop defines PCA as finding the direction that maximizes the variance of the projected data. For a single component, that turns into an eigenvalue problem: the optimal direction $\mathbf{u}_1$ satisfies

$\mathbf{S}\mathbf{u}_1 = \lambda_1 \mathbf{u}_1$

where $\mathbf{S}$ is the data's covariance matrix. In other words, $\mathbf{u}_1$ needs to be an eigenvector of $\mathbf{S}$, and the variance it captures is exactly the corresponding eigenvalue $\lambda_1$. The larger the eigenvalue, the more variance that direction captures, which is why the "first principal component" is always the eigenvector with the largest eigenvalue. The following components repeat the recipe, each one maximizing variance among the remaining directions, orthogonal to the previous ones.

Notice: at no point does this computation use the label y. PCA only looks at X, searching for where the data varies most. Keep that sentence in mind, it's the thread running through this whole post.

Reducing for real, with a catch

With 9 components (30% of the original 30 variables), the captured variance is already 99.99997%:

n_most_important = int(X.shape[1]*0.3)  # 9
scores = cross_val_score(model, X_pca[:, :n_most_important], y, cv=splitter)

The professor compares accuracy (original 30-variable space versus the first 9 PCA components) across five different classifiers:

Notice the effect isn't the same for everyone. KNN doesn't even notice the difference (PCA that keeps almost all the variance is basically a rotation plus a near-lossless cut, and Euclidean distance doesn't care about rotation). Gaussian Naive Bayes gets notably worse, which is a bit counterintuitive at first: it assumes the variables are independent of each other, and PCA's components are decorrelated by construction, so I'd have expected it to help, not hurt. But "decorrelated" isn't the same thing as "everyone matters equally": the 9 remaining components have wildly unequal variance among themselves (the first one alone carries the overwhelming share of the total variance), and Gaussian Naive Bayes fits one variance per variable per class, so it ends up trusting the higher-variance components more. The catch is that "higher variance" and "better at separating the classes" aren't the same thing, the exact trap the next section makes even clearer: dropping the lower-variance components can throw away precisely the axis where one class's mean differs from another's, which Gaussian Naive Bayes feels directly since it depends on those per-variable variances, unlike KNN, which only looks at total distance. And the decision tree gets slightly worse too, for the same reason I already saw in the decision trees post: a tree cuts one axis at a time, and PCA rotates the axes, so a decision boundary that used to align with one original variable can turn diagonal in the components, harder to carve with straight cuts.

Scikit-learn also lets you choose the number of components by fraction of variance instead of a fixed count:

pca = PCA(n_components=0.9999)
X_pca = pca.fit_transform(X)
print(X_pca.shape)

Output: (569, 5). Just 5 components already reach 99.99% of the variance, even fewer than the 9 picked by hand.

The warning: PCA doesn't know what's "important" to you

This section (the professor titled it "PCA fail" right in the notebook) is why I wrote "keep that sentence in mind" above. He builds an artificial dataset on purpose: 12 groups of points, arranged in two rows (one at $y\approx0.1$, another at $y\approx0.4$), spread along the $x$ axis from 0 to 1.1. The class alternates: the bottom row is one class, the top row is the other, regardless of $x$'s value.

model = Perceptron()
model.fit(X, y)
print(accuracy_score(y, model.predict(X)))

Output: 1.0. A hundred percent, because separating by row (the y value) is easy, it's just a horizontal line.

Now apply PCA with 1 component before training:

pca = PCA(n_components=1)
X_pca = pca.fit_transform(X)
model.fit(X_pca, y)
print(accuracy_score(y, model.predict(X_pca)))

Output: 0.47. Basically a coin flip.

What happened: the $x$ axis (0 to 1.1) is spread out far more than the $y$ axis (0.1 to 0.4), so the direction of maximum variance sits almost entirely along $x$. PCA, doing exactly what it promises, picks that direction as the one component. Except the information that separates the classes lives on the $y$ axis, the one with less variance, and PCA threw it out entirely. The algorithm has no way of knowing $y$ mattered more for classifying, because it never looked at the label.

Interactive: where the first component points

My own reconstruction of the same 100-point dataset (two groups, same row layout), with both principal components drawn on top (solid red = 1st component, dashed green = 2nd, each arrow scaled by the square root of the variance that component explains):

Without normalizing, component 1 (red) lies almost flat, following the $x$ axis, exactly the direction that does not separate the classes. Click "Normalized (z-score)": now both variables compete on equal footing, and component 1 rotates toward a diagonal, still not perfectly aligned with the real split (which is why the normalized result, 0.81 accuracy in the notebook, is better but not perfect), but much less blind to what matters than the unnormalized version.

PCA as real compression

aula10b switches to MNIST (70 thousand images, 784 pixels each) and uses PCA to find how many components are enough to retain 95% of the variance:

pca = PCA()
pca.fit(X_train)
cumsum = np.cumsum(pca.explained_variance_ratio_)
d = np.argmax(cumsum >= 0.95) + 1
print(d)

Output: d = 154. From 784 pixels to 154 components, almost an 80% size reduction, still keeping 95% of the variance.

Comparing accuracy, raw pixels against the 154 components:

The same pattern as the cancer dataset: the linear model doesn't care (or even improves a hair), the tree forest gets worse, because it again loses the axis alignment it needs to cut well.

And since PCA keeps each component's direction, it can walk back (with loss) and reconstruct an approximation of the original image:

X_train_recovered = pca.inverse_transform(X_train_reduced)

The reconstructed digits come out visibly "blurrier" than the original, but you can still tell which number is which, with only 154 numbers stored per image instead of 784. It's literally lossy compression, in the same spirit as a JPEG, except the "basis" used to describe the image was learned from the dataset itself instead of being generic.

Wrapping up

Practical application

Without normalizing, component 1 of the "PCA fail" dataset captures 83.5% of the total variance, almost everything. Normalized, it drops to 54.8%, nearly tied with component 2 (45.2%). That alone is a useful warning sign: whenever the first component overwhelmingly dominates the variance (like breast cancer's 98.2%, or here unnormalized, 83.5%), it's worth suspecting a large-scale variable is dominating the computation on its own, the same scale problem I already saw with KNN, just now affecting PCA itself instead of a distance.

# variance explained by component 1, with and without normalizing
ratio_unnormalized = 0.835  # x dominates, y barely counts
ratio_normalized = 0.548    # x and y compete on equal footing

The practical rule that sticks: whenever I'm about to run PCA on variables that aren't in the same unit (price in dollars next to age in years, say), normalizing first isn't optional, it's the same old caution, now applied somewhere new.