Does retaining 99% of variation preserve what a model needs?

The largest difference may not be the decisive one

A business model receives many features we want to compress. Retaining over 99% variance sounds as if less than 1% of useful information were lost. But data variation and predictive information differ. We derive principal component analysis, PCA, and construct a result optimal for reconstruction yet insufficient to distinguish classes. The aim is to interpret its objective correctly, not discredit the method.

Abstract. Four(±10,±1)points have diagonal covariance, eigenvalues 400/3 and 4/3. PC 1 retains 100/101≈99.0099% variance with mean squared reconstruction error 1 per point. Labels defined by second-coordinate sign collide under projection: no deterministic PC 1-only classifier matches more than 2 of 4 labels. This is not test accuracy. Rescaling feature 1 changes selected direction and can yield degenerate eigenvalues. Calculations use NumPy, no trained predictor.

The complete dataset and supervised question

X rows are(−10,−1),(−10,1),(10,−1),(10,1), synthetic dimensionless features with large and small amplitudes. Label 0 means second coordinate negative,1 positive. Both column means are zero. No train/test split or generalization estimate: four points prove high retained variance does not logically imply preservation of task-relevant information.

What PCA maximizes

Choose unit column direction v with two coordinates. Projection z=Xv gives one number per row. Centered sample variance is vᵀCv, C=XᵀX/(n−1), n=4. PC 1 maximizes it: where data spread most, not where labels are best predicted. Labels do not enter the objective.

XᵀX has diagonal 400,4 and cross-products 10−10−10+10=0. Divide by 3. Differentiating vᵀCv−λ(vᵀv−1) gives 2 Cv−2λv=0, so an eigenvector with variance equal to its eigenvalue. Since 400/3>4/3, PC 1 is coordinate 1, up to a sign irrelevant to reconstructed projection.

Retained 99.0099% divides by total variance, trace C, not class information. Coordinate 2 contributes below 1% variance yet entirely defines labels. PCA correctly solves its objective. The error is mentally replacing retained variance with retained predictive utility.

Why reconstruction really is good

One-component reconstruction X_hat=(Xv)vᵀ=Xvvᵀ is orthogonal projection. Reconstruction and residual are orthogonal, so squared total norm splits: minimum residual means maximum projected norm. Frobenius norm sums every matrix-entry square. This explains PCA as sensible compression for that particular error.

Each point loses only±1 in coordinate 2, squared residual 1, mean 1. Original squared sum 404 retains 400, again 100/101. Lost sample variance 4/3 differs because covariance divides by n−1 while per-point error divides by n. Explicit denominators reconcile them; this is not a per-coordinate mean.

Small information the projection removes

After projection, first two have z=−10 but opposite labels; last two share 10 with opposite labels. Deterministic g(z)must answer identically for identical input, matching at most one per pair,2/4 overall. This proves a finite representation limit, not trained 50% test accuracy. Row IDs or other features would change the premise and are excluded.

With both coordinates, the label is second-sign by definition, not real-world generalization or frequency of PCA failures. This counterexample refutes a universal guarantee. Low variance can be noise elsewhere; here it defines target. Task relation, not amplitude alone, determines importance.

Changing scale can change the component

Scale coordinate 1 by c. Covariance diagonal becomes 100 c²× 4/3 and 4/3. At c=.2, coordinate 1 retains 80%; at.05, coordinate 2 retains 80%. At.1 eigenvalues tie: no unique first direction. A solver still returns a basis, but arbitrary orientation within a tied eigenspace is not a discovered preferred direction.

Choosing c=.05 is not a general optimum; we used counterexample knowledge. Scaling defines method geometry. Standardization does not guarantee predictive direction and can tie variances here. Rescaling or whitening retained PC 1 cannot restore deleted coordinate: identical z remains identical under any function of z alone.

What to verify in a real pipeline

Official scikit-learn PCA docs read at 1.9.1 specify centering, SVD, explained variance and no automatic feature scaling. We used NumPy 2.5.3, not scikit-learn. Real predictors should compare compressed dimensions against task outcomes and no transform. Official pitfalls docs include PCA in test-data leakage concerns. Our four points are not a real predictive evaluation.

Package saves points, labels, covariance, eigenvectors, reconstruction, rescalings; assertions check 100/101, error 1, limit 2/4. No randomness. Eigenvector signs may vary without changing projection; degeneracy needs separate treatment. Large-matrix truncated/randomized SVD choices are not run. No EL-AI time, memory savings or quality inferred.

Retaining 99% variance can preserve little of what distinguishes required answers. Here reconstruction is correctly optimized while target lives in removed low-variance coordinate. This does not imply always retaining everything: align compression metrics with the final task. Variance percentage describes representation under chosen geometry; prediction needs separate evidence.

scikit-learn 1.9.1 PCA documentation, read 17 October 2026: centering, variance, SVD, scaling and solver caveats. NumPy, not scikit-learn, was executed..

scikit-learn 1.9.1 Common pitfalls,§12.2, read 17 October 2026. Real evaluation requires separation of fitting and testing; no test estimate claimed here..

from experiment import X
import numpy as np
print(np.cov(X,rowvar=False))
print(np.linalg.eigh(np.cov(X,rowvar=False)))

Code, data, and instructions · JSON. Educational calculations executed with Python 3.14.0; figures with Matplotlib 3.11.2. AI-assisted analysis, without claiming peer review or human review. Original illustrative ImageGen cover: it does not document EL-AI people, premises, or installations. Sources accessed 17 October 2026.