Principal Component Analysis Steps

Why This Matters

PCA is the workhorse of dimensionality reduction, and you'll encounter it everywhere in data science—from preprocessing high-dimensional datasets to building recommendation systems and visualizing complex data. But here's what exams really test: your understanding of the linear algebra mechanics underneath the algorithm. You're being tested on eigendecomposition, variance maximization, orthogonal projections, and matrix transformations—PCA just happens to be the perfect vehicle for all of these concepts.

Don't just memorize "standardize, then find eigenvectors." Know why each step exists and what linear algebra principle it demonstrates. When an FRQ asks you to explain why we use the covariance matrix or what eigenvalues actually represent, you need to connect the dots between the algorithm and the underlying mathematics. Master these connections, and PCA questions become straightforward applications of concepts you already understand.

---

Data Preparation: Setting Up the Matrix

Before any linear algebra magic happens, your data needs to be in the right form. Standardization ensures that the covariance matrix reflects true relationships between features, not artifacts of measurement scale.

Data Standardization

• Z-score transformation—converts each feature to have $\mu = 0$ and $\sigma = 1$ using $z = \frac{x - \mu}{\sigma}$
• Equal feature contribution ensures that variables measured in different units (dollars vs. counts) don't dominate the analysis due to scale alone
• Covariance matrix sensitivity makes this step non-negotiable; without standardization, PCA finds directions of maximum scale, not maximum variance

---

The Covariance Matrix: Capturing Relationships

The covariance matrix is the foundation of PCA—it encodes everything about how your features relate to each other. This symmetric matrix contains all the information needed to find directions of maximum spread in your data.

Covariance Matrix Calculation

• Pairwise relationships—element $C_{ij}$ of the covariance matrix measures how features $i$ and $j$ vary together, computed as $\text{Cov}(X_i, X_j) = \frac{1}{n-1}\sum(x_i - \bar{x}_i)(x_j - \bar{x}_j)$
• Symmetric positive semi-definite structure guarantees real, non-negative eigenvalues—a critical property for the next steps
• Diagonal entries represent each feature's variance; off-diagonal entries reveal correlations that PCA will exploit

---

Eigendecomposition: Finding the Principal Directions

This is where core linear algebra takes center stage. Eigenvectors of the covariance matrix point in directions of maximum variance; eigenvalues tell you how much variance each direction captures.

Eigenvalue and Eigenvector Computation

• Characteristic equation $\det(C - \lambda I) = 0$ yields eigenvalues $\lambda$, each representing variance along its corresponding eigenvector
• Eigenvectors define axes—these orthogonal vectors form the new coordinate system where your data will live after transformation
• Spectral theorem guarantees that symmetric matrices (like covariance matrices) have orthogonal eigenvectors and real eigenvalues

Sorting Eigenvectors by Eigenvalues

• Descending order ranking—the eigenvector with the largest eigenvalue captures the most variance and becomes PC1
• Variance hierarchy means each subsequent component captures progressively less information, enabling principled dimensionality reduction
• Matrix formation stacks the top $k$ eigenvectors as columns to create the projection matrix $W$

---

Dimensionality Reduction: Making the Cut

Choosing how many components to keep is both art and science. The goal is to retain enough variance to preserve meaningful structure while eliminating noise and redundancy.

Selecting Principal Components

• Explained variance threshold—common choices include retaining components until 90-95% cumulative variance is captured
• Elbow method plots eigenvalues and looks for a "bend" where additional components add diminishing returns
• Trade-off awareness is essential: too few components loses signal, too many defeats the purpose of dimensionality reduction

Calculating Explained Variance Ratio

• Individual ratio for component $i$ equals $\frac{\lambda_i}{\sum_{j=1}^{n} \lambda_j}$, showing each component's contribution
• Cumulative sum helps determine the minimum number of components needed to hit your variance threshold
• Scree plots visualize these ratios, making it easy to identify where variance drops off sharply

---

Projection: Transforming the Data

The final step applies everything you've computed. Projection is a linear transformation that maps your original data onto the subspace spanned by the selected principal components.

Projecting Data onto Principal Components

• Matrix multiplication $Z = XW$ transforms standardized data $X$ using projection matrix $W$ (columns are top eigenvectors)
• Dimensionality change—if $X$ is $n \times p$ and you keep $k$ components, $Z$ is $n \times k$ where $k < p$
• Orthogonal projection ensures the transformed features (principal components) are uncorrelated—no redundant information

Interpreting Results and Dimensionality Reduction

• Loadings analysis examines eigenvector entries to understand which original features contribute most to each PC
• Reconstruction is possible via $\hat{X} = ZW^T$, though information lost to discarded components cannot be recovered
• Downstream applications include visualization (2-3 PCs), noise reduction, and preprocessing for models sensitive to multicollinearity

---

Quick Reference Table

---

Self-Check Questions

1. Why must data be standardized before computing the covariance matrix, and what would happen to your principal components if you skipped this step with features on different scales?

2. Which two PCA steps both rely directly on the eigenvalues of the covariance matrix, and how does each step use them differently?

3. Compare and contrast the information contained in eigenvectors versus eigenvalues—if someone gave you only the eigenvectors, what could you determine about the data, and what would be missing?

4. If your first three principal components explain 95% of the variance, what does this tell you about the effective dimensionality of your original dataset? How might this inform your modeling choices?

5. Explain why the projected data $Z = XW$ has uncorrelated features. What property of the eigenvectors guarantees this, and why is it useful for downstream analysis?