10.1 Principal Component Analysis (PCA) and Dimensionality Reduction

Principal Component Analysis (PCA) is a powerful technique for reducing data complexity. It transforms high-dimensional data into a lower-dimensional space, preserving the most important information while minimizing noise and redundancy.

PCA finds the directions of maximum variance in the data, called principal components. These components are uncorrelated and ordered by importance, allowing us to focus on the most significant patterns in our dataset.

Principal Component Analysis Fundamentals

Eigenvectors and Eigenvalues

Top images from around the web for Eigenvectors and Eigenvalues

• 

  Principal Component Analysis View original

  Is this image relevant?

• 

  pca - Making sense of principal component analysis, eigenvectors & eigenvalues - Cross Validated View original

  Is this image relevant?

• 

  pca - Making sense of principal component analysis, eigenvectors & eigenvalues - Cross Validated View original

  Is this image relevant?

• 

  Principal Component Analysis View original

  Is this image relevant?

• 

  pca - Making sense of principal component analysis, eigenvectors & eigenvalues - Cross Validated View original

  Is this image relevant?

1 of 3

Top images from around the web for Eigenvectors and Eigenvalues

• 

  Principal Component Analysis View original

  Is this image relevant?

• 

  pca - Making sense of principal component analysis, eigenvectors & eigenvalues - Cross Validated View original

  Is this image relevant?

• 

  pca - Making sense of principal component analysis, eigenvectors & eigenvalues - Cross Validated View original

  Is this image relevant?

• 

  Principal Component Analysis View original

  Is this image relevant?

• 

  pca - Making sense of principal component analysis, eigenvectors & eigenvalues - Cross Validated View original

  Is this image relevant?

1 of 3

• Eigenvectors are special vectors that, when a linear transformation is applied, only change in scale (magnitude) but not in direction
• The scale factor by which an eigenvector changes is called its eigenvalue
• Eigenvectors are orthogonal (perpendicular) to each other when the matrix is symmetric
• Eigenvectors and eigenvalues are crucial in understanding the underlying structure and properties of a matrix

Covariance Matrix and Principal Components

• The covariance matrix measures the linear relationship between pairs of variables in a dataset
  • Positive covariance indicates variables increase together
  • Negative covariance indicates one variable increases while the other decreases
  • Zero covariance suggests no linear relationship between variables
• Principal components are new variables that are constructed as linear combinations of the original variables
  • They are orthogonal (uncorrelated) to each other
  • The first principal component captures the largest possible variance in the data
  • Subsequent principal components capture the remaining variance in decreasing order while maintaining orthogonality
• Principal components are derived from the eigenvectors of the covariance matrix
  • The eigenvector with the largest eigenvalue becomes the first principal component, and so on

PCA Evaluation and Interpretation

Variance Explained and Scree Plot

• The variance explained by each principal component indicates how much information it captures from the original data
  • It is calculated as the eigenvalue of the principal component divided by the sum of all eigenvalues
  • Expressed as a percentage, it shows the proportion of total variance accounted for by each component
• A scree plot is a graphical representation of the variance explained by each principal component
  • The x-axis represents the principal components in descending order of variance explained
  • The y-axis shows the eigenvalues or the percentage of variance explained
  • The "elbow" point in the scree plot suggests the optimal number of principal components to retain

Feature Selection and Dimensionality Reduction

• PCA can be used for feature selection by identifying the most important variables that contribute to the majority of the variance
  • Variables with high loadings (coefficients) on the top principal components are considered more important
  • Variables with low loadings across all principal components may be less relevant and can be discarded
• By selecting a subset of the top principal components, PCA achieves dimensionality reduction
  • It transforms the data from a high-dimensional space to a lower-dimensional space
  • The reduced dimensions capture the essence of the original data while minimizing information loss
  • Dimensionality reduction helps in data visualization, computational efficiency, and mitigating the curse of dimensionality

Applications of PCA

Data Compression and Noise Reduction

• PCA can be used for data compression by representing the data using a smaller number of principal components
  • The compressed data retains the most important information while requiring less storage space
  • This is particularly useful in fields like image and signal processing (e.g., JPEG compression)
• PCA can also help in noise reduction by separating the signal from the noise
  • The top principal components capture the meaningful patterns and structures in the data
  • The lower principal components often represent noise or less significant variations
  • By reconstructing the data using only the top components, noise can be effectively filtered out

Singular Value Decomposition (SVD) and Matrix Factorization

• Singular Value Decomposition (SVD) is a matrix factorization technique closely related to PCA
  • It decomposes a matrix into three matrices: $A = U \Sigma V^T$
  • The columns of $U$ are the left singular vectors (eigenvectors of $AA^T$)
  • The columns of $V$ are the right singular vectors (eigenvectors of $A^TA$)
  • $\Sigma$ is a diagonal matrix containing the singular values (square roots of eigenvalues)
• SVD has applications in various domains, such as:
  • Recommender systems (e.g., Netflix prize)
  • Latent Semantic Analysis (LSA) in natural language processing
  • Matrix completion and collaborative filtering