Principal Component Analysis (PCA) is a linear dimensionality-reduction technique. It identifies directions in a numerical dataset along which the observations vary most. Before applying PCA, features are commonly centered by subtracting their means; standardization may also be appropriate when features use different units or scales. PCA then identifies principal components, which are orthogonal directions formed as linear combinations of the original features. The first principal component captures the largest possible variance, while each subsequent component captures the largest remaining variance subject to being orthogonal to the previous components. PCA is commonly computed using eigenvalue decomposition of the covariance matrix or singular value decomposition of the centered data matrix. By retaining only selected components, PCA can reduce dimensionality, remove some redundancy, and support visualization. However, components may be difficult to interpret and PCA may discard information important for a particular prediction task.
Principal Component Analysis (PCA)
Principal Component Analysis is a statistical technique that transforms correlated numerical features into a smaller set of uncorrelated components that capture as much variance as possible.
X_centered = X - μ; Cov(X) = (1 / (n - 1)) X_centeredᵀ X_centered
Explanation
Variables
| Symbol | Meaning |
|---|---|
X | Original data matrix |
X_centered | Data matrix after subtracting feature means |
μ | Vector of feature means |
n | Number of observations |
Cov(X) | Sample covariance matrix of centered data |
Example or application
A dataset contains measurements of petal length, petal width, sepal length, and sepal width. PCA transforms these correlated measurements into principal components, allowing the observations to be visualized using two dimensions.
Text
PCA creates new coordinate directions that summarize the largest possible amounts of variation in numerical data.
Key Points
PCA is a linear dimensionality-reduction method.
Principal components are orthogonal linear combinations of original features.
The first component captures the greatest possible variance.
Later components capture remaining variance under orthogonality constraints.
PCA can be computed using eigenvalue decomposition or SVD.
Feature scaling may be important when variables use different units.
Applications
Data visualization
Noise reduction
Image compression
Exploratory data analysis
Preprocessing high-dimensional datasets
Learning Outcomes
Define Principal Component Analysis.
Explain how principal components are constructed.
Understand variance and orthogonality in PCA.
Identify situations where PCA is useful and where it may lose information.
Practice Questions
What is the main purpose of PCA?
What does the first principal component represent?
Why are principal components orthogonal?
When might standardization be important before applying PCA?
Related Concepts
Dimensionality Reduction
Covariance Matrix
Eigenvalues
Eigenvectors
Singular Value Decomposition
Common Mistakes
Assuming PCA always improves predictive performance.
Ignoring feature scales when variables use different units.
Treating principal components as original measured variables.
Assuming high explained variance guarantees high relevance to a target variable.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Jolliffe, I. T., and Cadima, J. (2016). Principal Component Analysis: A Review and Recent Developments.
- Bishop, C. M. (2006). Pattern Recognition and Machine Learning.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning