✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-087

Principal Component Analysis (PCA)

Principal Component Analysis is a statistical technique that transforms correlated numerical features into a smaller set of uncorrelated components that capture as much variance as possible.

X_centered = X - μ; Cov(X) = (1 / (n - 1)) X_centeredᵀ X_centered

Explanation

Principal Component Analysis (PCA) is a linear dimensionality-reduction technique. It identifies directions in a numerical dataset along which the observations vary most. Before applying PCA, features are commonly centered by subtracting their means; standardization may also be appropriate when features use different units or scales. PCA then identifies principal components, which are orthogonal directions formed as linear combinations of the original features. The first principal component captures the largest possible variance, while each subsequent component captures the largest remaining variance subject to being orthogonal to the previous components. PCA is commonly computed using eigenvalue decomposition of the covariance matrix or singular value decomposition of the centered data matrix. By retaining only selected components, PCA can reduce dimensionality, remove some redundancy, and support visualization. However, components may be difficult to interpret and PCA may discard information important for a particular prediction task.

Variables

SymbolMeaning

X

Original data matrix

X_centered

Data matrix after subtracting feature means

μ

Vector of feature means

n

Number of observations

Cov(X)

Sample covariance matrix of centered data

Example or application

A dataset contains measurements of petal length, petal width, sepal length, and sepal width. PCA transforms these correlated measurements into principal components, allowing the observations to be visualized using two dimensions.

Text

PCA creates new coordinate directions that summarize the largest possible amounts of variation in numerical data.

Key Points

  • PCA is a linear dimensionality-reduction method.

  • Principal components are orthogonal linear combinations of original features.

  • The first component captures the greatest possible variance.

  • Later components capture remaining variance under orthogonality constraints.

  • PCA can be computed using eigenvalue decomposition or SVD.

  • Feature scaling may be important when variables use different units.

Applications

  • Data visualization

  • Noise reduction

  • Image compression

  • Exploratory data analysis

  • Preprocessing high-dimensional datasets

Learning Outcomes

  • Define Principal Component Analysis.

  • Explain how principal components are constructed.

  • Understand variance and orthogonality in PCA.

  • Identify situations where PCA is useful and where it may lose information.

Practice Questions

  • What is the main purpose of PCA?

  • What does the first principal component represent?

  • Why are principal components orthogonal?

  • When might standardization be important before applying PCA?

Related Concepts

  • Dimensionality Reduction

  • Covariance Matrix

  • Eigenvalues

  • Eigenvectors

  • Singular Value Decomposition

Common Mistakes

  • Assuming PCA always improves predictive performance.

  • Ignoring feature scales when variables use different units.

  • Treating principal components as original measured variables.

  • Assuming high explained variance guarantees high relevance to a target variable.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Jolliffe, I. T., and Cadima, J. (2016). Principal Component Analysis: A Review and Recent Developments.
  • Bishop, C. M. (2006). Pattern Recognition and Machine Learning.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning