Skip to content
My Docs

17_Principal_Component_Analysis__Pca_

Principal Component Analysis (PCA)

Section titled “Principal Component Analysis (PCA)”

Category: Classic Machine Learning Algorithms
Type: AI/ML Concept
Generated on: 2025-08-26 10:56:04
For: Data Science, Machine Learning & Technical Interviews

PCA Cheatsheet: Principal Component Analysis

Section titled “PCA Cheatsheet: Principal Component Analysis”

1. Quick Overview

Section titled “1. Quick Overview”

What is it? Principal Component Analysis (PCA) is a dimensionality reduction technique used to transform a dataset with potentially correlated variables into a new set of uncorrelated variables called Principal Components (PCs). The first PC captures the most variance in the data, the second PC captures the second most, and so on.

Why is it important?

  • Reduces Dimensionality: Simplifies complex datasets, leading to faster computation and reduced storage.
  • Feature Extraction: Extracts the most important features, improving model performance and interpretability.
  • Noise Reduction: Filters out less important variations, focusing on the underlying structure of the data.
  • Visualization: Allows for data visualization in lower dimensions (2D or 3D), revealing patterns and clusters.

2. Key Concepts

Section titled “2. Key Concepts”
  • Variance: A measure of how spread out the data is. PCA aims to maximize the variance captured by each principal component.
  • Covariance: A measure of how two variables change together. PCA aims to find uncorrelated principal components, minimizing covariance between them.
  • Eigenvalues and Eigenvectors:
    • Eigenvector: A vector that, when multiplied by a matrix, only changes in scale (not direction). In PCA, eigenvectors represent the direction of the principal components.
    • Eigenvalue: The scale factor by which the eigenvector is scaled when multiplied by the matrix. It represents the amount of variance explained by the corresponding eigenvector (principal component).
  • Principal Components (PCs): The new set of uncorrelated variables. They are linear combinations of the original variables.
  • Explained Variance Ratio: The proportion of the total variance in the dataset that is explained by each principal component. Used to determine how many PCs to keep.
    • Formula: Explained Variance Ratio = Eigenvalue / Sum of all Eigenvalues
  • Data Standardization/Scaling: Crucial before applying PCA. Transforms data to have zero mean and unit variance. Prevents variables with larger scales from dominating the analysis.
  • Singular Value Decomposition (SVD): A matrix factorization technique often used to compute PCA. It decomposes the data matrix into three matrices: U, S, and V. The eigenvectors are found in V.

3. How It Works

Section titled “3. How It Works”

Step-by-Step:

  1. Data Preparation:

    • Standardize/Scale the data: Ensure each feature has zero mean and unit variance.
      • Formula: z = (x - μ) / σ (where x is the original value, μ is the mean, and σ is the standard deviation)

  2. Calculate the Covariance Matrix: Measures the relationships between all pairs of variables.

    • Formula: Cov(X, Y) = Σ [(Xi - μX) * (Yi - μY)] / (n - 1) (where X and Y are the variables, μX and μY are their means, and n is the number of data points)

  3. Compute Eigenvalues and Eigenvectors: Find the eigenvalues and eigenvectors of the covariance matrix.

    • A * v = λ * v (where A is the covariance matrix, v is the eigenvector, and λ is the eigenvalue)

  4. Sort Eigenvalues and Select Principal Components:

    • Sort the eigenvalues in descending order. The eigenvector corresponding to the largest eigenvalue is the first principal component, and so on.
    • Choose the top k eigenvectors based on the explained variance ratio. A common rule of thumb is to keep enough PCs to explain 90-95% of the variance.

  5. Project the Data: Transform the original data onto the new principal component axes.

    • Transformed Data = Original Data * Eigenvector Matrix

Diagram (ASCII Art):

Original Data (High Dimensional)
     |
     | Standardize/Scale
     V
Standardized Data
     |
     | Calculate Covariance Matrix
     V
Covariance Matrix
     |
     | Compute Eigenvalues & Eigenvectors
     V
Eigenvalues & Eigenvectors
     |
     | Sort Eigenvalues & Select Top k Eigenvectors
     V
Principal Components (Reduced Dimensionality)
     |
     | Project Original Data
     V
Transformed Data (Lower Dimensional)

Python Example (Scikit-learn):

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt


# Sample data (replace with your actual data)
data = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]])


# 1. Standardize the data
scaler = StandardScaler()
scaled_data = scaler.fit_transform(data)


# 2. Apply PCA
pca = PCA(n_components=2)  # Choose the number of components
pca.fit(scaled_data)


# 3. Transform the data
transformed_data = pca.transform(scaled_data)


# Explained variance ratio
print("Explained Variance Ratio:", pca.explained_variance_ratio_)


# Visualize the transformed data (if n_components=2)
if transformed_data.shape[1] == 2:
    plt.scatter(transformed_data[:, 0], transformed_data[:, 1])
    plt.xlabel("Principal Component 1")
    plt.ylabel("Principal Component 2")
    plt.title("PCA Transformed Data")
    plt.show()

4. Real-World Applications

Section titled “4. Real-World Applications”
  • Image Compression: Reducing the size of images by keeping only the most important principal components.
  • Facial Recognition: Identifying faces by extracting the most relevant features from facial images.
  • Genomics: Analyzing gene expression data to identify patterns and relationships between genes.
  • Finance: Reducing the dimensionality of financial data for risk management and portfolio optimization.
  • Recommendation Systems: Improving the performance of recommendation systems by reducing the number of features used to represent users and items.
  • Data Visualization: Visualizing high-dimensional data in 2D or 3D scatter plots to identify clusters and outliers. Example: Visualizing customer segments based on purchase history.

5. Strengths and Weaknesses

Section titled “5. Strengths and Weaknesses”

Strengths:

  • Dimensionality Reduction: Simplifies data and reduces computational cost.
  • Feature Extraction: Identifies the most important features.
  • Noise Reduction: Filters out irrelevant variations.
  • Improved Model Performance: Can improve the accuracy and efficiency of machine learning models.
  • Data Visualization: Facilitates visualization of high-dimensional data.

Weaknesses:

  • Information Loss: Some information is inevitably lost during dimensionality reduction.
  • Linearity Assumption: PCA assumes that the relationships between variables are linear. May not perform well on data with highly non-linear relationships.
  • Interpretability: The principal components may not be easily interpretable in terms of the original variables.
  • Sensitivity to Scaling: Data must be properly scaled before applying PCA.
  • Curse of Dimensionality Mitigation, Not Elimination: Reduces the effects but does not fundamentally solve the curse of dimensionality.

6. Interview Questions

Section titled “6. Interview Questions”
  • What is PCA and why is it used?
    • Answer: PCA is a dimensionality reduction technique used to transform a dataset into a new set of uncorrelated variables called principal components. It’s used to reduce dimensionality, extract features, reduce noise, and improve model performance.
  • Explain the steps involved in PCA.
    • Answer: (See “How It Works” section above)
  • What is an eigenvalue and an eigenvector? How are they related to PCA?
    • Answer: An eigenvector is a vector that, when multiplied by a matrix, only changes in scale. An eigenvalue is the scale factor. In PCA, eigenvectors represent the direction of the principal components, and eigenvalues represent the amount of variance explained by each component.
  • How do you decide how many principal components to keep?
    • Answer: You can look at the explained variance ratio for each component. A common approach is to keep enough components to explain a certain percentage of the variance (e.g., 90-95%). Scree plots (plotting eigenvalues) can also help identify an “elbow” point where adding more components provides diminishing returns.
  • Why is it important to scale the data before applying PCA?
    • Answer: Scaling ensures that variables with larger scales don’t dominate the analysis. Without scaling, variables with larger ranges will have a disproportionately large influence on the principal components.
  • What are some real-world applications of PCA?
    • Answer: (See “Real-World Applications” section above)
  • What are the limitations of PCA?
    • Answer: (See “Strengths and Weaknesses” section above)
  • How does PCA relate to Singular Value Decomposition (SVD)?
    • Answer: SVD is a matrix factorization technique that is often used to compute PCA. The eigenvectors used for PCA can be derived from the SVD of the data matrix.
  • What is the difference between PCA and LDA (Linear Discriminant Analysis)?
    • Answer: PCA is an unsupervised technique that aims to find the directions of maximum variance in the data. LDA is a supervised technique that aims to find the directions that best separate different classes in the data. PCA focuses on data representation, while LDA focuses on classification.

7. Further Reading

Section titled “7. Further Reading”
  • Related Concepts:
    • Linear Discriminant Analysis (LDA)
    • t-distributed Stochastic Neighbor Embedding (t-SNE)
    • Autoencoders (for non-linear dimensionality reduction)
    • Feature Selection Techniques
  • Resources: