Finite Sample Approximation Results for Principal Component Analysis: a Matrix Perturbation Approach

Abstract

Principal Component Analysis (PCA) is a standard tool for dimensional reduction of a set of n observations (samples), each with p variables. In this paper, using a matrix perturbation approach, we study the non-asymptotic relation between the eigenvalues and eigenvectors of PCA computed on a finite sample of size n, to those of the limiting population PCA as n → ∞. As in machine learning, we present a finite sample theorem which holds with high probability for the closeness between the leading eigenvalue and eigenvector of sample PCA and population PCA under a spiked covariance model. In addition, we also consider the relation between finite sample PCA and the asymptotic results in the joint limit p, n →∞, with p/n = c. We present a matrix perturbation view of the “phase transition phenomenon”, and a simple linear-algebra based derivation of the eigenvalue and eigenvector overlap in this asymptotic limit. Moreover, our analysis also applies for finite p, n where we show that although there is no sharp phase transition as in the infinite case, either as a function of noise level or as a function of sample size n, the eigenvector of sample PCA may exhibit a sharp ”loss of tracking”, suddenly losing its relation to the (true) eigenvector of the population PCA matrix. This occurs due to a crossover between the eigenvalue due to the signal and the largest eigenvalue due to noise, whose eigenvector points in a random direction.

2 Figures and Tables

0102020072008200920102011201220132014201520162017
Citations per Year

115 Citations

Semantic Scholar estimates that this publication has 115 citations based on the available data.

See our FAQ for additional information.

Cite this paper

@inproceedings{Nadler2008FiniteSA, title={Finite Sample Approximation Results for Principal Component Analysis: a Matrix Perturbation Approach}, author={Boaz Nadler}, year={2008} }