# Information-Theoretic Bounds and Phase Transitions in Clustering, Sparse PCA, and Submatrix Localization

@article{Banks2016InformationTheoreticBA, title={Information-Theoretic Bounds and Phase Transitions in Clustering, Sparse PCA, and Submatrix Localization}, author={Jessica E. Banks and Cristopher Moore and Roman Vershynin and Nicolas Verzelen and Jiaming Xu}, journal={IEEE Transactions on Information Theory}, year={2016}, volume={64}, pages={4872-4894} }

We study the problem of detecting a structured, low-rank signal matrix corrupted with additive Gaussian noise. This includes clustering in a Gaussian mixture model, sparse PCA, and submatrix localization. Each of these problems is conjectured to exhibit a sharp information-theoretic threshold, below which the signal is too weak for any algorithm to detect. We derive upper and lower bounds on these thresholds by applying the first and second moment methods to the likelihood ratio between these…

## 64 Citations

### Fundamental limits of symmetric low-rank matrix estimation

- Computer ScienceCOLT
- 2017

This paper considers the high-dimensional inference problem where the signal is a low-rank symmetric matrix which is corrupted by an additive Gaussian noise and compute the limit in the large dimension setting for the mutual information between the signal and the observations, while the rank of the signal remains constant.

### Fundamental limits of symmetric low-rank matrix estimation

- Computer ScienceProbability Theory and Related Fields
- 2018

This paper considers the high-dimensional inference problem where the signal is a low-rank symmetric matrix which is corrupted by an additive Gaussian noise and compute the limit in the large dimension setting for the mutual information between the signal and the observations, while the rank of the signal remains constant.

### Phase transitions in spiked matrix estimation: information-theoretic analysis

- Computer ScienceArXiv
- 2018

The minimal mean squared error is computed for the estimation of the low-rank signal and it is compared to the performance of spectral estimators and message passing algorithms.

### Rank-one matrix estimation with groupwise heteroskedasticity

- Computer Science, MathematicsArXiv
- 2021

This work proves asymptotically exact formulas for the minimum mean-squared error in estimating both the matrix and the latent variables of a rank-one matrix from Gaussian observations.

### Optimality and Sub-optimality of PCA for Spiked Random Matrices and Synchronization

- Computer ScienceArXiv
- 2016

The fundamental limitations of statistical methods are studied, including non-spectral ones, and it is shown that inefficient procedures can work below the threshold where PCA succeeds, whereas no known efficient algorithm achieves this.

### Statistical limits of spiked tensor models

- Computer ScienceAnnales de l'Institut Henri Poincaré, Probabilités et Statistiques
- 2020

The replica prediction from statistical physics is conjectured to give the exact information-theoretic threshold for any fixed $d$, and a new improvement to the second moment method for contiguity is introduced, on which the lower bounds are based.

### Reducibility and Computational Lower Bounds for Problems with Planted Sparse Structure

- Computer ScienceCOLT
- 2018

This work introduces several new techniques to give a web of average-case reductions showing strong computational lower bounds based on the planted clique conjecture using natural problems as intermediates, including tight lower bounds for Planted Independent Set, Planted Dense Subgraph, Sparse Spiked Wigner, and Sparse PCA.

### Optimality and Sub-optimality of PCA I: Spiked Random Matrix Models

- Computer ScienceThe Annals of Statistics
- 2018

The statistical limits of tests for the presence of a spike are studied, including non-spectral tests, and include the Gaussian Wigner ensemble, where it is shown that PCA achieves the optimal detection threshold for certain natural priors for the spike.

### Community Detection and Stochastic Block Models

- Computer ScienceFound. Trends Commun. Inf. Theory
- 2018

The recent developments that establish the fundamental limits for community detection in the stochastic block model are surveyed, both with respect to information-theoretic and computational thresholds, and for various recovery requirements such as exact, partial and weak recovery.

### Fundamental limits of low-rank matrix estimation: the non-symmetric case

- Computer Science
- 2017

This work considers the high-dimensional inference problem where the signal is a low-rank matrix which is corrupted by an additive Gaussian noise and compute the limit in the large dimension setting for the mutual information between the signal and the observations, as well as the matrix minimum mean square error.

## References

SHOWING 1-10 OF 53 REFERENCES

### Fundamental limits of symmetric low-rank matrix estimation

- Computer ScienceCOLT
- 2017

This paper considers the high-dimensional inference problem where the signal is a low-rank symmetric matrix which is corrupted by an additive Gaussian noise and compute the limit in the large dimension setting for the mutual information between the signal and the observations, while the rank of the signal remains constant.

### Statistical-Computational Tradeoffs in Planted Problems and Submatrix Localization with a Growing Number of Clusters and Submatrices

- Computer ScienceJ. Mach. Learn. Res.
- 2016

The results establish the minimax recovery limits, which are tight up to universal constants and hold even with a growing number of clusters/submatrices, and provide order-wise stronger performance guarantees for polynomial-time algorithms than previously known.

### Optimality and Sub-optimality of PCA for Spiked Random Matrices and Synchronization

- Computer ScienceArXiv
- 2016

The fundamental limitations of statistical methods are studied, including non-spectral ones, and it is shown that inefficient procedures can work below the threshold where PCA succeeds, whereas no known efficient algorithm achieves this.

### Phase transitions in sparse PCA

- Computer Science2015 IEEE International Symposium on Information Theory (ISIT)
- 2015

It is shown that both for low density and for large rank the problem undergoes a series of phase transitions suggesting existence of a region of parameters where estimation is information theoretically possible, but AMP (and presumably every other polynomial algorithm) fails.

### DO SEMIDEFINITE RELAXATIONS SOLVE SPARSE PCA UP TO THE INFORMATION LIMIT

- Computer Science
- 2015

It is proved that when the proposed SDP approach, at least in its standard usage, cannot recover the sparse spike, and empirical results suggesting that up to sparsity levels $k=O(\sqrt{n})$, recovery is possible by a simple covariance thresholding algorithm.

### Statistical limits of spiked tensor models

- Computer ScienceAnnales de l'Institut Henri Poincaré, Probabilités et Statistiques
- 2020

The replica prediction from statistical physics is conjectured to give the exact information-theoretic threshold for any fixed $d$, and a new improvement to the second moment method for contiguity is introduced, on which the lower bounds are based.

### Spectral redemption in clustering sparse networks

- Computer ScienceProceedings of the National Academy of Sciences
- 2013

A way of encoding sparse data using a “nonbacktracking” matrix, and it is shown that the corresponding spectral algorithm performs optimally for some popular generative models, including the stochastic block model.

### Minimax Localization of Structural Information in Large Noisy Matrices

- Computer ScienceNIPS
- 2011

The SNR required by several computationally tractable procedures for biclustering including element-wise thresholding, column/row average thresholding and a convex relaxation approach to sparse singular vector decomposition is characterized.

### Sparse PCA via Covariance Thresholding

- Computer ScienceJ. Mach. Learn. Res.
- 2016

A covariance thresholding algorithm that was recently proposed by Krauthgamer, Nadler and Vilenchik is analyzed and it is rigorously proved that the algorithm succeeds with high probability for k of order √n.

### Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula

- Computer ScienceNIPS
- 2016

It is shown how to rigorously prove the conjectured formula for the symmetric rank-one case, which allows to express the minimal mean-square-error and to characterize the detectability phase transitions in a large set of estimation problems ranging from community detection to sparse PCA.