# Information-Theoretic Bounds and Phase Transitions in Clustering, Sparse PCA, and Submatrix Localization

@article{Banks2018InformationTheoreticBA,
title={Information-Theoretic Bounds and Phase Transitions in Clustering, Sparse PCA, and Submatrix Localization},
author={Jessica E. Banks and Cristopher Moore and Roman Vershynin and Nicolas Verzelen and Jiaming Xu},
journal={IEEE Transactions on Information Theory},
year={2018},
volume={64},
pages={4872-4894}
}
• Published 18 July 2016
• Computer Science
• IEEE Transactions on Information Theory
We study the problem of detecting a structured, low-rank signal matrix corrupted with additive Gaussian noise. This includes clustering in a Gaussian mixture model, sparse PCA, and submatrix localization. Each of these problems is conjectured to exhibit a sharp information-theoretic threshold, below which the signal is too weak for any algorithm to detect. We derive upper and lower bounds on these thresholds by applying the first and second moment methods to the likelihood ratio between these…
64 Citations

### Fundamental limits of symmetric low-rank matrix estimation

• Computer Science
COLT
• 2017
This paper considers the high-dimensional inference problem where the signal is a low-rank symmetric matrix which is corrupted by an additive Gaussian noise and compute the limit in the large dimension setting for the mutual information between the signal and the observations, while the rank of the signal remains constant.

### Phase transitions in spiked matrix estimation: information-theoretic analysis

The minimal mean squared error is computed for the estimation of the low-rank signal and it is compared to the performance of spectral estimators and message passing algorithms.

### Rank-one matrix estimation with groupwise heteroskedasticity

• Computer Science, Mathematics
ArXiv
• 2021
This work proves asymptotically exact formulas for the minimum mean-squared error in estimating both the matrix and the latent variables of a rank-one matrix from Gaussian observations.

### Optimality and Sub-optimality of PCA for Spiked Random Matrices and Synchronization

• Computer Science
ArXiv
• 2016
The fundamental limitations of statistical methods are studied, including non-spectral ones, and it is shown that inefficient procedures can work below the threshold where PCA succeeds, whereas no known efficient algorithm achieves this.

### Statistical limits of spiked tensor models

• Computer Science
Annales de l'Institut Henri Poincaré, Probabilités et Statistiques
• 2020
The replica prediction from statistical physics is conjectured to give the exact information-theoretic threshold for any fixed $d$, and a new improvement to the second moment method for contiguity is introduced, on which the lower bounds are based.

### Reducibility and Computational Lower Bounds for Problems with Planted Sparse Structure

• Computer Science
COLT
• 2018
This work introduces several new techniques to give a web of average-case reductions showing strong computational lower bounds based on the planted clique conjecture using natural problems as intermediates, including tight lower bounds for Planted Independent Set, Planted Dense Subgraph, Sparse Spiked Wigner, and Sparse PCA.

### Optimality and Sub-optimality of PCA I: Spiked Random Matrix Models

• Computer Science
The Annals of Statistics
• 2018
The statistical limits of tests for the presence of a spike are studied, including non-spectral tests, and include the Gaussian Wigner ensemble, where it is shown that PCA achieves the optimal detection threshold for certain natural priors for the spike.

### Community Detection and Stochastic Block Models

• E. Abbe
• Computer Science
Found. Trends Commun. Inf. Theory
• 2018
The recent developments that establish the fundamental limits for community detection in the stochastic block model are surveyed, both with respect to information-theoretic and computational thresholds, and for various recovery requirements such as exact, partial and weak recovery.

### Finite Size Corrections and Likelihood Ratio Fluctuations in the Spiked Wigner Model

• Computer Science, Mathematics
ArXiv
• 2017
It is proved that below the reconstruction threshold, where it becomes impossible to reconstruct the spike, the log-likelihood ratio has fluctuations of constant order and converges in distribution to a Gaussian under both the planted and (under restrictions) the null model.

### Fundamental limits of low-rank matrix estimation: the non-symmetric case

This work considers the high-dimensional inference problem where the signal is a low-rank matrix which is corrupted by an additive Gaussian noise and compute the limit in the large dimension setting for the mutual information between the signal and the observations, as well as the matrix minimum mean square error.

## References

SHOWING 1-10 OF 53 REFERENCES

### Fundamental limits of symmetric low-rank matrix estimation

• Computer Science
COLT
• 2017
This paper considers the high-dimensional inference problem where the signal is a low-rank symmetric matrix which is corrupted by an additive Gaussian noise and compute the limit in the large dimension setting for the mutual information between the signal and the observations, while the rank of the signal remains constant.

### Mutual information in rank-one matrix estimation

• Computer Science
2016 IEEE Information Theory Workshop (ITW)
• 2016
It is proved that the Bethe mutual information always yields an upper bound to the exact mutual information, using an interpolation method proposed by Guerra and later refined by Korada and Macris, in the case of rank-one symmetric matrix estimation.

### Statistical-Computational Tradeoffs in Planted Problems and Submatrix Localization with a Growing Number of Clusters and Submatrices

• Computer Science
J. Mach. Learn. Res.
• 2016
The results establish the minimax recovery limits, which are tight up to universal constants and hold even with a growing number of clusters/submatrices, and provide order-wise stronger performance guarantees for polynomial-time algorithms than previously known.

### Optimality and Sub-optimality of PCA for Spiked Random Matrices and Synchronization

• Computer Science
ArXiv
• 2016
The fundamental limitations of statistical methods are studied, including non-spectral ones, and it is shown that inefficient procedures can work below the threshold where PCA succeeds, whereas no known efficient algorithm achieves this.

### Phase transitions in sparse PCA

• Computer Science
2015 IEEE International Symposium on Information Theory (ISIT)
• 2015
It is shown that both for low density and for large rank the problem undergoes a series of phase transitions suggesting existence of a region of parameters where estimation is information theoretically possible, but AMP (and presumably every other polynomial algorithm) fails.

### DO SEMIDEFINITE RELAXATIONS SOLVE SPARSE PCA UP TO THE INFORMATION LIMIT

• Computer Science
• 2015
It is proved that when the proposed SDP approach, at least in its standard usage, cannot recover the sparse spike, and empirical results suggesting that up to sparsity levels $k=O(\sqrt{n})$, recovery is possible by a simple covariance thresholding algorithm.

### Statistical limits of spiked tensor models

• Computer Science
Annales de l'Institut Henri Poincaré, Probabilités et Statistiques
• 2020
The replica prediction from statistical physics is conjectured to give the exact information-theoretic threshold for any fixed $d$, and a new improvement to the second moment method for contiguity is introduced, on which the lower bounds are based.

### Spectral redemption in clustering sparse networks

• Computer Science
Proceedings of the National Academy of Sciences
• 2013
A way of encoding sparse data using a “nonbacktracking” matrix, and it is shown that the corresponding spectral algorithm performs optimally for some popular generative models, including the stochastic block model.

### Minimax Localization of Structural Information in Large Noisy Matrices

• Computer Science
NIPS
• 2011
The SNR required by several computationally tractable procedures for biclustering including element-wise thresholding, column/row average thresholding and a convex relaxation approach to sparse singular vector decomposition is characterized.

### Sparse PCA via Covariance Thresholding

• Computer Science
J. Mach. Learn. Res.
• 2016
A covariance thresholding algorithm that was recently proposed by Krauthgamer, Nadler and Vilenchik is analyzed and it is rigorously proved that the algorithm succeeds with high probability for k of order √n.