Information-Theoretic Bounds and Phase Transitions in Clustering, Sparse PCA, and Submatrix Localization

@article{Banks2018InformationTheoreticBA,
  title={Information-Theoretic Bounds and Phase Transitions in Clustering, Sparse PCA, and Submatrix Localization},
  author={Jessica E. Banks and Cristopher Moore and Roman Vershynin and Nicolas Verzelen and Jiaming Xu},
  journal={IEEE Transactions on Information Theory},
  year={2018},
  volume={64},
  pages={4872-4894}
}
We study the problem of detecting a structured, low-rank signal matrix corrupted with additive Gaussian noise. This includes clustering in a Gaussian mixture model, sparse PCA, and submatrix localization. Each of these problems is conjectured to exhibit a sharp information-theoretic threshold, below which the signal is too weak for any algorithm to detect. We derive upper and lower bounds on these thresholds by applying the first and second moment methods to the likelihood ratio between these… 

Fundamental limits of symmetric low-rank matrix estimation

This paper considers the high-dimensional inference problem where the signal is a low-rank symmetric matrix which is corrupted by an additive Gaussian noise and compute the limit in the large dimension setting for the mutual information between the signal and the observations, while the rank of the signal remains constant.

Phase transitions in spiked matrix estimation: information-theoretic analysis

The minimal mean squared error is computed for the estimation of the low-rank signal and it is compared to the performance of spectral estimators and message passing algorithms.

Rank-one matrix estimation with groupwise heteroskedasticity

This work proves asymptotically exact formulas for the minimum mean-squared error in estimating both the matrix and the latent variables of a rank-one matrix from Gaussian observations.

Optimality and Sub-optimality of PCA for Spiked Random Matrices and Synchronization

The fundamental limitations of statistical methods are studied, including non-spectral ones, and it is shown that inefficient procedures can work below the threshold where PCA succeeds, whereas no known efficient algorithm achieves this.

Statistical limits of spiked tensor models

The replica prediction from statistical physics is conjectured to give the exact information-theoretic threshold for any fixed $d$, and a new improvement to the second moment method for contiguity is introduced, on which the lower bounds are based.

Reducibility and Computational Lower Bounds for Problems with Planted Sparse Structure

This work introduces several new techniques to give a web of average-case reductions showing strong computational lower bounds based on the planted clique conjecture using natural problems as intermediates, including tight lower bounds for Planted Independent Set, Planted Dense Subgraph, Sparse Spiked Wigner, and Sparse PCA.

Optimality and Sub-optimality of PCA I: Spiked Random Matrix Models

The statistical limits of tests for the presence of a spike are studied, including non-spectral tests, and include the Gaussian Wigner ensemble, where it is shown that PCA achieves the optimal detection threshold for certain natural priors for the spike.

Community Detection and Stochastic Block Models

  • E. Abbe
  • Computer Science
    Found. Trends Commun. Inf. Theory
  • 2018
The recent developments that establish the fundamental limits for community detection in the stochastic block model are surveyed, both with respect to information-theoretic and computational thresholds, and for various recovery requirements such as exact, partial and weak recovery.

Finite Size Corrections and Likelihood Ratio Fluctuations in the Spiked Wigner Model

It is proved that below the reconstruction threshold, where it becomes impossible to reconstruct the spike, the log-likelihood ratio has fluctuations of constant order and converges in distribution to a Gaussian under both the planted and (under restrictions) the null model.

Fundamental limits of low-rank matrix estimation: the non-symmetric case

This work considers the high-dimensional inference problem where the signal is a low-rank matrix which is corrupted by an additive Gaussian noise and compute the limit in the large dimension setting for the mutual information between the signal and the observations, as well as the matrix minimum mean square error.
...

References

SHOWING 1-10 OF 53 REFERENCES

Fundamental limits of symmetric low-rank matrix estimation

This paper considers the high-dimensional inference problem where the signal is a low-rank symmetric matrix which is corrupted by an additive Gaussian noise and compute the limit in the large dimension setting for the mutual information between the signal and the observations, while the rank of the signal remains constant.

Mutual information in rank-one matrix estimation

It is proved that the Bethe mutual information always yields an upper bound to the exact mutual information, using an interpolation method proposed by Guerra and later refined by Korada and Macris, in the case of rank-one symmetric matrix estimation.

Statistical-Computational Tradeoffs in Planted Problems and Submatrix Localization with a Growing Number of Clusters and Submatrices

The results establish the minimax recovery limits, which are tight up to universal constants and hold even with a growing number of clusters/submatrices, and provide order-wise stronger performance guarantees for polynomial-time algorithms than previously known.

Optimality and Sub-optimality of PCA for Spiked Random Matrices and Synchronization

The fundamental limitations of statistical methods are studied, including non-spectral ones, and it is shown that inefficient procedures can work below the threshold where PCA succeeds, whereas no known efficient algorithm achieves this.

Phase transitions in sparse PCA

It is shown that both for low density and for large rank the problem undergoes a series of phase transitions suggesting existence of a region of parameters where estimation is information theoretically possible, but AMP (and presumably every other polynomial algorithm) fails.

DO SEMIDEFINITE RELAXATIONS SOLVE SPARSE PCA UP TO THE INFORMATION LIMIT

It is proved that when the proposed SDP approach, at least in its standard usage, cannot recover the sparse spike, and empirical results suggesting that up to sparsity levels $k=O(\sqrt{n})$, recovery is possible by a simple covariance thresholding algorithm.

Statistical limits of spiked tensor models

The replica prediction from statistical physics is conjectured to give the exact information-theoretic threshold for any fixed $d$, and a new improvement to the second moment method for contiguity is introduced, on which the lower bounds are based.

Spectral redemption in clustering sparse networks

A way of encoding sparse data using a “nonbacktracking” matrix, and it is shown that the corresponding spectral algorithm performs optimally for some popular generative models, including the stochastic block model.

Minimax Localization of Structural Information in Large Noisy Matrices

The SNR required by several computationally tractable procedures for biclustering including element-wise thresholding, column/row average thresholding and a convex relaxation approach to sparse singular vector decomposition is characterized.

Sparse PCA via Covariance Thresholding

A covariance thresholding algorithm that was recently proposed by Krauthgamer, Nadler and Vilenchik is analyzed and it is rigorously proved that the algorithm succeeds with high probability for k of order √n.
...