Streaming Bayesian inference: Theoretical limits and mini-batch approximate message-passing

@article{Manoel2017StreamingBI,
  title={Streaming Bayesian inference: Theoretical limits and mini-batch approximate message-passing},
  author={Andre Manoel and Florent Krzakala and Eric W. Tramel and Lenka Zdeborov{\'a}},
  journal={2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton)},
  year={2017},
  pages={1048-1055}
}
In statistical learning for real-world large-scale data problems, one must often resort to “streaming” algorithms which operate sequentially on small batches of data. In this work, we present an analysis of the information-theoretic limits of mini-batch inference in the context of generalized linear models and low-rank matrix factorization. In a controlled Bayes-optimal setting, we characterize the optimal performance and phase transitions as a function of mini-batch size. We base part of our… 

Figures from this paper

Streaming Belief Propagation for Community Detection
TLDR
This work introduces a simple model for networks growing over time which it is referred to as streaming stochastic block model (StSBM) and proves that voting algorithms have fundamental limitations, and develops a streaming belief-propagation approach which proves optimality in certain regimes.
An Approximate Message Passing Framework for Side Information
TLDR
This paper develops a suite of algorithms, called AMP-SI, and derive denoisers for the BDD and BG models, and demonstrates the simplicity and applicability of this approach.
Rigorous State Evolution Analysis for Approximate Message Passing with Side Information
TLDR
This work provides rigorous performance guarantees for AMP-SI when there are statistical dependencies between the signal and SI pairs and the entries of the measurement matrix are independent and identically distributed Gaussian.
Approximate message-passing for convex optimization with non-separable penalties
TLDR
The connection between message-passing algorithms-typically used for approximate inference-and proximal methods for optimization, and show that the VAMP scheme is, as VAMP, similar in nature to the Peaceman-Rachford splitting, with the important difference that stepsizes are set adaptively.
An Analysis of State Evolution for Approximate Message Passing with Side Information
TLDR
This work provides rigorous performance guarantees for AMP-SI when the input signal and SI are drawn i.i.d. according to some joint distribution subject to finite moment constraints.
Mean-field inference methods for neural networks
  • Marylou Gabri'e
  • Computer Science
    Journal of Physics A: Mathematical and Theoretical
  • 2020
TLDR
A selection of classical mean-field methods and recent progress relevant for inference in neural networks are reviewed, and the principles of derivations of high-temperature expansions, the replica method and message passing algorithms are reminded, highlighting their equivalences and complementarities.
L1-Minimization Algorithm for Bayesian Online Compressed Sensing
TLDR
It is shown that reconstruction is possible even if prior knowledge about the generation of the signal is limited, by introduction of a Laplace prior and of an extra Kullback–Leibler divergence minimization step for hyper-parameter learning.
Learning capabilities of belief propagation based algorithms for sparse binary neural networks
TLDR
The learning performance of belief propagation (BP) based algorithms, applied to simple two layers sparse neural networks with discrete synapses, is analysed and encourages results, allowing to perfectly reconstruct the weights of the ‘teacher’ network that generated the training data.
Proof of State Evolution for AMP with Side Information
A common goal in many research areas is to reconstruct an unknown signal x from noisy linear measurements. Approximate message passing (AMP) is a class of low-complexity algorithms for efficiently
Decentralized Expectation Consistent Signal Recovery for Phase Retrieval
TLDR
A phase retrieval solution that aims to recover signals from noisy phaseless measurements by leveraging the core framework of GEC-SR, and proposes a novel decentralized algorithm called deGEC- SR, which exhibits excellent performance similar to G EC-SR but runs tens to hundreds of times faster.
...
1
2
...

References

SHOWING 1-10 OF 46 REFERENCES
Streaming Variational Inference for Bayesian Nonparametric Mixture Models
TLDR
This work works within this general framework and presents a streaming variational inference algorithm for NRM mixture models based on assumed density filtering, leading straightforwardly to expectation propagation for large-scale batch inference as well.
Online Learning of Nonparametric Mixture Models via Sequential Variational Approximation
TLDR
A Bayesian learning algorithm for DP mixture models that recursively transforms a given prior into an approximate posterior through sequential variational approximation, making it particularly suited for applications with massive data.
S-AMP: Approximate message passing for general matrix ensembles
TLDR
This work proposes a novel iterative estimation algorithm for linear observation models called S-AMP, which extends the approximate message-passing algorithm to general matrix ensembles with a well-defined large system size limit.
Phase Transitions and Sample Complexity in Bayes-Optimal Matrix Factorization
TLDR
This work compute the minimal mean-squared-error achievable, in principle, in any computational time, and the error that can be achieved by an efficient approximate message passing algorithm based on the asymptotic state-evolution analysis of the algorithm.
Streaming Variational Bayes
TLDR
SDA-Bayes is presented, a framework for streaming updates to the estimated posterior of a Bayesian posterior, with variational Bayes (VB) as the primitive, and the usefulness of the framework is demonstrated by fitting the latent Dirichlet allocation model to two large-scale document collections.
Constrained Low-rank Matrix Estimation: Phase Transitions, Approximate Message Passing and Applications
TLDR
The derivation of the TAP equations for models as different as the Sherrington-Kirkpatrick model, the restricted Boltzmann machine, the Hopfield model or vector (xy, Heisenberg and other) spin glasses are unify.
Generalized approximate message passing for estimation with random linear mixing
  • S. Rangan
  • Computer Science
    2011 IEEE International Symposium on Information Theory Proceedings
  • 2011
TLDR
G-AMP incorporates general measurement channels and shows that the asymptotic behavior of the G-AMP algorithm under large i.i.d. measurement channels is similar to the AWGN output channel case, and Gaussian transform matrices is described by a simple set of state evolution (SE) equations.
Statistical and computational phase transitions in spiked tensor estimation
TLDR
The performance of Approximate Message Passing is studied and it is shown that it achieves the MMSE for a large set of parameters, and that factorization is algorithmically “easy” in a much wider region than previously believed.
Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula
TLDR
It is shown how to rigorously prove the conjectured formula for the symmetric rank-one case, which allows to express the minimal mean-square-error and to characterize the detectability phase transitions in a large set of estimation problems ranging from community detection to sparse PCA.
Vector approximate message passing
TLDR
This paper considers a “vector AMP” (VAMP) algorithm and shows that VAMP has a rigorous scalar state-evolution that holds under a much broader class of large random matrices A: those that are right-rotationally invariant.
...
1
2
3
4
5
...