# Streaming Bayesian inference: Theoretical limits and mini-batch approximate message-passing

@article{Manoel2017StreamingBI, title={Streaming Bayesian inference: Theoretical limits and mini-batch approximate message-passing}, author={Andre Manoel and Florent Krzakala and Eric W. Tramel and Lenka Zdeborov{\'a}}, journal={2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton)}, year={2017}, pages={1048-1055} }

In statistical learning for real-world large-scale data problems, one must often resort to “streaming” algorithms which operate sequentially on small batches of data. In this work, we present an analysis of the information-theoretic limits of mini-batch inference in the context of generalized linear models and low-rank matrix factorization. In a controlled Bayes-optimal setting, we characterize the optimal performance and phase transitions as a function of mini-batch size. We base part of our…

## 12 Citations

Streaming Belief Propagation for Community Detection

- Computer ScienceNeurIPS
- 2021

This work introduces a simple model for networks growing over time which it is referred to as streaming stochastic block model (StSBM) and proves that voting algorithms have fundamental limitations, and develops a streaming belief-propagation approach which proves optimality in certain regimes.

An Approximate Message Passing Framework for Side Information

- Computer ScienceIEEE Transactions on Signal Processing
- 2019

This paper develops a suite of algorithms, called AMP-SI, and derive denoisers for the BDD and BG models, and demonstrates the simplicity and applicability of this approach.

Rigorous State Evolution Analysis for Approximate Message Passing with Side Information

- Computer Science
- 2020

This work provides rigorous performance guarantees for AMP-SI when there are statistical dependencies between the signal and SI pairs and the entries of the measurement matrix are independent and identically distributed Gaussian.

Approximate message-passing for convex optimization with non-separable penalties

- Computer ScienceArXiv
- 2018

The connection between message-passing algorithms-typically used for approximate inference-and proximal methods for optimization, and show that the VAMP scheme is, as VAMP, similar in nature to the Peaceman-Rachford splitting, with the important difference that stepsizes are set adaptively.

An Analysis of State Evolution for Approximate Message Passing with Side Information

- Computer Science2019 IEEE International Symposium on Information Theory (ISIT)
- 2019

This work provides rigorous performance guarantees for AMP-SI when the input signal and SI are drawn i.i.d. according to some joint distribution subject to finite moment constraints.

Mean-field inference methods for neural networks

- Computer ScienceJournal of Physics A: Mathematical and Theoretical
- 2020

A selection of classical mean-field methods and recent progress relevant for inference in neural networks are reviewed, and the principles of derivations of high-temperature expansions, the replica method and message passing algorithms are reminded, highlighting their equivalences and complementarities.

L1-Minimization Algorithm for Bayesian Online Compressed Sensing

- Computer ScienceEntropy
- 2017

It is shown that reconstruction is possible even if prior knowledge about the generation of the signal is limited, by introduction of a Laplace prior and of an extra Kullback–Leibler divergence minimization step for hyper-parameter learning.

Learning capabilities of belief propagation based algorithms for sparse binary neural networks

- Computer Science
- 2020

The learning performance of belief propagation (BP) based algorithms, applied to simple two layers sparse neural networks with discrete synapses, is analysed and encourages results, allowing to perfectly reconstruct the weights of the ‘teacher’ network that generated the training data.

Proof of State Evolution for AMP with Side Information

- Computer Science
- 2019

A common goal in many research areas is to reconstruct an unknown signal x from noisy linear measurements. Approximate message passing (AMP) is a class of low-complexity algorithms for efficiently…

Decentralized Expectation Consistent Signal Recovery for Phase Retrieval

- Computer ScienceIEEE Transactions on Signal Processing
- 2020

A phase retrieval solution that aims to recover signals from noisy phaseless measurements by leveraging the core framework of GEC-SR, and proposes a novel decentralized algorithm called deGEC- SR, which exhibits excellent performance similar to G EC-SR but runs tens to hundreds of times faster.

## References

SHOWING 1-10 OF 46 REFERENCES

Streaming Variational Inference for Bayesian Nonparametric Mixture Models

- Computer ScienceAISTATS
- 2015

This work works within this general framework and presents a streaming variational inference algorithm for NRM mixture models based on assumed density filtering, leading straightforwardly to expectation propagation for large-scale batch inference as well.

Online Learning of Nonparametric Mixture Models via Sequential Variational Approximation

- Computer ScienceNIPS
- 2013

A Bayesian learning algorithm for DP mixture models that recursively transforms a given prior into an approximate posterior through sequential variational approximation, making it particularly suited for applications with massive data.

S-AMP: Approximate message passing for general matrix ensembles

- Computer Science2014 IEEE Information Theory Workshop (ITW 2014)
- 2014

This work proposes a novel iterative estimation algorithm for linear observation models called S-AMP, which extends the approximate message-passing algorithm to general matrix ensembles with a well-defined large system size limit.

Phase Transitions and Sample Complexity in Bayes-Optimal Matrix Factorization

- Computer ScienceIEEE Transactions on Information Theory
- 2016

This work compute the minimal mean-squared-error achievable, in principle, in any computational time, and the error that can be achieved by an efficient approximate message passing algorithm based on the asymptotic state-evolution analysis of the algorithm.

Streaming Variational Bayes

- Computer ScienceNIPS
- 2013

SDA-Bayes is presented, a framework for streaming updates to the estimated posterior of a Bayesian posterior, with variational Bayes (VB) as the primitive, and the usefulness of the framework is demonstrated by fitting the latent Dirichlet allocation model to two large-scale document collections.

Constrained Low-rank Matrix Estimation: Phase Transitions, Approximate Message Passing and Applications

- Computer ScienceArXiv
- 2017

The derivation of the TAP equations for models as different as the Sherrington-Kirkpatrick model, the restricted Boltzmann machine, the Hopfield model or vector (xy, Heisenberg and other) spin glasses are unify.

Generalized approximate message passing for estimation with random linear mixing

- Computer Science2011 IEEE International Symposium on Information Theory Proceedings
- 2011

G-AMP incorporates general measurement channels and shows that the asymptotic behavior of the G-AMP algorithm under large i.i.d. measurement channels is similar to the AWGN output channel case, and Gaussian transform matrices is described by a simple set of state evolution (SE) equations.

Statistical and computational phase transitions in spiked tensor estimation

- Computer Science2017 IEEE International Symposium on Information Theory (ISIT)
- 2017

The performance of Approximate Message Passing is studied and it is shown that it achieves the MMSE for a large set of parameters, and that factorization is algorithmically “easy” in a much wider region than previously believed.

Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula

- Computer ScienceNIPS
- 2016

It is shown how to rigorously prove the conjectured formula for the symmetric rank-one case, which allows to express the minimal mean-square-error and to characterize the detectability phase transitions in a large set of estimation problems ranging from community detection to sparse PCA.

Vector approximate message passing

- Computer Science2017 IEEE International Symposium on Information Theory (ISIT)
- 2017

This paper considers a “vector AMP” (VAMP) algorithm and shows that VAMP has a rigorous scalar state-evolution that holds under a much broader class of large random matrices A: those that are right-rotationally invariant.