# Asynchronous Gibbs Sampling

@inproceedings{Terenin2020AsynchronousGS, title={Asynchronous Gibbs Sampling}, author={Alexander Terenin and Daniel P. Simpson and David Draper}, booktitle={AISTATS}, year={2020} }

Gibbs sampling is a Markov Chain Monte Carlo (MCMC) method often used in Bayesian learning. MCMC methods can be difficult to deploy on parallel and distributed systems due to their inherently sequential nature. We study asynchronous Gibbs sampling, which achieves parallelism by simply ignoring sequential requirements. This method has been shown to produce good empirical results for some hierarchical models, and is popular in the topic modeling community, but was also shown to diverge for other…

## 31 Citations

Anytime Monte Carlo

- Computer ScienceData-Centric Engineering
- 2021

An anytime framework is proposed to address the concern of length bias in sequential Monte Carlo algorithms, showing that for any MCMC algorithm, the length bias of the final state’s distribution due to the imposed real-time computing budget can be eliminated by using a multiple chain construction.

HOGWILD!-Gibbs can be PanAccurate

- Mathematics, Computer ScienceNeurIPS
- 2018

It is shown that the synchronous and asynchronous Gibbs samplers can be coupled so that the expected Hamming distance between their (multivariate) samples remains bounded by $O(\tau \log n),$ where $n$ is the number of variables in the graphical model, and $\tau$ is a measure of the asynchronicity.

Variance reduction for distributed stochastic gradient MCMC

- Computer ScienceArXiv
- 2020

This work derives variance bounds for distributed SGLD and introduces the concept of conducive gradients, zero-mean stochastic gradients that serve as a mechanism for sharing probabilistic information between workers, and shows both theoretically and empirically that it reduces variance, and hence improves convergence.

A survey of Monte Carlo methods for parameter estimation

- Computer ScienceEURASIP J. Adv. Signal Process.
- 2020

A thorough review of MC methods for the estimation of static parameters in signal processing applications is performed, describing many of the most relevant MCMC and IS algorithms, and their combined use.

Variational Inference for Shrinkage Priors: The R package vir

- Computer Science
- 2021

We present vir, an R package for variational inference with shrinkage priors. Our package implements variational and stochastic variational algorithms for linear and probit regression models, the use…

Communication-Efficient Distributed Statistical Inference

- Computer ScienceJournal of the American Statistical Association
- 2018

CSL provides a communication-efficient surrogate to the global likelihood that can be used for low-dimensional estimation, high-dimensional regularized estimation, and Bayesian inference and significantly improves the computational efficiency of Markov chain Monte Carlo algorithms even in a nondistributed setting.

Parallel Gibbs Sampler for Wavelet-Based Bayesian Compressive Sensing with High Reconstruction Accuracy

- Computer ScienceJ. Signal Process. Syst.
- 2020

This work proposes a two-stage parallel coefficient update scheme for wavelet-based BCS, where the first stage approximates the real distributions of the wavelet coefficients and the second stage computes the final estimate of the coefficients.

Federated stochastic gradient Langevin dynamics

- Computer ScienceUAI
- 2021

Stochastic gradient MCMC methods, such as stochastic gradient Langevin dynamics (SGLD), employ fast but noisy gradient estimates to enable large-scale posterior sampling. Although we can easily…

Parallel Wavelet-based Bayesian Compressive Sensing based on Gibbs Sampling

- Computer Science2018 IEEE International Workshop on Signal Processing Systems (SiPS)
- 2018

A new coefficient update scheme that updates coefficients in both stages based on data generated a few rounds ago is proposed for wavelet-based Bayesian compressive sensing, where the first stage approximates the real distributions of the wavelet coefficients and the second stage computes the final estimate of the coefficients.

Exploring Probability Measures with Markov Processes

- Computer Science, Mathematics
- 2020

A transparent characterisation of how one can construct a PDMP (within the class of trajectorially-reversible processes) which admits the desired invariant measure is developed, and actionable recommendations on how these processes should be designed in practice are offered.

## References

SHOWING 1-10 OF 56 REFERENCES

Techniques for proving Asynchronous Convergence results for Markov Chain Monte Carlo methods

- Computer ScienceArXiv
- 2017

This article describes how to apply theory on Asynchronous Gibbs sampling in a generic setting, to understand the asynchronous behavior of any MCMC algorithm, including those implemented using parameter servers, and those not based on Gibbs sampling.

Analyzing Hogwild Parallel Gaussian Gibbs Sampling

- Computer ScienceNIPS
- 2013

It is shown that if the Gaussian precision matrix is generalized diagonally dominant, then any Hogwild Gibbs sampler, with any update schedule or allocation of variables to processors, yields a stable sampling process with the correct sample mean.

Asymptotically Exact, Embarrassingly Parallel MCMC

- Computer ScienceUAI
- 2014

This paper presents a parallel Markov chain Monte Carlo (MCMC) algorithm in which subsets of data are processed independently, with very little communication, and proves that it generates asymptotically exact samples and empirically demonstrate its ability to parallelize burn-in and sampling in several models.

Slice Sampling

- MathematicsThe Annals of Statistics
- 2003

Markov chain sampling methods that adapt to characteristics of the distribution being sampled can be constructed using the principle that one can ample from a distribution by sampling uniformly from…

Parallel Gibbs Sampling: From Colored Fields to Thin Junction Trees

- Computer ScienceAISTATS
- 2011

This work proposes two methods to construct parallel Gibbs samplers guaranteed to draw from the targeted distribution, called the Chromatic sampler and the Splash sampler, a complementary strategy which can be used when the variables are tightly coupled.

Distributed Algorithms for Topic Models

- Computer ScienceJ. Mach. Learn. Res.
- 2009

This work describes distributed algorithms for two widely-used topic models, namely the Latent Dirichlet Allocation (LDA) model and the Hierarchical Dirichet Process (HDP) model, and proposes a model that uses a hierarchical Bayesian extension of LDA to directly account for distributed data.

Understanding Errors in Approximate Distributed Latent Dirichlet Allocation

- Computer ScienceIEEE Transactions on Knowledge and Data Engineering
- 2012

It is shown empirically that the bound is sufficiently tight to give a meaningful and intuitive measure of approximation error in AD-LDA, allowing the user to track the tradeoff between accuracy and efficiency while executing in parallel.

Sampling-Based Approaches to Calculating Marginal Densities

- Computer Science
- 1990

Abstract Stochastic substitution, the Gibbs sampler, and the sampling-importance-resampling algorithm can be viewed as three alternative sampling- (or Monte Carlo-) based approaches to the…

Bayes and big data: the consensus Monte Carlo algorithm

- Computer Science
- 2016

A useful definition of ‘big data’ is data that is too big to process comfortably on a single machine, either because of processor, memory, or disk bottlenecks, so there is a need for algorithms that perform distributed approximate Bayesian analyses with minimal communication.

Exponential convergence of Langevin distributions and their discrete approximations

- Mathematics
- 1996

In this paper we consider a continuous-time method of approximating a given distribution using the Langevin diusion dLtdWt 1 2 r log (Lt)dt. We ®nd conditions under this diusion converges…