• Corpus ID: 88520849

Asynchronous Gibbs Sampling

  title={Asynchronous Gibbs Sampling},
  author={Alexander Terenin and Daniel P. Simpson and David Draper},
Gibbs sampling is a Markov Chain Monte Carlo (MCMC) method often used in Bayesian learning. MCMC methods can be difficult to deploy on parallel and distributed systems due to their inherently sequential nature. We study asynchronous Gibbs sampling, which achieves parallelism by simply ignoring sequential requirements. This method has been shown to produce good empirical results for some hierarchical models, and is popular in the topic modeling community, but was also shown to diverge for other… 

Figures from this paper

Anytime Monte Carlo
An anytime framework is proposed to address the concern of length bias in sequential Monte Carlo algorithms, showing that for any MCMC algorithm, the length bias of the final state’s distribution due to the imposed real-time computing budget can be eliminated by using a multiple chain construction.
HOGWILD!-Gibbs can be PanAccurate
It is shown that the synchronous and asynchronous Gibbs samplers can be coupled so that the expected Hamming distance between their (multivariate) samples remains bounded by $O(\tau \log n),$ where $n$ is the number of variables in the graphical model, and $\tau$ is a measure of the asynchronicity.
Variance reduction for distributed stochastic gradient MCMC
This work derives variance bounds for distributed SGLD and introduces the concept of conducive gradients, zero-mean stochastic gradients that serve as a mechanism for sharing probabilistic information between workers, and shows both theoretically and empirically that it reduces variance, and hence improves convergence.
A survey of Monte Carlo methods for parameter estimation
A thorough review of MC methods for the estimation of static parameters in signal processing applications is performed, describing many of the most relevant MCMC and IS algorithms, and their combined use.
Variational Inference for Shrinkage Priors: The R package vir
We present vir, an R package for variational inference with shrinkage priors. Our package implements variational and stochastic variational algorithms for linear and probit regression models, the use
Communication-Efficient Distributed Statistical Inference
CSL provides a communication-efficient surrogate to the global likelihood that can be used for low-dimensional estimation, high-dimensional regularized estimation, and Bayesian inference and significantly improves the computational efficiency of Markov chain Monte Carlo algorithms even in a nondistributed setting.
Parallel Gibbs Sampler for Wavelet-Based Bayesian Compressive Sensing with High Reconstruction Accuracy
This work proposes a two-stage parallel coefficient update scheme for wavelet-based BCS, where the first stage approximates the real distributions of the wavelet coefficients and the second stage computes the final estimate of the coefficients.
Federated stochastic gradient Langevin dynamics
Stochastic gradient MCMC methods, such as stochastic gradient Langevin dynamics (SGLD), employ fast but noisy gradient estimates to enable large-scale posterior sampling. Although we can easily
Parallel Wavelet-based Bayesian Compressive Sensing based on Gibbs Sampling
A new coefficient update scheme that updates coefficients in both stages based on data generated a few rounds ago is proposed for wavelet-based Bayesian compressive sensing, where the first stage approximates the real distributions of the wavelet coefficients and the second stage computes the final estimate of the coefficients.
Exploring Probability Measures with Markov Processes
A transparent characterisation of how one can construct a PDMP (within the class of trajectorially-reversible processes) which admits the desired invariant measure is developed, and actionable recommendations on how these processes should be designed in practice are offered.


Techniques for proving Asynchronous Convergence results for Markov Chain Monte Carlo methods
This article describes how to apply theory on Asynchronous Gibbs sampling in a generic setting, to understand the asynchronous behavior of any MCMC algorithm, including those implemented using parameter servers, and those not based on Gibbs sampling.
Analyzing Hogwild Parallel Gaussian Gibbs Sampling
It is shown that if the Gaussian precision matrix is generalized diagonally dominant, then any Hogwild Gibbs sampler, with any update schedule or allocation of variables to processors, yields a stable sampling process with the correct sample mean.
Asymptotically Exact, Embarrassingly Parallel MCMC
This paper presents a parallel Markov chain Monte Carlo (MCMC) algorithm in which subsets of data are processed independently, with very little communication, and proves that it generates asymptotically exact samples and empirically demonstrate its ability to parallelize burn-in and sampling in several models.
Slice Sampling
Markov chain sampling methods that adapt to characteristics of the distribution being sampled can be constructed using the principle that one can ample from a distribution by sampling uniformly from
Parallel Gibbs Sampling: From Colored Fields to Thin Junction Trees
This work proposes two methods to construct parallel Gibbs samplers guaranteed to draw from the targeted distribution, called the Chromatic sampler and the Splash sampler, a complementary strategy which can be used when the variables are tightly coupled.
Distributed Algorithms for Topic Models
This work describes distributed algorithms for two widely-used topic models, namely the Latent Dirichlet Allocation (LDA) model and the Hierarchical Dirichet Process (HDP) model, and proposes a model that uses a hierarchical Bayesian extension of LDA to directly account for distributed data.
Understanding Errors in Approximate Distributed Latent Dirichlet Allocation
  • A. IhlerD. Newman
  • Computer Science
    IEEE Transactions on Knowledge and Data Engineering
  • 2012
It is shown empirically that the bound is sufficiently tight to give a meaningful and intuitive measure of approximation error in AD-LDA, allowing the user to track the tradeoff between accuracy and efficiency while executing in parallel.
Sampling-Based Approaches to Calculating Marginal Densities
Abstract Stochastic substitution, the Gibbs sampler, and the sampling-importance-resampling algorithm can be viewed as three alternative sampling- (or Monte Carlo-) based approaches to the
Bayes and big data: the consensus Monte Carlo algorithm
A useful definition of ‘big data’ is data that is too big to process comfortably on a single machine, either because of processor, memory, or disk bottlenecks, so there is a need for algorithms that perform distributed approximate Bayesian analyses with minimal communication.
Exponential convergence of Langevin distributions and their discrete approximations
In this paper we consider a continuous-time method of approximating a given distribution using the Langevin di€usion dLtˆdWt‡ 1 2 r log (Lt)dt. We ®nd conditions under this di€usion converges