• Corpus ID: 49397102

Tensor Monte Carlo: particle methods for the GPU era

  title={Tensor Monte Carlo: particle methods for the GPU era},
  author={Laurence Aitchison},
  • L. Aitchison
  • Published in NeurIPS 22 June 2018
  • Computer Science
Multi-sample, importance-weighted variational autoencoders (IWAE) give tighter bounds and more accurate uncertainty estimates than variational autoencoders (VAE) trained with a standard single-sample objective. However, IWAEs scale poorly: as the latent dimensionality grows, they require exponentially many samples to retain the benefits of importance weighting. While sequential Monte-Carlo (SMC) can address this problem, it is prohibitively slow because the resampling step imposes sequential… 

Figures from this paper

[Re] Tensor Monte Carlo: Particle Methods for the GPU Era

This work reproduces what they believe are the most important results presented in the Tensor Monte Carlo paper, and supplements the original paper by visualizing the TMC s̓ reconstruction and clustering capabilities, while contrasting them to the capabilities of the baseline, IWAE.

On importance-weighted autoencoders

It is argued that directly optimising the proposal distribution in importance sampling as in the reweighted wake-sleep (RWS) algorithm is preferable to optimising IWAE-type multi-sample objectives and an adaptive-importance sampling framework termed adaptive importance sampling for learning (AISLE) is introduced which slightly generalises the RWS algorithm.

Multiple Importance Sampling ELBO and Deep Ensembles of Variational Approximations

This work proposes the multiple importance sampling ELBO (MISELBO), a versatile yet simple framework that allows to unveil connections between VI and recent advances in the importance sampling literature, paving the way for further methodological advances.

De-Sequentialized Monte Carlo: a parallel-in-time particle smoother

This work proposes dSMC (de-Sequentialized Monte Carlo), a new particle smoother that is able to process T observations in O (log T ) time on parallel architecture, which compares favourably with standard particle smoothers, the complexity of which is linear in T.

Data augmentation in Bayesian neural networks and the cold posterior effect

It is found that thecold posterior effect persists even in these principled models incorporating data augmentation, suggesting that the cold posterior effect cannot be dismissed as an artifact of data augmented using incorrect likelihoods.

Differentiable Particle Filtering via Entropy-Regularized Optimal Transport

By leveraging optimal transport ideas, this work introduces a principled differentiable particle filter and provides convergence results and demonstrates this novel method on a variety of applications.

The divide-and-conquer sequential Monte Carlo algorithm: theoretical properties and limit theorems

We revisit the divide-and-conquer sequential Monte Carlo (DaC-SMC) algorithm and firmly establish it as a well-founded method by showing that it possesses the same basic properties as conventional

Product-form estimators: exploiting independence to scale up Monte Carlo

We introduce a class of Monte Carlo estimators that aim to overcome the rapid growth of variance with dimension often observed for standard estimators by exploiting the target’s independence

Variational Marginal Particle Filters

The variational marginal particle filter (VMPF), which is a differentiable and reparameterizable variational filtering objective for SSMs based on an unbiased estimator, is proposed and it is found that VMPF with biased gradients gives tighter bounds than previous objectives, and the unbiasedReparameterization gradients are sometimes beneficial.



Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives

A computationally efficient, unbiased drop-in gradient estimator that reduces the variance of the IWAE gradient, the reweighted wake-sleep update (RWS), and the jackknife variational inference (JVI) gradient (Nowozin, 2018).

Variational Inference for Monte Carlo Objectives

The first unbiased gradient estimator designed for importance-sampled objectives is developed, which is both simpler and more effective than the NVIL estimator proposed for the single-sample variational objective, and is competitive with the currently used biases.

Importance Weighted Autoencoders

The importance weighted autoencoder (IWAE), a generative model with the same architecture as the VAE, but which uses a strictly tighter log-likelihood lower bound derived from importance weighting, shows empirically that IWAEs learn richer latent space representations than VAEs, leading to improved test log- likelihood on density estimation benchmarks.

Auto-Encoding Variational Bayes

A stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case is introduced.

Ladder Variational Autoencoders

A new inference model is proposed, the Ladder Variational Autoencoder, that recursively corrects the generative distribution by a data dependent approximate likelihood in a process resembling the recently proposed Ladder Network.

Adam: A Method for Stochastic Optimization

This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.

Tighter Variational Bounds are Not Necessarily Better

We provide theoretical and empirical evidence that using tighter evidence lower bounds (ELBOs) can be detrimental to the process of learning an inference network by reducing the signal-to-noise ratio

Black Box Variational Inference

This paper presents a "black box" variational inference algorithm, one that can be quickly applied to many models with little additional derivation, based on a stochastic optimization of the variational objective where the noisy gradient is computed from Monte Carlo samples from the Variational distribution.

Variational Sequential Monte Carlo

The VSMC family is a variational family that can approximate the posterior arbitrarily well, while still allowing for efficient optimization of its parameters, and is demonstrated its utility on state space models, stochastic volatility models for financial data, and deep Markov models of brain neural circuits.

Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

A reparameterization of the weight vectors in a neural network that decouples the length of those weight vectors from their direction is presented, improving the conditioning of the optimization problem and speeding up convergence of stochastic gradient descent.