• Corpus ID: 17181633

MCMC for Variationally Sparse Gaussian Processes

@inproceedings{Hensman2015MCMCFV,
  title={MCMC for Variationally Sparse Gaussian Processes},
  author={James Hensman and Alexander G. de G. Matthews and Maurizio Filippone and Zoubin Ghahramani},
  booktitle={NIPS},
  year={2015}
}
Gaussian process (GP) models form a core part of probabilistic machine learning. Considerable research effort has been made into attacking three issues with GP models: how to compute efficiently when the number of data is large; how to approximate the posterior when the likelihood is not Gaussian and how to estimate covariance function parameter posteriors. This paper simultaneously addresses these, using a variational approximation to the posterior which is sparse in support of the function… 

Figures and Tables from this paper

On MCMC for variationally sparse Gaussian processes: A pseudo-marginal approach
TLDR
A pseudomarginal (PM) scheme is proposed that offers asymptotically exact inference as well as computational gains through doubly stochastic estimators for the intractable likelihood and large datasets.
A Tutorial on Sparse Gaussian Processes and Variational Inference
TLDR
This tutorial is to provide access to the basic matter for readers without prior knowledge in both GPs and VI, where pseudo-training examples are treated as optimization arguments of the approximate posterior that are jointly identified together with hyperparameters of the generative model.
Stein Variational Gaussian Processes
TLDR
SVGD provides a non-parametric alternative to variational inference which is substantially faster than MCMC but unhindered by parametric assumptions, and it is proved that for GP models with Lipschitz gradients the SVGD algorithm monotonically decreases the Kullback-Leibler divergence from the sampling distribution to the true posterior.
Rethinking Sparse Gaussian Processes: Bayesian Approaches to Inducing-Variable Approximations
TLDR
This work develops a fully Bayesian approach to scalable GP and deep GP models, and shows that treating both inducing locations and GP hyper-parameters in a Bayesian way, by inferring their full posterior, further significantly improves performance.
Sparse Orthogonal Variational Inference for Gaussian Processes
TLDR
A new interpretation of sparse variational approximations for Gaussian processes using inducing points which can lead to more scalable algorithms than previous methods and report state-of-the-art results on CIFAR-10 with purely GP-based models.
Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo
TLDR
This work provides evidence for the non-Gaussian nature of the posterior and applies the Stochastic Gradient Hamiltonian Monte Carlo method to generate samples, which results in significantly better predictions at a lower computational cost than its VI counterpart.
Sparse Orthogonal Variational Inference for Gaussian Processes
TLDR
A new interpretation of sparse variational approximations for Gaussian processes using inducing points is introduced, which can lead to more scalable algorithms than previous methods and report state-of-the-art results on CIFAR-10 among purely GP-based models.
Sparse Gaussian Processes Revisited: Bayesian Approaches to Inducing-Variable Approximations
TLDR
This work shows that, by revisiting old model approximations such as the fully-independent training conditionals endowed with powerful sampling-based inference methods, treating both inducing locations and GP hyper-parameters in a Bayesian way can improve performance significantly.
Online Gaussian Process regression with non-Gaussian likelihood
TLDR
A new algorithm for GP regression over data with non-Gaussian likelihood that does not require costly MCMC sampling, or variational Bayes optimization, and can work with streaming data, is analytically tractable, computationally efficient, and as accurate or better than Expectation Propagation, Laplace Approximation, and MCMC inference methods for non- Gaussian likelihood data.
Variational Fourier Features for Gaussian Processes
TLDR
This work hinges on a key result that there exist spectral features related to a finite domain of the Gaussian process which exhibit almost-independent covariances, and derives these expressions for Matern kernels in one dimension, and generalize to more dimensions using kernels with specific structures.
...
...

References

SHOWING 1-10 OF 42 REFERENCES
Automated Variational Inference for Gaussian Process Models
TLDR
An automated variational method for approximate inference in Gaussian process (GP) models whose posteriors are often intractable is developed, performing as well as the exact or hard-coded implementations while running orders of magnitude faster than the alternative MCMC sampling approaches.
Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models
TLDR
A novel re-parametrisation of variational inference for sparse GP regression and latent variable models that allows for an efficient distributed algorithm and shows that GPs perform better than many common models often used for big data.
Sparse On-Line Gaussian Processes
TLDR
An approach for sparse representations of gaussian process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations for large data sets is developed based on a combination of a Bayesian on-line algorithm and a sequential construction of a relevant subsample of data that fully specifies the prediction of the GP model.
Variational Bayesian Multinomial Probit Regression with Gaussian Process Priors
TLDR
This is the first time that a fully variational Bayesian treatment for multiclass GP classification has been developed without having to resort to additional explicit approximations to the nongaussian likelihood term.
Sparse Gaussian Processes using Pseudo-inputs
TLDR
It is shown that this new Gaussian process (GP) regression model can match full GP performance with small M, i.e. very sparse solutions, and it significantly outperforms other approaches in this regime.
Fast Sparse Gaussian Process Methods: The Informative Vector Machine
TLDR
A framework for sparse Gaussian process (GP) methods which uses forward selection with criteria based on information-theoretic principles, which allows for Bayesian model selection and is less complex in implementation is presented.
Variational Multinomial Logit Gaussian Process
TLDR
A variational approximation to the Gaussian process multi-class model is proposed, and criteria which can be used to select the inducing set are derived and the effectiveness of these criteria over random selection in an experiment is shown.
Sparse Spectrum Gaussian Process Regression
TLDR
The achievable trade-offs between predictive accuracy and computational requirements are compared, and it is shown that these are typically superior to existing state-of-the-art sparse approximations.
Slice sampling covariance hyperparameters of latent Gaussian models
TLDR
This paper presents a slice sampling approach that requires little tuning while mixing well in both strong- and weak-data regimes and shows good results with both strong and weak data regimes.
Fast Bayesian Inference for Non-Conjugate Gaussian Process Regression
We present a new variational inference algorithm for Gaussian process regression with non-conjugate likelihood functions, with application to a wide array of problems including binary and multi-class
...
...