• Corpus ID: 240420140

Bayes-Newton Methods for Approximate Bayesian Inference with PSD Guarantees

@article{Wilkinson2021BayesNewtonMF,
  title={Bayes-Newton Methods for Approximate Bayesian Inference with PSD Guarantees},
  author={William J. Wilkinson and Simo Sarkka and A. Solin},
  journal={ArXiv},
  year={2021},
  volume={abs/2111.01721}
}
We formulate natural gradient variational inference (VI), expectation propagation (EP), and posterior linearisation (PL) as generalisations of Newton’s method for optimising the parameters of a Bayesian posterior distribution. This viewpoint explicitly casts inference algorithms under the framework of numerical optimisation. We show that common approximations to Newton’s method from the optimisation literature, namely Gauss–Newton and quasi-Newton methods ( e.g. , the BFGS algorithm), are still… 
[PDF] Semantic Reader
1 Citations
Background Citations
1
Methods Citations
1

Figures and Tables from this paper

One Citation

Markovian Gaussian Process Variational Autoencoders

This work leverage the equivalent discrete state space representation of Markovian GPs to enable a linear-time GP solver via Kalman filtering and smoothing and shows via corrupt and missing frames tasks that this method performs favourably, especially on the latter where it outperforms RNN-based models.

GPJax: A Gaussian Process Framework in JAX

GPJax is a didactic GP library targeted at researchers who wish to develop novel GP methodology and provides users with a set of composable objects for constructing GP models that closely resemble the underlying maths that one would write on paper.

A Look at Improving Robustness in Visual- inertial SLAM by Moment Matching

This paper revisits the assumed density formulation of Bayesian filtering and employs a moment matching (unscented Kalman filtering) approach to both visual-inertial odometry and visual SLAM, and shows state-of-the-art results on EuRoC MAV drone data benchmark.

Short-term Prediction and Filtering of Solar Power Using State-Space Gaussian Processes

This work considers Gaussian processes (GPs) for modelling and predicting solar photovoltaic energy production in the UK by using the state-space form of GPs, combined with modern variational inference techniques, and develops a scalable GP inference model that is not only scalable to large datasets but can also handle continuous data streams via Kalman filtering.

References

SHOWING 1-10 OF 63 REFERENCES

Gaussian Kullback-Leibler approximate inference

Numerical results comparing G-KL and other deterministic Gaussian approximate inference methods are presented for: robust Gaussian process regression models with either Student-t or Laplace likelihoods, large scale Bayesian binary logistic regression models, and Bayesian sparse linear models for sequential experimental design.

Sparse Algorithms for Markovian Gaussian Processes

This work derives a general site-based approach to approximate inference, whereby the non-Gaussian likelihood is approximate with local Gaussian terms, called sites, and results in a suite of novel sparse extensions to algorithms from both the machine learning and signal processing literature, including variational inference, expectation propagation, and the classical nonlinear Kalman smoothers.

State Space Expectation Propagation: Efficient Inference Schemes for Temporal Gaussian Processes

This work forms approximate Bayesian inference in non-conjugate temporal and spatio-temporal Gaussian process models as a simple parameter update rule applied during Kalman smoothing, providing extensive empirical analysis demonstrating the efficacy of various algorithms under this unifying framework.

Stochastic Expectation Propagation

Stochastic expectation propagation is presented, called SEP, that maintains a global posterior approximation but updates it in a local way (like EP), and is ideally suited to performing approximate Bayesian learning in the large model, large dataset setting.

A Unifying Framework for Gaussian Process Pseudo-Point Approximations using Power Expectation Propagation

This paper develops a new pseudo-point approximation framework using Power Expectation Propagation (Power EP) that unifies a large number of these pseudo- point approximations and demonstrates that the new framework includes new Pseudo- point approximation methods that outperform current approaches on regression and classification tasks.

Online Model Selection Based on the Variational Bayes

By combining sequential model selection procedures, the online VB method provides a fully online learning method with a model selection mechanism and was able to adapt the model structure to dynamic environments.

Expectation propagation in the large data limit

In this limit of infinite data, it is proved that the iterations of both averaged EP and EP are simple: they behave like iterations of Newton's algorithm for finding the mode of a function.

Fast Variational Learning in State-Space Gaussian Process Models

This paper provides an efficient JAX implementation which exploits just-in-time compilation and allows for fast automatic differentiation through large for-loops and leads to fast and stable variational inference in state-space GP models that can be scaled to time series with millions of data points.

The Bayesian Learning Rule

We show that many machine-learning algorithms are specific instances of a single algorithm called the Bayesian learning rule. The rule, derived from Bayesian principles, yields a wide-range of…

Robust Gaussian Process Regression with a Student-t Likelihood

This paper illustrates the situations where standard EP fails to converge and review different modifications and alternative algorithms for improving the convergence and demonstrates that convergence problems may occur during the type-II maximum a posteriori (MAP) estimation of the hyperparameters.
...