# Bayes-Newton Methods for Approximate Bayesian Inference with PSD Guarantees

@article{Wilkinson2021BayesNewtonMF, title={Bayes-Newton Methods for Approximate Bayesian Inference with PSD Guarantees}, author={William J. Wilkinson and Simo Sarkka and A. Solin}, journal={ArXiv}, year={2021}, volume={abs/2111.01721} }

We formulate natural gradient variational inference (VI), expectation propagation (EP), and posterior linearisation (PL) as generalisations of Newton’s method for optimising the parameters of a Bayesian posterior distribution. This viewpoint explicitly casts inference algorithms under the framework of numerical optimisation. We show that common approximations to Newton’s method from the optimisation literature, namely Gauss–Newton and quasi-Newton methods ( e.g. , the BFGS algorithm), are still…

## One Citation

### Markovian Gaussian Process Variational Autoencoders

- Computer ScienceArXiv
- 2022

This work leverage the equivalent discrete state space representation of Markovian GPs to enable a linear-time GP solver via Kalman ﬁltering and smoothing and shows via corrupt and missing frames tasks that this method performs favourably, especially on the latter where it outperforms RNN-based models.

### GPJax: A Gaussian Process Framework in JAX

- Computer ScienceJ. Open Source Softw.
- 2022

GPJax is a didactic GP library targeted at researchers who wish to develop novel GP methodology and provides users with a set of composable objects for constructing GP models that closely resemble the underlying maths that one would write on paper.

### A Look at Improving Robustness in Visual- inertial SLAM by Moment Matching

- Computer Science2022 25th International Conference on Information Fusion (FUSION)
- 2022

This paper revisits the assumed density formulation of Bayesian filtering and employs a moment matching (unscented Kalman filtering) approach to both visual-inertial odometry and visual SLAM, and shows state-of-the-art results on EuRoC MAV drone data benchmark.

### Short-term Prediction and Filtering of Solar Power Using State-Space Gaussian Processes

- Computer Science, Environmental Science
- 2023

This work considers Gaussian processes (GPs) for modelling and predicting solar photovoltaic energy production in the UK by using the state-space form of GPs, combined with modern variational inference techniques, and develops a scalable GP inference model that is not only scalable to large datasets but can also handle continuous data streams via Kalman ﬁltering.

## References

SHOWING 1-10 OF 63 REFERENCES

### Gaussian Kullback-Leibler approximate inference

- Computer ScienceJ. Mach. Learn. Res.
- 2013

Numerical results comparing G-KL and other deterministic Gaussian approximate inference methods are presented for: robust Gaussian process regression models with either Student-t or Laplace likelihoods, large scale Bayesian binary logistic regression models, and Bayesian sparse linear models for sequential experimental design.

### Sparse Algorithms for Markovian Gaussian Processes

- Computer ScienceAISTATS
- 2021

This work derives a general site-based approach to approximate inference, whereby the non-Gaussian likelihood is approximate with local Gaussian terms, called sites, and results in a suite of novel sparse extensions to algorithms from both the machine learning and signal processing literature, including variational inference, expectation propagation, and the classical nonlinear Kalman smoothers.

### State Space Expectation Propagation: Efficient Inference Schemes for Temporal Gaussian Processes

- Computer ScienceICML
- 2020

This work forms approximate Bayesian inference in non-conjugate temporal and spatio-temporal Gaussian process models as a simple parameter update rule applied during Kalman smoothing, providing extensive empirical analysis demonstrating the efficacy of various algorithms under this unifying framework.

### Stochastic Expectation Propagation

- Computer ScienceNIPS
- 2015

Stochastic expectation propagation is presented, called SEP, that maintains a global posterior approximation but updates it in a local way (like EP), and is ideally suited to performing approximate Bayesian learning in the large model, large dataset setting.

### A Unifying Framework for Gaussian Process Pseudo-Point Approximations using Power Expectation Propagation

- Computer ScienceJ. Mach. Learn. Res.
- 2017

This paper develops a new pseudo-point approximation framework using Power Expectation Propagation (Power EP) that unifies a large number of these pseudo- point approximations and demonstrates that the new framework includes new Pseudo- point approximation methods that outperform current approaches on regression and classification tasks.

### Online Model Selection Based on the Variational Bayes

- Computer ScienceNeural Computation
- 2001

By combining sequential model selection procedures, the online VB method provides a fully online learning method with a model selection mechanism and was able to adapt the model structure to dynamic environments.

### Expectation propagation in the large data limit

- Computer Science
- 2015

In this limit of infinite data, it is proved that the iterations of both averaged EP and EP are simple: they behave like iterations of Newton's algorithm for finding the mode of a function.

### Fast Variational Learning in State-Space Gaussian Process Models

- Computer Science2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP)
- 2020

This paper provides an efficient JAX implementation which exploits just-in-time compilation and allows for fast automatic differentiation through large for-loops and leads to fast and stable variational inference in state-space GP models that can be scaled to time series with millions of data points.

### The Bayesian Learning Rule

- Computer ScienceArXiv
- 2021

We show that many machine-learning algorithms are specific instances of a single algorithm called the Bayesian learning rule. The rule, derived from Bayesian principles, yields a wide-range of…

### Robust Gaussian Process Regression with a Student-t Likelihood

- Computer ScienceJ. Mach. Learn. Res.
- 2011

This paper illustrates the situations where standard EP fails to converge and review different modifications and alternative algorithms for improving the convergence and demonstrates that convergence problems may occur during the type-II maximum a posteriori (MAP) estimation of the hyperparameters.