# Distributed Gaussian Processes

@inproceedings{Deisenroth2015DistributedGP, title={Distributed Gaussian Processes}, author={Marc Peter Deisenroth and Jun Wei Ng}, booktitle={ICML}, year={2015} }

To scale Gaussian processes (GPs) to large data sets we introduce the robust Bayesian Committee Machine (rBCM), a practical and scalable product-of-experts model for large-scale distributed GP regression. Unlike state-of-theart sparse GP approximations, the rBCM is conceptually simple and does not rely on inducing or variational parameters. The key idea is to recursively distribute computations to independent computational units and, subsequently, recombine them to form an overall result…

## 228 Citations

Generalized Robust Bayesian Committee Machine for Large-scale Gaussian Process Regression

- Computer ScienceICML
- 2018

This work first proves the inconsistency of typical aggregations using disjoint or random data partition, and then presents a consistent yet efficient aggregation model for large-scale GP.

Exact gaussian process regression with distributed computations

- Computer ScienceSAC
- 2019

The design and evaluation of a distributed method for exact GP inference is presented, that achieves true model parallelism using simple, high-level distributed computing frameworks and shows that exact inference at scale is not only feasible, but it also brings substantial benefits in terms of low error rates and accurate quantification of uncertainty.

Asynchronous Distributed Variational Gaussian Process for Regression

- Computer ScienceICML
- 2017

ADVGP is proposed, the first Asynchronous Distributed Variational Gaussian Process inference for regression, on the recent large-scale machine learning platform, PARAMETERSERVER, and greatly improves upon the efficiency of the existing variational methods.

Exact Gaussian Processes on a Million Data Points

- Computer ScienceNeurIPS
- 2019

A scalable approach for exact GPs is developed that leverages multi-GPU parallelization and methods like linear conjugate gradients, accessing the kernel matrix only through matrix multiplication, and is generally applicable, without constraints to grid data or specific kernel classes.

Deep Structured Mixtures of Gaussian Processes

- Computer ScienceAISTATS
- 2020

Deep structured mixtures of GP experts are introduced, a stochastic process model which allows exact posterior inference, has attractive computational and memory costs, and when used as GP approximation, captures predictive uncertainties consistently better than previous expert-based approximations.

Scalable Gaussian Process Using Inexact Admm for Big Data

- Computer ScienceICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- 2019

A novel scalable GP regression model for processing big datasets, using a large number of parallel computation units is proposed, which solves the classic maximum likelihood based hyper-parameter optimization problem by a carefully designed distributed alternating direction method of multipliers (ADMM).

Distributed Gaussian Processes Hyperparameter Optimization for Big Data Using Proximal ADMM

- Computer ScienceIEEE Signal Processing Letters
- 2019

This letter proposes an alternative distributed GP hyperparameter optimization scheme using the efficient proximal alternating direction method of multipliers, proposed by Hong et al. in 2016, and derives the closed-form solution for the local sub-problems.

Optimal recovery and uncertainty quantification for distributed Gaussian process regression

- Computer Science
- 2022

This work derives frequentist theoretical guarantees and limitations for a range of distributed methods for general GP priors in context of the nonparametric regression model, both for recovery and uncertainty quantiﬁcation.

Composite Gaussian Processes: Scalable Computation and Performance Analysis

- Computer Science
- 2018

This work derives an approximation based on a composite likelihood approach using a general belief updating framework, which leads to a recursive computation of the predictor as well as of learning the hyper-parameters.

Thoughts on Massively Scalable Gaussian Processes

- Computer ScienceArXiv
- 2015

The MSGP framework enables the use of Gaussian processes on billions of datapoints, without requiring distributed inference, or severe assumptions, and reduces the standard GP learning and inference complexity to O(n), and the standard test point prediction complexity to $O(1).

## References

SHOWING 1-10 OF 38 REFERENCES

Hierarchical Mixture-of-Experts Model for Large-Scale Gaussian Process Regression

- Computer ScienceArXiv
- 2014

This work proposes a practical and scalable Gaussian process model for large-scale nonlinear probabilistic regression that can handle arbitrarily large data sets, without explicit sparse approximations.

Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models

- Computer ScienceNIPS
- 2014

A novel re-parametrisation of variational inference for sparse GP regression and latent variable models that allows for an efficient distributed algorithm and shows that GPs perform better than many common models often used for big data.

Fast Allocation of Gaussian Process Experts

- Computer ScienceICML
- 2014

A scalable nonparametric Bayesian regression model based on a mixture of Gaussian process experts and the inducing points formalism underpinning sparse GP approximations that significantly outperforms six competitive baselines while requiring only a few hours of training.

Sparse Gaussian Processes using Pseudo-inputs

- Computer ScienceNIPS
- 2005

It is shown that this new Gaussian process (GP) regression model can match full GP performance with small M, i.e. very sparse solutions, and it significantly outperforms other approaches in this regime.

A framework for evaluating approximation methods for Gaussian process regression

- Computer ScienceJ. Mach. Learn. Res.
- 2013

Four different approximation algorithms are empirically investigated on four different prediction problems, and the quality of the predictions obtained as a function of the compute time taken are assessed.

Efficient Gaussian Process Inference for Short-Scale Spatio-Temporal Modeling

- Computer ScienceAISTATS
- 2012

The proposed Gaussian process inference scheme is compared to the standard approach using the sparse Cholesky decomposition and it is shown to be much faster and computationally feasible for 100–1000 times larger datasets.

Fast Forward Selection to Speed Up Sparse Gaussian Process Regression

- Computer ScienceAISTATS
- 2003

A method for the sparse greedy approximation of Bayesian Gaussian process regression, featuring a novel heuristic for very fast forward selection, which leads to a sufficiently stable approximation of the log marginal likelihood of the training data, which can be optimised to adjust a large number of hyperparameters automatically.

Sparse Spectrum Gaussian Process Regression

- Computer ScienceJ. Mach. Learn. Res.
- 2010

The achievable trade-offs between predictive accuracy and computational requirements are compared, and it is shown that these are typically superior to existing state-of-the-art sparse approximations.

Infinite Mixtures of Gaussian Process Experts

- Computer ScienceNIPS
- 2001

We present an extension to the Mixture of Experts (ME) model, where the individual experts are Gaussian Process (GP) regression models. Using an input-dependent adaptation of the Dirichlet Process,…

Variational Mixture of Gaussian Process Experts

- Computer ScienceNIPS
- 2008

A new generative mixture of experts model is presented, where each expert is still a Gaussian process but is reformulated by a linear model, which breaks the dependency among training outputs and enables us to use a much faster variational Bayesian algorithm for training.