• Corpus ID: 17686175

Distributed Gaussian Processes

@inproceedings{Deisenroth2015DistributedGP,
  title={Distributed Gaussian Processes},
  author={Marc Peter Deisenroth and Jun Wei Ng},
  booktitle={ICML},
  year={2015}
}
To scale Gaussian processes (GPs) to large data sets we introduce the robust Bayesian Committee Machine (rBCM), a practical and scalable product-of-experts model for large-scale distributed GP regression. Unlike state-of-theart sparse GP approximations, the rBCM is conceptually simple and does not rely on inducing or variational parameters. The key idea is to recursively distribute computations to independent computational units and, subsequently, recombine them to form an overall result… 

Figures and Tables from this paper

Generalized Robust Bayesian Committee Machine for Large-scale Gaussian Process Regression
TLDR
This work first proves the inconsistency of typical aggregations using disjoint or random data partition, and then presents a consistent yet efficient aggregation model for large-scale GP.
Exact gaussian process regression with distributed computations
TLDR
The design and evaluation of a distributed method for exact GP inference is presented, that achieves true model parallelism using simple, high-level distributed computing frameworks and shows that exact inference at scale is not only feasible, but it also brings substantial benefits in terms of low error rates and accurate quantification of uncertainty.
Asynchronous Distributed Variational Gaussian Process for Regression
TLDR
ADVGP is proposed, the first Asynchronous Distributed Variational Gaussian Process inference for regression, on the recent large-scale machine learning platform, PARAMETERSERVER, and greatly improves upon the efficiency of the existing variational methods.
Exact Gaussian Processes on a Million Data Points
TLDR
A scalable approach for exact GPs is developed that leverages multi-GPU parallelization and methods like linear conjugate gradients, accessing the kernel matrix only through matrix multiplication, and is generally applicable, without constraints to grid data or specific kernel classes.
Deep Structured Mixtures of Gaussian Processes
TLDR
Deep structured mixtures of GP experts are introduced, a stochastic process model which allows exact posterior inference, has attractive computational and memory costs, and when used as GP approximation, captures predictive uncertainties consistently better than previous expert-based approximations.
Scalable Gaussian Process Using Inexact Admm for Big Data
TLDR
A novel scalable GP regression model for processing big datasets, using a large number of parallel computation units is proposed, which solves the classic maximum likelihood based hyper-parameter optimization problem by a carefully designed distributed alternating direction method of multipliers (ADMM).
Distributed Gaussian Processes Hyperparameter Optimization for Big Data Using Proximal ADMM
TLDR
This letter proposes an alternative distributed GP hyperparameter optimization scheme using the efficient proximal alternating direction method of multipliers, proposed by Hong et al. in 2016, and derives the closed-form solution for the local sub-problems.
Optimal recovery and uncertainty quantification for distributed Gaussian process regression
TLDR
This work derives frequentist theoretical guarantees and limitations for a range of distributed methods for general GP priors in context of the nonparametric regression model, both for recovery and uncertainty quantification.
Composite Gaussian Processes: Scalable Computation and Performance Analysis
TLDR
This work derives an approximation based on a composite likelihood approach using a general belief updating framework, which leads to a recursive computation of the predictor as well as of learning the hyper-parameters.
Thoughts on Massively Scalable Gaussian Processes
TLDR
The MSGP framework enables the use of Gaussian processes on billions of datapoints, without requiring distributed inference, or severe assumptions, and reduces the standard GP learning and inference complexity to O(n), and the standard test point prediction complexity to $O(1).
...
...

References

SHOWING 1-10 OF 38 REFERENCES
Hierarchical Mixture-of-Experts Model for Large-Scale Gaussian Process Regression
TLDR
This work proposes a practical and scalable Gaussian process model for large-scale nonlinear probabilistic regression that can handle arbitrarily large data sets, without explicit sparse approximations.
Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models
TLDR
A novel re-parametrisation of variational inference for sparse GP regression and latent variable models that allows for an efficient distributed algorithm and shows that GPs perform better than many common models often used for big data.
Fast Allocation of Gaussian Process Experts
TLDR
A scalable nonparametric Bayesian regression model based on a mixture of Gaussian process experts and the inducing points formalism underpinning sparse GP approximations that significantly outperforms six competitive baselines while requiring only a few hours of training.
Sparse Gaussian Processes using Pseudo-inputs
TLDR
It is shown that this new Gaussian process (GP) regression model can match full GP performance with small M, i.e. very sparse solutions, and it significantly outperforms other approaches in this regime.
A framework for evaluating approximation methods for Gaussian process regression
TLDR
Four different approximation algorithms are empirically investigated on four different prediction problems, and the quality of the predictions obtained as a function of the compute time taken are assessed.
Efficient Gaussian Process Inference for Short-Scale Spatio-Temporal Modeling
TLDR
The proposed Gaussian process inference scheme is compared to the standard approach using the sparse Cholesky decomposition and it is shown to be much faster and computationally feasible for 100–1000 times larger datasets.
Fast Forward Selection to Speed Up Sparse Gaussian Process Regression
TLDR
A method for the sparse greedy approximation of Bayesian Gaussian process regression, featuring a novel heuristic for very fast forward selection, which leads to a sufficiently stable approximation of the log marginal likelihood of the training data, which can be optimised to adjust a large number of hyperparameters automatically.
Sparse Spectrum Gaussian Process Regression
TLDR
The achievable trade-offs between predictive accuracy and computational requirements are compared, and it is shown that these are typically superior to existing state-of-the-art sparse approximations.
Infinite Mixtures of Gaussian Process Experts
We present an extension to the Mixture of Experts (ME) model, where the individual experts are Gaussian Process (GP) regression models. Using an input-dependent adaptation of the Dirichlet Process,
Variational Mixture of Gaussian Process Experts
TLDR
A new generative mixture of experts model is presented, where each expert is still a Gaussian process but is reformulated by a linear model, which breaks the dependency among training outputs and enables us to use a much faster variational Bayesian algorithm for training.
...
...