# Sparse Spectrum Gaussian Process Regression

@article{LzaroGredilla2010SparseSG, title={Sparse Spectrum Gaussian Process Regression}, author={Miguel L{\'a}zaro-Gredilla and Joaquin Qui{\~n}onero Candela and Carl Edward Rasmussen and An{\'i}bal R. Figueiras-Vidal}, journal={J. Mach. Learn. Res.}, year={2010}, volume={11}, pages={1865-1881} }

We present a new sparse Gaussian Process (GP) model for regression. The key novel idea is to sparsify the spectral representation of the GP. This leads to a simple, practical algorithm for regression tasks. We compare the achievable trade-offs between predictive accuracy and computational requirements, and show that these are typically superior to existing state-of-the-art sparse approximations. We discuss both the weight space and function space representations, and note that the new…

## 389 Citations

Sparse Gaussian Process regression model based on ℓ1/2 regularization

- Computer ScienceApplied Intelligence
- 2013

A new sparse GP model is developed, referred to as GPHalf, which represents the GP as a generalized linear regression model, then uses the modified ℓ1/2 half thresholding algorithm to optimize the corresponding objective function, thus yielding a sparse GPModel, proof that the proposed model converges to a sparse solution.

Sparse Orthogonal Variational Inference for Gaussian Processes

- Computer Science
- 2019

A new interpretation of sparse variational approximations for Gaussian processes using inducing points which can lead to more scalable algorithms than previous methods and report state-of-the-art results on CIFAR-10 with purely GP-based models.

Sparse Orthogonal Variational Inference for Gaussian Processes

- Computer ScienceAISTATS
- 2020

A new interpretation of sparse variational approximations for Gaussian processes using inducing points is introduced, which can lead to more scalable algorithms than previous methods and report state-of-the-art results on CIFAR-10 among purely GP-based models.

Improving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputs

- Computer ScienceICML
- 2015

This work models the covariance function with a finite Fourier series approximation and treats it as a random variable, on which a variational distribution is placed, and applies the distribution to fit the data.

MCMC for Variationally Sparse Gaussian Processes

- Computer ScienceNIPS
- 2015

A Hybrid Monte-Carlo sampling scheme which allows for a non-Gaussian approximation over the function values and covariance parameters simultaneously, with efficient computations based on inducing-point sparse GPs.

Variational Inference for Sparse Spectrum Approximation in Gaussian Process Regression

- Computer Science
- 2015

This work models the covariance function with a finite Fourier series approximation and treats it as a random variable, on which a variational distribution is placed and transforms the random covariancefunction to fit the data.

Variational Inference for Sparse Spectrum Approximation in Gaussian Process Regression

- Computer Science
- 2015

This work models the covariance function with a finite Fourier series approximation and treats it as a random variable, on which a variational distribution is placed, and applies the distribution to fit the data.

A Generalized Stochastic Variational Bayesian Hyperparameter Learning Framework for Sparse Spectrum Gaussian Process Regression

- Computer ScienceAAAI
- 2017

The generalized framework of stochastic variational Bayesian sparse spectrum GP (sVBSSGP) models addresses their shortcomings by adopting a Bayesian treatment of the spectral frequencies to avoid overfitting, and exploiting local data for boosting the predictive performance.

Variational inference for sparse spectrum Gaussian process regression

- Computer ScienceStat. Comput.
- 2016

A variational Bayes algorithm for fitting sparse spectrum GP regression models that uses nonconjugate variational message passing to derive fast and efficient updates and a novel adaptive neighbourhood technique for obtaining predictive inference that is effective in dealing with nonstationarity are presented.

Sparse Spectrum Gaussian Process for Bayesian Optimisation

- Computer SciencePAKDD
- 2021

A novel sparse spectrum approximation of Gaussian process (GP) tailored for Bayesian optimization is proposed, which uses the entropy of the global maximum distribution from the posterior GP as the regularizer that needs to be maximized.

## References

SHOWING 1-10 OF 23 REFERENCES

Sparse Gaussian Processes using Pseudo-inputs

- Computer ScienceNIPS
- 2005

It is shown that this new Gaussian process (GP) regression model can match full GP performance with small M, i.e. very sparse solutions, and it significantly outperforms other approaches in this regime.

Sparse Online Gaussian Processes

- Computer Science
- 2008

This work develops an approach for sparse representations of Gaussian Process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations for large data sets by using an appealing parametrisation and projection techniques that use the RKHS norm.

Sparse Greedy Gaussian Process Regression

- Computer ScienceNIPS
- 2000

A simple sparse greedy technique to approximate the maximum a posteriori estimate of Gaussian Processes with much improved scaling behaviour in the sample size m, and shows applications to large scale problems.

A Unifying View of Sparse Approximate Gaussian Process Regression

- Computer ScienceJ. Mach. Learn. Res.
- 2005

A new unifying view, including all existing proper probabilistic sparse approximations for Gaussian process regression, relies on expressing the effective prior which the methods are using, and highlights the relationship between existing methods.

Sparse On-Line Gaussian Processes

- Computer ScienceNeural Computation
- 2002

An approach for sparse representations of gaussian process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations for large data sets is developed based on a combination of a Bayesian on-line algorithm and a sequential construction of a relevant subsample of data that fully specifies the prediction of the GP model.

Fast Forward Selection to Speed Up Sparse Gaussian Process Regression

- Computer ScienceAISTATS
- 2003

A method for the sparse greedy approximation of Bayesian Gaussian process regression, featuring a novel heuristic for very fast forward selection, which leads to a sufficiently stable approximation of the log marginal likelihood of the training data, which can be optimised to adjust a large number of hyperparameters automatically.

Sparse Spectral Sampling Gaussian Processes

- Computer Science
- 2007

Though it is especially well-suited for low dimensional problems, PSDS can reach and even outperform other state-of-the-art sparse GP methods on higher dimensional datasets if allowed to work in batch mode, learning the locations of the spectral samples.

Sparse multiscale gaussian process regression

- Computer ScienceICML '08
- 2008

This work performs gradient based optimisation of the marginal likelihood, which costs O(m2n) time, and compares the method to various other sparse g.p. methods, which outperforms the other methods, particularly for the case of very few basis functions, i.e. a very high sparsity ratio.

Variational Learning of Inducing Variables in Sparse Gaussian Processes

- Computer ScienceAISTATS
- 2009

A variational formulation for sparse approximations that jointly infers the inducing inputs and the kernel hyperparameters by maximizing a lower bound of the true log marginal likelihood.

The Generalized FITC Approximation

- Computer ScienceNIPS
- 2007

An efficient generalization of the sparse pseudo-input Gaussian process model developed by Snelson and Ghahramani is presented, applying it to binary classification problems and resulting in a numerically stable algorithm with O(NM2) training complexity.