Sparse Spectrum Gaussian Process Regression

@article{LzaroGredilla2010SparseSG,
  title={Sparse Spectrum Gaussian Process Regression},
  author={Miguel L{\'a}zaro-Gredilla and Joaquin Qui{\~n}onero Candela and Carl Edward Rasmussen and An{\'i}bal R. Figueiras-Vidal},
  journal={J. Mach. Learn. Res.},
  year={2010},
  volume={11},
  pages={1865-1881}
}
We present a new sparse Gaussian Process (GP) model for regression. The key novel idea is to sparsify the spectral representation of the GP. This leads to a simple, practical algorithm for regression tasks. We compare the achievable trade-offs between predictive accuracy and computational requirements, and show that these are typically superior to existing state-of-the-art sparse approximations. We discuss both the weight space and function space representations, and note that the new… 

Figures from this paper

Sparse Gaussian Process regression model based on ℓ1/2 regularization
TLDR
A new sparse GP model is developed, referred to as GPHalf, which represents the GP as a generalized linear regression model, then uses the modified ℓ1/2 half thresholding algorithm to optimize the corresponding objective function, thus yielding a sparse GPModel, proof that the proposed model converges to a sparse solution.
Sparse Orthogonal Variational Inference for Gaussian Processes
TLDR
A new interpretation of sparse variational approximations for Gaussian processes using inducing points which can lead to more scalable algorithms than previous methods and report state-of-the-art results on CIFAR-10 with purely GP-based models.
Sparse Orthogonal Variational Inference for Gaussian Processes
TLDR
A new interpretation of sparse variational approximations for Gaussian processes using inducing points is introduced, which can lead to more scalable algorithms than previous methods and report state-of-the-art results on CIFAR-10 among purely GP-based models.
Improving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputs
TLDR
This work models the covariance function with a finite Fourier series approximation and treats it as a random variable, on which a variational distribution is placed, and applies the distribution to fit the data.
MCMC for Variationally Sparse Gaussian Processes
TLDR
A Hybrid Monte-Carlo sampling scheme which allows for a non-Gaussian approximation over the function values and covariance parameters simultaneously, with efficient computations based on inducing-point sparse GPs.
Variational Inference for Sparse Spectrum Approximation in Gaussian Process Regression
TLDR
This work models the covariance function with a finite Fourier series approximation and treats it as a random variable, on which a variational distribution is placed and transforms the random covariancefunction to fit the data.
Variational Inference for Sparse Spectrum Approximation in Gaussian Process Regression
TLDR
This work models the covariance function with a finite Fourier series approximation and treats it as a random variable, on which a variational distribution is placed, and applies the distribution to fit the data.
A Generalized Stochastic Variational Bayesian Hyperparameter Learning Framework for Sparse Spectrum Gaussian Process Regression
TLDR
The generalized framework of stochastic variational Bayesian sparse spectrum GP (sVBSSGP) models addresses their shortcomings by adopting a Bayesian treatment of the spectral frequencies to avoid overfitting, and exploiting local data for boosting the predictive performance.
Variational inference for sparse spectrum Gaussian process regression
TLDR
A variational Bayes algorithm for fitting sparse spectrum GP regression models that uses nonconjugate variational message passing to derive fast and efficient updates and a novel adaptive neighbourhood technique for obtaining predictive inference that is effective in dealing with nonstationarity are presented.
Sparse Spectrum Gaussian Process for Bayesian Optimisation
TLDR
A novel sparse spectrum approximation of Gaussian process (GP) tailored for Bayesian optimization is proposed, which uses the entropy of the global maximum distribution from the posterior GP as the regularizer that needs to be maximized.
...
...

References

SHOWING 1-10 OF 23 REFERENCES
Sparse Gaussian Processes using Pseudo-inputs
TLDR
It is shown that this new Gaussian process (GP) regression model can match full GP performance with small M, i.e. very sparse solutions, and it significantly outperforms other approaches in this regime.
Sparse Online Gaussian Processes
TLDR
This work develops an approach for sparse representations of Gaussian Process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations for large data sets by using an appealing parametrisation and projection techniques that use the RKHS norm.
Sparse Greedy Gaussian Process Regression
TLDR
A simple sparse greedy technique to approximate the maximum a posteriori estimate of Gaussian Processes with much improved scaling behaviour in the sample size m, and shows applications to large scale problems.
A Unifying View of Sparse Approximate Gaussian Process Regression
TLDR
A new unifying view, including all existing proper probabilistic sparse approximations for Gaussian process regression, relies on expressing the effective prior which the methods are using, and highlights the relationship between existing methods.
Sparse On-Line Gaussian Processes
TLDR
An approach for sparse representations of gaussian process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations for large data sets is developed based on a combination of a Bayesian on-line algorithm and a sequential construction of a relevant subsample of data that fully specifies the prediction of the GP model.
Fast Forward Selection to Speed Up Sparse Gaussian Process Regression
TLDR
A method for the sparse greedy approximation of Bayesian Gaussian process regression, featuring a novel heuristic for very fast forward selection, which leads to a sufficiently stable approximation of the log marginal likelihood of the training data, which can be optimised to adjust a large number of hyperparameters automatically.
Sparse Spectral Sampling Gaussian Processes
TLDR
Though it is especially well-suited for low dimensional problems, PSDS can reach and even outperform other state-of-the-art sparse GP methods on higher dimensional datasets if allowed to work in batch mode, learning the locations of the spectral samples.
Sparse multiscale gaussian process regression
TLDR
This work performs gradient based optimisation of the marginal likelihood, which costs O(m2n) time, and compares the method to various other sparse g.p. methods, which outperforms the other methods, particularly for the case of very few basis functions, i.e. a very high sparsity ratio.
Variational Learning of Inducing Variables in Sparse Gaussian Processes
TLDR
A variational formulation for sparse approximations that jointly infers the inducing inputs and the kernel hyperparameters by maximizing a lower bound of the true log marginal likelihood.
The Generalized FITC Approximation
TLDR
An efficient generalization of the sparse pseudo-input Gaussian process model developed by Snelson and Ghahramani is presented, applying it to binary classification problems and resulting in a numerically stable algorithm with O(NM2) training complexity.
...
...