Bayesian Classification With Gaussian Processes

  title={Bayesian Classification With Gaussian Processes},
  author={Christopher K. I. Williams and David Barber},
  journal={IEEE Trans. Pattern Anal. Mach. Intell.},
We consider the problem of assigning an input vector to one of m classes by predicting P(c|x) for c=1,...,m. For a two-class problem, the probability of class one given x is estimated by /spl sigma/(y(x)), where /spl sigma/(y)=1/(1+e/sup -y/). A Gaussian process prior is placed on y(x), and is combined with the training data to obtain predictions for new x points. We provide a Bayesian treatment, integrating over uncertainty in y and in the parameters that control the Gaussian process prior the… 

Figures and Tables from this paper

Bayes Machines for binary classification

Sparse Gaussian Process Classification With Multiple Classes

This work shows how to generalize the binary classification informative vector machine (IVM) to multiple classes and is a principled approximation to Bayesian inference which yields valid uncertainty estimates and allows for hyperparameter adaption via marginal likelihood maximization.

Gaussian process models for robust regression, classification, and reinforcement learning

Gaussian process models constitute a class of probabilistic statistical models in which a Gaussian process is used to describe the Bayesian a priori uncertainty about a latent function, and it will be shown how this can be used to estimate value functions.

Multi-class Gaussian Process Classification with Noisy Inputs

The results obtained show that, although the classification error is similar across methods, the predicted distribution of the proposed methods is better, in terms of the test log-likelihood, than the predictive distribution of a classifier based on GPs that ignores input noise.

Twin Gaussian Processes for Binary Classification

A new Gaussian process model termed as twin Gaussian processes for binary classification, which allows for an explicit inference based on analytical methods, thereby avoiding the high computational cost caused by approximating the posterior with Gaussian distribution.

Approximations for Binary Gaussian Process Classification

We provide a comprehensive overview of many recent algorithms for approximate inference in Gaussian process models for probabilistic binary classification. The relationships between several

Exact Bayesian bin classification: a fast alternative to Bayesian classification and its application to neural response analysis

This work introduces a fast, exact alternative to Bayesian classification, which allows for the computation of feedback signals, which can be used as input to subsequent stages of inference, e.g. neural network training.

Bayesian Gaussian Process Classification with the EM-EP Algorithm

An approximate EM algorithm, the EM-EP algorithm, is presented, which is found to converge in practice and provides an efficient Bayesian framework for learning hyperparameters of the kernel.

Gaussian Processes for Object Categorization

This work shows that with an appropriate combination of kernels a significant boost in classification performance is possible, and indicates the utility of active learning with probabilistic predictive models, especially when the amount of training data labels that may be sought for a category is ultimately very small.

Variational Multinomial Logit Gaussian Process

A variational approximation to the Gaussian process multi-class model is proposed, and criteria which can be used to select the inducing set are derived and the effectiveness of these criteria over random selection in an experiment is shown.



Soft Classiication, A. K. A. Risk Estimation, via Penalized Log Likelihood and Smoothing Spline Analysis of Variance

A family of methods which are well suited for the estimation of the probability that the next example will be in class 1 are described, which combines results from Penalized log likelihood estimation, Smoothing splines, and Analysis of variance to get the PSA class of methods.

Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification

Software is now available that implements Gaussian process methods using covariance functions with hierarchical parameterizations, which can discover high-level properties of the data, such as which inputs are relevant to predicting the response.

Evaluation of gaussian processes and other methods for non-linear regression

It is shown that a Bayesian approach to learning in multi-layer perceptron neural networks achieves better performance than the commonly used early stopping procedure, even for reasonably short amounts of computation time.

Gaussian Processes for Bayesian Classification via Hybrid Monte Carlo

Gaussian processes are used to approximate the weight space integral analytically, so that only a small number of hyperparameters need be integrated over by MCMC methods.

Bayesian Methods for Backpropagation Networks

This chapter describes numerical techniques based on Gaussian approximations for implementation of powerful and practical methods for controlling, comparing, and using adaptive network models.

Gaussian Processes for Regression

This paper investigates the use of Gaussian process priors over functions, which permit the predictive Bayesian analysis for fixed values of hyperparameters to be carried out exactly using matrix operations.

Bayesian Learning for Neural Networks

This book demonstrates how Bayesian methods allow complex neural network models to be used without fear of the "overfitting" that can occur with traditional training methods.

Computing with Infinite Networks

For neural networks with a wide class of weight-priors, it can be shown that in the limit of an infinite number of hidden units the prior over functions tends to a Gaussian process. In this paper

Maximum likelihood estimation of models for residual covariance in spatial regression

We describe the maximum likelihood method for fitting the linear model when residuals are correlated and when the covariance among the residuals is determined by a parametric model containing unknown

Flexible Non-linear Approaches to Classification

Questions common to all members of the class, including how best to use the outputs to classify, how to fit the class of functions and also how to choose between classes are explored via some theory and three real examples.