• Corpus ID: 44085695

Dirichlet-based Gaussian Processes for Large-scale Calibrated Classification

@article{Milios2018DirichletbasedGP,
  title={Dirichlet-based Gaussian Processes for Large-scale Calibrated Classification},
  author={Dimitrios Milios and Raffaello Camoriano and Pietro Michiardi and Lorenzo Rosasco and Maurizio Filippone},
  journal={ArXiv},
  year={2018},
  volume={abs/1805.10915}
}
In this paper, we study the problem of deriving fast and accurate classification algorithms with uncertainty quantification. Gaussian process classification provides a principled approach, but the corresponding computational burden is hardly sustainable in large-scale problems and devising efficient alternatives is a challenge. In this work, we investigate if and how Gaussian process regression directly applied to the classification labels can be used to tackle this question. While in this case… 

Figures and Tables from this paper

Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration
TLDR
A natively multiclass calibration method applicable to classifiers from any model class, derived from Dirichlet distributions and generalising the beta calibration method from binary classification is proposed.
Classifier Calibration: How to assess and improve predicted class probabilities: a survey
TLDR
This paper provides both an introduction to and a detailed overview of the principles and practice of classifier calibration, including proper scoring rules and other evaluation metrics, visualisation approaches, a comprehensive account of post-hoc calibration methods for binary and multiclass classification, and several advanced topics.
Kernel Interpolation for Scalable Online Gaussian Processes
TLDR
This work shows how to use structured kernel interpolation to efficiently reuse computations for constant-time O(1) online updates with respect to the number of points n, while retaining exact inference.
A Survey on Uncertainty Estimation in Deep Learning Classification Systems from a Bayesian Perspective
TLDR
The topic of uncertainty estimation is introduced, and the peculiarities of such estimation when applied to classification systems are analyzed, and some of the properties that should be borne in mind when developing such metrics are reviewed.
Gaussian Processes with Differential Privacy
TLDR
This work introduces GPs with DP protection for both model inputs and outputs and proposes a method for hyperparameter learning using a private selection protocol applied to validation set log-likelihood.
Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning
TLDR
The I-Max concept for binning is derived, which maximizes the mutual information between labels and binned (quantized) logits, and outperforms the state of the art calibration methods on various evaluation metrics across different benchmark datasets and models, even when using only a small set of calibration data.
Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning
TLDR
This paper proposes an alternative data-efficient kernel density-based estimator for a reliable evaluation of the calibration performance and proves its asymptotically unbiasedness and consistency.
Meta-Cal: Well-controlled Post-hoc Calibration by Ranking
TLDR
This paper introduces two practical constraints to be taken into consideration for post-hoc calibration for multi-class classification under constraints, and presents Meta-Cal, which is built from a base calibrator and a ranking model.
TUAL INFORMATION MAXIMIZATION-BASED BINNING
TLDR
The I-Max concept for binning is derived, which maximizes the mutual information between labels and quantized logits and mitigates potential loss in ranking performance due to lossy quantization, and by disentangling the optimization of bin edges and representatives allows simultaneous improvement of ranking and calibration performance.
...
...

References

SHOWING 1-10 OF 41 REFERENCES
Scalable Gaussian Process Classification via Expectation Propagation
TLDR
This work describes here how to train Gaussian process classifiers efficiently using expectation propagation using a stochastic approximation of the gradient, and indicates that the method described is competitive with the variational approach.
Bayesian Classification With Gaussian Processes
TLDR
A Bayesian treatment is provided, integrating over uncertainty in y and in the parameters that control the Gaussian process prior the necessary integration over y is carried out using Laplace's approximation, and the method is generalized to multiclass problems (m>2) using the softmax function.
Assessing Approximate Inference for Binary Gaussian Process Classification
TLDR
This work reviews and compares Laplace's method and Expectation Propagation for approximate Bayesian inference in the binary Gaussian process classification model, and presents a comprehensive comparison of the approximations, their predictive performance and marginal likelihood estimates to results obtained by MCMC sampling.
On Calibration of Modern Neural Networks
TLDR
It is discovered that modern neural networks, unlike those from a decade ago, are poorly calibrated, and on most datasets, temperature scaling -- a single-parameter variant of Platt Scaling -- is surprisingly effective at calibrating predictions.
Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers
TLDR
Extensive experiments show that beta calibration is superior to logistic calibration for Naive Bayes and Adaboost and derive the method from first principles and show that fitting it is as easy as fitting a logistic curve.
Multi-output learning via spectral filtering
TLDR
A finite sample bound for the excess risk of the obtained estimator allows to prove consistency both for regression and multi-category classification and some promising results of the proposed algorithms are presented.
Ensemble Nystrom Method
TLDR
A new family of algorithms based on mixtures of Nystrom approximation, ensemble Nystrom algorithms, that yield more accurate low-rank approximations than the standard Nystrom method are introduced.
Gaussian Processes for Machine Learning
TLDR
The treatment is comprehensive and self-contained, targeted at researchers and students in machine learning and applied statistics, and deals with the supervised learning problem for both regression and classification.
Less is More: Nyström Computational Regularization
TLDR
A simple incremental variant of Nystrom Kernel Regularized Least Squares is suggested, where the subsampling level implements a form of computational regularization, in the sense that it controls at the same time regularization and computations.
How to Scale Up Kernel Methods to Be As Good As Deep Neural Nets
TLDR
This work develops methods to scale up kernel models to successfully tackle large-scale learning problems that are so far only approachable by deep learning architectures, and conducts extensive empirical studies on problems from image recognition and automatic speech recognition.
...
...