Corpus ID: 4697208

Multiple Adaptive Bayesian Linear Regression for Scalable Bayesian Optimization with Warm Start

@article{Perrone2017MultipleAB,
  title={Multiple Adaptive Bayesian Linear Regression for Scalable Bayesian Optimization with Warm Start},
  author={Valerio Perrone and Rodolphe Jenatton and Matthias W. Seeger and C. Archambeau},
  journal={arXiv: Machine Learning},
  year={2017}
}
Bayesian optimization (BO) is a model-based approach for gradient-free black-box function optimization. Typically, BO is powered by a Gaussian process (GP), whose algorithmic complexity is cubic in the number of evaluations. Hence, GP-based BO cannot leverage large amounts of past or related function evaluations, for example, to warm start the BO procedure. We develop a multiple adaptive Bayesian linear regression model as a scalable alternative whose complexity is linear in the number of… Expand

Figures from this paper

Scalable Hyperparameter Transfer Learning
TLDR
This work proposes a multi-task adaptive Bayesian linear regression model for transfer learning in BO, whose complexity is linear in the function evaluations: one Bayesianlinear regression model is associated to each black-box function optimization problem (or task), while transfer learning is achieved by coupling the models through a shared deep neural net. Expand
Mixed-Variable Bayesian Optimization
TLDR
MiVaBo is introduced, a novel BO algorithm for the efficient optimization of mixed-variable functions combining a linear surrogate model based on expressive feature representations with Thompson sampling, making MiVaBo the first BO method that can handle complex constraints over the discrete variables. Expand
BOHB: Robust and Efficient Hyperparameter Optimization at Scale
TLDR
This work proposes a new practical state-of-the-art hyperparameter optimization method, which consistently outperforms both Bayesian optimization and Hyperband on a wide range of problem types, including high-dimensional toy functions, support vector machines, feed-forward neural networks, Bayesian Neural networks, deep reinforcement learning, and convolutional neural networks. Expand
BANANAS: Bayesian Optimization with Neural Networks for Neural Architecture Search
TLDR
This work designs a NAS algorithm that performs Bayesian optimization using a neural network model, and develops a path-based encoding scheme to featurize the neural architectures that are used to train the neural network models. Expand
Probabilistic Matrix Factorization for Automated Machine Learning
TLDR
This paper uses probabilistic matrix factorization techniques and acquisition functions from Bayesian optimization to identify high-performing pipelines across a wide range of datasets, significantly outperforming the current state-of-the-art. Expand
Amazon SageMaker Automatic Model Tuning: Scalable Black-box Optimization
TLDR
Amazon SageMaker Automatic Model Tuning (AMT), a fully managed system for black-box optimization at scale, which finds the best version of a machine learning model by repeatedly training it with different hyperparameter configurations. Expand
Auto-Differentiating Linear Algebra
TLDR
This work details how a number of matrix decompositions (Cholesky, LQ, symmetric eigen) can be implemented as differentiable operators in MXNet, running on CPU and GPU in single and double precision. Expand
Learning the Learning Rate for Gradient Descent by Gradient Descent
This paper introduces an algorithm inspired from the work of Franceschi et al. (2017) for automatically tuning the learning rate while training neural networks. We formalize this problem asExpand
Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization
Black-box model-based optimization (MBO) problems, where the goal is to find 1 a design input that maximizes an unknown objective function, are ubiquitous 2 in a wide range of domains, such as theExpand
Meta-Learning: A Survey
TLDR
This chapter provides an overview of the state of the art in meta-learning, the science of systematically observing how different machine learning approaches perform on a wide range of learning tasks and then learning from this experience, or meta-data, to learn new tasks much faster than otherwise possible. Expand
...
1
2
...

References

SHOWING 1-10 OF 23 REFERENCES
Bayesian Optimization with Robust Bayesian Neural Networks
TLDR
This work presents a general approach for using flexible parametric models (neural networks) for Bayesian optimization, staying as close to a truly Bayesian treatment as possible and obtaining scalability through stochastic gradient Hamiltonian Monte Carlo, whose robustness is improved via a scale adaptation. Expand
Sparse Gaussian Processes for Bayesian Optimization
TLDR
This work introduces a new updating scheme for the online GP that accounts for the authors' preference during optimization for regions with better performance, and applies this method to optimize the performance of a free-electron laser. Expand
Scalable Bayesian Optimization Using Deep Neural Networks
TLDR
This work shows that performing adaptive basis function regression with a neural network as the parametric form performs competitively with state-of-the-art GP-based approaches, but scales linearly with the number of data rather than cubically, which allows for a previously intractable degree of parallelism. Expand
Bayesian Optimization with Tree-structured Dependencies
TLDR
A novel surrogate model for Bayesian optimization is introduced which combines independent Gaussian Processes with a linear model that encodes a tree-based dependency structure and can transfer information between overlapping decision sequences. Expand
Multi-Task Bayesian Optimization
TLDR
This paper proposes an adaptation of a recently developed acquisition function, entropy search, to the cost-sensitive, multi-task setting and demonstrates the utility of this new acquisition function by leveraging a small dataset to explore hyper-parameter settings for a large dataset. Expand
Initializing Bayesian Hyperparameter Optimization via Meta-Learning
TLDR
This paper mimics a strategy human domain experts use: speed up optimization by starting from promising configurations that performed well on similar datasets, and substantially improves the state of the art for the more complex combined algorithm selection and hyperparameter optimization problem. Expand
Collaborative hyperparameter tuning
TLDR
A generic method to incorporate knowledge from previous experiments when simultaneously tuning a learning algorithm on new problems at hand is proposed and is demonstrated in two experiments where it outperforms standard tuning techniques and single-problem surrogate-based optimization. Expand
Probabilistic Matrix Factorization for Automated Machine Learning
TLDR
This paper uses probabilistic matrix factorization techniques and acquisition functions from Bayesian optimization to identify high-performing pipelines across a wide range of datasets, significantly outperforming the current state-of-the-art. Expand
Efficient Transfer Learning Method for Automatic Hyperparameter Tuning
TLDR
This work proposes a fast and effective algorithm for automatic hyperparameter tuning that can generalize across datasets and empirically shows the superiority of the method on a large number of synthetic and real-world datasets for tuning hyperparameters of logistic regression and ensembles of classifiers. Expand
Taking the Human Out of the Loop: A Review of Bayesian Optimization
TLDR
This review paper introduces Bayesian optimization, highlights some of its methodological aspects, and showcases a wide range of applications. Expand
...
1
2
3
...