Corpus ID: 18014762

Using Meta-Learning to Initialize Bayesian Optimization of Hyperparameters

@inproceedings{Feurer2014UsingMT,
  title={Using Meta-Learning to Initialize Bayesian Optimization of Hyperparameters},
  author={Matthias Feurer and Jost Tobias Springenberg and F. Hutter},
  booktitle={MetaSel@ECAI},
  year={2014}
}
Model selection and hyperparameter optimization is crucial in applying machine learning to a novel dataset. Recently, a sub-community of machine learning has focused on solving this problem with Sequential Model-based Bayesian Optimization (SMBO), demonstrating substantial successes in many applications. However, for expensive algorithms the computational overhead of hyperparameter optimization can still be prohibitive. In this paper we explore the possibility of speeding up SMBO by… Expand
Initializing Bayesian Hyperparameter Optimization via Meta-Learning
TLDR
This paper mimics a strategy human domain experts use: speed up optimization by starting from promising configurations that performed well on similar datasets, and substantially improves the state of the art for the more complex combined algorithm selection and hyperparameter optimization problem. Expand
Two-Stage Transfer Surrogate Model for Automatic Hyperparameter Optimization
TLDR
This work presents a model that transfers knowledge of the performance of an algorithm on given other data sets to automatically accelerate the hyperparameter optimization for a new data set and is outperforming the state of the art methods. Expand
Learning hyperparameter optimization initializations
TLDR
Empirical evidence is given that the proposed initialization strategy for hyperparameter optimization provides an improvement over the state of the art for initialization strategies and further methods that are able to transfer knowledge between data sets. Expand
Hyperparameter Search Space Pruning - A New Component for Sequential Model-Based Hyperparameter Optimization
TLDR
This work proposes to discard regions of the search space that are unlikely to contain better hyperparameter configurations by transferring knowledge from past experiments on other data sets as well as taking into account the evaluations already done on the current data set. Expand
Scalable Gaussian process-based transfer surrogates for hyperparameter optimization
TLDR
This work proposes to learn individual surrogate models on the observations of each data set and then combine all surrogates to a joint one using ensembling techniques, and extends the framework to directly estimate the acquisition function in the same setting, using a novel technique which is name the “transfer acquisition function”. Expand
Learning Data Set Similarities for Hyperparameter Optimization Initializations
TLDR
This work further improve the initialization techniques for sequential model-based optimization, the current state of the art hyperparameter optimization framework by using the few evaluations on the new data sets to create new features that allow a better prediction of the data set similarity. Expand
Sequential Model-Free Hyperparameter Tuning
TLDR
This work adapts the sequential model-based optimization by replacing its surrogate model and acquisition function with one policy that is optimized for the task of hyperparameter tuning and proposes a similarity measure for data sets that yields more comprehensible results than those using meta-features. Expand
Meta-learning for automatic model selection and hyperparameter optimisation
Model selection and hyperparameter optimisation is an important step in creating well-performing machine learning models. Optimizing hyperparameters for a given task can be enhanced with informationExpand
Using a thousand optimization tasks to learn hyperparameter search strategies
TLDR
The diversity of the TaskSet and the method for learning hyperparameter lists are used to empirically explore the generalization of these lists to new optimization tasks in a variety of settings including ImageNet classification with Resnet50 and LM1B language modeling with transformers. Expand
Amortized Auto-Tuning: Cost-Efficient Transfer Optimization for Hyperparameter Recommendation
TLDR
A thorough analysis of the multi-task multi-fidelity Bayesian optimization framework, which leads to the best instantiation—AmorTized Auto-Tuning (AT2) and an offline-computed 27-task Hyperparameter Recommendation (HyperRec) database to serve the community. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 38 REFERENCES
Multi-Task Bayesian Optimization
TLDR
This paper proposes an adaptation of a recently developed acquisition function, entropy search, to the cost-sensitive, multi-task setting and demonstrates the utility of this new acquisition function by leveraging a small dataset to explore hyper-parameter settings for a large dataset. Expand
Collaborative hyperparameter tuning
TLDR
A generic method to incorporate knowledge from previous experiments when simultaneously tuning a learning algorithm on new problems at hand is proposed and is demonstrated in two experiments where it outperforms standard tuning techniques and single-problem surrogate-based optimization. Expand
Practical Bayesian Optimization of Machine Learning Algorithms
TLDR
This work describes new algorithms that take into account the variable cost of learning algorithm experiments and that can leverage the presence of multiple cores for parallel experimentation and shows that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization for many algorithms. Expand
Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms
TLDR
This work considers the problem of simultaneously selecting a learning algorithm and setting its hyperparameters, going beyond previous work that attacks these issues separately and shows classification performance often much better than using standard selection and hyperparameter optimization methods. Expand
Algorithms for Hyper-Parameter Optimization
TLDR
This work contributes novel techniques for making response surface models P(y|x) in which many elements of hyper-parameter assignment (x) are known to be irrelevant given particular values of other elements. Expand
Efficient Transfer Learning Method for Automatic Hyperparameter Tuning
TLDR
This work proposes a fast and effective algorithm for automatic hyperparameter tuning that can generalize across datasets and empirically shows the superiority of the method on a large number of synthetic and real-world datasets for tuning hyperparameters of logistic regression and ensembles of classifiers. Expand
Exploiting correlation and budget constraints in Bayesian multi-armed bandit optimization
TLDR
A Bayesian approach is introduced for the problem of finding the maximizer of a nonlinear smooth function, that can only be evaluated point-wise, subject to constraints on the number of permitted function evaluations, and it empirically outperforms both the existing frequentist counterpart and other Bayesian optimization methods. Expand
Combining meta-learning and search techniques to select parameters for support vector machines
TLDR
This work investigates the combination of meta-learning with two search algorithms: Particle Swarm Optimization and Tabu Search and showed that, on average, the proposed hybrid methods obtained lower error rates when compared to their components applied in isolation. Expand
Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters
TLDR
A library of benchmarks from the prominent application of hyperparameter optimization is introduced and used to compare Spearmint, TPE, and SMAC, three recent BO methods for hyperparameters optimization. Expand
Hyperopt-Sklearn: Automatic Hyperparameter Configuration for Scikit-Learn
TLDR
This work improves on best-known scores for the model space for both MNIST and Convex Shapes, and uses Hyperopt to define a search space that encompasses many standard components and common patterns of composing them together. Expand
...
1
2
3
4
...