• Corpus ID: 237353100

Approximate Bayesian Optimisation for Neural Networks

  title={Approximate Bayesian Optimisation for Neural Networks},
  author={Nadhir Hassen and Irina Rish},
A body of work has been done to automate machine learning algorithms and to highlight the importance of model choice. Automating the process of choosing the best forecasting model and its corresponding parameters can result to improve a wide range of real-world applications. Bayesian optimisation (BO) uses a black-box optimisation methods to propose solutions according to an exploration-exploitation trade-off criterion through acquisition functions. BO framework imposes two key ingredients: a… 

Figures from this paper



Bayesian Optimization with Robust Bayesian Neural Networks

This work presents a general approach for using flexible parametric models (neural networks) for Bayesian optimization, staying as close to a truly Bayesian treatment as possible and obtaining scalability through stochastic gradient Hamiltonian Monte Carlo, whose robustness is improved via a scale adaptation.

Scalable Hyperparameter Transfer Learning

This work proposes a multi-task adaptive Bayesian linear regression model for transfer learning in BO, whose complexity is linear in the function evaluations: one Bayesianlinear regression model is associated to each black-box function optimization problem (or task), while transfer learning is achieved by coupling the models through a shared deep neural net.

Bayesian Optimization with Tree-structured Dependencies

A novel surrogate model for Bayesian optimization is introduced which combines independent Gaussian Processes with a linear model that encodes a tree-based dependency structure and can transfer information between overlapping decision sequences.

Efficient Bayesian Experimental Design for Implicit Models

This work devise a novel experimental design framework for implicit models that improves upon previous work in two ways, and uses Likelihood-Free Inference by Ratio Estimation to approximate posterior distributions, instead of the traditional approximate Bayesian computation or synthetic likelihood methods.

Bayesian Optimization by Density Ratio Estimation

This paper casts the computation of EI as a binary classification problem, building on the well-known link between class-probability estimation (CPE) and density ratio estimation (DRE) and the lesser- known link between density ratios and EI.

BOHB: Robust and Efficient Hyperparameter Optimization at Scale

This work proposes a new practical state-of-the-art hyperparameter optimization method, which consistently outperforms both Bayesian optimization and Hyperband on a wide range of problem types, including high-dimensional toy functions, support vector machines, feed-forward neural networks, Bayesian Neural networks, deep reinforcement learning, and convolutional neural networks.

BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search

A thorough analysis of the "BO + neural predictor framework" is given by identifying five main components: the architecture encoding, neural predictor, uncertainty calibration method, acquisition function, and acquisition function optimization, and developing a novel path-based encoding scheme for neural architectures.

Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning)

The treatment is comprehensive and self-contained, targeted at researchers and students in machine learning and applied statistics, and includes detailed algorithms for supervised-learning problem for both regression and classification.

Adam: A Method for Stochastic Optimization

This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.

RoBO : A Flexible and Robust Bayesian Optimization Framework in Python

The BSD-licensed python package ROBO, released with this paper, offers the only available implementations of Bayesian optimization with Bayesian neural networks, multi-task optimization, and fast Bayesian hyperparameter optimization on large datasets (Fabolas).