• Corpus ID: 235417525

Model Selection for Bayesian Autoencoders

@inproceedings{Tran2021ModelSF,
  title={Model Selection for Bayesian Autoencoders},
  author={Ba-Hien Tran and Simone Rossi and Dimitrios Milios and Pietro Michiardi and Edwin V. Bonilla and Maurizio Filippone},
  booktitle={Neural Information Processing Systems},
  year={2021}
}
We develop a novel method for carrying out model selection for Bayesian autoencoders (BAEs) by means of prior hyper-parameter optimization. Inspired by the common practice of type-II maximum likelihood optimization and its equivalence to Kullback-Leibler divergence minimization, we propose to optimize the distributional sliced-Wasserstein distance (DSWD) between the output of the autoencoder and the empirical data distribution. The advantages of this formulation are that we can estimate the… 

All You Need is a Good Functional Prior for Bayesian Deep Learning

This work proposes a novel and robust framework to match their prior with the functional prior of neural networks based on the minimization of their Wasserstein distance, and provides vast experimental evidence that coupling these priors with scalable Markov chain Monte Carlo sampling offers systematically large performance improvements over alternative choices of priors and state-of-the-art approximate Bayesian deep learning approaches.

Taming Diffusion Times in Score-based Generative Models: Trade-offs and Solutions

  • Computer Science
  • 2022
This work shows how an auxiliary model can be used to bridge the gap between the ideal and the simulated forward dynamics, followed by a standard reverse diffusion process, and suggests a new method to improve quality and efficiency of both training and sampling, by adopting smaller diffusion times.

How Much is Enough? A Study on Diffusion Times in Score-based Generative Models

This work shows how an auxiliary model can be used to bridge the gap between the ideal and the simulated forward dynamics, followed by a standard reverse diffusion process, and suggests a new method to improve quality and efficiency of both training and sampling, by adopting smaller diffusion times.

References

SHOWING 1-10 OF 83 REFERENCES

Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection

This work proposes a new probabilistic, unsupervised approach to this problem based on a Bayesian variational autoencoder model, which estimates a full posterior distribution over the decoder parameters using stochastic gradient Markov chain Monte Carlo, instead of fitting a point estimate.

Resampled Priors for Variational Autoencoders

Learning Accept/Reject Sampling (LARS) is proposed, a method for constructing richer priors using rejection sampling with a learned acceptance function, and it is demonstrated that LARS priors improve VAE performance on several standard datasets both when they are learned jointly with the rest of the model andWhen they are fitted to a pretrained model.

From Variational to Deterministic Autoencoders

It is shown, in a rigorous empirical study, that the proposed regularized deterministic autoencoders are able to generate samples that are comparable to, or better than, those of VAEs and more powerful alternatives when applied to images as well as to structured data such as molecules.

All You Need is a Good Functional Prior for Bayesian Deep Learning

This work proposes a novel and robust framework to match their prior with the functional prior of neural networks based on the minimization of their Wasserstein distance, and provides vast experimental evidence that coupling these priors with scalable Markov chain Monte Carlo sampling offers systematically large performance improvements over alternative choices of priors and state-of-the-art approximate Bayesian deep learning approaches.

VAE with a VampPrior

This paper proposes to extend the variational auto-encoder (VAE) framework with a new type of prior called "Variational Mixture of Posteriors" prior, or VampPrior for short, which consists of a mixture distribution with components given by variational posteriors conditioned on learnable pseudo-inputs.

Probabilistic Auto-Encoder

The PAE combines the advantages of an Auto-Encoder, i.e. it is fast and easy to train and achieves small reconstruction error, with the desired properties of a generative model, such as high sample quality and good performance in downstream tasks.

What Are Bayesian Neural Network Posteriors Really Like?

It is shown that BNNs can achieve significant performance gains over standard training and deep ensembles, and a single long HMC chain can provide a comparable representation of the posterior to multiple shorter chains, and posterior tempering is not needed for near-optimal performance.

Adam: A Method for Stochastic Optimization

This work introduces Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments, and provides a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework.

Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning

This work presents a scalable marginal-likelihood estimation method to select both hyperparameters and network architectures, based on the training data alone, and it outperforms cross-validation and manualtuning on standard regression and image classification datasets, especially in terms of calibration and out-of-distribution detection.

Bayesian Deep Learning and a Probabilistic Perspective of Generalization

It is shown that deep ensembles provide an effective mechanism for approximate Bayesian marginalization, and a related approach is proposed that further improves the predictive distribution by marginalizing within basins of attraction, without significant overhead.
...