• Corpus ID: 220666060

Generalizing Variational Autoencoders with Hierarchical Empirical Bayes

@article{Cheng2020GeneralizingVA,
  title={Generalizing Variational Autoencoders with Hierarchical Empirical Bayes},
  author={Wei Cheng and Gregory Darnell and Sohini Ramachandran and Lorin Crawford},
  journal={ArXiv},
  year={2020},
  volume={abs/2007.10389}
}
Variational Autoencoders (VAEs) have experienced recent success as data-generating models by using simple architectures that do not require significant fine-tuning of hyperparameters. However, VAEs are known to suffer from over-regularization which can lead to failure to escape local maxima. This phenomenon, known as posterior collapse, prevents learning a meaningful latent encoding of the data. Recent methods have mitigated this issue by deterministically moment-matching an aggregated… 

Figures from this paper

References

SHOWING 1-10 OF 28 REFERENCES
From Variational to Deterministic Autoencoders
TLDR
It is shown, in a rigorous empirical study, that the proposed regularized deterministic autoencoders are able to generate samples that are comparable to, or better than, those of VAEs and more powerful alternatives when applied to images as well as to structured data such as molecules.
Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
TLDR
This paper investigates posterior collapse from the perspective of training dynamics and proposes an extremely simple modification to VAE training to reduce inference lag: depending on the model's current mutual information between latent variable and observation, the inference network is optimized before performing each model update.
Resampled Priors for Variational Autoencoders
TLDR
Learning Accept/Reject Sampling (LARS) is proposed, a method for constructing richer priors using rejection sampling with a learned acceptance function, and it is demonstrated that LARS priors improve VAE performance on several standard datasets both when they are learned jointly with the rest of the model andWhen they are fitted to a pretrained model.
Diagnosing and Enhancing VAE Models
TLDR
This work rigorously analyzes the VAE objective, and uses the corresponding insights to develop a simple VAE enhancement that requires no additional hyperparameters or sensitive tuning, all while retaining desirable attributes of the original VAE architecture.
Adversarial Autoencoders
TLDR
This paper shows how the adversarial autoencoder can be used in applications such as semi-supervised classification, disentangling style and content of images, unsupervised clustering, dimensionality reduction and data visualization, and performed experiments on MNIST, Street View House Numbers and Toronto Face datasets.
Preventing Posterior Collapse with delta-VAEs
TLDR
This paper proposes an alternative that utilizes the most powerful generative models as decoders, whilst optimising the variational lower bound all while ensuring that the latent variables preserve and encode useful information.
Auto-Encoding Variational Bayes
TLDR
A stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case is introduced.
The Mutual Autoencoder: Controlling Information in Latent Code Representations
TLDR
This work proposes a method for explicitly controlling the amount of information stored in the latent code, which can learn codes ranging from independent to nearly deterministic while benefiting from decoder capacity.
beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework
Learning an interpretable factorised representation of the independent data generative factors of the world without supervision is an important precursor for the development of artificial
Neural Discrete Representation Learning
TLDR
Pairing these representations with an autoregressive prior, the model can generate high quality images, videos, and speech as well as doing high quality speaker conversion and unsupervised learning of phonemes, providing further evidence of the utility of the learnt representations.
...
...