Corpus ID: 49742003

Avoiding Latent Variable Collapse With Generative Skip Models

@article{Dieng2019AvoidingLV,
  title={Avoiding Latent Variable Collapse With Generative Skip Models},
  author={Adji B. Dieng and Yoon Kim and Alexander M. Rush and David M. Blei},
  journal={ArXiv},
  year={2019},
  volume={abs/1807.04863}
}
Variational autoencoders learn distributions of high-dimensional data. [...] Key Result Compared to existing VAE architectures, we show that generative skip models maintain similar predictive performance but lead to less collapse and provide more meaningful representations of the data.Expand
Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
TLDR
This paper investigates posterior collapse from the perspective of training dynamics and proposes an extremely simple modification to VAE training to reduce inference lag: depending on the model's current mutual information between latent variable and observation, the inference network is optimized before performing each model update. Expand
Posterior Collapse and Latent Variable Non-identifiability
  • 2020
Variational autoencoders (VAEs) model high-dimensional data by positing low-dimensional latent variables that are mapped through a flexible, implicit distribution parametrized by a neural network.Expand
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
TLDR
This paper introduces the Bidirectional-Inference Variational Autoencoder (BIVA), characterized by a skip-connected generative model and an inference network formed by a bidirectional stochastic inference path, and shows that BIVA reaches state-of-the-art test likelihoods, generates sharp and coherent natural images, and uses the hierarchy of latent variables to capture different aspects of the data distribution. Expand
Preventing Posterior Collapse with Levenshtein Variational Autoencoder
TLDR
Levenstein VAE produces more informative latent representations than alternative approaches to preventing posterior collapse and is closely related to optimizing a bound on the intractable Kullback-Leibler divergence of an LD-based kernel density estimator from the model distribution. Expand
PREVENTING POSTERIOR COLLAPSE WITH δ-VAES
Due to the phenomenon of “posterior collapse,” current latent variable generative models pose a challenging design choice that either weakens the capacity of the decoder or requires altering theExpand
A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text
TLDR
A simple fix for posterior collapse is investigated which yields surprisingly effective results and is used to argue that the typical surrogate objective for VAEs may not be sufficient or necessarily appropriate for balancing the goals of representation learning and data distribution modeling. Expand
Discretized Bottleneck in VAE: Posterior-Collapse-Free Sequence-to-Sequence Learning
TLDR
This paper proposes a principled approach to eliminate the posterior-collapse issue in latent space by applying a discretized bottleneck in the latent space and imposes a shared discrete latent space where each input is learned to choose a combination of shared latent atoms as its latent representation. Expand
Sequential Learning and Regularization in Variational Recurrent Autoencoder
TLDR
Experiments on language model and sentiment classification show that the proposed method mitigates the issue of posterior collapse and learns the meaningful latent features to improve the inference and generation for semantic representation. Expand
Characterizing and Avoiding Problematic Global Optima of Variational Autoencoders
TLDR
This paper demonstrates that both issues stem from the fact that the global optima of the VAE training objective often correspond to undesirable solutions and presents a novel inference method, LiBI, mitigating the problems identified in the analysis. Expand
Reweighted Expectation Maximization
TLDR
A new EM-based algorithm for fitting deep generative models called reweighted expectation maximization (REM) is proposed and compared to the VAE and the IWAE on several density estimation benchmarks and found it leads to significantly better performance as measured by log-likelihood. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 36 REFERENCES
Spherical Latent Spaces for Stable Variational Autoencoders
TLDR
This work experiments with another choice of latent distribution, namely the von Mises-Fisher (vMF) distribution, which places mass on the surface of the unit hypersphere and shows that they learn richer and more nuanced structures in their latent representations than their Gaussian counterparts. Expand
Ladder Variational Autoencoders
TLDR
A new inference model is proposed, the Ladder Variational Autoencoder, that recursively corrects the generative distribution by a data dependent approximate likelihood in a process resembling the recently proposed Ladder Network. Expand
Fixing a Broken ELBO
TLDR
This framework derives variational lower and upper bounds on the mutual information between the input and the latent variable, and uses these bounds to derive a rate-distortion curve that characterizes the tradeoff between compression and reconstruction accuracy. Expand
Importance Weighted Autoencoders
TLDR
The importance weighted autoencoder (IWAE), a generative model with the same architecture as the VAE, but which uses a strictly tighter log-likelihood lower bound derived from importance weighting, shows empirically that IWAEs learn richer latent space representations than VAEs, leading to improved test log- likelihood on density estimation benchmarks. Expand
Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
TLDR
Adversarial Variational Bayes (AVB), a technique for training Variational Autoencoders with arbitrarily expressive inference models by introducing an auxiliary discriminative network that allows to rephrase the maximum-likelihood-problem as a two-player game, hence establishing a principled connection between VAEs and Generative Adversarial Networks (GANs). Expand
Tackling Over-pruning in Variational Autoencoders
TLDR
The epitomic variational autoencoder (eVAE) is proposed, which makes efficient use of model capacity and generalizes better than VAE and helps prevent inactive units since each group is pressured to explain the data. Expand
Auto-Encoding Variational Bayes
TLDR
A stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case is introduced. Expand
Learning Deep Latent Gaussian Models with Markov Chain Monte Carlo
TLDR
This paper proposes a different approach to deep latent Gaussian models: rather than use a variational approximation, this work uses Markov chain Monte Carlo (MCMC), which yields higher held-out likelihoods, produces sharper images, and does not suffer from the variational overpruning effect. Expand
How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks
TLDR
This work proposes three advances in training algorithms of variational autoencoders, for the first time allowing to train deep models of up to five stochastic layers, using a structure similar to the Ladder network as the inference model and shows state-of-the-art log-likelihood results for generative modeling on several benchmark datasets. Expand
VAE with a VampPrior
TLDR
This paper proposes to extend the variational auto-encoder (VAE) framework with a new type of prior called "Variational Mixture of Posteriors" prior, or VampPrior for short, which consists of a mixture distribution with components given by variational posteriors conditioned on learnable pseudo-inputs. Expand
...
1
2
3
4
...