• Corpus ID: 173188595

On the Necessity and Effectiveness of Learning the Prior of Variational Auto-Encoder

@article{Xu2019OnTN,
  title={On the Necessity and Effectiveness of Learning the Prior of Variational Auto-Encoder},
  author={Haowen Xu and Wenxiao Chen and Jinlin Lai and Zhihan Li and Youjian Zhao and Dan Pei},
  journal={ArXiv},
  year={2019},
  volume={abs/1905.13452}
}
Using powerful posterior distributions is a popular approach to achieving better variational inference. However, recent works showed that the aggregated posterior may fail to match unit Gaussian prior, thus learning the prior becomes an alternative way to improve the lower-bound. In this paper, for the first time in the literature, we prove the necessity and effectiveness of learning the prior when aggregated posterior does not match unit Gaussian prior, analyze why this situation may happen… 

A Contrastive Learning Approach for Training Variational Autoencoder Priors

TLDR
Noise contrastive priors are proposed that improve the generative performance of state-of-the-art VAEs by a large margin on the MNIST, CIFAR-10, CelebA 64, and CelebA HQ 256 datasets.

NCP-VAE: Variational Autoencoders with Noise Contrastive Priors

TLDR
Noise contrastive priors are proposed that improve the generative performance of state-of-the-art VAEs by a large margin on the MNIST, CIFAR-10, CelebA 64, and CelebA HQ 256 datasets.

dpVAEs: Fixing Sample Generation for Regularized VAEs

TLDR
dpVAE fixes sample generation for regularized VAEs by decoupling the representation space from the generation space, and thereby reaping the representation learning benefits of the regularizations without sacrificing the sample generation.

RecVAE: A New Variational Autoencoder for Top-N Recommendations with Implicit Feedback

TLDR
The Recommender VAE (RecVAE) model is proposed that significantly outperforms previously proposed autoencoder-based models, including Mult-VAE and RaCT, across classical collaborative filtering datasets, and is presented with a detailed ablation study to assess new developments.

Dirichlet Process Prior for Student's t Graph Variational Autoencoders

TLDR
A novel prior distribution for GVAE is presented, called Dirichlet process (DP) construction for Student’s t (St) distribution, which allows the latent variables to adapt their complexity during learning and then cooperates with heavy-tailed St distribution to approach sufficient node representation.

RENs: Relevance Encoding Networks

TLDR
RENs is a novel probabilistic VAE-based framework that uses the automatic relevance determination (ARD) prior in the latent space to learn the data-specific bottleneck dimensionality and leverages the concept of DeepSets to capture permutation invariant statistical properties in both data and latent spaces for relevance determination.

A Contextual Latent Space Model: Subsequence Modulation in Melodic Sequence

TLDR
A contextual latent space model (CLSM) is proposed in order for users to be able to explore subsequence generation with a sense of direction in the generation space, e.g., interpolation, as well as exploring variations -- semantically similar possible subsequences.

Graph Generation with Variational Recurrent Neural Network

TLDR
This paper introduces Graph Variational Recurrent Neural Network (GraphVRNN), a probabilistic autoregressive model for graph generation that can capture the joint distributions of graph structures and the underlying node attributes.

References

SHOWING 1-10 OF 37 REFERENCES

VAE with a VampPrior

TLDR
This paper proposes to extend the variational auto-encoder (VAE) framework with a new type of prior called "Variational Mixture of Posteriors" prior, or VampPrior for short, which consists of a mixture distribution with components given by variational posteriors conditioned on learnable pseudo-inputs.

Variational Autoencoder with Implicit Optimal Priors

TLDR
The density ratio trick is introduced to estimate this KL divergence without modeling the aggregated posterior explicitly, and experiments show that the VAE with this implicit optimal prior achieves high density estimation performance.

Learnable Explicit Density for Continuous Latent Space and Variational Inference

TLDR
The decompose the learning of VAEs into layerwise density estimation, and argue that having a flexible prior is beneficial to both sample generation and inference, and analyze the family of inverse autoregressive flows (inverse AF), showing that with further improvement, inverse AF could be used as universal approximation to any complicated posterior.

Distribution Matching in Variational Inference

TLDR
It is concluded that at present, VAE-GAN hybrids have limited applicability: they are harder to scale, evaluate, and use for inference compared to VAEs; and they do not improve over the generation quality of GANs.

Nonparametric Variational Auto-Encoders for Hierarchical Representation Learning

TLDR
This work proposes hierarchical non-parametric variational autoencoders, which combines tree-structured Bayesian nonparametric priors with VAEs, to enable infinite flexibility of the latent representation space.

Approximate Inference for Deep Latent Gaussian Mixtures

TLDR
The authors' deep Latent Gaussian mixture model (DLGMM) generalizes previous work such as Factor Mixture Analysis and Deep Gaussian Mixtures to arbitrary differentiable inter-layer transformations and describes learning and inference for not only the traditional mixture model but also Dirichlet Process mixtures.

Resampled Priors for Variational Autoencoders

TLDR
Learning Accept/Reject Sampling (LARS) is proposed, a method for constructing richer priors using rejection sampling with a learned acceptance function, and it is demonstrated that LARS priors improve VAE performance on several standard datasets both when they are learned jointly with the rest of the model andWhen they are fitted to a pretrained model.

Importance Weighted Autoencoders

TLDR
The importance weighted autoencoder (IWAE), a generative model with the same architecture as the VAE, but which uses a strictly tighter log-likelihood lower bound derived from importance weighting, shows empirically that IWAEs learn richer latent space representations than VAEs, leading to improved test log- likelihood on density estimation benchmarks.

Ladder Variational Autoencoders

TLDR
A new inference model is proposed, the Ladder Variational Autoencoder, that recursively corrects the generative distribution by a data dependent approximate likelihood in a process resembling the recently proposed Ladder Network.

BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling

TLDR
This paper introduces the Bidirectional-Inference Variational Autoencoder (BIVA), characterized by a skip-connected generative model and an inference network formed by a bidirectional stochastic inference path, and shows that BIVA reaches state-of-the-art test likelihoods, generates sharp and coherent natural images, and uses the hierarchy of latent variables to capture different aspects of the data distribution.