A Batch Normalized Inference Network Keeps the KL Vanishing Away

@inproceedings{Zhu2020ABN,
  title={A Batch Normalized Inference Network Keeps the KL Vanishing Away},
  author={Qile Zhu and Wei Bi and Xiaojiang Liu and Xiyao Ma and Xiaolin Li and Dapeng Oliver Wu},
  booktitle={ACL},
  year={2020}
}
Variational Autoencoder (VAE) is widely used as a generative model to approximate a model’s posterior on latent variables by combining the amortized variational inference and deep neural networks. However, when paired with strong autoregressive decoders, VAE often converges to a degenerated local optimum known as “posterior collapse”. Previous approaches consider the Kullback–Leibler divergence (KL) individual for each datapoint. We propose to let the KL follow a distribution across the whole… Expand
Improving Variational Autoencoder for Text Modelling with Timestep-Wise Regularisation
TLDR
This paper proposes a simple, generic architecture called Timestep-Wise Regularisation VAE (TWR-VAE), which can effectively avoid posterior collapse and can be applied to any RNN-based VAE models. Expand
Stochastic Graph Recurrent Neural Network
TLDR
A novel neural architecture that applies stochastic latent variables to simultaneously capture the evolution in node attributes and topology is proposed, and a simple and interpretable structure is proposed based on the lower bound of KL-divergence. Expand
Learning Agent Representations for Ice Hockey
TLDR
A new approach for agent representations, based on a Markov game model, that is tailored towards applications in professional ice hockey, and provides a contextualized player representation with a hierarchy of latent variables that effectively prevents latent posterior collapse. Expand
Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness
  • Dazhong Shen, Chuan Qin, Chao Wang, Hengshu Zhu, Enhong Chen, Hui Xiong
  • Computer Science
  • IJCAI
  • 2021
TLDR
This paper proposes an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space, and thus the representation can be learned in a meaningful and compact manner and can outperform state-of-the-art baselines on both likelihood estimation and underlying classification tasks. Expand
On the Low-density Latent Regions of VAE-based Language Models
By representing semantics in latent spaces, Variational autoencoders (VAEs) have been proven powerful in modelling and generating signals such as image and text, even without supervision. However,Expand
A Novel Hybrid Method for KPI Anomaly Detection Based on VAE and SVDD
Key performance indicator (KPI) anomaly detection is the underlying core technology in Artificial Intelligence for IT operations (AIOps). It has an important impact on subsequent anomaly location andExpand
ADA-INCVAE: Improved data generation using variational autoencoder for imbalanced classification
  • Kai Huang, Xiaoguo Wang
  • Computer Science
  • Applied Intelligence
  • 2021
TLDR
An effective data generation algorithm for minority samples called the Adaptive Increase dimension of Variational AutoEncoder (ADA-INCVAE) is proposed, which could help the classifier to effectively improve F1-measure and G-mean, which verifies the effectiveness of synthetic data generated by ADA- INCVAE. Expand
Deep transfer learning mechanism for fine-grained cross-domain sentiment classification
TLDR
DTLM provides a transfer mechanism to better transfer sentiment across domains by incorporating BERT(Bidirextional Encoder Representations from Transformers) and KL (Kullback-Leibler) divergence and design a domain adaptive model using KL divergence to eliminate the difference of feature distribution between the source domain and target domain. Expand
GMAIR: Unsupervised Object Detection Based on Spatial Attention and Gaussian Mixture
TLDR
This paper presents a framework, GMAIR, for unsupervised object detection that incorporates spatial attention and a Gaussian mixture in a unified deep generative model and evaluates the model on MultiMNIST and Fruit2D datasets and shows that it achieves competitive results on localization and clustering compared to state-of-the-art methods. Expand
On the Latent Holes of VAEs for Text Generation
  • Ruizhe Li, Xutan Peng, Chenghua Lin
  • Computer Science
  • ArXiv
  • 2021
TLDR
This paper provides the first focused study on the discontinuities (aka. holes) in the latent space of Variational Auto-Encoders (VAEs), a phenomenon which has been shown to have a detrimental effect on model capacity, by proposing a highly efficient Tree-based Decoder-Centric (TDC) algorithm for latent hole identification. Expand
...
1
2
...

References

SHOWING 1-10 OF 47 REFERENCES
Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
TLDR
This paper investigates posterior collapse from the perspective of training dynamics and proposes an extremely simple modification to VAE training to reduce inference lag: depending on the model's current mutual information between latent variable and observation, the inference network is optimized before performing each model update. Expand
Importance Weighted Autoencoders
TLDR
The importance weighted autoencoder (IWAE), a generative model with the same architecture as the VAE, but which uses a strictly tighter log-likelihood lower bound derived from importance weighting, shows empirically that IWAEs learn richer latent space representations than VAEs, leading to improved test log- likelihood on density estimation benchmarks. Expand
Spherical Latent Spaces for Stable Variational Autoencoders
TLDR
This work experiments with another choice of latent distribution, namely the von Mises-Fisher (vMF) distribution, which places mass on the surface of the unit hypersphere and shows that they learn richer and more nuanced structures in their latent representations than their Gaussian counterparts. Expand
MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders
Variational Autoencoder (VAE), a simple and effective deep generative model, has led to a number of impressive empirical successes and spawned many advanced variants and theoretical investigations.Expand
A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text
TLDR
A simple fix for posterior collapse is investigated which yields surprisingly effective results and is used to argue that the typical surrogate objective for VAEs may not be sufficient or necessarily appropriate for balancing the goals of representation learning and data distribution modeling. Expand
VAE with a VampPrior
TLDR
This paper proposes to extend the variational auto-encoder (VAE) framework with a new type of prior called "Variational Mixture of Posteriors" prior, or VampPrior for short, which consists of a mixture distribution with components given by variational posteriors conditioned on learnable pseudo-inputs. Expand
Avoiding Latent Variable Collapse With Generative Skip Models
TLDR
Compared to existing VAE architectures, it is shown that generative skip models maintain similar predictive performance but lead to less collapse and provide more meaningful representations of the data. Expand
Ladder Variational Autoencoders
TLDR
A new inference model is proposed, the Ladder Variational Autoencoder, that recursively corrects the generative distribution by a data dependent approximate likelihood in a process resembling the recently proposed Ladder Network. Expand
InfoVAE: Information Maximizing Variational Autoencoders
TLDR
It is shown that this model can significantly improve the quality of the variational posterior and can make effective use of the latent features regardless of the flexibility of the decoding distribution, and it is demonstrated that the models outperform competing approaches on multiple performance metrics. Expand
Preventing Posterior Collapse with delta-VAEs
TLDR
This paper proposes an alternative that utilizes the most powerful generative models as decoders, whilst optimising the variational lower bound all while ensuring that the latent variables preserve and encode useful information. Expand
...
1
2
3
4
5
...