• Corpus ID: 189928477

Fixing Gaussian Mixture VAEs for Interpretable Text Generation

  title={Fixing Gaussian Mixture VAEs for Interpretable Text Generation},
  author={Wenxian Shi and Hao Zhou and Ning Miao and Shenjian Zhao and Lei Li},
Variational auto-encoder (VAE) with Gaussian priors is effective in text generation. To improve the controllability and interpretability, we propose to use Gaussian mixture distribution as the prior for VAE (GMVAE), since it includes an extra discrete latent variable in addition to the continuous one. Unfortunately, training GMVAE using standard variational approximation often leads to the mode-collapse problem. We theoretically analyze the root cause --- maximizing the evidence lower bound of… 

Figures and Tables from this paper

Variational Auto-Encoder for text generation
The Variational Auto-Encoder Recurrent Neural Network (VAE-RNNLM) is proposed, which designed to explicitly capture such global features as continuous latent variable in neural network generative models with latent variables.
APo-VAE: Text Generation in Hyperbolic Space
Extensive experiments in language modeling, unaligned style transfer, and dialog-response generation demonstrate the effectiveness of the proposed APo-VAE model over VAEs in Euclidean latent space, thanks to its superb capabilities in capturing latent language hierarchies in hyperbolic space.
Regularizing Transformers With Deep Probabilistic Layers
This work uses a Gaussian Mixture Variational Autoencoder (GMVAE) as a regularizer layer and proves its effectiveness not only in Transformers but also in the most relevant encoder-decoder based LM, seq2seq with and without attention.
FD-VAE: A Feature Driven VAE Architecture for Flexible Synthetic Data Generation
A novel VAE architecture that uses Gaussian Mixture Models to structure the latent space into meaningful partitions, and it allows us to generate data with any desired combination of features, even when that specific combination has been never seen in the training examples is presented.
Stylized Text Generation: Approaches and Applications
This tutorial delves deep into machine learning methods, including embedding learning techniques to represent style, adversarial learning, and reinforcement learning with cycle consistency to match content but to distinguish different styles.
Interpretable Embeddings From Molecular Simulations Using Gaussian Mixture Variational Autoencoders
This work incorporates physical intuition into the prior by employing a Gaussian mixture variational autoencoder (GMVAE), which encourages the separation of metastable states within the embedding, and demonstrates the enhanced clustering effect of the GMVAE prior compared to standard VAEs.
Detecting Outlier Machine Instances Through Gaussian Mixture Variational Autoencoder With One Dimensional CNN
DOMI is proposed, a novel unsupervised model that combines Gaussian mixture VAE with 1D-CNN, to detect outlier machine instances that captures the normal patterns of machine instances by learning their latent representations that consider the shape characteristics, reconstruct input data by the learned representations, and apply reconstruction probabilities to determine outliers.
Database and Expert Systems Applications: 31st International Conference, DEXA 2020, Bratislava, Slovakia, September 14–17, 2020, Proceedings, Part I
Details of Keynote Talks Knowledge Graph for Drug Discovery are presented, which aims to provide real-time information about the pharmacological properties of various drugs and provide clues to the development of new drugs.


Topic-Guided Variational Auto-Encoder for Text Generation
Experimental results show that the TGVAE outperforms its competitors on both unconditional and conditional text generation, which can also generate semantically-meaningful sentences with various topics.
A Hybrid Convolutional Variational Autoencoder for Text Generation
A novel hybrid architecture that blends fully feed-forward convolutional and deconvolutional components with a recurrent language model is proposed that helps to avoid the issue of the VAE collapsing to a deterministic model.
Neural Variational Inference for Text Processing
This paper introduces a generic variational inference framework for generative and conditional models of text, and constructs an inference network conditioned on the discrete text input to provide the variational distribution.
Spherical Latent Spaces for Stable Variational Autoencoders
This work experiments with another choice of latent distribution, namely the von Mises-Fisher (vMF) distribution, which places mass on the surface of the unit hypersphere and shows that they learn richer and more nuanced structures in their latent representations than their Gaussian counterparts.
Deep Recurrent Generative Decoder for Abstractive Text Summarization
A new framework for abstractive text summarization based on a sequence-to-sequence oriented encoder-decoder model equipped with a deep recurrent generative decoder (DRGN) achieves improvements over the state-of-the-art methods.
Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders
This work presents a novel framework based on conditional variational autoencoders that capture the discourse-level diversity in the encoder and uses latent variables to learn a distribution over potential conversational intents and generates diverse responses using only greedy decoders.
Toward Controlled Generation of Text
A new neural generative model is proposed which combines variational auto-encoders and holistic attribute discriminators for effective imposition of semantic structures inGeneric generation and manipulation of text.
Adversarially Regularized Autoencoders
This work proposes a flexible method for training deep latent variable models of discrete structures based on the recently-proposed Wasserstein autoencoder (WAE), and shows that the latent representation can be trained to perform unaligned textual style transfer, giving improvements both in automatic/human evaluation compared to existing methods.
Unsupervised Discrete Sentence Representation Learning for Interpretable Neural Dialog Generation
Two novel models are presented, DI-VAE and DI-VST, that improve VAEs and can discover interpretable semantics via either auto encoding or context predicting and enhance encoder-decoder models with interpretable generation.
Generating Sentences from a Continuous Space
This work introduces and study an RNN-based variational autoencoder generative model that incorporates distributed latent representations of entire sentences that allows it to explicitly model holistic properties of sentences such as style, topic, and high-level syntactic features.