# Generating Sentences from a Continuous Space

@inproceedings{Bowman2016GeneratingSF,
title={Generating Sentences from a Continuous Space},
author={Samuel R. Bowman and L. Vilnis and Oriol Vinyals and Andrew M. Dai and R. J{\'o}zefowicz and Samy Bengio},
booktitle={CoNLL},
year={2016}
}
The standard recurrent neural network language model (RNNLM) generates sentences one word at a time and does not work from an explicit global sentence representation. [...] Key Method By examining paths through this latent space, we are able to generate coherent novel sentences that interpolate between known sentences. We present techniques for solving the difficult learning problem presented by this model, demonstrate its effectiveness in imputing missing words, explore many interesting properties of the model…Expand
1,492 Citations
A topic-driven language model for learning to generate diverse sentences
• Computer Science
• Neurocomputing
• 2019
A neural language model for generating diverse sentences conditioned on a given topic distribution that takes the advantages of variational autoencoders with convolutional neural network and long short-term memory architecture and decodes the samples of topics obtained from latent space to generate each sentence. Expand
Generating Sentences from Disentangled Syntactic and Semantic Spaces
The proposed method explicitly models syntactic information in the VAE’s latent space by using the linearized tree sequence, leading to better performance of language generation and the advantage of sampling in the disentangled syntactic and semantic latent spaces. Expand
Latent Space Expanded Variational Autoencoder for Sentence Generation
• Computer Science
• IEEE Access
• 2019
The experimental results show that the LSE-VAE improves the reconstruction ability of the VAE, generates plausible and more diverse sentences, and learns a larger informative latent space than theVAE with the properties of continuity and smoothness. Expand
An RNN Model for Generating Sentences with a Desired Word at a Desired Position
• Computer Science
• 2020
This work proposes a generative model of sentences named Coupled-RNN that employs two RNN's to generate sentences backwards and forwards respectively starting from a desired word, and inject position embeddings into the model to solve the problem of position information loss. Expand
Language as a Latent Variable: Discrete Generative Models for Sentence Compression
• Computer Science
• EMNLP
• 2016
This work forms a variational auto-encoder for inference in a deep generative model of text in which the latent representation of a document is itself drawn from a discrete language model distribution and shows that generative formulations of both abstractive and extractive compression yield state-of-the-art results when trained on a large amount of supervised data. Expand
Sentence Bottleneck Autoencoders from Transformer Language Models
Representation learning for text via pretraining a language model on a large corpus has become a standard starting point for building NLP systems. This approach stands in contrast to autoencoders,Expand
Language coverage and generalization in RNN-based continuous sentence embeddings for interacting agents
• Computer Science
• ArXiv
• 2019
It is found that RNN-based embeddings underfit the training data and cover only a small subset of the language defined by theCFG, and a new embedding method based on arithmetic coding, AriEL, that is not data-driven and that efficiently encodes in continuous space any sentence from the CFG is proposed. Expand
Discrete Autoencoders for Sequence Models
• Computer Science, Mathematics
• ArXiv
• 2018
This work proposes to improve the representation in sequence models by augmenting current approaches with an autoencoder that is forced to compress the sequence through an intermediate discrete latent space, and introduces an improved semantic hashing technique. Expand
Can Unconditional Language Models Recover Arbitrary Sentences?
• Computer Science
• NeurIPS
• 2019
This work introduces a pair of effective complementary methods for feeding representations into pretrained unconditional language models and a corresponding set of methods to map sentences into and out of this representation space, the \textit{reparametrized sentence space}. Expand
Syntax-Infused Variational Autoencoder for Text Generation
• Computer Science, Mathematics
• ACL
• 2019
A syntax-infused variational autoencoder (SIVAE), that integrates sentences with their syntactic trees to improve the grammar of generated sentences and can be used for unsupervised paraphrasing given different syntactic tree templates. Expand

#### References

SHOWING 1-10 OF 51 REFERENCES
A Hierarchical Neural Autoencoder for Paragraphs and Documents
• Computer Science
• ACL
• 2015
This paper introduces an LSTM model that hierarchically builds an embedding for a paragraph from embeddings for sentences and words, then decodes this embedding to reconstruct the original paragraph and evaluates the reconstructed paragraph using standard metrics to show that neural models are able to encode texts in a way that preserve syntactic, semantic, and discourse coherence. Expand
Skip-Thought Vectors
We describe an approach for unsupervised learning of a generic, distributed sentence encoder. Using the continuity of text from books, we train an encoder-decoder model that tries to reconstruct theExpand
• Computer Science
• IJCAI
• 2015
Both qualitative and quantitative analysis shows that AdaSent can automatically form and select the representations suitable for the task at hand during training, yielding superior classification performance over competitor models on 5 benchmark data sets. Expand
Neural Variational Inference for Text Processing
• Computer Science, Mathematics
• ICML
• 2016
This paper introduces a generic variational inference framework for generative and conditional models of text, and constructs an inference network conditioned on the discrete text input to provide the variational distribution. Expand
Deep Unordered Composition Rivals Syntactic Methods for Text Classification
• Computer Science
• ACL
• 2015
This work presents a simple deep neural network that competes with and, in some cases, outperforms such models on sentiment analysis and factoid question answering tasks while taking only a fraction of the training time. Expand
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
The m-RNN model directly models the probability distribution of generating a word given previous words and an image, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval. Expand
Sequence to Sequence Learning with Neural Networks
• Computer Science
• NIPS
• 2014
This paper presents a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure, and finds that reversing the order of the words in all source sentences improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier. Expand
A Recurrent Latent Variable Model for Sequential Data
• Computer Science
• NIPS
• 2015
It is argued that through the use of high-level latent random variables, the variational RNN (VRNN)1 can model the kind of variability observed in highly structured sequential data such as natural speech. Expand
Show and tell: A neural image caption generator
• Computer Science
• 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
• 2015
This paper presents a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. Expand
Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
The dynamic memory network (DMN), a neural network architecture which processes input sequences and questions, forms episodic memories, and generates relevant answers, is introduced. Expand