# Generating Sentences from a Continuous Space

@inproceedings{Bowman2016GeneratingSF,
title={Generating Sentences from a Continuous Space},
author={Samuel R. Bowman and Luke Vilnis and Oriol Vinyals and Andrew M. Dai and Rafal J{\'o}zefowicz and Samy Bengio},
booktitle={CoNLL},
year={2016}
}
• Published in CoNLL 19 November 2015
• Computer Science
The standard recurrent neural network language model (RNNLM) generates sentences one word at a time and does not work from an explicit global sentence representation. [] Key Method By examining paths through this latent space, we are able to generate coherent novel sentences that interpolate between known sentences. We present techniques for solving the difficult learning problem presented by this model, demonstrate its effectiveness in imputing missing words, explore many interesting properties of the model…
1,715 Citations

## Figures and Tables from this paper

Generating Sentences from Disentangled Syntactic and Semantic Spaces
• Computer Science
ACL
• 2019
The proposed method explicitly models syntactic information in the VAE’s latent space by using the linearized tree sequence, leading to better performance of language generation and the advantage of sampling in the disentangled syntactic and semantic latent spaces.
Latent Space Expanded Variational Autoencoder for Sentence Generation
• Computer Science
IEEE Access
• 2019
The experimental results show that the LSE-VAE improves the reconstruction ability of the VAE, generates plausible and more diverse sentences, and learns a larger informative latent space than theVAE with the properties of continuity and smoothness.
An RNN Model for Generating Sentences with a Desired Word at a Desired Position
• Computer Science
• 2020
This work proposes a generative model of sentences named Coupled-RNN that employs two RNN's to generate sentences backwards and forwards respectively starting from a desired word, and inject position embeddings into the model to solve the problem of position information loss.
Language as a Latent Variable: Discrete Generative Models for Sentence Compression
• Computer Science
EMNLP
• 2016
This work forms a variational auto-encoder for inference in a deep generative model of text in which the latent representation of a document is itself drawn from a discrete language model distribution and shows that generative formulations of both abstractive and extractive compression yield state-of-the-art results when trained on a large amount of supervised data.
Sentence Bottleneck Autoencoders from Transformer Language Models
• Computer Science
EMNLP
• 2021
The construction of a sentence-level autoencoder from a pretrained, frozen transformer language model that achieves better quality than previous methods that extract representations from pretrained transformers on text similarity tasks, style transfer, and single-sentence classification tasks in the GLUE benchmark, while using fewer parameters than large pretrained models.
Language coverage and generalization in RNN-based continuous sentence embeddings for interacting agents
• Computer Science
ArXiv
• 2019
It is found that RNN-based embeddings underfit the training data and cover only a small subset of the language defined by theCFG, and a new embedding method based on arithmetic coding, AriEL, that is not data-driven and that efficiently encodes in continuous space any sentence from the CFG is proposed.
Discrete Autoencoders for Sequence Models
• Computer Science
ArXiv
• 2018
This work proposes to improve the representation in sequence models by augmenting current approaches with an autoencoder that is forced to compress the sequence through an intermediate discrete latent space, and introduces an improved semantic hashing technique.
Can Unconditional Language Models Recover Arbitrary Sentences?
• Computer Science
NeurIPS
• 2019
This work introduces a pair of effective complementary methods for feeding representations into pretrained unconditional language models and a corresponding set of methods to map sentences into and out of this representation space, the \textit{reparametrized sentence space}.
Syntax-Infused Variational Autoencoder for Text Generation
• Computer Science
ACL
• 2019
A syntax-infused variational autoencoder (SIVAE), that integrates sentences with their syntactic trees to improve the grammar of generated sentences and can be used for unsupervised paraphrasing given different syntactic tree templates.

## References

SHOWING 1-10 OF 51 REFERENCES
A Hierarchical Neural Autoencoder for Paragraphs and Documents
• Computer Science
ACL
• 2015
This paper introduces an LSTM model that hierarchically builds an embedding for a paragraph from embeddings for sentences and words, then decodes this embedding to reconstruct the original paragraph and evaluates the reconstructed paragraph using standard metrics to show that neural models are able to encode texts in a way that preserve syntactic, semantic, and discourse coherence.
Skip-Thought Vectors
• Computer Science
NIPS
• 2015
We describe an approach for unsupervised learning of a generic, distributed sentence encoder. Using the continuity of text from books, we train an encoder-decoder model that tries to reconstruct the
• Computer Science
IJCAI
• 2015
Both qualitative and quantitative analysis shows that AdaSent can automatically form and select the representations suitable for the task at hand during training, yielding superior classification performance over competitor models on 5 benchmark data sets.
Neural Variational Inference for Text Processing
• Computer Science
ICML
• 2016
This paper introduces a generic variational inference framework for generative and conditional models of text, and constructs an inference network conditioned on the discrete text input to provide the variational distribution.
Deep Unordered Composition Rivals Syntactic Methods for Text Classification
• Computer Science
ACL
• 2015
This work presents a simple deep neural network that competes with and, in some cases, outperforms such models on sentiment analysis and factoid question answering tasks while taking only a fraction of the training time.
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Computer Science
ICLR
• 2015
The m-RNN model directly models the probability distribution of generating a word given previous words and an image, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval.
Sequence to Sequence Learning with Neural Networks
• Computer Science
NIPS
• 2014
This paper presents a general end-to-end approach to sequence learning that makes minimal assumptions on the sequence structure, and finds that reversing the order of the words in all source sentences improved the LSTM's performance markedly, because doing so introduced many short term dependencies between the source and the target sentence which made the optimization problem easier.
A Recurrent Latent Variable Model for Sequential Data
• Computer Science
NIPS
• 2015
It is argued that through the use of high-level latent random variables, the variational RNN (VRNN)1 can model the kind of variability observed in highly structured sequential data such as natural speech.
Show and tell: A neural image caption generator
• Computer Science
2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
• 2015
This paper presents a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image.
Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• Computer Science
ICML
• 2016
The dynamic memory network (DMN), a neural network architecture which processes input sequences and questions, forms episodic memories, and generates relevant answers, is introduced.