Adversarial Decomposition of Text Representation

@article{Romanov2019AdversarialDO,
  title={Adversarial Decomposition of Text Representation},
  author={Alexey Romanov and Anna Rumshisky and Anna Rogers and David Donahue},
  journal={ArXiv},
  year={2019},
  volume={abs/1808.09042}
}
In this paper, we present a method for adversarial decomposition of text representation. [] Key Method The model uses adversarial-motivational training and includes a special motivational loss, which acts opposite to the discriminator and encourages a better decomposition. Furthermore, we evaluate the obtained meaning embeddings on a downstream task of paraphrase detection and show that they significantly outperform the embeddings of a regular autoencoder.

Figures and Tables from this paper

An Evaluation of Disentangled Representation Learning for Texts
TLDR
A systematic investigation of previously proposed loss functions for such models are conducted and they are evaluated on a highly-structured and synthetic natural language dataset that is well-suited for the task of disentangled representation learning, as well as two other parallel style transfer datasets.
Multi-type Disentanglement without Adversarial Training
TLDR
A unified distribution-controlling method, which provides each specific style value (the value of style types, e.g., positive sentiment, or past tense) with a unique representation, which contributes a solid theoretical basis to avoid adversarial training in multi-type disentanglement.
Self-conditioning pre-trained language models
TLDR
It is found that the activation of a surprisingly small amount of units is sufficient to steer text generation (as little as 3 units in a model with 345M parameters) and the approach is able to achieve gender parity at a lower perplexity.
Deep Learning for Text Style Transfer: A Survey
TLDR
A systematic survey of the research on neuralText style transfer, spanning over 100 representative articles since the first neural text style transfer work in 2017, is presented.
Style versus Content: A distinction without a (learnable) difference?
TLDR
The results of the experiments reveal the inherent trade-off between the multiple style transfer objectives which indicates that style cannot be usefully separated from content within these style-transfer systems.
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
TLDR
An end-to-end neural model for zero-shot abstractive text summarization of paragraphs, and a benchmark task, ROCSumm, based on ROCStories, a subset for which human summaries were collected, is introduced.
Style Transfer for Texts: Retrain, Report Errors, Compare with Rewrites
TLDR
This paper shows that standard assessment methodology for style transfer has several significant problems, and suggests taking BLEU between input and human-written reformulations into consideration for benchmarks, and proposes three new architectures that outperform state of the art in terms of this metric.
Deep Learning for Text Attribute Transfer: A Survey.
TLDR
This article collects all related academic works since the first appearance in 2017, and reveals that existing methods are indeed based on a combination of several loss functions with each of which serving a certain goal.
Attribute Alignment: Controlling Text Generation from Pre-trained Language Models
TLDR
This work proposes a simple and flexible method for controlling text generation by aligning disentangled attribute representations, and shows large performance gains over previous methods while retaining fluency and diversity.
Continual Learning for Text Classification with Information Disentanglement Based Regularization
TLDR
This work proposes an information disentanglement based regularization method for continual learning on text classification that first disentangles text hidden spaces into representations that are generic to all tasks and representations specific to each individual task, and further regularizes these representations differently to better constrain the knowledge required to generalize.
...
...

References

SHOWING 1-10 OF 47 REFERENCES
Disentangled Representation Learning for Text Style Transfer
TLDR
This paper proposes a simple, yet effective approach, which incorporates auxiliary objectives: a multi-task classification objective, and dual adversarial objectives for label prediction and bag-ofwords prediction, respectively, which shows that the style and content are indeed disentangled in the latent space, using this approach.
Style Transfer in Text: Exploration and Evaluation
TLDR
Two models are explored to learn style transfer with non-parallel data to learn separate content representations and style representations using adversarial networks, and a novel evaluation metrics which measure two aspects of style transfer: transfer strength and content preservation.
Disentangled Representation Learning for Non-Parallel Text Style Transfer
TLDR
A simple yet effective approach is proposed, which incorporates auxiliary multi-task and adversarial objectives, for style prediction and bag-of-words prediction, respectively, and this disentangled latent representation learning can be applied to style transfer on non-parallel corpora.
Toward Controlled Generation of Text
TLDR
A new neural generative model is proposed which combines variational auto-encoders and holistic attribute discriminators for effective imposition of semantic structures inGeneric generation and manipulation of text.
Adversarially Regularized Autoencoders
TLDR
This work proposes a flexible method for training deep latent variable models of discrete structures based on the recently-proposed Wasserstein autoencoder (WAE), and shows that the latent representation can be trained to perform unaligned textual style transfer, giving improvements both in automatic/human evaluation compared to existing methods.
A Simple but Tough-to-Beat Baseline for Sentence Embeddings
Adversarial Autoencoders
TLDR
This paper shows how the adversarial autoencoder can be used in applications such as semi-supervised classification, disentangling style and content of images, unsupervised clustering, dimensionality reduction and data visualization, and performed experiments on MNIST, Street View House Numbers and Toronto Face datasets.
Adversarially Regularized Autoencoders for Generating Discrete Structures
TLDR
This work considers a simple approach for handling these two challenges jointly, employing a discrete structure autoencoder with a code space regularized by generative adversarial training, and demonstrates empirically how key properties of the data are captured in the model's latent space.
An efficient framework for learning sentence representations
TLDR
This work reformulates the problem of predicting the context in which a sentence appears as a classification problem, and proposes a simple and efficient framework for learning sentence representations from unlabelled data.
Context-aware Natural Language Generation with Recurrent Neural Networks
TLDR
Two novel approaches are proposed which encode the contexts into a continuous semantic representation and then decode the semantic representation into text sequences with recurrent neural networks, addressing the problem of long-range dependency caused by lengthy sequences.
...
...