Variational Inference for Learning Representations of Natural Language Edits

@article{MarreseTaylor2021VariationalIF,
  title={Variational Inference for Learning Representations of Natural Language Edits},
  author={Edison Marrese-Taylor and Machel Reid and Yutaka Matsuo},
  journal={ArXiv},
  year={2021},
  volume={abs/2004.09143}
}
Document editing has become a pervasive component of production of information, with version control systems enabling edits to be efficiently stored and applied. In light of this, the task of learning distributed representations of edits has been recently proposed. With this in mind, we propose a novel approach that employs variational inference to learn a continuous latent space of vector representations to capture the underlying semantic information with regard to the document editing process… 

Figures and Tables from this paper

Text Editing by Command

TLDR
This work proposes a novel text editing task, and introduces WikiDocEdits, a dataset of single-sentence edits crawled from Wikipedia, and shows that the Interactive Editor, a transformer-based model trained on this dataset, outperforms baselines and obtains positive results in both automatic and human evaluations.

Learning to Model Editing Processes

TLDR
This work proposes modeling editing processes, modeling the whole process of iteratively generating sequences, and forms a conceptual framework to describe the likelihood of multi-step edits, and describes neural models that can learn a generative model of sequences based on these multi- step edits.

Verba Volant, Scripta Volant: Understanding Post-publication Title Changes in News Outlets

TLDR
It is found that an effective headline post-publication edit should occur within the first ten hours after the initial release to ensure that the previous, potentially misleading, information does not fully propagate over the social network.

References

SHOWING 1-10 OF 51 REFERENCES

Learning to Represent Edits

We introduce the problem of learning distributed representations of edits. By combining a "neural editor" with an "edit encoder", our models learn to represent the salient information of an edit and

Generating Sentences from a Continuous Space

TLDR
This work introduces and study an RNN-based variational autoencoder generative model that incorporates distributed latent representations of entire sentences that allows it to explicitly model holistic properties of sentences such as style, topic, and high-level syntactic features.

Translator2Vec: Understanding and Representing Human Post-Editors

TLDR
This paper releases and analyzes a new dataset with document-level post-editing action sequences, including edit operations from keystrokes, mouse actions, and waiting times, and shows that action sequences are informative enough to identify post-Editors accurately, compared to baselines that only look at the initial and final text.

WikiAtomicEdits: A Multilingual Corpus of Wikipedia Edits for Modeling Language and Discourse

TLDR
A corpus of 43 million atomic edits across 8 languages is released to show that the language generated during editing differs from the language that is observed in standard corpora, and that models trained on edits encode different aspects of semantics and discourse than models training on raw text.

Generating Sentences by Editing Prototypes

TLDR
A new generative language model for sentences that first samples a prototype sentence from the training corpus and then edits it into a new sentence that gives rise to a latent edit vector that captures interpretable semantics such as sentence similarity and sentence-level analogies is proposed.

StRE: Self Attentive Edit Quality Prediction in Wikipedia

TLDR
This paper proposes Self Attentive Revision Encoder (StRE) which leverages orthographic similarity of lexical units toward predicting the quality of new edits and deploys deep encoders to generate representations of the edits from its text content, which they then leverage to infer quality.

Variational Neural Machine Translation

TLDR
This paper builds a neural posterior approximator conditioned on both the source and the target sides, and equip it with a reparameterization technique to estimate the variational lower bound, and shows that the proposed variational neural machine translation achieves significant improvements over the vanilla neural machinetranslation baselines.

A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text

TLDR
A simple fix for posterior collapse is investigated which yields surprisingly effective results and is used to argue that the typical surrogate objective for VAEs may not be sufficient or necessarily appropriate for balancing the goals of representation learning and data distribution modeling.

Variational Attention for Sequence-to-Sequence Models

TLDR
This paper proposes a variational attention mechanism for VED, where the attention vector is also modeled as Gaussian distributed random variables, and alleviates the bypassing phenomenon as it increases the diversity of generated sentences.

Auto-Encoding Variational Bayes

TLDR
A stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case is introduced.
...