Tracking State Changes in Procedural Text: a Challenge Dataset and Models for Process Paragraph Comprehension

@inproceedings{Dalvi2018TrackingSC,
  title={Tracking State Changes in Procedural Text: a Challenge Dataset and Models for Process Paragraph Comprehension},
  author={Bhavana Dalvi and Lifu Huang and Niket Tandon and Wen-tau Yih and Peter Clark},
  booktitle={North American Chapter of the Association for Computational Linguistics},
  year={2018}
}
We present a new dataset and models for comprehending paragraphs about processes (e.g., photosynthesis), an important genre of text describing a dynamic world. [] Key Method We find that previous models that have worked well on synthetic data achieve only mediocre performance on ProPara, and introduce two new neural models that exploit alternative mechanisms for state prediction, in particular using LSTM input encoding and span prediction. The new models improve accuracy by up to 19%. The dataset and models…
[PDF] Semantic Reader
86 Citations
Highly Influential Citations
18
Background Citations
61
Methods Citations
27
Results Citations
4

Figures and Tables from this paper

86 Citations

Reasoning about Actions and State Changes by Injecting Commonsense Knowledge

This paper shows how the predicted effects of actions in the context of a paragraph can be improved in two ways: by incorporating global, commonsense constraints (e.g., a non-existent entity cannot be destroyed), and by biasing reading with preferences from large-scale corpora.

Understanding Procedural Text Using Interactive Entity Networks

This paper proposes a novel Interactive Entity Network (IEN), which is a recurrent network with memory equipped cells for state tracking that outperforms state-of-the-art models by precisely capturing the interactions of multiple entities and explicitly leverage the relationship between entity interactions and subsequent state changes.

Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Text

This work presents a new model (XPAD) that biases effect predictions towards those that explain more of the actions in the paragraph and are more plausible with respect to background knowledge, and extends an existing benchmark dataset for procedural text comprehension, ProPara, by adding the new task of explaining actions by predicting their dependencies.

Be Consistent! Improving Procedural Text Comprehension using Label Consistency

This work presents a new learning framework that leverages label consistency during training, allowing consistency bias to be built into the model and significantly improves prediction performance over prior state-of-the-art systems.

Effective Use of Transformer Networks for Entity Tracking

This paper tests standard lightweight approaches for prediction with pre-trained transformers, and finds that these approaches underperforms even simple baselines, and shows that much stronger results can be attained by restructuring the input to guide the model to focus on a particular entity.

What Happened? Leveraging VerbNet to Predict the Effects of Actions in Procedural Text

This work leverages VerbNet to build a rulebase of the preconditions and effects of actions, and uses it along with commonsense knowledge of persistence to answer questions about change in paragraphs describing processes.

Building Dynamic Knowledge Graphs from Text using Machine Reading Comprehension

A neural machine-reading model that constructs dynamic knowledge graphs recurrently for each step of the described procedure, and uses them to track the evolving states of participant entities to present some evidence that the model’s knowledge graphs help it to impose commonsense constraints on its predictions.

Knowledge-Aware Procedural Text Understanding with Multi-Stage Training

A novel KnOwledge-Aware proceduraL text understAnding (KoaLa) model is proposed, which effectively leverages multiple forms of external knowledge in this task of procedural text understanding and achieves state-of-the-art performance in comparison to various baselines.

Procedural Reading Comprehension with Attribute-Aware Context Flow

An algorithm for procedural reading comprehension is introduced by translating the text into a general formalism that represents processes as a sequence of transitions over entity attributes (e.g., location, temperature).

Predicting State Changes in Procedural Text using Analogical Question Answering

A novel approach that uses analogical question answering to predict what state changes affect entities in a paragraph describing a process, starting from the hypothesis that human level QA requires multiple layers of rich, relational representations.
...

References

SHOWING 1-10 OF 27 REFERENCES

What Happened? Leveraging VerbNet to Predict the Effects of Actions in Procedural Text

This work leverages VerbNet to build a rulebase of the preconditions and effects of actions, and uses it along with commonsense knowledge of persistence to answer questions about change in paragraphs describing processes.

Bidirectional Attention Flow for Machine Comprehension

The BIDAF network is introduced, a multi-stage hierarchical process that represents the context at different levels of granularity and uses bi-directional attention flow mechanism to obtain a query-aware context representation without early summarization.

Simple and Effective Multi-Paragraph Reading Comprehension

It is shown that it is possible to significantly improve performance by using a modified training scheme that teaches the model to ignore non-answer containing paragraphs, which involves sampling multiple paragraphs from each document, and using an objective function that requires themodel to produce globally correct output.

Constructing Datasets for Multi-hop Reading Comprehension Across Documents

A novel task to encourage the development of models for text understanding across multiple documents and to investigate the limits of existing methods, in which a model learns to seek and combine evidence — effectively performing multihop, alias multi-step, inference.

Modeling Biological Processes for Reading Comprehension

This paper focuses on a new reading comprehension task that requires complex reasoning over a single document, and demonstrates that answering questions via predicted structures substantially improves accuracy over baselines that use shallower representations.

Query-Reduction Networks for Question Answering

Query-Reduction Network (QRN), a variant of Recurrent Neural Network (RNN) that effectively handles both short-term and long-term sequential dependencies to reason over multiple facts, is proposed.

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

It is shown that, in comparison to other recently introduced large-scale datasets, TriviaQA has relatively complex, compositional questions, has considerable syntactic and lexical variability between questions and corresponding answer-evidence sentences, and requires more cross sentence reasoning to find answers.

A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task

A thorough examination of this new reading comprehension task by creating over a million training examples by pairing CNN and Daily Mail news articles with their summarized bullet points, and showing that a neural network can be trained to give good performance on this task.

SQuAD: 100,000+ Questions for Machine Comprehension of Text

A strong logistic regression model is built, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%).

Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks

This work argues for the usefulness of a set of proxy tasks that evaluate reading comprehension via question answering, and classify these tasks into skill sets so that researchers can identify (and then rectify) the failings of their systems.