Discourse in Multimedia: A Case Study in Extracting Geometry Knowledge from Textbooks

@article{Sachan2019DiscourseIM,
  title={Discourse in Multimedia: A Case Study in Extracting Geometry Knowledge from Textbooks},
  author={Mrinmaya Sachan and Kumar Avinava Dubey and Eduard H. Hovy and Tom Michael Mitchell and Dan Roth and Eric P. Xing},
  journal={Computational Linguistics},
  year={2019},
  volume={Just Accepted},
  pages={1-35}
}
To ensure readability, text is often written and presented with due formatting. These text formatting devices help the writer to effectively convey the narrative. At the same time, these help the readers pick up the structure of the discourse and comprehend the conveyed information. There have been a number of linguistic theories on discourse structure of text. However, these theories only consider unformatted text. Multimedia text contains rich formatting features which can be leveraged for… Expand
Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning
TLDR
This work constructs a new largescale benchmark, Geometry3K, consisting of 3,002 geometry problems with dense annotation in formal language, and proposes a novel geometry solving approach with formal language and symbolic reasoning, called Interpretable Geometry Problem Solver (InterGPS). Expand
AI2D-RST: A multimodal corpus of 1000 primary school science diagrams
TLDR
A multi-layer annotation schema that provides a rich description of diagram elements into perceptual units, the connections set up by diagrammatic elements such as arrows and lines, and the discourse relations between diagram elements, which are described using Rhetorical Structure Theory (RST). Expand
Classifying Diagrams and Their Parts using Graph Neural Networks: A Comparison of Crowd-Sourced and Expert Annotations
TLDR
The results show that the identity of diagram elements can be learned from their layout features, while the expert annotations provide better representations of diagram types. Expand
Emerging materials intelligence ecosystems propelled by machine learning
The age of cognitive computing and artificial intelligence (AI) is just dawning. Inspired by its successes and promises, several AI ecosystems are blossoming, many of them within the domain ofExpand
GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning
TLDR
A Neural Geometric Solver (NGS) is introduced to address geometric problems by comprehensively parsing multimodal information and generating interpretable programs, and multiple self-supervised auxiliary tasks on NGS are added to enhance cross-modal semantic representation. Expand

References

SHOWING 1-10 OF 187 REFERENCES
Integrating Text Formatting and Text Generation
TLDR
This paper presents a model for representing the architecture of documents for natural language generation from a specialized sublanguage (in the sense of Z. Harris) of natural language, and proposes a brief survey of a system of formatted text generation, based on this model. Expand
Automatic Generation of Formatted Text
TLDR
This work describes how work on the automated planning of multisentence text and on the display of information in a multimedia system led to the insight that text formatting devices such as footnotes, italicized regions, enumerations, etc., can be planned automatically by a text structure planning process. Expand
Discourse indicators for content selection in summarization
TLDR
The results establish the usefulness of discourse features and find that lexical overlap provides a simple and cheap alternative to discourse for computing text structure with comparable performance for the task of content selection. Expand
Towards Constructive Text, Diagram, and Layout Generation for Information Presentation
TLDR
It is demonstrated that layout offers a rich resource for achieving presentational coherence, alongside more traditional resources such as text-formatting and the text-internal marking of discourse connections, and an integrated approach to layout, text, and diagram generation is introduced. Expand
Pattern Matching and Discourse Processing in Information Extraction from Japanese Text
TLDR
A Japanese information extraction system that merges information using a pattern matcher and discourse processor that approaches human performance is reported on. Expand
AL FRESCO: Enjoying The Combination of NLP and Hypermedia for Information Exploration
Integration of natural language with other communicative modalities has a number of different motivations. One is that there are things that in absolute terms, "in nature", humans best communicateExpand
Rhetorical relations for information retrieval
TLDR
A language model modification is presented that considers rhetorical relations when estimating the relevance of a document to a query and shows that certain rhetorical relations can benefit retrieval effectiveness notably. Expand
Discourse segmentation in aid of document summarization
  • B. Boguraev, Mary S. Neff
  • Computer Science
  • Proceedings of the 33rd Annual Hawaii International Conference on System Sciences
  • 2000
TLDR
Evaluated against the corpus used in the development of the baseline summarizer, summaries derived either by means of segmentation analysis alone, or by a mix of strategies for combining salience calculation and topic shift detection, are shown to be of comparable, and under certain conditions even better quality. Expand
Recalling and Summarizing Complex Discourse
In this paper we investigate the properties of complex semantic information processing involved in the comprehension, (re-)production and summarizing of longer narrative discourse. In the theoreticalExpand
Structure and Rules in Automated Multimedia Presentation Planning
TLDR
A prototype planning system that performs the information-to-media allocation is described, arguing that since media allocation rules depend on the characteristics of the information to be presented, they can only be applied once the overall discourse structure has been essentially planned out and the individual portions of information have become apparent. Expand
...
1
2
3
4
5
...