Leveraging Knowledge Bases in LSTMs for Improving Machine Reading

@inproceedings{Yang2017LeveragingKB,
  title={Leveraging Knowledge Bases in LSTMs for Improving Machine Reading},
  author={Bishan Yang and Tom Michael Mitchell},
  booktitle={ACL},
  year={2017}
}
This paper focuses on how to take advantage of external knowledge bases (KBs) to improve recurrent neural networks for machine reading. [] Key Method To effectively integrate background knowledge with information from the currently processed text, our model employs an attention mechanism with a sentinel to adaptively decide whether to attend to background knowledge and which information from KBs is useful. Experimental results show that our model achieves accuracies that surpass the previous state-of-the-art…

Figures and Tables from this paper

Towards Knowledge Enhanced Language Model for Machine Reading Comprehension
TLDR
A novel deep model KCF-NET is proposed, which incorporates knowledge graph representations with context as the basis for predicting answers by leveraging capsule network to encode the intrinsic spatial relationship in triples of KG.
Knowledge Adaptive Neural Network for Natural Language Inference
TLDR
This paper proposes knowledge adaptive neural network (KANN) that adaptively incorporates commonsense knowledge at sentence encoding and inference stages, and is comparable to if not better than the recent neural network based approaches on NLI.
Learning to Tag OOV Tokens by Integrating Contextual Representation and Background Knowledge
TLDR
A novel knowledge-enhanced slot tagging model to integrate contextual representation of input text and the large-scale lexical background knowledge and uses multi-level graph attention to explicitly model lexical relations is proposed.
Knowledge-Aware and Retrieval-Based Models for Distantly Supervised Relation Extraction
TLDR
A knowledge-aware attention model is proposed, which can leverage the semantic knowledge in the KB to select the valid sentences and formalize distantly supervised RE as relation retrieval instead of relation classification to leverage the Semantic knowledge further.
KW-ATTN: Knowledge Infused Attention for Accurate and Interpretable Text Classification
TLDR
It is shown that KW-ATTN outperforms baseline models using only words as well as other approaches using concepts by classification accuracy, which indicates that high-level concepts help model prediction.
Incorporating External Knowledge into Machine Reading for Generative Question Answering
TLDR
A new neural model, Knowledge-Enriched Answer Generator (KEAG), which is able to compose a natural answer by exploiting and aggregating evidence from all four information sources available: question, passage, vocabulary and knowledge.
Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension
TLDR
This work introduces KT-NET, which employs an attention mechanism to adaptively select desired knowledge from KBs, and then fuses selected knowledge with BERT to enable context- and knowledge-aware predictions.
VIRTUAL KNOWLEDGE BASE
TLDR
A neural module, DrKIT, that traverses textual data like a KB, softly following paths of relations between mentions of entities in the corpus, cutting the gap between text-based and KB-based state-of-the-art by 70%.
Contextualized Knowledge-aware Attentive Neural Network: Enhancing Answer Selection with Knowledge
TLDR
A context-knowledge interaction learning framework, Knowledge-aware Neural Network, which learns the QA sentence representations by considering a tight interaction with the external knowledge from KG and the textual information and develops two kinds of knowledge-aware attention mechanism to summarize both the context-based and knowledge-based interactions between questions and answers.
...
...

References

SHOWING 1-10 OF 51 REFERENCES
Learning Structured Embeddings of Knowledge Bases
TLDR
A learning process based on an innovative neural network architecture designed to embed any of these symbolic representations into a more flexible continuous vector space in which the original knowledge is kept and enhanced would allow data from any KB to be easily used in recent machine learning methods for prediction and information retrieval.
Key-Value Memory Networks for Directly Reading Documents
TLDR
This work introduces a new method, Key-Value Memory Networks, that makes reading documents more viable by utilizing different encodings in the addressing and output stages of the memory read operation.
A Neural Knowledge Language Model
TLDR
A Neural Knowledge Language Model (NKLM) which combines symbolic knowledge provided by a knowledge graph with the RNN language model, and shows that the NKLM significantly improves the perplexity while generating a much smaller number of unknown words.
End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures
TLDR
A novel end-to-end neural model to extract entities and relations between them and compares favorably to the state-of-the-art CNN based model (in F1-score) on nominal relation classification (SemEval-2010 Task 8).
End-to-end learning of semantic role labeling using recurrent neural networks
TLDR
This work proposes to use deep bi-directional recurrent network as an end-to-end system for SRL, which takes only original text information as input feature, without using any syntactic knowledge.
Named Entity Recognition with Bidirectional LSTM-CNNs
TLDR
A novel neural network architecture is presented that automatically detects word- and character-level features using a hybrid bidirectional LSTM and CNN architecture, eliminating the need for most feature engineering.
Connecting Language and Knowledge Bases with Embedding Models for Relation Extraction
This paper proposes a novel approach for relation extraction from free text which is trained to jointly use information from the text and from existing knowledge. Our model is based on scoring
Event Extraction via Dynamic Multi-Pooling Convolutional Neural Networks
TLDR
A word-representation model to capture meaningful semantic regularities for words and a framework based on a convolutional neural network to capture sentence-level clues are introduced.
Reasoning With Neural Tensor Networks for Knowledge Base Completion
TLDR
An expressive neural tensor network suitable for reasoning over relationships between two entities given a subset of the knowledge base is introduced and performance can be improved when entities are represented as an average of their constituting word vectors.
Towards Universal Paraphrastic Sentence Embeddings
TLDR
This work considers the problem of learning general-purpose, paraphrastic sentence embeddings based on supervision from the Paraphrase Database, and compares six compositional architectures, finding that the most complex architectures, such as long short-term memory (LSTM) recurrent neural networks, perform best on the in-domain data.
...
...