• Corpus ID: 220301599

Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge

@article{Verga2020FactsAE,
  title={Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge},
  author={Pat Verga and Haitian Sun and Livio Baldini Soares and William W. Cohen},
  journal={ArXiv},
  year={2020},
  volume={abs/2007.00849}
}
Massive language models are the core of modern NLP modeling and have been shown to encode impressive amounts of commonsense and factual information. However, that knowledge exists only within the latent parameters of the model, inaccessible to inspection and interpretation, and even worse, factual information memorized from the training corpora is likely to become stale as the world changes. Knowledge stored as parameters will also inevitably exhibit all of the biases inherent in the source… 

Figures and Tables from this paper

Modifying Memories in Transformer Models
TLDR
This paper proposes a new task of explicitly modifying specific factual knowledge in Transformer models while ensuring the model performance does not degrade on the unmodified facts, and benchmarked several approaches that provide natural baseline performances on this task.
Relational World Knowledge Representation in Contextual Language Models: A Review
TLDR
This work proposes to organize knowledge representation strategies in LMs by the level of KB supervision provided, from no KB supervision at all to entity- and relation-level supervision, and provides a high-level, extensible taxonomy for knowledge representation in L Ms.
CoCoLM: Complex Commonsense Enhanced Language Model with Discourse Relations
TLDR
This paper proposes a general language model named CoCoLM, which through the careful training over a large-scale eventuality knowledge graph ASER, successfully teaches pre-trained language models (i.e., BERT and RoBERTa) rich multi-hop commonsense knowledge among eventualities.
CoCoLM: COmplex COmmonsense Enhanced Language Model
TLDR
Through the careful training over a large-scale eventuality knowledge graphs ASER, the proposed general language model CoCoLM successfully teaches pre-trained language models (i.e., BERT and RoBERTa) rich complex commonsense knowledge among eventualities.
Does entity abstraction help generative Transformers reason?
TLDR
The results suggest that the benefit of explicit abstraction is significant in formally defined logical reasoning settings requiring many reasoning hops, but point to the notion that it is less beneficial for NLP tasks having less formal logical structure.
Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey
TLDR
A comprehensive survey of the literature on this emerging and fast-growing field Knowledge Enhanced Pretrained Language Models (KE-PLMs) is provided and three taxonomies are introduced to categorize existing work.
Question Answering over Knowledge Bases by Leveraging Semantic Parsing and Neuro-Symbolic Reasoning
TLDR
A semantic parsing and reasoning-based Neuro-Symbolic Question Answering system that achieves state-of-the-art performance on QALD-9 and LC-QuAD 1.0 and integrates multiple, reusable modules that are trained specifically for their individual tasks and do not require end-to-end training data.
JAKET: Joint Pre-training of Knowledge Graph and Language Understanding
TLDR
A novel joint pre-training framework, JAKET, to model both the knowledge graph and language, which enables the pre-trained model to easily adapt to unseen knowledge graphs in new domains.
...
1
2
3
4
...

References

SHOWING 1-10 OF 48 REFERENCES
Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling
TLDR
This work introduces the knowledge graph language model (KGLM), a neural language model with mechanisms for selecting and copying facts from a knowledge graph that are relevant to the context that enable the model to render information it has never seen before, as well as generate out-of-vocabulary tokens.
Language Models as Knowledge Bases?
TLDR
An in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-the-art pretrained language models finds that BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge.
Guessing What's Plausible But Remembering What's True: Accurate Neural Reasoning for Question-Answering
TLDR
This work proposes here a novel KB embedding scheme that supports generalization, but also allows accurate logical reasoning with a KB, and introduces two new mechanisms for KB reasoning: neural retrieval over a set of embedded triples, and “memorization” of highly specific information with a compact sketch structure.
REALM: Retrieval-Augmented Language Model Pre-Training
TLDR
The effectiveness of Retrieval-Augmented Language Model pre-training (REALM) is demonstrated by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA) and is found to outperform all previous methods by a significant margin, while also providing qualitative benefits such as interpretability and modularity.
Memory Networks
TLDR
This work describes a new class of learning models called memory networks, which reason with inference components combined with a long-term memory component; they learn how to use these jointly.
Differentiable Reasoning over a Virtual Knowledge Base
TLDR
A neural module, DrKIT, that traverses textual data like a virtual KB, softly following paths of relations between mentions of entities in the corpus, which improves accuracy by 9 points on 3-hop questions in the MetaQA dataset and is very efficient.
Scalable Neural Methods for Reasoning With a Symbolic Knowledge Base
We describe a novel way of representing a symbolic knowledge base (KB) called a sparse-matrix reified KB. This representation enables neural modules that are fully differentiable, faithful to the
Key-Value Memory Networks for Directly Reading Documents
TLDR
This work introduces a new method, Key-Value Memory Networks, that makes reading documents more viable by utilizing different encodings in the addressing and output stages of the memory read operation.
Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text
TLDR
A novel model is proposed, GRAFT-Net, for extracting answers from a question-specific subgraph containing text and Knowledge Bases entities and relations that is competitive with the state-of-the-art when tested using either KBs or text alone, and vastly outperforms existing methods in the combined setting.
Tracking the World State with Recurrent Entity Networks
TLDR
The EntNet sets a new state-of-the-art on the bAbI tasks, and is the first method to solve all the tasks in the 10k training examples setting, and can generalize past its training horizon.
...
1
2
3
4
5
...