Language Models as Knowledge Bases?

@inproceedings{Petroni2019LanguageMA,
  title={Language Models as Knowledge Bases?},
  author={Fabio Petroni and Tim Rockt{\"a}schel and Patrick Lewis and A. Bakhtin and Yuxiang Wu and Alexander H. Miller and S. Riedel},
  booktitle={EMNLP},
  year={2019}
}
Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models may also be storing relational knowledge present in the training data, and may be able to answer queries structured as “fill-in-the-blank” cloze statements. Language models have many advantages over structured knowledge bases: they require no schema engineering, allow practitioners to query about an open class of… Expand
Prompt Tuning or Fine-Tuning-Investigating Relational Knowledge in Pre-Trained Language Models
  • 2021
Extracting relational knowledge from large pre-trained language models by a clozestyle sentence serving as a query has shown promising results. In particular, language models can be queried similarExpand
REALM: Retrieval-Augmented Language Model Pre-Training
TLDR
The effectiveness of Retrieval-Augmented Language Model pre-training (REALM) is demonstrated by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA) and is found to outperform all previous methods by a significant margin, while also providing qualitative benefits such as interpretability and modularity. Expand
Language Models are Open Knowledge Graphs
TLDR
This paper shows how to construct knowledge graphs (KGs) from pre-trained language models (e.g., BERT, GPT-2/3), without human supervision, and proposes an unsupervised method to cast the knowledge contained within language models into KGs. Expand
How Context Affects Language Models' Factual Predictions
TLDR
This paper reports that augmenting pre-trained language models in this way dramatically improves performance and that the resulting system, despite being unsupervised, is competitive with a supervised machine reading baseline. Expand
Relational world knowledge representation in contextual language models: A review
TLDR
It is concluded that LMs and KBs are complementary representation tools, as KBs provide a high standard of factual precision which can in turn be flexibly and expressively modeled by LMs, and provide suggestions for future research in this direction. Expand
KnowlyBERT - Hybrid Query Answering over Language Models and Knowledge Graphs
TLDR
This work creates a hybrid query answering system on top of knowledge graphs in combination with the masked language model BERT to achieve high precision query results, which outperforms state-of-the-art techniques by boosting their precision by over 30% in the authors' large Wikidata experiment. Expand
Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
TLDR
This work proposes a simple yet effective weakly supervised pretraining objective, which explicitly forces the model to incorporate knowledge about real-world entities, and consistently outperforms BERT on four entity-related question answering datasets. Expand
K-XLNet: A General Method for Combining Explicit Knowledge with Language Model Pretraining
TLDR
The experimental results show that solely by adding external knowledge to transformer can improve the learning performance on many NLP tasks. Expand
Static Embeddings as Efficient Knowledge Bases?
TLDR
It is shown that, when restricting the output space to a candidate set, simple nearest neighbor matching using static embeddings performs better than PLMs, while just using 0.3% of energy for training. Expand
How to Query Language Models?
TLDR
This work proposes to query LMs by example and suggests that LMs contain more factual and commonsense edge than previously assumed—if the authors query the model in the right way. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 43 REFERENCES
Language Models are Unsupervised Multitask Learners
TLDR
It is demonstrated that language models begin to learn these tasks without any explicit supervision when trained on a new dataset of millions of webpages called WebText, suggesting a promising path towards building language processing systems which learn to perform tasks from their naturally occurring demonstrations. Expand
Improving Language Understanding by Generative Pre-Training
TLDR
The general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied. Expand
T-REx: A Large Scale Alignment of Natural Language with Knowledge Base Triples
TLDR
T-REx, a dataset of large scale alignments between Wikipedia abstracts and Wikidata triples, is presented, which is two orders of magnitude larger than the largest available alignments dataset and covers 2.5 times more predicates. Expand
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
TLDR
A benchmark of nine diverse NLU tasks, an auxiliary dataset for probing models for understanding of specific linguistic phenomena, and an online platform for evaluating and comparing models, which favors models that can represent linguistic knowledge in a way that facilitates sample-efficient learning and effective knowledge-transfer across tasks. Expand
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
TLDR
This work presents CommonsenseQA: a challenging new dataset for commonsense question answering, which extracts from ConceptNet multiple target concepts that have the same semantic relation to a single source concept. Expand
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
TLDR
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks. Expand
Context-Aware Representations for Knowledge Base Relation Extraction
TLDR
It is demonstrated that for sentence-level relation extraction it is beneficial to consider other relations in the sentential context while predicting the target relation and to combine the context representations with an attention mechanism to make the final prediction. Expand
A Survey of Reinforcement Learning Informed by Natural Language
TLDR
The time is right to investigate a tight integration of natural language understanding into Reinforcement Learning in particular, and the state of the field is surveyed, including work on instruction following, text games, and learning from textual domain knowledge. Expand
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
TLDR
There is substantial room for improvement in NLI systems, and the HANS dataset can motivate and measure progress in this area, which contains many examples where the heuristics fail. Expand
Dissecting Contextual Word Embeddings: Architecture and Representation
TLDR
There is a tradeoff between speed and accuracy, but all architectures learn high quality contextual representations that outperform word embeddings for four challenging NLP tasks, suggesting that unsupervised biLMs, independent of architecture, are learning much more about the structure of language than previously appreciated. Expand
...
1
2
3
4
5
...