Language Models as Knowledge Bases?

@inproceedings{Petroni2019LanguageMA,
  title={Language Models as Knowledge Bases?},
  author={F. Petroni and Tim Rockt{\"a}schel and Patrick Lewis and A. Bakhtin and Yuxiang Wu and Alexander H. Miller and S. Riedel},
  booktitle={EMNLP},
  year={2019}
}
Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models may also be storing relational knowledge present in the training data, and may be able to answer queries structured as “fill-in-the-blank” cloze statements. Language models have many advantages over structured knowledge bases: they require no schema engineering, allow practitioners to query about an open class of… Expand
253 Citations

Paper Mentions

KILT: a Benchmark for Knowledge Intensive Language Tasks
  • 23
  • PDF
Language Models are Open Knowledge Graphs
  • 4
  • PDF
How Context Affects Language Models' Factual Predictions
  • 19
  • PDF
Relational world knowledge representation in contextual language models: A review
  • PDF
Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries
  • 4
  • Highly Influenced
  • PDF
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • 60
  • PDF
Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
  • 44
  • PDF
Static Embeddings as Efficient Knowledge Bases?
  • Highly Influenced
  • PDF
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 43 REFERENCES
Language Models are Unsupervised Multitask Learners
  • 2,679
  • PDF
Improving Language Understanding by Generative Pre-Training
  • 1,929
  • Highly Influential
  • PDF
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
  • 1,265
  • PDF
T-REx: A Large Scale Alignment of Natural Language with Knowledge Base Triples
  • 43
  • Highly Influential
  • PDF
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  • 15,760
  • Highly Influential
  • PDF
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
  • 182
  • PDF
A Survey of Reinforcement Learning Informed by Natural Language
  • 52
  • PDF
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
  • 253
  • PDF
Dissecting Contextual Word Embeddings: Architecture and Representation
  • 187
  • Highly Influential
  • PDF
...
1
2
3
4
5
...