Corpus ID: 211204736

REALM: Retrieval-Augmented Language Model Pre-Training

@article{Guu2020REALMRL,
  title={REALM: Retrieval-Augmented Language Model Pre-Training},
  author={Kelvin Guu and Kenton Lee and Z. Tung and Panupong Pasupat and Ming-Wei Chang},
  journal={ArXiv},
  year={2020},
  volume={abs/2002.08909}
}
Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus… Expand
Studying Strategically: Learning to Mask for Closed-book QA
How Context Affects Language Models' Factual Predictions
Learning Dense Representations of Phrases at Scale
Cross-Thought for Sentence Encoder Pre-training
Synthetic Target Domain Supervision for Open Retrieval QA
Distilling Knowledge from Reader to Retriever for Question Answering
Language Models are Open Knowledge Graphs
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 50 REFERENCES
Language Models as Knowledge Bases?
Language Models are Unsupervised Multitask Learners
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Knowledge Enhanced Contextual Word Representations
Learning Recurrent Span Representations for Extractive Question Answering
Latent Retrieval for Weakly Supervised Open Domain Question Answering
Skip-Thought Vectors
End-To-End Memory Networks
A Retrieve-and-Edit Framework for Predicting Structured Outputs
A Discrete Hard EM Approach for Weakly Supervised Question Answering
...
1
2
3
4
5
...