Cross-Document Language Modeling
@article{Caciularu2021CrossDocumentLM, title={Cross-Document Language Modeling}, author={Avi Caciularu and Arman Cohan and Iz Beltagy and Matthew E. Peters and Arie Cattan and I. Dagan}, journal={ArXiv}, year={2021}, volume={abs/2101.00406} }
We introduce a new pretraining approach for language models that are geared to support multi-document NLP tasks. Our crossdocument language model (CD-LM) improves masked language modeling for these tasks with two key ideas. First, we pretrain with multiple related documents in a single input, via cross-document masking, which encourages the model to learn cross-document and long-range relationships. Second, extending the recent Longformer model, we pretrain with long contexts of several… Expand
Figures and Tables from this paper
References
SHOWING 1-10 OF 44 REFERENCES
Multilevel Text Alignment with Cross-Document Attention
- Computer Science
- EMNLP
- 2020
- 1
- Highly Influential
- PDF
Revisiting Joint Modeling of Cross-document Entity and Event Coreference Resolution
- Computer Science
- ACL
- 2019
- 25
- PDF
Streamlining Cross-Document Coreference Resolution: Evaluation and Modeling
- Computer Science
- ArXiv
- 2020
- 4
- PDF
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
- Computer Science
- BlackboxNLP@EMNLP
- 2018
- 1,246
- PDF
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
- Computer Science
- ICML
- 2020
- 128
- PDF