Corpus ID: 230437704

Cross-Document Language Modeling

@article{Caciularu2021CrossDocumentLM,
  title={Cross-Document Language Modeling},
  author={Avi Caciularu and Arman Cohan and Iz Beltagy and Matthew E. Peters and Arie Cattan and I. Dagan},
  journal={ArXiv},
  year={2021},
  volume={abs/2101.00406}
}

• Avi Caciularu, Arman Cohan, +3 authors I. Dagan
• Published 2021
• Computer Science
• ArXiv

We introduce a new pretraining approach for language models that are geared to support multi-document NLP tasks. Our crossdocument language model (CD-LM) improves masked language modeling for these tasks with two key ideas. First, we pretrain with multiple related documents in a single input, via cross-document masking, which encourages the model to learn cross-document and long-range relationships. Second, extending the recent Longformer model, we pretrain with long contexts of several… Expand

View PDF on arXiv

Save to Library

Create Alert

Cite

Launch Research Feed