Corpus ID: 230437704

Cross-Document Language Modeling

@article{Caciularu2021CrossDocumentLM,
  title={Cross-Document Language Modeling},
  author={Avi Caciularu and Arman Cohan and Iz Beltagy and Matthew E. Peters and Arie Cattan and I. Dagan},
  journal={ArXiv},
  year={2021},
  volume={abs/2101.00406}
}
We introduce a new pretraining approach for language models that are geared to support multi-document NLP tasks. Our crossdocument language model (CD-LM) improves masked language modeling for these tasks with two key ideas. First, we pretrain with multiple related documents in a single input, via cross-document masking, which encourages the model to learn cross-document and long-range relationships. Second, extending the recent Longformer model, we pretrain with long contexts of several… Expand

Figures and Tables from this paper

References

SHOWING 1-10 OF 44 REFERENCES
Multilevel Text Alignment with Cross-Document Attention
  • 1
  • Highly Influential
  • PDF
Pre-training via Paraphrasing
  • 28
  • PDF
Revisiting Joint Modeling of Cross-document Entity and Event Coreference Resolution
  • 25
  • PDF
Streamlining Cross-Document Coreference Resolution: Evaluation and Modeling
  • 4
  • PDF
Semantic Text Matching for Long-Form Documents
  • 30
  • Highly Influential
  • PDF
Longformer: The Long-Document Transformer
  • 183
  • PDF
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
  • 1,246
  • PDF
Hierarchical Document Encoder for Parallel Corpus Mining
  • 9
  • PDF
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
  • 128
  • PDF
Hierarchical Transformers for Multi-Document Summarization
  • 65
  • PDF
...
1
2
3
4
5
...