Pretrained Language Models for Sequential Sentence Classification

@article{Cohan2019PretrainedLM,
  title={Pretrained Language Models for Sequential Sentence Classification},
  author={Arman Cohan and Iz Beltagy and Daniel King and Bhavana Dalvi and Daniel S. Weld},
  journal={ArXiv},
  year={2019},
  volume={abs/1909.04054}
}
As a step toward better document-level understanding, we explore classification of a sequence of sentences into their corresponding categories, a task that requires understanding sentences in context of the document. Recent successful models for this task have used hierarchical models to contextualize sentence representations, and Conditional Random Fields (CRFs) to incorporate dependencies between subsequent labels. In this work, we show that pretrained language models, BERT (Devlin et al… Expand
25 Citations

Figures and Tables from this paper

Sequential Span Classification with Neural Semi-Markov CRFs for Biomedical Abstracts
  • 1
  • Highly Influenced
  • PDF
Sequential Sentence Classification in Research Papers using Cross-Domain Multi-Task Learning
  • Highly Influenced
  • PDF
Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
  • 182
  • PDF
On Generating Extended Summaries of Long Documents
  • PDF
Self-Attention Guided Copy Mechanism for Abstractive Summarization
  • 4
  • PDF
...
1
2
3
...

References

SHOWING 1-10 OF 27 REFERENCES
Language Model Pre-training for Hierarchical Document Representations
  • 11
  • PDF
Deep contextualized word representations
  • 5,437
  • PDF
SciBERT: A Pretrained Language Model for Scientific Text
  • 376
  • PDF
Improving Language Understanding by Generative Pre-Training
  • 1,929
  • PDF
Unified Language Model Pre-training for Natural Language Understanding and Generation
  • 356
  • PDF
Neural Summarization by Extracting Sentences and Words
  • 475
  • PDF
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  • 15,760
  • PDF
Sequence to Sequence Learning with Neural Networks
  • 11,931
  • PDF
Universal Language Model Fine-tuning for Text Classification
  • 1,478
  • PDF
A Supervised Approach to Extractive Summarisation of Scientific Papers
  • 40
  • Highly Influential
  • PDF
...
1
2
3
...