Longformer: The Long-Document Transformer
- Iz Beltagy, Matthew E. Peters, Arman Cohan
- Computer ScienceArXiv
- 10 April 2020
Following prior work on long-sequence transformers, the Longformer is evaluated on character-level language modeling and achieves state-of-the-art results on text8 and enwik8 and pretrain Longformer and finetune it on a variety of downstream tasks.
SciBERT: A Pretrained Language Model for Scientific Text
- Iz Beltagy, Kyle Lo, Arman Cohan
- Computer ScienceConference on Empirical Methods in Natural…
- 1 March 2019
SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks and demonstrates statistically significant improvements over BERT.
A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents
- Arman Cohan, Franck Dernoncourt, Nazli Goharian
- Computer ScienceNorth American Chapter of the Association for…
- 1 April 2018
This work proposes the first model for abstractive summarization of single, longer-form documents (e.g., research papers), consisting of a new hierarchical encoder that models the discourse structure of a document, and an attentive discourse-aware decoder to generate the summary.
SciBERT: Pretrained Contextualized Embeddings for Scientific Text
- Iz Beltagy, Arman Cohan, Kyle Lo
- Computer ScienceArXiv
- 26 March 2019
SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks and demonstrates statistically significant improvements over BERT.
SPECTER: Document-level Representation Learning using Citation-informed Transformers
- Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, Daniel S. Weld
- Computer ScienceAnnual Meeting of the Association for…
- 15 April 2020
This work proposes SPECTER, a new method to generate document-level embedding of scientific papers based on pretraining a Transformer language model on a powerful signal of document- level relatedness: the citation graph, and shows that Specter outperforms a variety of competitive baselines on the benchmark.
CEDR: Contextualized Embeddings for Document Ranking
- Sean MacAvaney, Andrew Yates, Arman Cohan, Nazli Goharian
- Computer ScienceAnnual International ACM SIGIR Conference on…
- 15 April 2019
This work investigates how two pretrained contextualized language models (ELMo and BERT) can be utilized for ad-hoc document ranking and proposes a joint approach that incorporates BERT's classification vector into existing neural models and shows that it outperforms state-of-the-art ad-Hoc ranking baselines.
Structural Scaffolds for Citation Intent Classification in Scientific Publications
- Arman Cohan, Waleed Ammar, Madeleine van Zuylen, Field Cady
- Computer ScienceNorth American Chapter of the Association for…
- 1 April 2019
This work proposes structural scaffolds, a multitask model to incorporate structural information of scientific papers into citations for effective classification of citation intents, which achieves a new state-of-the-art on an existing ACL anthology dataset with a 13.3% absolute increase in F1 score.
Depression and Self-Harm Risk Assessment in Online Forums
- Andrew Yates, Arman Cohan, Nazli Goharian
- Computer ScienceConference on Empirical Methods in Natural…
- 1 September 2017
This work introduces a large-scale general forum dataset consisting of users with self-reported depression diagnoses matched with control users, and proposes methods for identifying posts in support communities that may indicate a risk of self-harm, and demonstrates that this approach outperforms strong previously proposed methods.
Fact or Fiction: Verifying Scientific Claims
- David Wadden, Kyle Lo, Hannaneh Hajishirzi
- Computer ScienceConference on Empirical Methods in Natural…
- 30 April 2020
We introduce scientific claim verification, a new task to select abstracts from the research literature containing evidence that supports or refutes a given scientific claim, and to identify…
Pretrained Language Models for Sequential Sentence Classification
- Arman Cohan, Iz Beltagy, Daniel King, Bhavana Dalvi, Daniel S. Weld
- Computer ScienceConference on Empirical Methods in Natural…
- 9 September 2019
This work constructs a joint sentence representation that allows BERT Transformer layers to directly utilize contextual information from all words in all sentences, and achieves state-of-the-art results on four datasets, including a new dataset of structured scientific abstracts.
...
...