• Publications
  • Influence
SciBERT: A Pretrained Language Model for Scientific Text
TLDR
SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks and demonstrates statistically significant improvements over BERT. Expand
Longformer: The Long-Document Transformer
TLDR
Following prior work on long-sequence transformers, the Longformer is evaluated on character-level language modeling and achieves state-of-the-art results on text8 and enwik8 and pretrain Longformer and finetune it on a variety of downstream tasks. Expand
Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks
TLDR
It is consistently found that multi-phase adaptive pretraining offers large gains in task performance, and it is shown that adapting to a task corpus augmented using simple data selection strategies is an effective alternative, especially when resources for domain-adaptive pretraining might be unavailable. Expand
SciBERT: Pretrained Contextualized Embeddings for Scientific Text
TLDR
SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks and demonstrates statistically significant improvements over BERT. Expand
ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing
TLDR
ScispaCy, a new Python library and models for practical biomedical/scientific text processing, which heavily leverages the spaCy library is described, which detail the performance of two packages of models released in scispa Cy and demonstrate their robustness on several tasks and datasets. Expand
Construction of the Literature Graph in Semantic Scholar
TLDR
This paper reduces literature graph construction into familiar NLP tasks, point out research challenges due to differences from standard formulations of these tasks, and report empirical results for each task. Expand
Probabilistic Soft Logic for Semantic Textual Similarity
TLDR
This work applies PSL to the task of Semantic Textual Similarity (STS), and shows that PSL gives improved results compared to a previous approach based on Markov Logic Networks (MLNs) and a purely distributional approach. Expand
Representing Meaning with a Combination of Logical and Distributional Models
TLDR
This article adopts a hybrid approach that combines logical and distributional semantics using probabilistic logic, specifically Markov Logic Networks and releases a lexical entailment data set of 10,213 rules extracted from the SICK data set, which is a valuable resource for evaluating lexical entailsment systems. Expand
SciREX: A Challenge Dataset for Document-Level Information Extraction
TLDR
SciREX is introduced, a document level IE dataset that encompasses multiple IE tasks, including salient entity identification and document level N-ary relation identification from scientific articles, and a neural model is developed as a strong baseline that extends previous state-of-the-art IE models to document-level IE. Expand
Pretrained Language Models for Sequential Sentence Classification
TLDR
This work constructs a joint sentence representation that allows BERT Transformer layers to directly utilize contextual information from all words in all sentences, and achieves state-of-the-art results on four datasets, including a new dataset of structured scientific abstracts. Expand
...
1
2
3
...