• Publications
  • Influence
Deep Contextualized Word Representations
A new type of deep contextualized word representation is introduced that models both complex characteristics of word use and how these uses vary across linguistic contexts, allowing downstream models to mix different types of semi-supervision signals. Expand
Longformer: The Long-Document Transformer
Following prior work on long-sequence transformers, the Longformer is evaluated on character-level language modeling and achieves state-of-the-art results on text8 and enwik8 and pretrain Longformer and finetune it on a variety of downstream tasks. Expand
AllenNLP: A Deep Semantic Natural Language Processing Platform
AllenNLP is designed to support researchers who want to build novel language understanding models quickly and easily and provides a flexible data API that handles intelligent batching and padding, and a modular and extensible experiment framework that makes doing good science easy. Expand
Linguistic Knowledge and Transferability of Contextual Representations
It is found that linear models trained on top of frozen contextual representations are competitive with state-of-the-art task-specific models in many cases, but fail on tasks requiring fine-grained linguistic knowledge. Expand
Dissecting Contextual Word Embeddings: Architecture and Representation
There is a tradeoff between speed and accuracy, but all architectures learn high quality contextual representations that outperform word embeddings for four challenging NLP tasks, suggesting that unsupervised biLMs, independent of architecture, are learning much more about the structure of language than previously appreciated. Expand
Relationships between Water Vapor Path and Precipitation over the Tropical Oceans
The relationship between water vapor path W and surface precipitation rate P over tropical oceanic regions is analyzed using 4 yr of gridded daily SSM/I satellite microwave radiometer data. A tightExpand
Semi-supervised sequence tagging with bidirectional language models
A general semi-supervised approach for adding pre- trained context embeddings from bidirectional language models to NLP systems and apply it to sequence labeling tasks, surpassing previous systems that use other forms of transfer or joint learning with additional labeled data and task specific gazetteers. Expand
Knowledge Enhanced Contextual Word Representations
After integrating WordNet and a subset of Wikipedia into BERT, the knowledge enhanced BERT (KnowBert) demonstrates improved perplexity, ability to recall facts as measured in a probing task and downstream performance on relationship extraction, entity typing, and word sense disambiguation. Expand
Understanding the origin and analysis of sediment-charcoal records with a simulation model
Interpreting sediment-charcoal records is challenging because there is little information linking charcoal production from fires to charcoal accumulation in lakes. We present a numerical modelExpand
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks
The empirical results across diverse NLP tasks with two state-of-the-art models show that the relative performance of fine-tuning vs. feature extraction depends on the similarity of the pretraining and target tasks. Expand