Skip to search formSkip to main contentSkip to account menu

RoBERTa: A Robustly Optimized BERT Pretraining Approach

It is found that BERT was significantly undertrained, and can match or exceed the performance of every model published after it, and the best model achieves state-of-the-art results on GLUE, RACE and SQuAD.
View PDF on arXiv (opens in a new tab)

Unsupervised Cross-lingual Representation Learning at Scale

It is shown that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks, and the possibility of multilingual modeling without sacrificing per-language performance is shown for the first time.
View on ACL (opens in a new tab)

Finding Deceptive Opinion Spam by Any Stretch of the Imagination

This work develops and compares three approaches to detecting deceptive opinion spam, and develops a classifier that is nearly 90% accurate on the authors' gold-standard opinion spam dataset, and reveals a relationship between deceptive opinions and imaginative writing.
View on ACL (opens in a new tab)

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks and supports distributed training across multiple GPUs and machines.
View on ACL (opens in a new tab)

Understanding Back-Translation at Scale

This work broadens the understanding of back-translation and investigates a number of methods to generate synthetic source sentences, finding that in all but resource poor settings back-translations obtained via sampling or noised beam outputs are most effective.
View on ACL (opens in a new tab)

Phrase-Based & Neural Unsupervised Machine Translation

This work investigates how to learn to translate when having access to only large monolingual corpora in each language, and proposes two model variants, a neural and a phrase-based model, which are significantly better than methods from the literature, while being simpler and having fewer hyper-parameters.
View on ACL (opens in a new tab)

Recipes for Building an Open-Domain Chatbot

Human evaluations show the best models outperform existing approaches in multi-turn dialogue on engagingness and humanness measurements, and the limitations of this work are discussed by analyzing failure cases of the models.
View on ACL (opens in a new tab)

Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences

This work uses unsupervised learning to train a deep contextual language model on 86 billion amino acids across 250 million protein sequences spanning evolutionary diversity, and finds that without prior knowledge, information emerges in the learned representations on fundamental properties of proteins such as secondary structure, contacts, and biological activity.
View on Publisher (opens in a new tab)

Scaling Neural Machine Translation

This paper shows that reduced precision and large batch training can speedup training by nearly 5x on a single 8-GPU machine with careful tuning and implementation.
View on ACL (opens in a new tab)

Negative Deceptive Opinion Spam

This work creates and study the first dataset of deceptive opinion spam with negative sentiment reviews, and finds that standard n-gram text categorization techniques can detect negative deceptive opinions spam with performance far surpassing that of human judges.
View on ACL (opens in a new tab)
...
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy (opens in a new tab), Terms of Service (opens in a new tab), and Dataset License (opens in a new tab)