Skip to search formSkip to main contentSkip to account menu
A Simple and Effective Model for Answering Multi-span Questions
TLDR
This work suggests a new approach for tackling multi-span questions, based on sequence tagging, which differs from previous approaches for answering span questions, and shows that this approach leads to an absolute improvement and slightly eclipses the current state-of-the-art results on the entire DROP dataset.
View on ACL
The Turking Test: Can Language Models Understand Instructions?
TLDR
The Turking Test is presented, which examines a model's ability to follow natural language instructions of varying complexity, and reveals that a large pretrained language model performs poorly across all tasks.
View PDF on arXiv
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
TLDR
Evaluation of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters finds that model performance and calibration both improve with scale, but are poor in absolute terms.
View PDF on arXiv
SCROLLS: Standardized CompaRison Over Long Language Sequences
TLDR
This work introduces SCROLLS, a suite of tasks that require reasoning over long texts, and examines existing long-text datasets, and handpick ones where the text is naturally long, while prioritizing tasks that involve synthesizing information across the input.
View PDF on arXiv
Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language
TLDR
This work presents Cryptonite, a large-scale dataset based on cryptic crosswords, which is both linguistically complex and naturally sourced, and on par with the accuracy of a rule-based clue solver.
View on ACL
How Optimal is Greedy Decoding for Extractive Question Answering?
Fine-tuned language models use greedy decoding to answer reading comprehension questions with relative success. However, this approach does not ensure that the answer is a span in the given passage,…
View PDF on arXiv
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy, Terms of Service, and Dataset License