Skip to search formSkip to main contentSkip to account menu

Measuring Massive Multitask Language Understanding

While most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average, however, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy.
View PDF on arXiv (opens in a new tab)

Aligning AI With Shared Human Values

With the ETHICS dataset, it is found that current language models have a promising but incomplete understanding of basic ethical knowledge, and it provides a steppingstone toward AI that is aligned with human values.
View PDF on arXiv (opens in a new tab)

Measuring Coding Challenge Competence With APPS

APPS is introduced, a benchmark for code generation that measures the ability of models to take an arbitrary natural language specification and generate satisfactory Python code and shows that machine learning models are now beginning to learn how to code.
View PDF on arXiv (opens in a new tab)

Measuring Mathematical Problem Solving With the MATH Dataset

This work introduces MATH, a new dataset of 12, 500 challenging competition mathematics problems which can be used to teach models to generate answer derivations and explanations, and shows that accuracy remains relatively low, even with enormous Transformer models.
View PDF on arXiv (opens in a new tab)

CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review

It is found that Transformer models have nascent performance, but that this performance is strongly influenced by model design and training dataset size, so there is still substantial room for improvement.
View PDF on arXiv (opens in a new tab)

Interpreting Black Box Models via Hypothesis Testing

This work reframe black box model interpretability as a multiple hypothesis testing problem, and proposes two testing methods: one that provably controls the false discovery rate but which is not yet feasible for large-scale applications, and an approximate testing method which can be applied to real-world data sets.
View on ACM (opens in a new tab)

Limitations of Post-Hoc Feature Alignment for Robustness

It is shown that this approach only significantly helps with a narrow set of distribution shifts and several settings in which it even degrades performance are identified, calling into question the utility of this approach and Unsupervised Domain Adaptation more broadly for improving robustness in practice.
View on IEEE (opens in a new tab)

Interpreting Black Box Models with Statistical Guarantees

A multiple hypothesis testing framework for finding important features by testing whether the model prediction is significantly different from what would be expected if the features were replaced with randomly-sampled counterfactuals is derived.
View PDF on arXiv (opens in a new tab)

Streaming Complexity of SVMs

It is shown that, for both problems, for dimensions $d=1,2$, one can obtain streaming algorithms with space polynomially smaller than SGD for strongly convex functions like the bias-regularized SVM, and polynomial lower bounds for both point estimation and optimization are proved.
View PDF on arXiv (opens in a new tab)

Discovering Latent Knowledge in Language Models Without Supervision

It is shown that despite using no supervision and no model outputs, the method can recover diverse knowledge represented in large language models: across 6 models and 10 question-answering datasets, it outperforms zero-shot accuracy by 4% on average.
View PDF on arXiv (opens in a new tab)
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy (opens in a new tab), Terms of Service (opens in a new tab), and Dataset License (opens in a new tab)