Corpus ID: 235293699

Comparing Test Sets with Item Response Theory

@inproceedings{Vania2021ComparingTS,
  title={Comparing Test Sets with Item Response Theory},
  author={Clara Vania and Phu Mon Htut and William R. Huang and Dhara Mungra and Richard Yuanzhe Pang and Jason Phang and Haokun Liu and Kyunghyun Cho and Sam Bowman},
  booktitle={ACL/IJCNLP},
  year={2021}
}
Recent years have seen numerous NLP datasets introduced to evaluate the performance of fine-tuned models on natural language understanding tasks. Recent results from large pretrained models, though, show that many of these datasets are largely saturated and unlikely to be able to detect further progress. What kind of datasets are still effective at discriminating among strong models, and what kind of datasets should we expect to be able to detect future improvements? To measure this uniformly… Expand

Figures and Tables from this paper

References

SHOWING 1-10 OF 68 REFERENCES
Building an Evaluation Scale using Item Response Theory
When Do You Need Billions of Words of Pretraining Data?
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds
Know What You Don't Know: Unanswerable Questions for SQuAD
NewsQA: A Machine Comprehension Dataset
...
1
2
3
4
5
...