Automated Mining of Leaderboards for Empirical AI Research

  title={Automated Mining of Leaderboards for Empirical AI Research},
  author={Salomon Kabongo KABENAMUALU and Jennifer D’Souza and S{\"o}ren Auer},
With the rapid growth of research publications, empowering scientists to keep oversight over the scientific progress is of paramount importance. In this regard, the Leaderboards facet of information organization provides an overview on the state-of-the-art by aggregating empirical results from various studies addressing the same research challenge. Crowdsourcing efforts like PapersWithCode among others are devoted to the construction of Leaderboards predominantly for various subdomains in… Expand

Figures and Tables from this paper


Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction
This model is a first step towards automatic leaderboard construction, e.g., in the NLP domain, aimed at automatically extracting task, dataset, metric and score from NLP papers, towards the automatic construction of leaderboards. Expand
End-to-End NLP Knowledge Graph Construction
This paper applies the SciNLP-KG framework to 30,000 NLP papers from ACL Anthology to build a large-scale KG, which can facilitate automatically constructing scientific leaderboards for the NLP community and indicates that the resulting KG contains high-quality information. Expand
Crowdsourcing Scholarly Discourse Annotations
This work proposes a method to crowdsource structured scholarly knowledge from paper authors with a web-based user interface supported by artificial intelligence that enables authors to select key sentences for annotation. Expand
AxCell: Automatic Extraction of Results from Machine Learning Papers
AxCell, an automatic machine learning pipeline for extracting results from papers using several novel components, including a table segmentation subtask, to learn relevant structural knowledge that aids extraction significantly improves the state of the art for results extraction. Expand
Open Research Knowledge Graph: Next Generation Infrastructure for Semantic Scholarly Knowledge
The first steps towards a knowledge graph based infrastructure that acquires scholarly knowledge in machine actionable form are presented thus enabling new possibilities for scholarly knowledge curation, publication and processing. Expand
TaPas: Weakly Supervised Table Parsing via Pre-training
TaPas is presented, an approach to question answering over tables without generating logical forms that outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA and performing on par with the state of theart on WikiSQL and WikiTQ, but with a simpler model architecture. Expand
TDMSci: A Specialized Corpus for Scientific Literature Entity Tagging of Tasks Datasets and Metrics
A new corpus that contains domain expert annotations for Task (T), Dataset (D), Metric (M) entities 2,000 sentences extracted from NLP papers is presented and made publicly available to the community. Expand
SciBERT: A Pretrained Language Model for Scientific Text
SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks and demonstrates statistically significant improvements over BERT. Expand
Towards an Open Research Knowledge Graph
In his NASIG vision session, Sören Auer introduced attendees to knowledge graphs and explained how they could make scientific research more discoverable, efficient, and collaborative. Expand
SciREX: A Challenge Dataset for Document-Level Information Extraction
SciREX is introduced, a document level IE dataset that encompasses multiple IE tasks, including salient entity identification and document level N-ary relation identification from scientific articles, and a neural model is developed as a strong baseline that extends previous state-of-the-art IE models to document-level IE. Expand