TREC-COVID: rationale and structure of an information retrieval shared task for COVID-19

  title={TREC-COVID: rationale and structure of an information retrieval shared task for COVID-19},
  author={Kirk Roberts and Tasmeer Alam and Steven Bedrick and Dina Demner-Fushman and Kyle Lo and Ian Soboroff and Ellen M. Voorhees and Lucy Lu Wang and William R. Hersh},
  journal={Journal of the American Medical Informatics Association : JAMIA},
  pages={1431 - 1436}
Abstract TREC-COVID is an information retrieval (IR) shared task initiated to support clinicians and clinical research during the COVID-19 pandemic. IR for pandemics breaks many normal assumptions, which can be seen by examining 9 important basic IR research questions related to pandemic situations. TREC-COVID differs from traditional IR shared task evaluations with special considerations for the expected users, IR modality considerations, topic development, participant requirements, assessment… 

Tables from this paper

Searching for scientific evidence in a pandemic: An overview of TREC-COVID

On the Quality of the TREC-COVID IR Test Collections

The quality of the resulting TREC-COVID test collections are examined, and a critique of the state-of-the-art in building reusable IR test collections is offered.

Information Retrieval in an Infodemic: The Case of COVID-19 Publications

This work presents an information retrieval methodology for effectively finding relevant publications for different information needs using traditional information retrieval models, as well as modern neural natural language processing algorithms for an infodemic.

Question Answering Systems for Covid-19

The survey of QA systems-CovidQA, CAiRE (Center for Artificial Intelligence Research)-COVID system, CO-search semantic search engine, COVIDASK, RECORD (Research Engine for COVID Open Research Dataset) available for CO VID-19 are described.

Impact of detecting clinical trial elements in exploration of COVID-19 literature

This study finds that the relational concept selection filters the original retrieved collection in a way that decreases the proportion of unjudged documents and increases the precision, which means that the user is likely to be exposed to a larger number of relevant documents.

Interpretable Self-supervised Multi-task Learning for COVID-19 Information Retrieval and Extraction

This study proposes an interpretable self-supervised multi-task learning model to jointly and effectively tackle the tasks of information retrieval (IR) and extraction (IE) during the current emergency health crisis situation and shows that the model effectively leverage the multi- task and self- supervised learning to improve generalization, data efficiency and robustness to the ongoing dataset shift problem.

Pandemic Literature Search: Finding Information on COVID-19

This work investigates how to better rank information for pandemic information retrieval and proposes a novel end-to-end method for neural retrieval that could lead to a search system that aids scientists, clinicians, policymakers and others in finding reliable answers from the scientific literature.

SLEDGE-Z: A Zero-Shot Baseline for COVID-19 Literature Search

This work presents a zero-shot ranking algorithm that adapts to COVID-related scientific literature, and uses a neural re-ranking model pre-trained on scientific text (SciBERT), and filters the target document collection.

Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Dataset

We present Covidex, a search engine that exploits the latest neural ranking models to provide information access to the COVID-19 Open Research Dataset curated by the Allen Institute for AI. Our



CORD-19: The COVID-19 Open Research Dataset

The mechanics of dataset construction are described, highlighting challenges and key design decisions, an overview of how CORD-19 has been used, and several shared tasks built around the dataset are described.

UQ IElab at TREC 2019 Decision Track

The solution addressed this challenge by extending the Entity Query Feature Expansion model (EQFE), a knowledge base (KB) query expansion method, which showed that Wikipedia and the Consumer Health Vocabulary resource can be effective as basis for consumer health search query expansion, within the EQFE method.

State-of-the-art in biomedical literature retrieval for clinical cases: a survey of the TREC 2014 CDS track

An overview of the task, a survey of the information retrieval methods employed by the participants, an analysis of the results, and a discussion on the future directions for this challenging yet important task are provided.

Rapidly Deploying a Neural Search Engine for the COVID-19 Open Research Dataset

The Neural Covidex is a search engine that exploits the latest neural ranking architectures to provide information access to the COVID-19 Open Research Dataset (CORD-19) curated by the Allen

Overview of the TREC 2014 Clinical Decision Support Track

The focus of the 2014 track was the retrieval of biomedical articles relevant for answering generic clinical questions about medical records, using short case reports, such as those published in biomedical articles, as idealized representations of actual medical records.

2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text

The 2010 i2b2/VA Workshop on Natural Language Processing Challenges for Clinical Records presented three tasks, which showed that machine learning approaches could be augmented with rule-based systems to determine concepts, assertions, and relations.

TREC GENOMICS Track Overview

The first year of TREC Genomics Track featured two tasks: ad hoc retrieval and information extraction. Both tasks centered around the Gene Reference into Function (GeneRIF) resource of the National

Anserini: Enabling the Use of Lucene for Information Retrieval Research

Anserini provides wrappers and extensions on top of core Lucene libraries that allow researchers to use more intuitive APIs to accomplish common research tasks, and aims to provide the best of both worlds to better align information retrieval practice and research.

Overview of the TREC 2019 Precision Medicine Track.

The potential impact of precision medicine in cancer, where lifesaving treatments for particular patients could prove ineffective or even deadly for other patients based entirely upon the particular genetic mutations in the HHS Public Access Author manuscript, is closely focused.

Retrieval evaluation with incomplete information

It is shown that current evaluation measures are not robust to substantially incomplete relevance judgments, and a new measure is introduced that is both highly correlated with existing measures when complete judgments are available and more robust to incomplete judgment sets.