Our Work

Semantic Scholar Publications

Our team of world-class researchers is dedicated to studying information overload and developing groundbreaking AI-powered tools to overcome it. Our team is part of the Allen Institute for AI, a nonprofit research institute founded by Microsoft co-founder Paul Allen to develop AI that benefits the common good.

10 Total Publications
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Clear
Threddy: An Interactive System for Personalized Thread-based Exploration and Organization of Scientific Literature
Hyeonsu B. Kang, Joseph Chee Chang, Yongsung Kim, Aniket Kittur
No items found.
  • UIST
  • October 29, 2022
TLDR

A tool integrated into users’ reading process that helps them with leveraging authors’ existing summarization of threads, typically in introduction or related work sections, in order to situate their own work’s contributions is developed.

Best Paper Award
MultiVerS: Improving scientific claim verification with weak supervision and full-document context
David Wadden, Kyle Lo, Lucy Lu Wang, Arman Cohan, Iz Beltagy, Hannaneh Hajishirzi
No items found.
  • NAACL Findings
  • July 10, 2022
TLDR

This work presents M ULTI V ER S, which predicts a fact-checking label and identifies rationales in a multitask fashion based on a shared encoding of the claim and full document context, which allows the model to perform weakly-supervised domain adaptation by training on scientific documents labeled using high-precision heuristics.

Best Paper Award
Few-Shot Self-Rationalization with Natural Language Prompts
Ana Marasović, Iz Beltagy, Doug Downey, Matthew E. Peters
No items found.
  • Findings of NAACL
  • July 10, 2022
TLDR

This work identifies the right prompting approach by extensively exploring natural language prompts on FEB and demonstrates that making progress on few-shot self-rationalization is possible, and presents FEB—a stan-dardized collection of four existing English-language datasets and associated metrics.

Best Paper Award
Long Context Question Answering via Supervised Contrastive Learning
Avi Caciularu, Ido Dagan, Jacob Goldberger, Arman Cohan
No items found.
  • NAACL
  • July 10, 2022
TLDR

This work proposes a novel method for equipping long-context QA models with an additional sequence-level objective for better identification of the supporting evidence, via an additional contrastive supervision signal in finetuning.

Best Paper Award
MultiCite: Modeling realistic citations requires moving beyond the single-sentence single-label setting
Anne Lauscher, Brandon Ko, Bailey Kuehl, Sophie Johnson, Arman Cohan, David Jurgens, Kyle Lo
No items found.
  • NAACL
  • July 10, 2022
TLDR

We highlight three understudied phenomena for citation context analysis and release MultiCite, a new dataset of 12.6K citation contexts from 1.2K computational linguistics papers that fully models these phenomena.

Best Paper Award
Literature-Augmented Clinical Outcome Prediction
Aakanksha Naik, S. Parasa, Sergey Feldman, Lucy Lu Wang, Tom Hope
No items found.
  • Findings of NAACL
  • July 10, 2022
TLDR

A novel system that automatically retrieves patient-specific literature based on intensive care (ICU) patient information, aggregates relevant papers and fuses them with internal admission notes to form outcome predictions, which is able to substantially boost predictive accuracy on three challenging tasks in comparison to strong recent baselines.

Best Paper Award
Paragraph-based Transformer Pre-training for Multi-Sentence Inference
Luca Di Liello, Siddhant Garg, Luca Soldaini, Alessandro Moschitti
No items found.
  • NAACL
  • July 10, 2022
TLDR

This paper shows that popular pre-trained transformers perform poorly when used for fine-tuning on multi-candidate inference tasks, and proposes a new pre-training objective that models the paragraph-level semantics across multiple input sentences.

Best Paper Award
Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities
Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg, Margo Schlanger, Doug Downey
No items found.
  • arXiv
  • June 22, 2022
TLDR

Multi-LexSum, a collection of 9,280 expert-authored summaries drawn from ongoing CRLC writing, is introduced, demonstrating that despite the high-quality summaries in the training data, state-of-the-art summarization models perform poorly on this task.

Best Paper Award
Data Governance in the Age of Large-Scale Data-Driven Language Technology
Yacine Jernite, Huu Nguyen, Stella Rose Biderman, A. Rogers, Maraim Masoud, V. Danchev, Samson Tan, A. Luccioni, Nishant Subramani, Gérard Dupont, Jesse Dodge, Kyle Lo, Zeerak Talat, Isaac Johnson, Dragomir R. Radev, Somaieh Nikpoor, Jorg Frohberg, Aaron Gokaslan, Peter Henderson, Rishi Bommasani, Margaret Mitchell
No items found.
  • FAccT
  • June 21, 2022
TLDR

The framework presented is a multi-party international governance structure focused on language data, and incorporating technical and organizational tools needed to support its work.

Best Paper Award
VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groups
Zejiang Shen, Kyle Lo, Lucy Lu Wang, Bailey Kuehl, Daniel S. Weld, Doug Downey
No items found.
  • TACL
  • June 1, 2022
TLDR

We introduce new methods for incorporating VIsual LAyout (VILA) structures, e.g., the grouping of page texts into text lines or text blocks, into language models to further improve performance on automated scientific document understanding.

Best Paper Award
Penguins Don't Fly: Reasoning about Generics through Instantiations and Exceptions
Emily Allaway, Jena D. Hwang, Chandra Bhagavatula, K. McKeown, Doug Downey, Yejin Choi
No items found.
  • arXiv
  • May 23, 2022
TLDR

A novel framework to generate pragmatically relevant true and false instances of a generic, which outperforms few-shot generation from GPT-3 and high-lights the importance of constrained decoding for this task and the implications of generics EXEMPLARS for language inference tasks.

Best Paper Award
Generating Scientific Claims for Zero-Shot Scientific Fact Checking
Dustin Wright, David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Isabelle Augenstein, Lucy Lu Wang
No items found.
  • ACL
  • May 22, 2022
TLDR

This work proposes scientific claim generation, the task of generating one or more atomic and verifiable claims from scientific sentences, and demonstrates its usefulness in zero-shot fact checking for biomedical claims, and proposes CLAIMGEN-BART, a new supervised method for generating claims supported by the literature, as well as KBIN, a novel methods for generating claim negations.

Best Paper Award
ACCoRD: A Multi-Document Approach to Generating Diverse Descriptions of Scientific Concepts
Sonia K. Murthy, Kyle Lo, Daniel King, Chandra Bhagavatula, Bailey Kuehl, Sophie Johnson, Jon Borchardt, Daniel S. Weld, Tom Hope, Doug Downey
No items found.
  • arXiv
  • May 14, 2022
TLDR

ACCoRD, an end-to-end system tack-ling the novel task of generating sets of descriptions of scientific concepts, is presented and a user study is conducted demonstrating that users prefer descriptions produced by the system, and users prefer multiple descriptions to a single “best” description.

Best Paper Award
Scim: Intelligent Faceted Highlights for Interactive, Multi-Pass Skimming of Scientific Papers
Raymond Fok, Andrew Head, Jonathan Bragg, Kyle Lo, Marti A. Hearst, Daniel S. Weld
No items found.
  • arXiv
  • May 9, 2022
TLDR

Scim is presented, an AI-augmented reading interface designed to help researchers skim papers by automatically identifying, classifying, and highlighting salient sentences, organized into rhetorical facets rooted in common information needs.

Best Paper Award
Scaling Creative Inspiration with Fine-Grained Functional Facets of Product Ideas
Tom Hope, Ronen Tamari, Hyeonsu Kang, Daniel Hershcovich, J. Chan, A. Kittur, Dafna Shahaf
No items found.
  • CHI
  • May 1, 2022
TLDR

This work proposes a novel computational representation that automatically breaks up products into fine-grained functional facets, and designs similarity metrics that support granular matching between functional facets across ideas, and uses them to build a novel functional search capability that enables expressive queries for mechanisms and purposes.

Best Paper Award
From Who You Know to What You Read: Augmenting Scientific Recommendations with Implicit Social Networks
Hyeonsu Kang, Rafal Kocielnik, Andrew Head, Jiangjiang Yang, Matt Latzke, A. Kittur, Daniel S. Weld, Doug Downey, Jonathan Bragg
No items found.
  • CHI
  • April 30, 2022
TLDR

This work introduces multiple new methods for augmenting recommendations with textual relevance messages that highlight knowledge-graph connections between recommended papers and a user’s publication and interaction history and develops a novel method that highlights connections with proxy authors of interest to users.

Best Paper Award
S2AMP: A High-Coverage Dataset of Scholarly Mentorship Inferred from Publications
Shaurya Rohatgi, Doug Downey, Daniel King, Sergey Feldman
No items found.
  • JCDL
  • April 22, 2022
TLDR

This work contributes two datasets to the study of mentorship, one of which has over 300,000 ground truth academic mentor-mentee pairs obtained from multiple diverse, manually-curated sources, and linked to the Semantic Scholar (S2) knowledge graph.

Best Paper Award
Bursting Scientific Filter Bubbles: Boosting Innovation via Novel Author Discovery
Jason Portenoy, Marissa Radensky, Jevin D. West, E. Horvitz, Daniel S. Weld, Tom Hope
No items found.
  • CHI
  • April 13, 2022
TLDR

We construct a faceted representation of authors with information gleaned from their papers and inferred author personas, and use it to develop an approach that locates commonalities ("bridges") and contrasts between scientists. This approach helps users discover authors considered useful for generating novel research directions.

Best Paper Award
Infrastructure for rapid open knowledge network development
Michael Cafarella, Michael Anderson, Iz Beltagy, Arie Cattan, Sarah Chasins, Ido Dagan, Doug Downey, Oren Etzioni, Sergey Feldman, Tian Gao, Tom Hope, Kexin Huang, Sophie Johnson, Daniel King, Kyle Lo, Yuze Lou, Matthew Shapiro, Dinghao Shen, Shivashankar Subramanian, Lucy Lu Wang, Yuning Wang, Yitong Wang, Daniel Weld, Jenny Vo-Phamhi, Anna Zeng, Jiayun Zou
No items found.
  • AI Magazine
  • March 31, 2022
TLDR

A National Science Foundation Convergence Accelerator project is described to build a set of Knowledge Network Programming Infrastructure systems to address the issue of frustratingly slow building, using, and scaling large knowledge networks.

Best Paper Award
CiteRead: Integrating Localized Citation Contexts into Scientific Paper Reading
Napol Rachatasumrit, Jonathan Bragg, Amy X. Zhang, Daniel S. Weld
No items found.
  • IUI
  • March 21, 2022
TLDR

A novel paper reading experience that integrates relevant information about follow-on work directly into a paper, allowing readers to learn about newer papers and see how a paper is discussed by its citing papers in the context of the reference paper.

Best Paper Award
Don't Say What You Don't Know: Improving the Consistency of Abstractive Summarization by Constraining Beam Search
Daniel King, Zejiang Shen, Nishant Subramani, Daniel S. Weld, Iz Beltagy, Doug Downey
No items found.
  • arXiv
  • March 16, 2022
TLDR

PINOCCHIO is presented, a new decoding method that improves the consistency of a transformer-based abstractive summarizer by constraining beam search to avoid hallucinations.

Best Paper Award
LIMEADE: From AI Explanations to Advice Taking
B. Lee, Doug Downey, Kyle Lo, Daniel S. Weld
No items found.
  • arXiv
  • March 9, 2022
TLDR

From AI Explanations to Advice Taking: From AI explanations to advice taking is a guide for decision-making in the rapidly changing environment.

Best Paper Award
Paper Plain: Making Medical Research Papers Approachable to Healthcare Consumers with Natural Language Processing
Tal August, Lucy Lu Wang, Jonathan Bragg, Marti A. Hearst, Andrew Head, Kyle Lo
No items found.
  • arXiv
  • February 28, 2022
TLDR

To improve access to medical papers, we introduce a novel interactive interface-Paper Plain-with four features powered by natural language processing: definitions of unfamiliar terms, in-situ plain language section summaries, a collection of key questions that guide readers to answering passages, and plain language summaries of the answering passages.

Best Paper Award
A Search Engine for Discovery of Scientific Challenges and Directions
D. Lahav, Jon Saad-Falcon, Bailey Kuehl, Sophie Johnson, S. Parasa, N. Shomron, Duen Horng Chau, Diyi Yang, E. Horvitz, Daniel S. Weld, Tom Hope
No items found.
  • AAAI
  • February 21, 2022
TLDR

Our goal is to bolster the ability of researchers and clinicians to keep track of difficulties, limitations and emerging hypotheses.

Best Paper Award
FLEX: Unifying Evaluation for Few-Shot NLP
Jonathan Bragg, Arman Cohan, Kyle Lo, Iz Beltagy
No items found.
  • NeurIPS
  • December 6, 2021
TLDR

Few-shot NLP research lacks a unified, challenging-yet-realistic evaluation setup. In response, we introduce FLEX, a rigorous few-shot learning NLP benchmark and public leaderboard measuring four transfer types. We also present UniFew, a simple, competitive baseline that does not rely on heavy prompt engineering or complex meta-learning methods.

Best Paper Award
Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity
Sheshera Mysore, Arman Cohan, Tom Hope
No items found.
  • NAACL
  • November 16, 2021
TLDR

We present Aspire, a new scientific document similarity model based on matching fine-grained aspects.

Best Paper Award
Towards Personalized Descriptions of Scientific Concepts
Sonia K. Murthy, Daniel King, Tom Hope, Daniel S. Weld, Doug Downey
No items found.
  • EMNLP 2021 • WiNLP
  • November 11, 2021
TLDR

This paper proposes generating personalized scientific concept descriptions that are tailored to the user’s expertise and context and outlines a complete architecture for the task and releases an expert-annotated resource, ACCoRD.

Best Paper Award
MS2: Multi-Document Summarization of Medical Studies
Jay DeYoung, Iz Beltagy, Madeleine van Zuylen, Bailey Kuehl, Lucy Lu Wang
No items found.
  • EMNLP
  • November 7, 2021
TLDR

This work releases MS^2 (Multi-Document Summarization of Medical Studies ), a dataset of over 470k documents and 20K summaries derived from the scientific literature that facilitates the development of systems that can assess and aggregate contradictory evidence across multiple studies , and is the first large-scale, publicly available multi-document summarization dataset in the biomedical domain.

Best Paper Award
CDLM: Cross-Document Language Modeling
Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew E. Peters, Arie Cattan, Ido Dagan
No items found.
  • Findings of EMNLP
  • November 7, 2021
TLDR

A new pretrained language model for cross document tasks.

Best Paper Award
SciA11y: Converting Scientific Papers to Accessible HTML
Lucy Lu Wang, Isabel Cachola, Jonathan Bragg, Evie (Yu-Yen) Cheng, Chelsea Hess Haupt, Matt Latzke, Bailey Kuehl, Madeleine van Zuylen, Linda M. Wagner, Daniel S. Weld
No items found.
  • ASSETS
  • October 18, 2021
TLDR

We present SciA11y, a system that renders inaccessible scientific paper PDFs into HTML.

Best Paper Award
SciCo: Hierarchical Cross-Document Coreference for Scientific Concepts
Arie Cattan, Sophie Johnson, Daniel S. Weld, Ido Dagan, Iz Beltagy, Doug Downey, Tom Hope
No items found.
  • AKBC
  • October 4, 2021
TLDR

An extension of cross-document coreference with a referential hierarchy over mention clusters, in the scientific document domain. New task, dataset and models with applications in faceted document retrieval and knowledge base construction.

Best Paper Award
Scientific Language Models for Biomedical Knowledge Base Completion: An Empirical Study
Rahul Nadkarni, David Wadden, Iz Beltagy, Noah A. Smith, Hannaneh Hajishirzi, Tom Hope
No items found.
  • AKBC
  • October 4, 2021
TLDR

Integrating scientific language models and graph embeddings for boosting drug discovery.

Best Paper Award
S2AND: A Benchmark and Evaluation System for Author Name Disambiguation
Shivashankar Subramanian, Daniel King, Doug Downey, Sergey Feldman
No items found.
  • JCDL
  • September 27, 2021
TLDR

In response to this challenge, we present S2AND, a unified benchmark dataset for AND on scholarly papers, as well as an open-source reference model implementation.

Best Paper Award
PAWLS: PDF Annotation With Labels and Structure
Mark Neumann, Zejiang Shen, Sam Skjonsberg
No items found.
  • Demo • ACL
  • August 2, 2021
TLDR

PAWLS is a new annotation tool designed specifically for the PDF document format. PAWLS supports span-based textual annotation, N-ary relations and freeform, non-textual bounding boxes, all of which can be exported in convenient formats for training multi-modal machine learning models.

Best Paper Award
Explaining Relationships Between Scientific Documents
Kelvin Luu, Xinyi Wu, Rik Koncel-Kedziorski, Kyle Lo, Isabel Cachola, Noah A. Smit
No items found.
  • ACL
  • August 2, 2021
TLDR

We address the task of citation text generation: given a pair of scientific documents, explain their relationship in natural language text in the manner of a citation from one text to the other.

Best Paper Award
ParsiNLU: A Suite of Language Understanding Challenges for Persian
Daniel Khashabi, Arman Cohan, Siamak Shakeri, et al.
No items found.
  • TACL
  • July 1, 2021
TLDR

We introduce ParsiNLU, the first benchmark in Persian language that includes a range of high-level tasks -- Reading Comprehension, Textual Entailment, etc. These datasets are collected in a multitude of ways, often involving manual annotations by native speakers.

Best Paper Award
Overview and Insights from the SciVer Shared Task on Scientific Claim Verification
David Wadden, Kyle Lo
No items found.
  • SDP Workshop • NAACL
  • June 10, 2021
TLDR

We present an overview of the SCIVER shared task. In addition to surveying the participating systems, we provide several insights into modeling approaches to support continued progress and future research on scientific claim verification.

Best Paper Award
Extracting a Knowledge Base of Mechanisms from COVID-19 Papers
Aida Amini, T. Hope, David Wadden, Madeleine van Zuylen, E. Horvitz, Roy Schwartz, Hannaneh Hajishirzi
No items found.
  • NAACL
  • June 6, 2021
TLDR

To navigate the collection of COVID19 papers from different domains, we present a KB of mechanisms relating to COVID19, to support domain-agnostic search and exploration of general activities, functions, influences and associations in these papers.

Best Paper Award
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, Matt Gardner
No items found.
  • NAACL
  • June 6, 2021
TLDR

Qasper is a dataset of 5049 questions over 1585 NLP papers designed to facilitate document-grounded, information-seeking QA. Existing models that do well on other QA tasks do not perform well on these questions.

Best Paper Award
Simplified Data Wrangling with ir_datasets
Sean MacAvaney, Andrew Yates, Sergey Feldman, Doug Downey, Arman Cohan, Nazli Goharian
No items found.
  • arXiv
  • May 10, 2021
TLDR

A new robust and lightweight tool for acquiring, managing, and performing typical operations over datasets used in IR, primarily focus on textual datasets used for ad-hoc search.

Best Paper Award
Augmenting Scientific Papers with Just-in-Time, Position-Sensitive Definitions of Terms and Symbols
Andrew Head, Kyle Lo, Dongyeop Kang, Raymond Fok, Sam Skjonsberg, Daniel S. Weld, Marti A. Hearst
No items found.
  • CHI
  • May 8, 2021
TLDR

We introduce ScholarPhi, an augmented reading interface that brings definitions of technical terms and symbols to readers when and where they need them most.

Best Paper Award
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Gagan Bansal, Tongshuang (Sherry) Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Túlio Ribeiro, Daniel S. Weld
No items found.
  • CHI
  • May 8, 2021
TLDR

This work conducts mixed-method user studies on three datasets, where an AI with accuracy comparable to humans helps participants solve a task (explaining itself in some conditions), and observes complementary improvements from AI augmentation that were not increased by explanations.

Best Paper Award
What Do We Mean by “Accessibility Research”?: A Literature Survey of Accessibility Papers in CHI and ASSETS from 1994 to 2019
K. Mack, Emma J. McDonnell, Dhruv Jain, Lucy Lu Wang, Jon Froehlich, Leah Findlater
No items found.
  • CHI
  • May 8, 2021
TLDR

Accessibility research has grown substantially in the past few decades, yet there has been no literature review of the field. To understand current and historical trends, we created and analyzed a dataset of accessibility papers appearing at CHI and ASSETS since ASSETS' founding in 1994.

Best Paper Award
CODE: COMPILER-BASED NEURON-AWARE ENSEMBLE TRAINING
E. Trainiti, Thanapon Noraset, David Demeter, Doug Downey, Simone Campanoni
No items found.
  • Proceedings of Machine Learning and Systems
  • May 1, 2021
TLDR

CODE introduces neuron-level analyses and transformations aimed at identifying and removing redundant computation from the networks that compose the ensemble that enables CODE to train large DNN ensembles in a fraction of the time and memory footprint needed by current techniques.

Best Paper Award
Searching for Scientific Evidence in a Pandemic: An Overview of TREC-COVID
Kirk Roberts, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, Kyle Lo, I. Soboroff, E. Voorhees, Lucy Lu Wang, W. Hersh
No items found.
  • arXiv
  • May 1, 2021
TLDR

This paper provides a comprehensive overview of the structure and results of TREC-COVID, an information retrieval (IR) shared task to evaluate search on scientific literature related to COVID-19.

Best Paper Award
Improving the Accessibility of Scientific Documents: Current State, User Needs, and a System Solution to Enhance Scientific PDF Accessibility for Blind and Low Vision Users
Lucy Lu Wang, Isabel Cachola, Jonathan Bragg, Evie Yu-Yen Cheng, Chelsea Hess Haupt, Matt Latzke, Bailey Kuehl, Madeleine van Zuylen, Linda M. Wagner, Daniel S. Weld
No items found.
  • arXiv
  • April 30, 2021
TLDR

The majority of scientific papers are distributed in PDF, which pose challenges for accessibility, especially for blind and low vision (BLV) readers. We characterize the scope of this problem...

Best Paper Award
LayoutParser: A Unified Toolkit for Deep Learning Based Document Image Analysis
Zejiang Shen, Ruochen Zhang, Melissa Dell, B. Lee, Jacob Carlson, Weining Li
No items found.
  • arXiv
  • March 29, 2021
TLDR

An open-source library for streamlining the usage of deep learning in document image analysis research and applications.

Best Paper Award
Gender trends in computer science authorship
Lucy Lu Wang, Gabriel Stanovsky, Luca Weihs, Oren Etzioni
No items found.
  • CACM
  • March 1, 2021
TLDR

An analysis of 2.87 million computer science papers reveals that, if current trends continue, parity between the number of male and female authors will not be reached in this century. With optimistic projection models, gender parity is forecast to be reached by 2100 in CS, but projected to be reached within two to three decades in the biomedical literature.

Best Paper Award
Optimizing AI for Teamwork
Gagan Bansal, Besmira Nushi, Ece Kamar, E. Horvitz, Daniel S. Weld
No items found.
  • AAAI
  • February 2, 2021
TLDR

It is argued that AI systems should be trained in a human-centered manner, directly optimized for team performance, and the benefit of modeling teamwork during training through improvements in expected team utility across datasets, considering parameters such as human skill and the cost of mistakes.

Best Paper Award
On Generating Extended Summaries of Long Documents
Sajad Sotudeh, Arman Cohan, Nazli Goharian
No items found.
  • AAAI • Scientific Document Understanding Workshop
  • February 2, 2021
TLDR

In this paper, we present a new method for generating extended summaries of long papers.

Best Paper Award
GENIE: A Leaderboard for Human-in-the-Loop Evaluation of Text Generation
Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A. Smith, Daniel S. Weld
No items found.
  • arXiv
  • January 17, 2021
TLDR

This work introduces GENIE, an extensible human evaluation leaderboard, which brings the ease of leaderboards to text generation tasks. GENIE automatically posts leaderboard submissions to crowdsourcing platforms and presents both manual and automatic metrics on the leaderboard.

Best Paper Award
Text mining approaches for dealing with the rapidly expanding literature on COVID-19
Lucy Lu Wang, Kyle Lo
No items found.
  • Briefings in Bioinformatics
  • December 7, 2020
TLDR

This review discusses the corpora, modeling resources, systems and shared tasks that have been introduced for COVID-19, and lists 39 systems that provide functionality such as search, discovery, visualization and summarization over the COVID-19 literature.

Best Paper Award
Mitigating Biases in CORD-19 for Analyzing COVID-19 Literature
Anshul Kanakia, Kuansan Wang, Yuxiao Dong, Boya Xie, Kyle Lo, Zhihong Shen, Lucy Lu Wang, Chiyuan Huang, Darrin Eide, Sebastian Kohlmeier, Chieh-Han Wu
No items found.
  • Frontiers in Research Metrics and Analytics
  • November 23, 2020
TLDR

The results suggest that while CORD-19 exhibits a strong tilt toward recent and topically focused articles, the knowledge being explored to attack the pandemic encompasses a much longer time span and is very interdisciplinary.

Best Paper Award
PySBD: Pragmatic Sentence Boundary Disambiguation
Nipun Sadvilkar, M. Neumann
No items found.
  • EMNLP • NLP-OSS Workshop
  • November 19, 2020
TLDR

This work adapts the Golden Rules Set (a language specific set of sentence boundary exemplars) originally implemented as a ruby gem pragmatic segmenter to Python, ported to Python with additional improvements and functionality.

Best Paper Award
Document-Level Definition Detection in Scholarly Documents: Existing Models, Error Analyses, and Future Directions
Dongyeop Kang, Andrew Head, Risham Sidhu, Kyle Lo, Daniel S. Weld, Marti A. Hearst
No items found.
  • EMNLP • SDP workshop
  • November 19, 2020
TLDR

The task of definition detection is important for scholarly papers, because papers often make use of technical terminology that may be unfamiliar to readers. We develop a new definition detection system, HEDDEx, that utilizes syntactic features, transformer encoders, and heuristic filters, and evaluate it on a standard sentence-level benchmark.

Best Paper Award
Fact or Fiction: Verifying Scientific Claims
David Wadden, Kyle Lo, Lucy Lu Wang, Shanchuan Lin, Madeleine van Zuylen, Arman Cohan, Hannaneh Hajishirzi
No items found.
  • EMNLP
  • November 16, 2020
TLDR

We construct SciFact, a dataset of 1.4K expert-written scientific claims paired with evidence-containing abstracts annotated with labels and rationales. We develop baseline models for SciFact, and demonstrate that these models benefit from combined training on a large dataset of claims about Wikipedia articles, together with the new SciFact data.

Best Paper Award
TLDR: Extreme Summarization of Scientific Documents
Isabel Cachola, Kyle Lo, Arman Cohan, Daniel S. Weld
No items found.
  • Findings of EMNLP
  • November 16, 2020
TLDR

We introduce TLDR generation for scientific papers, a new automatic summarization task with high source compression and provide a new dataset and models for effective generation of TLDRs.

Best Paper Award
SciSight: Combining faceted navigation and research group detection for COVID-19 exploratory scientific search
Tom Hope, Jason Portenoy, Kishore Vasan, Jonathan Borchardt, Eric Horvitz, Daniel S. Weld, Marti A. Hearst, Jevin D. West
No items found.
  • EMNLP • Demo
  • November 16, 2020
TLDR

SciSight is a novel framework for exploratory search of COVID-19 research that integrates two key capabilities: first, exploring interactions between biomedical facets (e.g., proteins, genes, drugs, diseases, patient characteristics); and second, discovering groups of researchers and how they are connected.

Best Paper Award
SLEDGE-Z: A Zero-Shot Baseline for COVID-19 Literature Search
S. MacAvaney, Arman Cohan, N. Goharian
No items found.
  • EMNLP
  • November 16, 2020
TLDR

We present a zero-shot ranking algorithm that adapts to COVID-related scientific literature. Our approach filters training data from another collection down to medical-related queries, uses a neural reranking model pre-trained on scientific text (SciBERT), and filters the target document collection.

Best Paper Award
MedICaT: A Dataset of Medical Images, Captions, and Textual References
Sanjay Subramanian, Lucy Lu Wang, Sachin Mehta, Ben Bogin, Madeleine van Zuylen, Sravanthi Parasa, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi
No items found.
  • Findings of EMNLP
  • November 16, 2020
TLDR

To address challenges in figure retrieval and figure-to-text alignment, we introduce MedICaT, a dataset of medical images in context.

Best Paper Award
ABNIRML: Analyzing the Behavior of Neural IR Models
Sean MacAvaney, Sergey Feldman, Nazli Goharian, Doug Downey, Arman Cohan
No items found.
  • arXiv
  • November 2, 2020
TLDR

A new comprehensive framework for Analyzing the Behavior of Neural IR ModeLs (ABNIRML), which includes new types of diagnostic tests that allow us to probe several characteristics---such as sensitivity to word order---that are not addressed by previous techniques.

Best Paper Award
Generative Data Augmentation for Commonsense Reasoning
Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, J. Wang, Chandra Bhagavatula, Yejin Choi, Doug Downey
No items found.
  • Findings of EMNLP
  • October 6, 2020
TLDR

This work investigates G-DAUG^C, a novel generative data augmentation method that aims to achieve more accurate and robust learning in the low-resource setting, and demonstrates that it produces a diverse set of fluent training examples, and that its selection and training approaches are important for performance.

Best Paper Award
Modelling kidney disease using ontology: insights from the Kidney Precision Medicine Project
E. Ong, L. Lu Wang, J. Schaub, J. O’Toole, B. Steck, A. Rosenberg, F. Dowd, J. Hansen, L. Barisoni, S. Jain, I. D. de Boer, M. Valerius, S. Waikar, C. Park, D. Crawford, T. Alexandrov, C. Anderton, C. Stoeckert, C. Weng, et al
No items found.
  • Nature Reviews Nephrology
  • September 16, 2020
TLDR

Ontologies are critical to support the types of big data analysis necessary for kidney precision medicine, where heterogeneous clinical, imaging and biopsy data from diverse sources must be combined to define a patient's phenotype.

Best Paper Award
High-Precision Extraction of Emerging Concepts from Scientific Literature
Daniel King, Doug Downey, Daniel S. Weld
No items found.
  • SIGIR
  • July 25, 2020
TLDR

A novel, unsupervised method for extracting scientific concepts from papers, based on the intuition that each scientific concept is likely to be introduced or popularized by a single paper that is disproportionately cited by subsequent papers mentioning the concept.

Best Paper Award
CORD-19: The Covid-19 Open Research Dataset
L. Lu Wang, K. Lo, Y. Chandrasekhar, R. Reas, J. Yang, D. Eide, K. Funk, R. Kinney, Z. Liu, W. Merrill, P. Mooney, D. Murdick, D. Rishi, J. Sheehan, Z. Shen, B. Stilson, A. D Wade, K. Wang, C. Wilhelm, B. Xie, D.Raymond, D. S Weld, O. Etzioni, S. Kohlmeier
No items found.
  • ACL • NLP-Covid
  • July 9, 2020
TLDR

The Covid-19 Open Research Dataset (CORD-19) is a growing 1 resource of scientific papers on Covid-19 and related historical coronavirus research. CORD-19 is designed to facilitate the development of text mining and information retrieval systems over its rich collection of metadata and structured full text papers.

Best Paper Award
SUPP. AI: finding evidence for supplement-drug interactions
Lucy Lu Wang, Oyvind Tafjord, Arman Cohan, Sarthak Jain, Sam Skjonsberg, Carissa Schoenick, Nick Botner, Waleed Ammar
No items found.
  • ACL• Demo
  • July 9, 2020
TLDR

SUPP.AI is an attempt to close the information gap on dietary supplements by making up-to-date evidence on SDIs more discoverable for researchers, clinicians, and consumers.

Best Paper Award
Stolen Probability: A Structural Weakness of Neural Language Models
David Demeter, Gregory Kimmel, Doug Downey
No items found.
  • ACL
  • July 5, 2020
TLDR

We show that the softmax output common in neural language models leads to a limitation: some words (in particular, those with an embedding interior to the convex hull of the embedding space) can never be assigned high probability by the model, no matter what the context.

Best Paper Award
SPECTER: Document-level Representation Learning using Citation-informed Transformers
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, Daniel S. Weld
No items found.
  • ACL
  • July 5, 2020
TLDR

This work proposes SPECTER, a new method to generate document-level embedding of scientific papers based on pretraining a Transformer language model on a powerful signal of document- level relatedness: the citation graph, and shows that Specter outperforms a variety of competitive baselines on the benchmark.

Best Paper Award
Language (Re)modelling: Towards Embodied Language Understanding
Ronen Tamari, Chen Shani, Tom Hope, Miriam R. L. Petruck, Omri Abend, Dafna Shahaf
No items found.
  • ACL
  • July 5, 2020
TLDR

We bring together ideas from cognitive science and AI/NLU, arguing that grounding by analogical inference and executable simulation will greatly benefit NLU systems. We propose a system architecture along with a roadmap towards realizing this vision.

Best Paper Award
SciREX: A Challenge Dataset for Document-Level Information Extraction
Sarthak Jain, Madeleine van Zuylen, Hannaneh Hajishirzi, Iz Beltagy
No items found.
  • ACL
  • July 5, 2020
TLDR

We introduce a new dataset called SciREX that requires understanding of the whole document to annotate entities, and their document-level relationships that usually span beyond sentences or even sections.

Best Paper Award
S2ORC: The Semantic Scholar Open Research Corpus
Kyle Lo, Lucy Lu Wang, Mark E Neumann, Rodney Michael Kinney, Daniel S. Weld
No items found.
  • ACL
  • July 5, 2020
TLDR

We introduce S2ORC, a large contextual citation graph of English-language academic papers from multiple scientific domains; the corpus consists of 81.1M papers, 380.5M citation edges, and associated paper metadata.

Best Paper Award
TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection
Ellen M. Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, Lucy Lu Wang
No items found.
  • arXiv
  • May 9, 2020
TLDR

TREC-COVID is a community evaluation designed to build a test collection that captures the information needs of biomedical researchers using the scientific literature during a pandemic.

Best Paper Award
TREC-COVID: Rationale and Structure of an Information Retrieval Shared Task for COVID-19
Kirk Roberts, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, Kyle Lo, Ian Soboroff, Ellen M. Voorhees, Lucy Lu Wang, William R. Hersh
No items found.
  • JAMIA
  • May 4, 2020
TLDR

This article presents a brief description of the rationale and structure of TREC-COVID, a still-ongoing IR evaluation. TREC-COVID is creating a new paradigm for search evaluation in rapidly evolving crisis scenarios.

Best Paper Award
Ranking Significant Discrepancies in Clinical Reports
Sean MacAvaney, Arman Cohan, Nazli Goharian, Ross Filice
No items found.
  • ECIR
  • April 14, 2020
TLDR

A novel ranking approach, consisting of textual and ontological overlaps between the preliminary and final versions of reports, is proposed, which allows medical practitioners to easily identify and learn from the reports in which their interpretation most substantially differed from that of the attending physician.

Best Paper Award
Longformer: The Long-Document Transformer
Iz Beltagy, Matthew E. Peters, Arman Cohan
No items found.
  • arXiv
  • April 10, 2020
TLDR

We introduce the Longformer, with an attention mechanism that scales linearly with sequence length, achieving state-of-the-art results on multiple character-level language modeling and document-level tasks.

Best Paper Award
Just Add Functions: A Neural-Symbolic Language Model
David Demeter, Doug Downey
No items found.
  • arXiv
  • December 11, 2019
TLDR

We present a model based on pretrained language models for classifying sentences in context of other sentences. Achieves SOTA results on 4 datasets on 2 different domains. We also release a challenging dataset of 2K discourse facets in CS domain.

Best Paper Award
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, Omer Levy
No items found.
  • EMNLP
  • November 3, 2019
TLDR

The approach extends BERT by masking contiguous random spans, rather than random tokens, and training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it.

Best Paper Award
SciBERT: A Pretrained Language Model for Scientific Text
Iz Beltagy, Kyle Lo, Arman Cohan
No items found.
  • EMNLP
  • November 3, 2019
TLDR

SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks and demonstrates statistically significant improvements over BERT.

Best Paper Award
Pretrained Language Models for Sequential Sentence Classification
Arman Cohan, Iz Beltagy, Daniel King, Bhavana Dalvi, Daniel S. Weld
No items found.
  • EMNLP
  • November 3, 2019
TLDR

We present a model based on pretrained language models for classifying sentences in context of other sentences. Achieves SOTA results on 4 datasets on 2 different domains. We also release a challenging dataset of 2K discourse facets in CS domain.

Best Paper Award
GrapAL: Connecting the Dots in Scientific Literature
Christine Betts, Joanna Power, Waleed Ammar
No items found.
  • ACL
  • October 4, 2019
TLDR

The basic elements of GrapAL are described, how to use it, and several use cases such as finding experts on a given topic for peer reviewing, discovering indirect connections between biomedical entities, and computing citation-based metrics are described.

Best Paper Award
ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing
Mark Neumann, Daniel King, Iz Beltagy, Waleed Ammar
No items found.
  • ACL • BioNLP Workshop
  • July 28, 2019
TLDR

We created a spaCy pipeline for biomedical and scientific text processing. The core models include dependency parsing, part of speech tagging, and named entity recognition models retrained on general biomedical text, and custom tokenization. We also release four specific named entity recognition models for more focused biomedical entity recognition. Additionally, we include optional components for abbreviation resolution, simple entity linking to UMLS, and sentence splitting.

Best Paper Award
Ontology-Aware Clinical Abstractive Summarization
Sean MacAvaney, Sajad Sotudeh, Arman Cohan, Nazli Goharian, Ish Talati, Ross W. Filice
No items found.
  • SIGIR
  • July 21, 2019
TLDR

A sequence-to-sequence abstractive summarization model augmented with domain-specific ontological information to enhance content selection and summary generation is proposed and significantly outperforms the current state-of-the-art on this task in terms of rouge scores.

Best Paper Award
CEDR: Contextualized Embeddings for Document Ranking
Sean MacAvaney, Andrew Yates, Arman Cohan, Nazli Goharian
No items found.
  • SIGIR
  • July 21, 2019
TLDR

This work investigates how two pretrained contextualized language models (ELMo and BERT) can be utilized for ad-hoc document ranking and proposes a joint approach that incorporates BERT's classification vector into existing neural models and shows that it outperforms state-of-the-art ad-Hoc ranking baselines.

Best Paper Award
Quantifying Sex Bias in Clinical Studies at Scale With Automated Data Extraction
Sergey Feldman, Waleed Ammar, Kyle Lo, Elly Trepman, Madeleine van Zuylen, Oren Etzioni
No items found.
  • JAMA
  • July 3, 2019
TLDR

We extracted counts of women and men from over 40k published clinical trial articles and found substantial underrepresentation of female participants in 7 of 11 disease categories, especially HIV/AIDS, chronic kidney diseases, and cardiovascular diseases.

Best Paper Award
Combining Distant and Direct Supervision for Neural Relation Extraction
Iz Beltagy, Kyle Lo, Waleed Ammar
No items found.
  • NAACL
  • June 2, 2019
TLDR

An effective multitask learning setup for reducing distant supervision noise by leveraging sentence-level supervision is proposed, and a novel neural architecture for collecting signals from multiple input sentences, which combines the benefits of attention and maxpooling is introduced.

Best Paper Award
Structural Scaffolds for Citation Intent Classification in Scientific Publications
Arman Cohan, Waleed Ammar, Madeleine van Zuylen, Field Cady
No items found.
  • NAACL
  • June 2, 2019
TLDR

We propose a new scaffolding model for classifying citation intents using two auxiliary tasks to handle low-resouce training data. We additionally propose SciCite, a multi-domain dataset of citation intents.

Best Paper Award
Construction of the Literature Graph in Semantic Scholar
Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew E. Peters, et al.
No items found.
  • NAACL-HLT
  • July 5, 2018
TLDR

This paper introduces the Semantic Scholar literature graph, consisting of more than 280M nodes, representing papers, authors, entities and various interactions between them. [acknowledgements: TAGME entity linker (https://tagme.d4science.org/)]

Best Paper Award
Citation Count Analysis for Papers with Preprints
Sergey Feldman, Kyle Lo, Waleed Ammar
No items found.
  • ArXiv
  • July 5, 2018
TLDR

We explore the degree to which papers prepublished on arXiv garner more citations, in an attempt to paint a sharper picture of fairness issues related to prepublishing. We observe that papers submitted to arXiv before acceptance have, on average, 65% more citations in the following year compared to papers submitted after, even after accounting for variables such as venue and author influentialness.

Best Paper Award
Extracting Scientific Figures with Distantly Supervised Neural Networks
Noah Siegel, Nicholas Lourie, Russell Power and Waleed Ammar
No items found.
  • JCDL
  • June 25, 2018
TLDR

In this paper, we induce high-quality training labels for the task of figure extraction in a large number of scientific documents, with no human intervention.

Best Paper Award
Content-Based Citation Recommendation
Chandra Bhagavatula, Sergey Feldman, Russell Power, Waleed Ammar
No items found.
  • NAACL-HLT
  • June 25, 2018
TLDR

We embed a given query document into a vector space, then use its nearest neighbors as candidates, and rerank the candidates using a discriminative model trained to distinguish between observed and unobserved citations.

Best Paper Award
A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications
Dongyeop Kang, Waleed Ammar, Bhavana Dalvi Mishra, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, Roy Schwartz
No items found.
  • NAACL-HLT
  • June 25, 2018
TLDR

We present the first public dataset of scientific peer reviews available for research purposes, containing 14.7K paper drafts and the corresponding accept/reject decisions in top-tier venues.

Best Paper Award
Ontology Alignment in the Biomedical Domain Using Entity Definitions and Context
Lucy L. Wang, Chandra Bhagavatula, M. Neumann, Kyle Lo, Chris Wilhelm, Waleed Ammar
No items found.
  • ACL • Proceedings of the BioNLP 2018 Workshop
  • June 20, 2018
TLDR

This ontology matcher can be used to generate alignments between entities in two biomedical ontologies. The matcher uses entity definitions and usage context retrieved from the Semantic Scholar corpus to assist in entity matching.

Best Paper Award
Semi-supervised sequence tagging with bidirectional language models
Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power
No items found.
  • ACL
  • July 1, 2017
TLDR

A general semi-supervised approach for adding pretrained context embeddings from bidirectional language models to NLP systems and apply it to sequence labeling tasks, surpassing previous systems that use other forms of transfer or joint learning with additional labeled data and task specific gazetteers.

Best Paper Award
AI zooms in on highly influential citations
Oren Etzioni
No items found.
  • Nature
  • June 25, 2017
TLDR

The number of times a paper is cited is a poor proxy for its impact (see P. Stephan et al. Nature 544, 411–412; 2017). I suggest relying ins...

Best Paper Award
Ontology Aware Token Embeddings for Prepositional Phrase Attachment
Pradeep Dasigi, Waleed Ammar, Chris Dyer, and Eduard Hovy
No items found.
  • ACL
  • June 25, 2017
TLDR

Using context-sensitive embeddings in a model for predicting prepositional phrase (PP) attachments improves the accuracy of the PP attachment model by 5.4% absolute points, which amounts to a 34.

Best Paper Award
End-to-End Neural Ad-hoc Ranking with Kernel Pooling
Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power
No items found.
  • SIGIR
  • June 25, 2017
TLDR

K-NRM uses a translation matrix that models word-level similarities via word embeddings, a new kernel-pooling technique that uses kernels to extract multi-level soft match features, and a learning-to-rank layer that combines those features into the final ranking score.

Best Paper Award
Learning to Predict Citation-Based Impact Measures
Luca Weihs and Oren Etzioni
No items found.
  • JCDL
  • June 25, 2017
TLDR

This work finds that existing probabilistic models for paper citations can predict measures of scientific impact for papers and authors, namely citation rates and h-indices, with surprising accuracy, even 10 years into the future.

Best Paper Award
The AI2 system at SemEval-2017 Task 10 (ScienceIE): semi-supervised end-to-end entity and relation extraction
Waleed Ammar, Matthew E. Peters, Chandra Bhagavatula, and Russell Power
No items found.
  • SemEval
  • June 25, 2017
TLDR

Our submission to SemEval 2017 Task 10 (ScienceIE) shared task placed 1st in end-to-end entity and relation extraction and 2nd in relation-only extraction. We find that pretraining neural forward and backward language model produces word representations that can drastically improve model performance. This finding resulted in the later development of ELMo contextualized embeddings.

Best Paper Award
Explicit Semantic Ranking for Academic Search via Knowledge Graph Embedding
Chenyan Xiong, Russell Power and Jamie Callan
No items found.
  • WWW
  • June 25, 2017
TLDR

Explicit Semantic Ranking is introduced, a new ranking technique that leverages knowledge graph embedding that represents queries and documents in the entity space and ranks them based on their semantic connections from their knowledgegraph embedding.

Best Paper Award
PDFFigures 2.0: Mining Figures from Research Papers
Christopher Clark and Santosh Divvala
No items found.
  • JCDL
  • June 25, 2016
TLDR

An algorithm that extracts figures, tables, and captions from documents called “PDFFigures 2.0” that analyzes the structure of individual pages by detecting captions, graphical elements, and chunks of body text, and then locates figures and tables by reasoning about the empty regions within that text.

Best Paper Award

Experience a smarter way to search and discover scholarly research.

Create Your Account