• Corpus ID: 236087830

Overview and Insights from the SciVer Shared Task on Scientific Claim Verification

@article{Wadden2021OverviewAI,
  title={Overview and Insights from the SciVer Shared Task on Scientific Claim Verification},
  author={David Wadden and Kyle Lo},
  journal={ArXiv},
  year={2021},
  volume={abs/2107.08188}
}
We present an overview of the SCIVER shared task, presented at the 2nd Scholarly Document Processing (SDP) workshop at NAACL 2021. In this shared task, systems were provided a scientific claim and a corpus of research abstracts, and asked to identify which articles SUPPORT or REFUTE the claim as well as provide evidentiary sentences justifying those labels. 11 teams made a total of 14 submissions to the shared task leaderboard, leading to an improvement of more than +23 F1 on the primary task… 

Tables from this paper

LongChecker: Improving scientific claim verification by modeling full-abstract context
TLDR
It is found that identifying the relationship between a claim and a rationale reporting a scientific finding often requires understanding the context in which the rationale appears, and LONGCHECKER achieves better performance on cases requiring this type of understanding.
MultiVerS: Improving scientific claim verification with weak supervision and full-document context
TLDR
This work presents M ULTI V ER S, which predicts a fact-checking label and identifies rationales in a multitask fashion based on a shared encoding of the claim and full document context, which allows the model to perform weakly-supervised domain adaptation by training on scientific documents labeled using high-precision heuristics.
Overview of the Second Workshop on Scholarly Document Processing
TLDR
The 2nd Workshop on Scholarly Document Processing (SDP) at NAACL 2021 aimed to connect researchers and practitioners from different communities working with scientific literature and data and created a premier meeting point to facilitate discussions on open problems in SDP.

References

SHOWING 1-10 OF 21 REFERENCES
Fact or Fiction: Verifying Scientific Claims
We introduce scientific claim verification, a new task to select abstracts from the research literature containing evidence that supports or refutes a given scientific claim, and to identify
QMUL-SDS at SCIVER: Step-by-Step Binary Classification for Scientific Claim Verification
TLDR
This paper presents team QMUL-SDS’s participation in the SCIVER shared task and proposes an approach that performs scientific claim verification by doing binary classifications step-by-step.
A Paragraph-level Multi-task Learning Model for Scientific Fact-Verification
TLDR
This work proposes a novel, paragraphlevel, multi-task learning model for the SCIFACT task by directly computing a sequence of contextualized sentence embeddings from a BERT model and jointly training the model on rationale selection and stance prediction.
Scientific Claim Verification with VerT5erini
This work describes the adaptation of a pretrained sequence-to-sequence model to the task of scientific claim verification in the biomedical domain. We propose a system called VerT5erini that
Evidence-based Verification for Real World Information Needs
TLDR
A novel claim verification dataset with instances derived from search-engine queries, yielding 10,987 claims annotated with evidence that represent real-world information needs that enables systems to use evidence extraction to summarize a rationale for an end-user while maintaining the accuracy when predicting a claim's veracity.
FEVER: a Large-scale Dataset for Fact Extraction and VERification
TLDR
This paper introduces a new publicly available dataset for verification against textual sources, FEVER, which consists of 185,445 claims generated by altering sentences extracted from Wikipedia and subsequently verified without knowledge of the sentence they were derived from.
e-FEVER: Explanations and Summaries forAutomated Fact Checking
TLDR
This paper demonstrates the capability of a large pre-trained language model (GPT-3) to automatically generate explanations for fact checks, and shows that the machine-generated explanations can themselves serve as evidence for automatically making true/false determinations.
SciBERT: A Pretrained Language Model for Scientific Text
TLDR
SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks and demonstrates statistically significant improvements over BERT.
Adapting Open Domain Fact Extraction and Verification to COVID-FACT through In-Domain Language Modeling
TLDR
SciKGAT is proposed to combine the advantages of open-domain literature search, state-of-the-art fact verification systems and in-domain medical knowledge through language modeling, and shows that improvement thrives from the in- domain language model by picking up more related evidence pieces and accurate fact verification.
Fine-grained Fact Verification with Kernel Graph Attention Network
TLDR
This paper presents Kernel Graph Attention Network (KGAT), which conducts more fine-grained fact verification with kernel-based attentions and significantly outperforms existing fact verification models on FEVER, a large-scale benchmark for fact verification.
...
...