• Corpus ID: 233210295

Predicting the Reproducibility of Social and Behavioral Science Papers Using Supervised Learning Models

  title={Predicting the Reproducibility of Social and Behavioral Science Papers Using Supervised Learning Models},
  author={Jian Wu and Rajal Nivargi and Sree Sai Teja Lanka and Arjun Manoj Menon and Sai Ajay Modukuri and Nishanth Nakshatri and Xin Wei and Zhuoer Wang and James Caverlee and Sarah Michele Rajtmajer and C. Lee Giles},
In recent years, significant effort has been invested verifying the reproducibility and robustness of research claims in social and behavioral sciences (SBS), much of which has involved resourceintensive replication projects. In this paper, we investigate prediction of the reproducibility of SBS papers using machine learning methods based on a set of features. We propose a framework that extracts five types of features from scholarly work that can be used to support assessments of… 

Figures and Tables from this paper


Mathematically aggregating experts’ predictions of possible futures
The main goal of this research is to explore how measures of knowledge and behaviour of individuals can be leveraged to produce a better performing combined group judgment.
Using prediction markets to predict the outcomes in the Defense Advanced Research Projects Agency's next-generation social science programme
This study sets up prediction markets for hypotheses tested in the Defense Advanced Research Projects Agency's (DARPA) Next Generation Social Science (NGS2) programme and asks participants to bet on the expected effect size for each hypothesis.
Acknowledgement Entity Recognition in CORD-19 Papers
An acknowledgement extraction system based on open-source text mining software is developed and evaluated using manually labeled data and it is found that only up to 50–60% of named entities are actually acknowledged.
Analytic-thinking predicts hoax beliefs and helping behaviors in response to the COVID-19 pandemic
Abstract The COVID-19 outbreak was labeled a global pandemic by the WHO in March of 2020. During that same month, the number of confirmed cases and the death rate grew exponentially in the United
CiteScore: Advances, Evolution, Applications, and Limitations
Elsevier’s (Scopus) CiteScore, a journal-based metric (JBM), has been assigned to more journals than Clarivate Analytics’ Journal Impact Factor (JIF), including journals indexed by Scopus that do not
Estimating the deep replicability of scientific findings using human and artificial intelligence
An artificial intelligence model is trained to estimate a paper’s replicability using ground truth data on studies that had passed or failed manual replication tests, and its generalizability is tested on an extensive set of out-of-sample studies.
Replicability, Robustness, and Reproducibility in Psychological Science
For psychology, the 2010s might be characterized as a decade of active confrontation as systematic and multi-site replication projects assessed current understanding and observed surprising failures to replicate many published findings.
Tracking self-citations in academic publishing
The view is that hiding self-citation data is indefensible and needlessly confuses any attempts to understand the bibliometric impact of one’s work and urges academics to embrace visibility of citation data in a community of peers, which relies on nuance and openness rather than curated scorekeeping.
Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories
Citations are increasingly used as performance indicators in research policy and within the research system. Usually, citations are assumed to reflect the impact of the research or its quality. What