Corpus ID: 235422083

Do peers share the same criteria for assessing grant applications?

@article{Hug2021DoPS,
  title={Do peers share the same criteria for assessing grant applications?},
  author={Sven E. Hug and M. Ochsner},
  journal={ArXiv},
  year={2021},
  volume={abs/2106.07386}
}
This study examines a basic assumption of peer review, namely, the idea that there is a consensus on evaluation criteria among peers, which is a necessary condition for the reliability of peer judgements. Empirical evidence indicating that there is no consensus or more than one consensus would offer an explanation for the disagreement effect, the low inter-rater reliability consistently observed in peer review. To investigate this basic assumption, we have surveyed all humanities scholars in… Expand

Figures and Tables from this paper

References

SHOWING 1-10 OF 68 REFERENCES
Risk evaluation in peer review of grant applications
The process of peer review is used to identify the most scientifically meritorious research projects for funding. Impact and innovation are among the criteria used to determine overall merit. AExpand
Criteria for assessing grant applications: a systematic review
TLDR
This systematic review identifies and synthesizes studies that examine grant peer review criteria in an empirical and inductive manner and finds that some of the criteria peers use to evaluate grant applications do not conform to the fairness doctrine and the ideal of impartiality. Expand
The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation
The reliability of peer review of scientific documents and the evaluative criteria scientists use to judge the work of their peers are critically reexamined with special attention to the consistentlyExpand
Does reviewing experience reduce disagreement in proposals evaluation? Insights from Marie Skłodowska-Curie and COST Actions
We have limited understanding of why reviewers tend to strongly disagree when scoring the same research proposal. Thus far, research that explored disagreement has focused on the characteristics ofExpand
Bias in peer review
TLDR
This review provides a brief description of the function, history, and scope of peer review, and characterizes and examines the empirical, methodological, and normative claims of bias in peer review research; and assesses possible alternatives to the status quo. Expand
The future of societal impact assessment using peer review: pre-evaluation training, consensus building and inter-reviewer reliability
There are strong political reasons underpinning the desire to achieve a high level of inter-reviewer reliability (IRR) within peer review panels. Achieving a high level of IRR is synonymous with anExpand
Metascience on Peer Review: Testing the Effects of a Study’s Originality and Statistical Significance in a Field Experiment
TLDR
The general aim of this experiment was to demonstrate the feasibility and value of metascientific experiments on the peer-review process and thereby encourage research that will lead to understanding its mechanisms and determinants, effectively contextualizing it in psychological theories of various biases, and developing practical procedures to increase its utility. Expand
The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications
TLDR
A retrospective multi-level regression analysis of individual unblinded evaluations of 725 biomedical research funding applications by 1,044 reviewers found that reviewer and applicant seniority could influence this relationship, suggesting social networks could have subtle influences on reviewer scoring. Expand
'Your comments are meaner than your score': score calibration talk influences intra- and inter-panel variability during scientific grant peer review.
TLDR
Results suggest that although reviewers within a single panel agree more following collaborative discussion, different panels agree less after discussion, and Score Calibration Talk plays a pivotal role in scoring variability during peer review. Expand
Grants Peer Review in Theory and Practice
Grants peer review is a family of ex ante methods used by federal agencies to select research proposalsforfunding. This article draws on Chubin and Hackett's 1990 book, Peerless Science: Peer ReviewExpand
...
1
2
3
4
5
...