Arbitrariness in the peer review process

@article{Brezis2020ArbitrarinessIT,
  title={Arbitrariness in the peer review process},
  author={Elise S. Brezis and Aliaksandr Birukou},
  journal={Scientometrics},
  year={2020},
  volume={123},
  pages={393-411}
}
The purpose of this paper is to analyze the causes and effects of arbitrariness in the peer review process. This paper focuses on two main reasons for the arbitrariness in peer review. The first is that referees are not homogenous and display homophily in their taste and perception of innovative ideas. The second element is that reviewers are different in the time they allocate for peer review. Our model replicates the NIPS experiment of 2014, showing that the ratings of peer review are not… Expand

Figures, Tables, and Topics from this paper

Disagreement and Agonistic Chance in Peer Review
The purpose of grant peer review is to identify the most excellent and promising research projects. However, sociologists of science and STS scholars have shown that peer review tends to promoteExpand
Do peers share the same criteria for assessing grant applications?
TLDR
This study examines a basic assumption of peer review, namely, the idea that there is a consensus on evaluation criteria among peers, which is a necessary condition for the reliability of peer judgements, and identifies two consensus classes, two consensus-close classes, and a consensus-far class. Expand
Bibliometrically Disciplined Peer Review: on Using Indicators in Research Evaluation
Evaluation of research uses peer review and bibliometrics, and the debate about their balance in research evaluation continues. Both approaches have supporters, and both approaches are criticized. InExpand
The oracles of science: On grant peer review and competitive funding
From a purely epistemological point of view, evaluating and predicting the future success of new research projects is often considered very difficult. Is it possible to forecast important findingsExpand
What is meaningful research and how should we measure it?
TLDR
It is claimed that, rather than promoting meaningful research, purely metric-based research evaluation schemes potentially lead to a dystopian academic reality, leaving no space for creativity and intellectual initiative. Expand
A Model of Winners Allocation
  • Yongjie Yang
  • Computer Science
  • AAAI
  • 2021
TLDR
A model of winners allocation is proposed to find two disjoint winning committees from respectively the two elections that are subjected to certain reasonable restrictions and the implication relationships among these properties are investigated. Expand

References

SHOWING 1-10 OF 40 REFERENCES
Improving the Peer review process: Capturing more information and enabling high-risk/high-return research
The Black-Scholes model and the peer-review process are combined to offer more insight into the apparent value of research projects. In doing so high-risk/high-return research is found to be moreExpand
On peer review in computer science: analysis of its effectiveness and suggestions for improvement
TLDR
The development, definition and rationale of a theoretical model for peer review processes are reported on to support the identification of appropriate metrics to assess the processes main characteristics in order to render peer review more transparent and understandable. Expand
Arbitrariness of peer review: A Bayesian analysis of the NIPS experiment
TLDR
A Bayesian analysis of an experiment on peer review, by introducing a hidden parameter which measures the probability that a submission meets basic quality criteria, suggested the total acceptance rate should be increased in order to decrease arbitrariness estimates in future review processes. Expand
Peer Review or Lottery? A Critical Analysis of Two Different Forms of Decision-making Mechanisms for Allocation of Research Grants
At present, peer review is the most common method used by funding agencies to make decisions about resource allocation. But how reliable, efficient, and fair is it in practice? The ex ante evaluationExpand
Focal randomisation: An optimal mechanism for the evaluation of R&D projects
In most countries, governments intervene in the process of R&D by financing a substantial part of it. The mechanism employed for choosing the projects to be financed is a committee composed ofExpand
Conservatism and risk-taking in peer review: Emerging ERC practices
This article explores the newly founded European Research Council's (ERC) peer review system and its ability to sustain its mission to promote excellent, groundbreaking research. The article exploresExpand
Scientometrics of peer review
TLDR
A special issue on “Scientometrics of peer review” is introduced, which collects papers originally presented at workshops and conferences organised by the COST ACTION TD1306 “New frontiers of peer reviews” to promote the establishment ofpeer review as an interdisciplinary field of research and stimulate further quantitative research. Expand
Breaking the grant cycle: on the rational allocation of public resources to scientific research projects
The thesis presents a reformative criticism of science funding by peer review. The criticism is based on epistemological scepticism, regarding the ability of scientific peers, or any other agent, toExpand
Focal Randomization: An optimal mechanism for the evaluation of R&D
In most countries, governments intervene in the process of R&D by financing a substantial part of it. The mechanism employed for choosing the projects to be financed is a committee composed ofExpand
Alternatives to Peer Review: Novel Approaches for Research Evaluation
TLDR
This paper discusses five approaches, including reputation-based ones, that come out of the research carried out by the LiquidPub project and research groups collaborated with LiquidPub, and discusses pros and cons of the proposed approaches. Expand
...
1
2
3
4
...