Arbitrariness in the peer review process

  title={Arbitrariness in the peer review process},
  author={Elise S. Brezis and Aliaksandr Birukou},
The purpose of this paper is to analyze the causes and effects of arbitrariness in the peer review process. This paper focuses on two main reasons for the arbitrariness in peer review. The first is that referees are not homogenous and display homophily in their taste and perception of innovative ideas. The second element is that reviewers are different in the time they allocate for peer review. Our model replicates the NIPS experiment of 2014, showing that the ratings of peer review are not… 


  • MohiniSakir Ahmed
  • Medicine
    Central Asian Journal of Medical Hypotheses and Ethics
  • 2022
This short review looks at the ideal peer review along with inherent challenges and makes suggestions on how novice and intermediate reviewers can improve their contributions as well as introduces various resources relevant for peer review.

Disagreement and Agonistic Chance in Peer Review

The purpose of grant peer review is to identify the most excellent and promising research projects. However, sociologists of science and STS scholars have shown that peer review tends to promote

Do peers share the same criteria for assessing grant applications?

This study examines a basic assumption of peer review, namely, the idea that there is a consensus on evaluation criteria among peers, which is a necessary condition for the reliability of peer judgements, and identifies two consensus classes, two consensus-close classes, and a consensus-far class.

Bibliometrically Disciplined Peer Review: on Using Indicators in Research Evaluation

Evaluation of research uses peer review and bibliometrics, and the debate about their balance in research evaluation continues. Both approaches have supporters, and both approaches are criticized. In

The oracles of science: On grant peer review and competitive funding

From a purely epistemological point of view, evaluating and predicting the future success of new research projects is often considered very difficult. Is it possible to forecast important findings

Peer review analyze: A novel benchmark resource for computational analysis of peer reviews

This work presents a first of its kind multi-layered dataset of 1199 open peer review texts manually annotated at the sentence level across the four layers, and introduces four novel tasks which could serve as an indicator of the exhaustiveness of a peer review and can be a step towards the automatic judgment of review quality.

You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism

The Isotonic Mechanism is introduced, a simple andcient approach to improving imprecise raw scores by leveraging certain information that the owner is incentivized to provide and it is proved that the adjusted scores provided by this owner-assisted mechanism are significantly more accurate than the raw Scores provided by the reviewers.

Gender differences in research grant allocation - a mixed picture

Gender bias in grant allocation is a deviation from the principle that scientific merit should guide grant decisions. However, most studies on gender bias in grant allocation focus on gender

When zero may not be zero: A cautionary note on the use of inter‐rater reliability in evaluating grant peer review

Considerable attention has focused on studying reviewer agreement via inter‐rater reliability (IRR) as a way to assess the quality of the peer review process. Inspired by a recent study that reported

Analyzing sentiments in peer review reports: Evidence from two science funding agencies

The results suggest that manual analysis of review sentiments can provide a reliable proxy of grant reviewer opinions, whereas the two SA algorithms can be useful only in some specific situations.



On peer review in computer science: analysis of its effectiveness and suggestions for improvement

The development, definition and rationale of a theoretical model for peer review processes are reported on to support the identification of appropriate metrics to assess the processes main characteristics in order to render peer review more transparent and understandable.

Arbitrariness of peer review: A Bayesian analysis of the NIPS experiment

A Bayesian analysis of an experiment on peer review, by introducing a hidden parameter which measures the probability that a submission meets basic quality criteria, suggested the total acceptance rate should be increased in order to decrease arbitrariness estimates in future review processes.

Peer Review or Lottery? A Critical Analysis of Two Different Forms of Decision-making Mechanisms for Allocation of Research Grants

At present, peer review is the most common method used by funding agencies to make decisions about resource allocation. But how reliable, efficient, and fair is it in practice? The ex ante evaluation

Conservatism and risk-taking in peer review: Emerging ERC practices

This article explores the newly founded European Research Council's (ERC) peer review system and its ability to sustain its mission to promote excellent, groundbreaking research. The article explores

Scientometrics of peer review

A special issue on “Scientometrics of peer review” is introduced, which collects papers originally presented at workshops and conferences organised by the COST ACTION TD1306 “New frontiers of peer reviews” to promote the establishment ofpeer review as an interdisciplinary field of research and stimulate further quantitative research.

Breaking the grant cycle: on the rational allocation of public resources to scientific research projects

The thesis presents a reformative criticism of science funding by peer review. The criticism is based on epistemological scepticism, regarding the ability of scientific peers, or any other agent, to

Focal Randomization: An optimal mechanism for the evaluation of R&D

In most countries, governments intervene in the process of R&D by financing a substantial part of it. The mechanism employed for choosing the projects to be financed is a committee composed of

Alternatives to Peer Review: Novel Approaches for Research Evaluation

This paper discusses five approaches, including reputation-based ones, that come out of the research carried out by the LiquidPub project and research groups collaborated with LiquidPub, and discusses pros and cons of the proposed approaches.

Evaluating alternative systems of peer review: a large-scale agent-based modelling approach to scientific publication

A previously developed agent-based model of the scientific publication and peer- review system calibrated with empirical data is used to compare the efficiency of five alternative peer-review systems with the conventional system and recommends prioritizing a system of review-sharing to create a sustainable scientific publication-review system.