On peer review in computer science: analysis of its effectiveness and suggestions for improvement

@article{Ragone2013OnPR,
  title={On peer review in computer science: analysis of its effectiveness and suggestions for improvement},
  author={Azzurra Ragone and Katsiaryna Mirylenka and Fabio Casati and Maurizio Marchese},
  journal={Scientometrics},
  year={2013},
  volume={97},
  pages={317-356}
}
In this paper we focus on the analysis of peer reviews and reviewers behaviour in a number of different review processes. More specifically, we report on the development, definition and rationale of a theoretical model for peer review processes to support the identification of appropriate metrics to assess the processes main characteristics in order to render peer review more transparent and understandable. Together with known metrics and techniques we introduce new ones to assess the overall… 

Assisting Decision Making in Scholarly Peer Review: A Preference Learning Perspective

TLDR
A novel, multi-faceted generic evaluation framework for ranking submissions based on peer reviews that takes into account effectiveness, effectiveness, efficiency and fairness is introduced.

Understanding Peer Review of Software Engineering Papers

TLDR
This work aims at understanding how reviewers, including those who have won awards for reviewing, perform their reviews of software engineering papers to identify both what makes a good reviewing approach and what make a good paper.

Reimagining peer review as an expert elicitation process

Journal peer review regulates the flow of ideas through an academic discipline and thus has the power to shape what a research community knows, actively investigates, and recommends to policymakers

Innovations in peer review in scholarly publishing: a meta-summary

TLDR
The aim of this literature review was to aid the development of the inventory of peer review innovations by identifying innovations in peer review reported in the scholarly literature and by providing a summary of the different approaches.

“Flawed, but like democracy we don’t have a better system”: The Experts’ Insights on the Peer Review Process of Evaluating Security Papers

TLDR
This paper presents the first qualitative study to examine the peer review process in the computer security field, and identifies several issues in the security review system that need to be improved.

What We Still Don’t Know About Peer Review

TLDR
It is argued in this article that empirical research on PRP has not been addressed in a comprehensive way and a lack of integration among the methodological approaches to PRP results in a partial comprehension of this important process.

Arbitrariness in the peer review process

TLDR
The model replicates the NIPS experiment of 2014, showing that the ratings of peer review are not robust, and that altering reviewers leads to a dramatic impact on the ranking of the papers.

Does single blind peer review hinder newcomers?

Several fields of research are characterized by the coexistence of two different peer review modes to select quality contributions for scientific venues, namely double blind (DBR) and single blind

Ranking Scientific Papers Using Preference Learning

TLDR
This work proposes a novel approach to paper ranking based on Gaussian Process Preference Learning (GPPL) and evaluates it on peer review data from the ACL-2018 conference, demonstrating the superiority of the GPPL-based approach over prior work.

Does single blind peer review hinder newcomers?

TLDR
Evidence is found that SBR relates to a lower ration of contributions from newcomers to the venue and particularly newcomers that are otherwise experienced of publishing in other computer science conferences, suggesting the possible existence of ingroup–outgroup behaviors that may harm knowledge advancement in the long run.

References

SHOWING 1-10 OF 69 REFERENCES

A Quantitative Analysis of Peer Review

TLDR
A number of unexpected results were found, in particular the low correlation between peer review outcome and impact in time of the accepted contributions and the presence of a high level of randomness in the analyzed peer review processes.

Reliability of reviewers' ratings when using public peer review: a case study

TLDR
The results of the study show that inter‐rater reliability is low (kappa coefficient) or reasonable (Intraclass Correlation Coefficient) in public peer review.

Capitalizing on order effects in the bids of peer-reviewed conferences to secure reviews by expert referees

TLDR
It is shown that order effects lead to unconscious favoring of early-submitted papers to the detriment of later-Submitted papers, and it is advised to counterbalance order effects during the bidding phase of peer review by promoting the submissions with fewer bids to potential referees.

Measuring the quality of editorial peer review.

TLDR
Until the objectives of peer-review are properly defined, it will remain almost impossible to assess or improve its effectiveness, and research needed to understand the broader effects of peer review poses many methodologic problems and would require the cooperation of many parts of the scientific community.

Effects of editorial peer review: a systematic review.

CONTEXT Editorial peer review is widely used to select submissions to journals for publication and is presumed to improve their usefulness. Sufficient research on peer review has been published to

Effect of open peer review on quality of reviews and on reviewers'recommendations: a randomised trial

TLDR
Asking reviewers to consent to being identified to the author had no important effect on the quality of the review, the recommendation regarding publication, or the time taken to review, but it significantly increased the likelihood of reviewers declining to review.

Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of Board of Trustees' decisions

TLDR
Bibliometric analysis showed that the selection procedure followed by the Boehringer Ingelheim Fonds is thus highly valid: research articles by B.I.F. fellows are cited considerably more often than the “average' paper (average citation rate) published in the journal sets corresponding to the fields “Multidisciplinary', “Molecular Biology & Genetics', and “Biology & Biochemistry' in Essential Science Indicators (ESI).

Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial.

TLDR
Neither blinding reviewers to the authors and origin of the paper nor requiring them to sign their reports had any effect on rate of detection of errors, and such measures are unlikely to improve the quality of peer review reports.

Peer review of grant applications in biology and medicine. Reliability, fairness, and validity

TLDR
Bibliometric analysis provides evidence that the decisions of a public funding organization for basic project-based research are in line with the future publication success of applicants and argues for an expansion of approaches and methodologies in peer review research by increasingly focusing on process rather than outcome.

Committee peer review at an international research foundation: predictive validity and fairness of selection decisions on post-graduate fellowship applications

TLDR
Investigation of committee peer review for awarding long-term fellowships to post-graduate researchers as practised by the Boehringer Ingelheim Fonds shows that the selection procedure is thus highly valid and achieves its aim to select as fellowship recipients the best junior scientists.
...