On peer review in computer science: analysis of its effectiveness and suggestions for improvement

@article{Ragone2013OnPR,
  title={On peer review in computer science: analysis of its effectiveness and suggestions for improvement},
  author={Azzurra Ragone and Katsiaryna Mirylenka and Fabio Casati and Maurizio Marchese},
  journal={Scientometrics},
  year={2013},
  volume={97},
  pages={317-356}
}
In this paper we focus on the analysis of peer reviews and reviewers behaviour in a number of different review processes. More specifically, we report on the development, definition and rationale of a theoretical model for peer review processes to support the identification of appropriate metrics to assess the processes main characteristics in order to render peer review more transparent and understandable. Together with known metrics and techniques we introduce new ones to assess the overall… Expand
What We Still Don’t Know About Peer Review
TLDR
It is argued in this article that empirical research on PRP has not been addressed in a comprehensive way and a lack of integration among the methodological approaches to PRP results in a partial comprehension of this important process. Expand
Arbitrariness in the peer review process
TLDR
The model replicates the NIPS experiment of 2014, showing that the ratings of peer review are not robust, and that altering reviewers leads to a dramatic impact on the ranking of the papers. Expand
Does single blind peer review hinder newcomers?
Several fields of research are characterized by the coexistence of two different peer review modes to select quality contributions for scientific venues, namely double blind (DBR) and single blindExpand
Ranking Scientific Papers Using Preference Learning
TLDR
This work proposes a novel approach to paper ranking based on Gaussian Process Preference Learning (GPPL) and evaluates it on peer review data from the ACL-2018 conference, demonstrating the superiority of the GPPL-based approach over prior work. Expand
Does single blind peer review hinder newcomers?
TLDR
Evidence is found that SBR relates to a lower ration of contributions from newcomers to the venue and particularly newcomers that are otherwise experienced of publishing in other computer science conferences, suggesting the possible existence of ingroup–outgroup behaviors that may harm knowledge advancement in the long run. Expand
A longitudinal examination of SIGITE conference submission data, 2007-2012
TLDR
Examination of submission data for the SIGITE conference between the years 2007-2012 examines which external factors and which internal characteristics of the submissions are related to eventual reviewer ratings. Expand
Assessing peer review by gauging the fate of rejected manuscripts: the case of the Journal of Artificial Societies and Social Simulation
TLDR
Being exposed to more than one round of reviews before rejection, having received a more detailed reviewer report and being subjected to higher inter-reviewer disagreement were all associated with the number of citations received when the manuscript was eventually published, indicating that peer review could contribute to increasing the quality even of rejected manuscripts. Expand
Mechanism change in a simulation of peer review: from junk support to elitism
TLDR
It is argued that exploration of the parameter space is not enough if the authors want to support theoretical statements with simulation, and that exploration at the level of mechanisms is needed, which supports prudence in the application of simulation results based on single mechanisms. Expand
Scientometrics of peer review
TLDR
A special issue on “Scientometrics of peer review” is introduced, which collects papers originally presented at workshops and conferences organised by the COST ACTION TD1306 “New frontiers of peer reviews” to promote the establishment ofpeer review as an interdisciplinary field of research and stimulate further quantitative research. Expand
Are non-monetary rewards effective in attracting peer reviewers? A natural experiment
TLDR
The results suggest that non-monetary rewards are not necessarily effective, as in some cases they may actually discourage the most intrinsically motivated and competent reviewers. Expand
...
1
2
3
...

References

SHOWING 1-10 OF 68 REFERENCES
A Quantitative Analysis of Peer Review
In this paper we focus on the analysis of peer reviews and reviewers behaviour in a number of different review processes. More specifically, we report on the development, definition and rationale ofExpand
Reliability of reviewers' ratings when using public peer review: a case study
TLDR
The results of the study show that inter‐rater reliability is low (kappa coefficient) or reasonable (Intraclass Correlation Coefficient) in public peer review. Expand
Capitalizing on order effects in the bids of peer-reviewed conferences to secure reviews by expert referees
TLDR
It is shown that order effects lead to unconscious favoring of early-submitted papers to the detriment of later-Submitted papers, and it is advised to counterbalance order effects during the bidding phase of peer review by promoting the submissions with fewer bids to potential referees. Expand
Measuring the quality of editorial peer review.
TLDR
Until the objectives of peer-review are properly defined, it will remain almost impossible to assess or improve its effectiveness, and research needed to understand the broader effects of peer review poses many methodologic problems and would require the cooperation of many parts of the scientific community. Expand
Effects of editorial peer review: a systematic review.
CONTEXT Editorial peer review is widely used to select submissions to journals for publication and is presumed to improve their usefulness. Sufficient research on peer review has been published toExpand
Does Editorial Peer Review Work?
  • S. Lock
  • Medicine, Computer Science
  • Annals of Internal Medicine
  • 1994
For an activity seemingly so important for science, editorial peer review has received scant research. Introduced with the first two scientific journals [1], it did not become universal until afterExpand
Effect of open peer review on quality of reviews and on reviewers'recommendations: a randomised trial
TLDR
Asking reviewers to consent to being identified to the author had no important effect on the quality of the review, the recommendation regarding publication, or the time taken to review, but it significantly increased the likelihood of reviewers declining to review. Expand
Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of Board of Trustees' decisions
TLDR
Bibliometric analysis showed that the selection procedure followed by the Boehringer Ingelheim Fonds is thus highly valid: research articles by B.I.F. fellows are cited considerably more often than the “average' paper (average citation rate) published in the journal sets corresponding to the fields “Multidisciplinary', “Molecular Biology & Genetics', and “Biology & Biochemistry' in Essential Science Indicators (ESI). Expand
Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial.
TLDR
Neither blinding reviewers to the authors and origin of the paper nor requiring them to sign their reports had any effect on rate of detection of errors, and such measures are unlikely to improve the quality of peer review reports. Expand
Peer review of grant applications in biology and medicine. Reliability, fairness, and validity
TLDR
Bibliometric analysis provides evidence that the decisions of a public funding organization for basic project-based research are in line with the future publication success of applicants and argues for an expansion of approaches and methodologies in peer review research by increasingly focusing on process rather than outcome. Expand
...
1
2
3
4
5
...