A Simulation of Disagreement for Control of Rational Cheating in Peer Review

@article{Grimaldo2013ASO,
  title={A Simulation of Disagreement for Control of Rational Cheating in Peer Review},
  author={F. Grimaldo and Mario Paolucci},
  journal={Adv. Complex Syst.},
  year={2013},
  volume={16}
}
Understanding the peer review process could help research and shed light on the mechanisms that underlie crowdsourcing. In this paper, we present an agent-based model of peer review built on three entities — the paper, the scientist and the conference. The system is implemented on a BDI platform (Jason) that allows to define a rich model of scoring, evaluating and selecting papers for conferences. Then, we propose a programme committee update mechanism based on disagreement control that is able… Expand
Mechanism change in a simulation of peer review: from junk support to elitism
TLDR
It is argued that exploration of the parameter space is not enough if the authors want to support theoretical statements with simulation, and that exploration at the level of mechanisms is needed, which supports prudence in the application of simulation results based on single mechanisms. Expand
Reputation or peer review? The role of outliers
TLDR
An agent-based model of paper publication and consumption suggests that a reputation system could perform better than peer review as a scientific information filter for quality except when research is multi-disciplinary and in a field where outliers exist. Expand
Do Editors Have A Silver Bullet? An Agent-Based Model Of Peer Review
TLDR
An agent-based model of peer review that looks at the effect of different editorial policies of referee selection shows that in case of randomness of referee judgment, any editorial policy is detrimental for peer review. Expand
Can editors save peer review from peer reviewers?
TLDR
It is found that the biggest hazard to the quality of published literature is not selfish rejection of high-quality manuscripts but indifferent acceptance of low-quality ones. Expand
A scoping review of simulation models of peer review
TLDR
This scoping review is an attempt to understand the current state of simulation studies of peer review, and develops a proposed taxonomy of model features that include model type and type of modeled peer review system, to help distinguish between the modeling approaches. Expand
Attitudes of referees in a multidisciplinary journal: An empirical analysis
TLDR
It is found that senior researchers are harsher in their judgments than junior researchers, and the latter accept requests to review more often and are faster in reporting, and common standards of evaluation can be established even in multidisciplinary communities. Expand
Is three better than one? simulating the effect of reviewer selection and behavior on the quality and efficiency of peer review
TLDR
It is found that, when reviewer reliability is random or reviewers behave strategically, involving more than one reviewer per submission reduces evaluation bias, however, if scientists review scrupulously, multiple reviewers require an abnormal resource drain at the system level. Expand
Hidden connections: Network effects on editorial decisions in four computer science journals
TLDR
Examination of the influence of authors’ reputation on editorial bias in scholarly journals found that more reputed authors were less likely to be rejected by editors when they submitted papers receiving negative reviews. Expand
By the Numbers: Track Record, Flawed Reviews, Journal Space, and the Fate of Talented Authors
TLDR
Even small decreases in the correlations, and small increases in the weight given to track record, quickly skew the outcomes of the peer review process, favouring authors who develop a track record of publications in the first cycles of journal publication while excluding many equally-talented or more-Talented authors from publishing. Expand
On Performance of Peer Review for Academic Journals: Analysis Based on Distributed Parallel System
A simulation model based on parallel systems is established, aiming to explore the relation between the number of submissions and the overall standard of academic journals within a similar disciplineExpand

References

SHOWING 1-10 OF 47 REFERENCES
Peer-review in a world with rational scientists: Toward selection of the average
TLDR
It is found that a small fraction of incorrect (selfish or rational) referees is sufficient to drastically lower the quality of the published (accepted) scientific standard. Expand
A Proposal for Agent Simulation of Peer Review
TLDR
This paper develops and presents a model of the peer review process, and proposes a working implementation of a subset of the general model, developed with Jason, a language that implements the BDI architecture for multi agent systems. Expand
Saint Matthew strikes again: An agent-based model of peer review and the scientific community structure
TLDR
It is found that referee behavior drastically affects peer review and an equal distribution of the reviewing effort is beneficial only if the scientific community is homogeneous and referee reliability is the rule. Expand
Is Peer Review a Game of Chance?
Abstract Peer review is the standard that journals and granting agencies use to ensure the scientific quality of their publications and funded projects. The peer-review process continues to beExpand
Modeling peer review: an agent-based approach
TLDR
A framework in which alternatives to the current peer review system can be studied quantitatively using agent-based modeling is proposed, showing how these alternatives produce different results in terms of speed of publication, quality control, reviewers' effort, and authors' impact. Expand
For an Integrated Approach to Agent-Based Modeling of Science
  • N. Payette
  • Computer Science
  • J. Artif. Soc. Soc. Simul.
  • 2011
TLDR
It is argued that an agent-based model of the scientific process should be constructed with normative claims in mind: i.e. that it should be useful for scientific policy making. Expand
Reputation for Innovating Social Networks
TLDR
Simulation-based studies of the role of reputation as a more tolerant form of social capital than familiarity networks, and a computational system (REPAGE) for forming and exchanging reputation information are reported. Expand
Crowdsourcing, Open Innovation and Collective Intelligence in the Scientific Method - A Research Agenda and Operational Framework
TLDR
A research agenda is proposed that investigates a) the applicability of Crowdsourcing to fundamental science and b) the impact of distributed agent principles from Artificial Intelligence research on the robustness ofrowdsourcing. Expand
A Quantitative Analysis of Peer Review
In this paper we focus on the analysis of peer reviews and reviewers behaviour in a number of different review processes. More specifically, we report on the development, definition and rationale ofExpand
Chance and consensus in peer review.
An experiment in which 150 proposals submitted to the National Science Foundation were evaluated independently by a new set of reviewers indicates that getting a research grant depends to aExpand
...
1
2
3
4
5
...