Improving the peer-review process for grant applications: reliability, validity, bias, and generalizability.

@article{Marsh2008ImprovingTP,
  title={Improving the peer-review process for grant applications: reliability, validity, bias, and generalizability.},
  author={Herbert W. Marsh and Upali W. Jayasinghe and Nigel W. Bond},
  journal={The American psychologist},
  year={2008},
  volume={63 3},
  pages={
          160-8
        }
}
Peer review is a gatekeeper, the final arbiter of what is valued in academia, but it has been criticized in relation to traditional psychological research criteria of reliability, validity, generalizability, and potential biases. Despite a considerable literature, there is surprisingly little sound peer-review research examining these criteria or strategies for improving the process. This article summarizes the authors' research program with the Australian Research Council, which receives… 
Are peer-reviews of grant proposals reliable? An analysis of Economic and Social Research Council (ESRC) funding applications
Peer-review is widely used throughout academia, most notably in the publication of journal articles and the allocation of research grants. Yet peer-review has been subject to much criticism,
Peer-review of grant proposals. An analysis of Economic and Social Research Council grant applications
Peer-review is widely used throughout academia, most notably in the publication of journal articles and the allocation of research grants. Yet peer-review has been subject to much criticism,
Bias in peer review
TLDR
This review provides a brief description of the function, history, and scope of peer review, and characterizes and examines the empirical, methodological, and normative claims of bias in peer review research; and assesses possible alternatives to the status quo.
ADVANCES IN INFORMATION SCIENCE Bias in Peer Review
Research on bias in peer review examines scholarly communication and funding processes to assess the epistemic and social legitimacy of the mechanisms by which knowledge communities vet and
Bias in peer review: a case study.
TLDR
Analysis of three datasets providing information on the attributes of authors and reviewers and review outcomes finds no evidence of bias in terms of gender, or the language or prestige of author and reviewer institutions in any of the three datasets, but some weak evidence of regional bias in all three.
Assessment of potential bias in research grant peer review in Canada
TLDR
Evidence of bias in peer review of operating grants that is of sufficient magnitude to change application scores from fundable to nonfundable is found and should be addressed by training and policy changes in research funding.
Personal attributes of authors and reviewers, social bias and the outcomes of peer review: a case study
TLDR
Analysis of three datasets providing information on the attributes of authors and reviewers and review outcomes finds no evidence of bias in terms of gender, or the language or prestige of author and reviewer institutions in any of the three datasets, but some weak evidence of regional bias in all three.
Gender Effects in the Peer Reviews of Grant Proposals: A Comprehensive Meta-Analysis Comparing Traditional and Multilevel Approaches
Peer review is valued in higher education, but also widely criticized in terms of potential biases, particularly gender. We evaluate gender differences in peer reviews of grant applications,
The future of societal impact assessment using peer review: pre-evaluation training, consensus building and inter-reviewer reliability
There are strong political reasons underpinning the desire to achieve a high level of inter-reviewer reliability (IRR) within peer review panels. Achieving a high level of IRR is synonymous with an
...
...

References

SHOWING 1-10 OF 43 REFERENCES
Peer review process: Assessments by applicant-nominated referees are biased, inflated, unreliable and invalid
Abstract How trustworthy are peer reviews by applicant-nominated assessors (ANAs)? For Australian Research Council (ARC) proposals (N = 2,330) with at least one ANA and one assessor nominated by the
Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of Board of Trustees' decisions
TLDR
Bibliometric analysis showed that the selection procedure followed by the Boehringer Ingelheim Fonds is thus highly valid: research articles by B.I.F. fellows are cited considerably more often than the “average' paper (average citation rate) published in the journal sets corresponding to the fields “Multidisciplinary', “Molecular Biology & Genetics', and “Biology & Biochemistry' in Essential Science Indicators (ESI).
The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation
Abstract The reliability of peer review of scientific documents and the evaluative criteria scientists use to judge the work of their peers are critically reexamined with special attention to the
Peer-review practices of psychological journals: The fate of published articles, submitted again
Abstract A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines.
Multiple Evaluations of Grant Proposals by Independent Assessors: Confirmatory Factor Analysis Evaluations of Reliability, Validity, and Structure.
TLDR
Tests of whether proposal and researcher ratings reflected one or two latent variables, based on the internal structure of the assessment ratings and relations with external criteria (academic rank, prior funding, and publications), supported a one-factor solution.
A multilevel cross‐classified modelling approach to peer review of grant proposals: the effects of assessor and researcher attributes on assessor ratings
Summary. The peer review of grant proposals is very important to academics from all disciplines. Although there is limited research on the reliability of assessments for grant proposals, previously
Peer Review in the Funding of Research in Higher Education: The Australian Experience
In this article we evaluate the peer review process used to fund Australian university research across all disciplines. Peer reviews of research proposals (2,989 proposals, 6,233 external reviewers)
A new reader trial approach to peer review in funding research grants: An Australian experiment
TLDR
In comparison to the traditional peer review approach, the new reader system is substantially more reliable, timely, and cost efficient - and applicable to many peer review situations.
The Peer Review Process Used to Evaluate Manuscripts Submitted to Academic Journals: Interjudgmental Reliability
AbstractThe purpose of the present investigation was to assess the agreement between two independent reviews of each of 278 manuscripts submitted to the Journal of Educational Psychology on an
Grants Peer Review in Theory and Practice
Grants peer review is a family of ex ante methods used by federal agencies to select research proposalsforfunding. This article draws on Chubin and Hackett's 1990 book, Peerless Science: Peer Review
...
...