A Dataset on Malicious Paper Bidding in Peer Review

@article{Jecmen2022ADO,
  title={A Dataset on Malicious Paper Bidding in Peer Review},
  author={Steven Jecmen and Minji Yoon and Vincent Conitzer and Nihar B. Shah and Fei Fang},
  journal={ArXiv},
  year={2022},
  volume={abs/2207.02303}
}
In conference peer review, reviewers are often asked to provide “bids” on each submitted paper that express their interest in reviewing that paper. A paper assignment algorithm then uses these bids (along with other data) to compute a high-quality assignment of reviewers to papers. However, this process has been exploited by malicious reviewers who strategically bid in order to unethically manipulate the paper assignment, crucially undermining the peer review process. For example, these… 

Figures and Tables from this paper

Tradeoffs in Preventing Manipulation in Paper Bidding for Reviewer Assignment

TLDR
Certain desirable properties that algorithms for addressing bid manipulation should satisfy are enumerated and a high-level analysis of various approaches to preventing this manipulation is conducted along with directions for future investigation.

References

SHOWING 1-10 OF 27 REFERENCES

Making Paper Reviewing Robust to Bid Manipulation Attacks

TLDR
This paper develops a novel approach for paper bidding and assignment that is much more robust against bid manipulation attacks and shows empirically that this approach provides robustness even when dishonest reviewers collude, have full knowledge of the assignment system's internal workings, and have access to the system’s inputs.

Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments

TLDR
A (randomized) algorithm for reviewer assignment is presented that can optimally solve the reviewer-assignment problem under any given constraints on the probability of assignment for any reviewer-paper pair.

Tradeoffs in Preventing Manipulation in Paper Bidding for Reviewer Assignment

TLDR
Certain desirable properties that algorithms for addressing bid manipulation should satisfy are enumerated and a high-level analysis of various approaches to preventing this manipulation is conducted along with directions for future investigation.

On Good and Fair Paper-Reviewer Assignment

TLDR
This paper performs a detailed study on conflict-of-interest (COI) types and discusses several issues related to using COI, which, it is hoped, can raise some open discussions among researchers on the COI study.

A SUPER* Algorithm to Optimize Paper Bidding in Peer Review

TLDR
An algorithm called SUPER*, inspired by the A* algorithm, is presented, which considerably outperforms baselines deployed in existing systems, consistently reducing the number of papers with fewer than requisite bids by 50-75% or more, and is also robust to various real world complexities.

A robust model for paper reviewer assignment

TLDR
The proposed automatic paper-reviewer recommendation system obtains performance gains over state-of-the-art reviewer recommendation systems in terms of expertise, authority, diversity, and, most importantly, relevance as judged by human experts.

Matching Papers and Reviewers at Large Conferences

TLDR
A novel reviewer– paper matching approach that was recently deployed in the 35th AAAI Conference on Artificial Intelligence (AAAI 2021), and has since been adopted by other conferences including AAAI 2022 and ICML 2022 is studied.

Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design

TLDR
It is shown both empirically (on real conference data) and theoretically (under certain natural conditions) that dividing reviewers uniformly at random is near-optimal.

An algorithm to determine peer-reviewers

TLDR
An algorithm to automatically determine the most appropriate reviewers for a manuscript by way of a co-authorship network data structure and a relative-rank particle-swarm algorithm is presented.

Paper Matching with Local Fairness Constraints

TLDR
This paper introduces two new algorithms, FairIR and FairFlow, for computing fair matchings that approximately optimize the new formulation of paper matching that achieves higher objective scores, FairFlow achieves competitive fairness, and both are capable of more evenly allocating reviewers.