• Corpus ID: 182952827

Stretching the Effectiveness of MLE from Accuracy to Bias for Pairwise Comparisons

  title={Stretching the Effectiveness of MLE from Accuracy to Bias for Pairwise Comparisons},
  author={Jingyan Wang and Nihar B. Shah and Ramamoorthi Ravi},
  booktitle={International Conference on Artificial Intelligence and Statistics},
A number of applications (e.g., AI bot tournaments, sports, peer grading, crowdsourcing) use pairwise comparison data and the Bradley-Terry-Luce (BTL) model to evaluate a given collection of items (e.g., bots, teams, students, search results). Past work has shown that under the BTL model, the widely-used maximum-likelihood estimator (MLE) is minimax-optimal in estimating the item parameters, in terms of the mean squared error. However, another important desideratum for designing estimators is… 

Figures and Tables from this paper

A Tale of HodgeRank and Spectral Method: Target Attack Against Rank Aggregation Is the Fixed Point of Adversarial Game

The effectiveness of the suggested target attack strategies is demonstrated by a series of toy simulations and several real-world data experiments, showing that the proposed methods could achieve the attacker's goal and the leading candidate of the perturbed ranking list is the designated one by the adversary.

Integrating Rankings into Quantized Scores in Peer Review

The approach addresses the two aforementioned challenges by ensuring that rankings are incorporated into the updates scores in the same manner for all papers, thereby mitigating arbitrariness, and allowing to seamlessly use existing interfaces and workflows designed for scores.

Lagrangian Inference for Ranking Problems

This paper provides a novel approach to inferring the ranking systems of movies’ ranking using a large-scale data set with strong theoretical guarantees showing the optimality of the proposed method.

Building community together: towards equitable CSCL practices and processes

  • C. RoséS. Järvelä
  • Education
    International Journal of Computer-Supported Collaborative Learning
  • 2020
In the midst of these shifting sands, the September edition of the International Journal of Computer-Supported Collaborative learning is offered, comprising four full articles and one squib, all of which speak to these themes, with a consistent deep inspection of processes of collaboration.



Spectral Method and Regularized MLE Are Both Optimal for Top-$K$ Ranking

It is demonstrated that under a natural random sampling model, the spectral method alone, or the regularized MLE alone, is minimax optimal in terms of the sample complexity - the number of paired comparisons needed to ensure exact top-K identification, for the fixed dynamic range regime.

Minimax-optimal Inference from Partial Rankings

It is shown that even if one applies the mismatched maximum likelihood estimator that assumes independence (on pairwise comparisons that are now dependent due to rank-breaking), minimax optimal performance is still achieved up to a logarithmic factor.

Optimal Sample Complexity of M-wise Data for Top-K Ranking

This work examines an M-wise comparison model that builds on the Plackett-Luce model where for each sample, M items are ranked according to their perceived utilities modeled as noisy observations of their underlying true utilities.

Accelerated Spectral Ranking

This paper designs a provably faster spectral ranking algorithm, which it is called accelerated spectral ranking (ASR), that is also consistent under the MNL/BTL models, and gives the first general sample complexity bounds for recovering the parameters of theMNL model from multiway comparisons under any (connected) comparison graph.

Overcoming Calibration Problems in Pattern Labeling with Pairwise Ratings: Application to Personality Traits

By performing extensive simulations, an empirical law is determined for the scaling of the number of pairs needed as a function of thenumber of videos in order to achieve a given accuracy of score reconstruction and it is shown that the pairwise method is affordable.

The Impact of Regularization on High-dimensional Logistic Regression

This paper studies regularized logistic regression (RLR), where a convex regularizer that encourages the desired structure is added to the negative of the log-likelihood function, and provides a precise analysis of the performance of RLR via the solution of a system of six nonlinear equations.

Online Rank Elicitation for Plackett-Luce: A Dueling Bandits Approach

The approach is based on constructing a surrogate probability distribution over rankings based on a sorting procedure, for which the pairwise marginals provably coincide with the marginals of the Plackett-Luce distribution.

MM algorithms for generalized Bradley-Terry models

The Bradley-Terry model for paired comparisons is a simple and muchstudied means to describe the probabilities of the possible outcomes when individuals are judged against one another in pairs. Among

Asymptotic Behavior of Likelihood Methods for Exponential Families when the Number of Parameters Tends to Infinity

Consider a sample of size n from a regular exponential family in Pn dimensions. Let 6, denote the maximum likelihood estimator, and consider the case where Pn tends to infinity with n and where {(On}

Nonuniformity of P-values Can Occur Early in Diverging Dimensions

It is established that the conventional p-values in generalized linear models become invalid in diverging dimensions and can occur early in nonlinear models.