Modeling and Correcting Bias in Sequential Evaluation

@article{Wang2022ModelingAC,
  title={Modeling and Correcting Bias in Sequential Evaluation},
  author={Jingyan Wang and Ashwin Pananjady},
  journal={ArXiv},
  year={2022},
  volume={abs/2205.01607}
}
We consider the problem of sequential evaluation, in which an evaluator observes candidates in a sequence and assigns scores to these candidates in an online, irrevocable fashion. Motivated by the psychology literature that has studied sequential bias in such settings—namely, dependencies between the evaluation outcome and the order in which the candidates appear—we propose a natural model for the evaluator’s rating process that captures the lack of calibration inherent to such a task. We… 

Figures and Tables from this paper

References

SHOWING 1-10 OF 65 REFERENCES

Save the last dance II: unwanted serial position effects in figure skating judgments.

The findings presented here replicate those of Bruine de Bruin (2005) [Save the last dance for me: unwanted serial position effects in jury evaluations], using a larger dataset, and addressing potential concerns about BruineDe Bruin's analyses.

Objectivity and Effect of Order of Appearance in Judging of Synchronized Swimming Meets

To determine the objectivity of judges and whether the scores of swimmers were affected by the order in which they competed data were analyzed for 87 AAU and 84 World meet swimmers who swam 6 and 5

Spearman's Footrule as a Measure of Disarray

SUMMARY Spearman's measure of disarray D is the sum of the absolute values of the difference between the ranks. We treat D as a metric on the set of permutations. The limiting mean, variance and

Sequential effects in Olympic synchronized diving scores

Investigation of sequential effects during synchronized diving events at the 2012 and 2016 Olympic Games finds that the specific direction of this sequential bias may depend on the particular sport, which has implications for how judges might approach their roles in a context-dependent manner.

Symmetric binary B-Trees: Data structure and maintenance algorithms

  • R. Bayer
  • Computer Science
    Acta Informatica
  • 2004
Symmetric B-Trees are a modification of B-trees described previously by Bayer and McCreight and properly contains the balanced trees.

A generative model for rank data based on insertion sort algorithm

Olympic medals as fruits of comparison? Assimilation and contrast in sequential performance judgments.

Sequential performance judgments were influenced by previously judged performances, and the direction of this influence depended on the degree of perceived similarity between the successive performances.

Ensuring Sustainable Evaluation: How to Improve Quality of Evaluating Grant Proposals?

To increase the unbiasedness and fairness of the evaluation process, reviewers’ work should be analysed and, within a certain panel, all grant proposals should be evaluated by the same set of reviewers, which would help to eliminate the distorting influence of the selection of a very severe or very lenient expert.

Lenience breeds strictness: The generosity-erosion effect in hiring decisions

The generosity-erosion effect is introduced, which states that evaluators might be harsher in their assessment of candidates after grading previous candidates generously, which means that for each candidate graded generously, the probability for subsequent candidates to pass decreased.

You never forget your first? Impact of interview timing on institutional rank order

To analyze the impact of interview date on the applicant rank for Neurology residencies in the United States, a database of more than 1,000 cases over a 25-year period showed an association between interview date and rank.
...