Can training improve the quality of inferences made by raters in competency modeling? A quasi-experiment.

@article{Lievens2007CanTI,
  title={Can training improve the quality of inferences made by raters in competency modeling? A quasi-experiment.},
  author={Filip Lievens and Juan I. Sanchez},
  journal={The Journal of applied psychology},
  year={2007},
  volume={92 3},
  pages={
          812-9
        }
}
A quasi-experiment was conducted to investigate the effects of frame-of-reference training on the quality of competency modeling ratings made by consultants. Human resources consultants from a large consulting firm were randomly assigned to either a training or a control condition. The discriminant validity, interrater reliability, and accuracy of the competency ratings were significantly higher in the training group than in the control group. Further, the discriminant validity and interrater… 
View on PubMed
users.ugent.be
58 Citations
Highly Influential Citations
2
Background Citations
28
Methods Citations
9
Results Citations
3

Tables from this paper

58 Citations

Rater Training: Understanding Effects of Training Content, Practice Ratings, and Feedback

Despite the popularity of frame‐of‐reference training (FORT), it is not clear how different structural elements of FORT work in concert to improve rating accuracy. Furthermore, past rater…

The Impact of Training on the Accuracy of Teacher-Completed Direct Behavior Ratings (DBRs)

The purpose of this study was to investigate the effects of three levels of training (direct, indirect, and none) on teachers’ ability to accurately rate video of student behavior. Direct and…

Lack of consensus among competency ratings of the same occupation: noise or substance?

Drawing from job crafting, creativity, and role theories to uncover situational factors related to differences among competency ratings of the same occupation showed that 25% of the variance associated with differences in competency Rating was related to the complexity, the context, and primarily the nature of the occupation's work activities.

Sources of Variance Underlying Competency Ratings 2

Although rating differences among incumbents of the same occupation have traditionally been viewed as error variance in the work analysis domain, such differences might often capture substantive…

A Cognitively-Based Competency Model for Small Unit Counter-IED Performance

A cognitively-based competency model developed to understand the types of tasks performed at the small unit level for counter-IED operations for the purpose of developing assessment items is described.

Does effectiveness in performance appraisal improve with rater training?

The main results show that training based on performance dimensions and the creation of a common framework, in addition to the training that includes the four programmes, increases the level of knowledge of performance and its dimensions.

A Critical Review of the Science and Practice of Competency Modeling

As an alternative to traditional job analysis, the practice of competency modeling may be appealing to scholars and practitioners of human resource development (HRD) to serve as the foundation for…

EFFECTS OF DESCRIPTOR SPECIFICITY AND OBSERVABILITY ON INCUMBENT WORK ANALYSIS RATINGS

Judgments regarding the requirements of jobs and the requirements of individuals performing those jobs comprise the critical groundwork on which human resource practices are built. Yet, such…

Rater training revisited: An updated meta-analytic review of frame-of-reference training

The present study updates Woehr and Huffcutt's (1994) rater training meta-analysis and demonstrates that frame-of-reference (FOR) training is an effective method of improving rating accuracy. The…

Disappointing Interventions and Weak Criteria: Carving Out a Solution Is Still Possible

In their focal article, Adler and his colleagues (2016) elaborate on the pros and cons of abolishing the performance appraisal process in organizations. Sherman-Garr (2014) contends that this trend…
...

References

SHOWING 1-10 OF 45 REFERENCES

Inferring Competencies 2 Easing the Inferential Leap in Competency Modeling : The Effects of Task-Related Information and Subject Matter Expertise

This study begins to close this practice‐science gap through 3 studies, which employ generalizability analysis to shed light on the quality of inferences made in competency modeling and the effects of incorporating elements of traditional job analysis into Competency modeling to raise thequality of competency inferences.

The impact of raters' cognition on judgment accuracy: An extension to the job analysis domain

Eighty-six incumbents of three different jobs produced job-analytic ratings using either a decomposed (task-based) or a holistic (job-based) rating strategy. Approximately half of them received rater…

Rater training for performance appraisal: A quantitative review

A substantial amount of research in the performance appraisal literature has focused on rater training as a means of improving performance ratings. Unfortunately the value of this research is…

Strategies in Rater Training

There is no evidence to support the belief that training raters to change rating distributions will increase accuracy or validity. Such training may merely promote a temporary and situation-specific…

THE PRACTICE OF COMPETENCY MODELING

The purpose of this article is to define and explain a trend that has caused a great deal of confusion among HR researchers, practitioners, and consumers of HR-related services: competency modeling.…

A new look at within-source interrater reliability of 360-degree feedback ratings.

This study investigated within-source interrater reliability of supervisor, peer, and subordinate feedback ratings made for managerial development. Raters provided 360-degree feedback ratings on a…

Effects of training and information on the accuracy and reliability of job evaluations.

Subjects evaluated 23 benchmark jobs on 10 dimensions after either receiving training or no training in job evaluation and after being given only a title, only a job description, or both a title and…

Understanding frame-of-reference training: the impact of training on the recall of performance information

Research on the cognitive mechanisms underlying frame-of-reference (FOR) rater training is extended by examining the impact of FOR training on the recall of performance information. It was…

Self-presentation processes in job analysis: a field experiment investigating inflation in abilities, tasks, and competencies.

Results indicated that ability statements were more subject to inflation than were task statements across all rating scales, and frequency and importance ratings of global competency statements were generally higher than decomposed ability and task scales, but required-at-entry judgments demonstrated the opposite relationship.

Further investigation of common knowledge effects on job analysis ratings.

Previous research (Smith & Hakel, 1979) raised the possibility that the Position Analysis Questionnaire (PAQ) only captures common knowledge, or stereotypes, about jobs, Cornelius, DeNisi, and…