Understanding The Latent Structure Of Job Performance Ratings

Journal of Applied Psychology
2000, Vol. 85, No. 6, 956-970

Copyright 2000 by the American Psychological Association, Inc.
0021-9010/00/S5.00 DOT: 10.1037//0021-9010.85.6.956

Understanding the Latent Structure of Job Performance Ratings
Steven E. Scullen

Michael K. Mount
University of Iowa

North Carolina State University

Maynard Goff
Personnel Decisions International

Thisstudy quantified the effects of 5 factors postulated to influence performance ratings: the ratee's
general level of performance, the ratee's performance on a specific dimension, the rater's idiosyncratic
rating tendencies, the rater's organizational perspective, and random measurement error. Two large data
sets, consisting of managers (« = 2,350 and n = 2,142) who received developmental ratingson 3
performance dimensions from 7 raters (2 bosses, 2 peers, 2 subordinates, and self) were used. Results
indicated that idiosyncratic rater effects (62% and 53%) accounted for over half of the rating variance in
both data sets. The combined effects of general and dimensional ratee performance (21% and 25%) were
less than half the size of the idiosyncratic rater effects. Smallperspective-related effects were f ound in
boss and subordinate ratings but not in peer ratings. Average random error effects in the 2 data sets were
11% and 18%.

Job performance is a fundamentally important construct in organizational practice and research. From a practical perspective, it
plays a central role in most personnel decisions, such as meritbased compensation, promotion, and retention. It isalso used as an
important source of developmental feedback. From a theoretical
perspective, researchers have long been interested in understanding the causal mechanisms that lead to effective job performance.
Although job performance has been measured in many ways
(e.g., volume of sales, quantity or quality of items produced,
absences, number of promotions), the most frequently used measure isa supervisory performance rating. In recent years, multirater
or 360° feedback programs have gained popularity. This means
that peer, subordinate, and self-ratings also play an important role
in the assessment of job performance. Given the significance of the
job performance construct and the growing dependence on ratings
from multiple sources as a means for measuring that construct, it
isimportant to identify and quantify the factors that influence
performance ratings.
The present study examines the latent structure of job performance ratings. We present a model that contains five factors that
are postulated to affect ratings in a multirater feedback program. In

doing so, we address two major objectives. The first is to measure
the effects associated with each of the fivefactors in the model.
Other research has examined subsets of the five factors we investigate here, but to our knowledge, none has investigated the effects
of all five of the factors simultaneously. This is important, because
failure to investigate each of these effects provides an incomplete
representation of the latent structure of performance ratings. The
second major objective is to examinethe influence of each of the
five factors on ratings made by raters from four different perspectives (boss, peer, subordinate, and self) and on three dimensions of
managerial performance. This allows us to determine whether the
effects are consistent across different types of raters and different
aspects of performance.

Factors That Influence Performance Ratings
Wherry's theory of rating(Wherry & Bartlett, 1982) indicates
that there are three broad types of factors that influence performance ratings: the ratee's actual job performance, various rater
biases in the perception and recall of that performance, and measurement error.' The model we developed for this study is based
on those general distinctions. It was not our intention, however, to
model Wherry's detailed...