Journal of Applied Psychology Copyright 1996 by the American Psychological Association, Inc.

1996, Vol. 81, No. 5, 557-574 0021-9010/96/J3.00

Comparative Analysis of the Reliability of Job Performance Ratings

Chockalingam Viswesvaran Deniz S. Ones
Florida International University University of Houston

Frank L. Schmidt
University of Iowa

This study used meta-analytic methods to compare the interrater and intrarater reliabil-
ities of ratings of 10 dimensions of job performance used in the literature; ratings of
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

overall job performance were also examined. There was mixed support for the notion
This document is copyrighted by the American Psychological Association or one of its allied publishers.

that some dimensions are rated more reliably than others. Supervisory ratings appear to
have higher interrater reliability than peer ratings. Consistent with H. R. Rothstein
(1990), mean interrater reliability of supervisory ratings of overall job performance was
found to be .52. In all cases, interrater reliability is lower than intrarater reliability, indi-
cating that the inappropriate use of intrarater reliability estimates to correct for biases
from measurement error leads to biased research results. These findings have important
implications for both research and practice.

Several measures of job performance have been used
over the years as criterion measures (cf. Campbell, 1990;
Campbell, Gasser, & Oswald, 1996; Cleveland, Murphy,
& Williams, 1989). Attempts have also been made to
identify the specifications for these criteria (Blum &
Naylor, 1968; Brogden, 1946; Dunnette, 1963; Stuit &
Wilson, 1946; Toops, 1944). For example, Blum and
Naylor (1968) identified 11 dimensions or characteristics
on which the different criteria can be evaluated, whereas
Brogden (1946) identified relevance, reliability, and
practicality as desired characteristics for criteria. Reli-
ability of criteria has been included as an important con-
sideration by all authors writing about job performance
measurement.
Indeed, for a measure to have any research or admin-
Chockalingam Viswesvaran, Department of Psychology,
Florida International University; Deniz S. Ones, Department of
Management, University of Houston; Frank L. Schmidt, De-
partment of Management and Organizations, University of
Iowa. The order of authorship is arbitrary, and all three authors
contributed equally. Deniz S. Ones is now at the Department of
Psychology, University of Minnesota.
An earlier version of this article was presented at a sympo-
sium called "Reliability and Accuracy Issues in Measuring Job
Performance," which was chaired by Frank L. Schmidt at the
10th Annual Meeting of the Society of Industrial and Organiza-
tional Psychology, Orlando, Florida, May 1995.
Correspondence concerning this article should be addressed
to Chockalingam Viswesvaran, Department of Psychology,
Florida International University, Miami, Florida 33199. Elec-
tronic mail may be sent via Internet to vish@servax.fiu.edu.
557
istrative use, it must have some reliability. Low-reliability
results in systematic reduction in the magnitude of ob-
served relationships and can therefore distort theory test-
ing. The recent resurgence of interest in criteria (Austin
& Villanova, 1992) and in developing a theory of job per-
formance (e.g., Campbell, 1990; Campbell, McCloy, Op-
pier, & Sager, 1993; McCloy, Campbell, & Cudeck, 1994;
Schmidt & Hunter, 1992) also emphasizes the impor-
tance of the reliability of criterion measures. A thorough
investigation of the criterion domain ought to include an
examination of the reliability of dimensions of job per-
formance. The focus of this article is the reliability of job
performance ratings.
Of the different ways to measure job performance, per-
formance ratings are the most prevalent. Ratings are sub-
jective evaluations that can be obtained from supervisors,
peers, subordinates, self, or customers, with supervisors
being the most commonly used source (Cascio, 1991;
Cleveland et al., 1989) and peers constituting the second
most commonly used source. For example, Bernardin
and Beatty (1984) found in a survey of human resource
managers that over 90% of the respondents used supervi-
sory ratings as their primary source of performance rat-
ings and peers were the second most widely used source
of ratings.
Constructing comprehensive and valid theories of hu-
man motivation and work behavior is predicated on the
reliable measurement of constructs. Given the centrality
of the construct of job performance in industrial and or-
ganizational psychology (Campbell et al., 1996), and
given that ratings are the most commonly used source for
 558 VISWESVARAN, ONES, AND SCHMIDT
measuring job performance, it is important to estimate
precisely the reliability of job performance ratings. Fur-
thermore, competing cognitive process mechanisms have
been postulated (e.g., Borman, 1979; Wohlers & London,
1989) to explain the convergence in ratings between two
raters. An accurate evaluation of these competing mech-
anisms will facilitate and enhance understanding of the
psychology underlying the rating or the evaluative judg-
ment process in general, and job performance in partic-
ular. Finally, many human resource practices recom-
mended to practitioners in organizations are predicated
on the reliable measurement of job performance. As
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
such, both from a theoretical perspective (i.e., to analyze
This document is copyrighted by the American Psychological Association or one of its allied publishers.
and build theories contributing to the science of indus-
trial and organizational psychology) and from a practice
perspective, a comparative analysis of reliability of job
performance ratings is warranted.
The primary purpose of this study was to investigate
the reliability of peer and supervisory ratings of various
job performance dimensions. Meta-analytic principles
(Hunter & Schmidt, 1990) were used to cumulate reli-
ability estimates across studies. Reliability estimates of
various job performance dimensions can be compared
to identify which dimensions are reliably rated so as to
identify and improve through training the dimensions
rated with low reliability. A second objective of this study
was to compare interrater agreement and intrarater con-
sistency in the reliability of ratings. A third and final ob-
jective of this study was to compile and present reliability
distributions that can be used in future meta-analyses in-
volving job performance ratings.
Comparing Reliability Across Dimensions
Supervisory and peer ratings have been used to assess
individuals on many dimensions of job performance.
Comparing the reliability of ratings of different dimen-
sions enables an empirical test of the hypothesis that cer-
tain dimensions of job performance are easier to evaluate
than others (cf. Wohlers & London, 1989). In essence,
the thrust of this hypothesis is that some dimensions of
job performance are easier than others to evaluate be-
cause they are easier to observe and clearer standards of
evaluation are available. Wohlers and London (1989)
suggested that dimensions of performance such as ad-
ministrative competence, leadership, and communica-
tion competence are more difficult to evaluate than di-
mensions such as output and errors.
Similarly, Borman (1979) found that "raters evaluated
ratees significantly more accurately on some dimensions
than on others, and that for most part these differences
were consistent across formats" (p. 419). Borman
(1979) also stated that the rank order accuracy on the
different dimensions in his study were similar to the rank
order accuracy in an earlier study by Borman, Hough,
and Dunnette (1976). The rank order correlation was
.88 for assessing managers and .54 for recruiters. That
is, the rank order accuracy of different dimensions were
consistent across rating formats, studies and samples,
and jobs. Borman (1979) noted that this consistent di-
mension effect, even across a variety of formats (and if
we may add, across jobs and samples), may be due to
something inherent in the nature of the dimensions that
makes them either difficult or easy for raters. Further-
more, Borman suggested that "accuracy is highest on
those dimensions for which actors provided the least am-
biguous, most consistent performances, perhaps because
they, as well as the student raters, understood those par-
ticular dimensions better than some of the other dimen-
sions" (p. 420).
The hypothesis that certain dimensions of job perfor-
mance are easier to evaluate than others is also found in
personality literature (e.g., Christensen, 1974). This line
of thought is also found in literature on social psychology.
For example, Bandura (1977), as well as Salancik and
Pfeffer (1978), posited from a social information-pro-
cessing framework that when there are no clear interpret-
able signs of behavior or where the standards of evalua-
tion are ambiguous, the interrater agreement will be
lower in comparison with where there are clear interpret-
able signs and standards are unambiguous. This is also
hypothesized to be true when certain dimensions of job
performance have rare occurrence (low base rate) or
have greater salience in memory (e.g., accidents).
Comparing the reliability of dimensions facilitates an
empirical test of the hypothesis (Borman, 1979; Wohlers
& London, 1989) of a gradient in reliabilities across job
performance dimensions. Such knowledge will facilitate
an understanding of the rating processes.
Comparing Different Types
of Reliability Estimates
Comparing the different types of reliability estimates
(coefficient of equivalence, coefficient of stability, etc.)
for each dimension of job performance is also valuable.
Reliability of a measure is defined as the ratio of the true
to observed variance (Nunnally, 1978). Different types
of reliability coefficients assign different sources of vari-
ance to measurement error. In general, the most fre-
quently used reliability coefficients associated with crite-
rion ratings can be broadly classified into two categories:
interrater and intrarater. In the context of performance
measurement, interrater reliability assesses the extent to
which different raters agree on the performance of
different individuals. As such, individual raters' idiosyn-
cratic perceptions of job performance are considered to
be part of measurement error. Intrarater reliability, on
 RELIABILITY OF RATINGS
the other hand, assigns any specific error unique to the
individual rater to true variance. That is, each rater's id-
iosyncratic perceptions of job performance is relegated
to the true variance component. Both coefficient alpha
and the coefficient of stability (rate-rerate reliability with
the same rater) are forms of intrarater reliability. Intrar-
ater reliability is most frequently indexed by coefficient
alpha computed on ratings from a single rater on the basis
of the correlations or covariances among different rating
items or dimensions. Coefficient alpha assesses the extent
to which the different items used to measure a criterion
are indeed assessing the same criterion.1 Rate-rerate re-
liability computed using data from the same rater at two
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
points in time assesses the extent to which there is consis-
tency in performance appraisal ratings of a given rater
over time. Both of these indices of intrarater reliability,
coefficient alpha and coefficient of stability (over short
period of times when it is assumed that true performance
does not change), estimate what the correlation would be
if the same rater rerated the same employees (Cronbach,
1951).2
Thus, different types of reliability estimates assign
different sources of variance to measurement error vari-
ance. When a single judge rates a job performance di-
mension with a set of items, coefficient alpha may be
computed on the basis of the correlations or covariances
among the items. Coefficient alpha, which is a measure
of equivalence of the items, assigns the item specific vari-
ance and variance because of random responses in rat-
ings to measurement error variance. Influences unique
to the particular rating occasion and unique to the rater
are not assigned to measurement error but incorporated
into the true variance. When job performance is assessed
by the same rater with the same set of items at two differ-
ent points in time, the resulting coefficient of stability
(rate-rerate reliability) assigns variance because of tran-
sient errors in rating (i.e., variance from mental states
and other factors in raters that vary over days) to mea-
surement error variance (Schmidt & Hunter, 1996).
Thus, by comparing the different reliability estimates
for the same dimension of job performance, one can
gauge the magnitude of a particular source of error in
ratings involving that dimension of job performance.
Such knowledge can be valuable in designing rating for-
mats and rater training programs.
Constructing Artifact Distributions
Constructing artifact distributions for different dimen-
sions of job performance also serves meta-analytic cu-
mulations involving ratings of job performance. The re-
liability distributions reported here could be used in
meta-analyses of studies involving ratings of perfor-
mance.3 Also, some published meta-analyses involving
559
ratings have erroneously combined estimates of in-
terrater and intrarater reliability in one artifact distribu-
tion as if these estimates were equivalent. With the in-
creasing emphasis on precision of estimates to be used in
theory testing (Schmidt & Hunter, 1996; Viswesvaran &
Ones, 1995), it is imperative that future meta-analyses
use the appropriate reliability estimates. By providing es-
timates of different reliability coefficients for each dimen-
sion of job performance, this article aims to provide a
useful source of reference to researchers.
Thus, the primary purpose in this article is to cumu-
late the reliabilities of job performance ratings with the
principles of psychometric meta-analysis (Hunter &
Schmidt, 1990) and to compare the reliability of the rat-
ings of different dimensions. Comparing the reliability of
different dimensions enables an evaluation of the hypoth-
esis (Borman, 1979; Wohlers & London, 1989) that eval-
uation difficulty varies across dimensions. A secondary
purpose in this article is to compare the magnitude of
the different sources of errors (by comparing interrater
reliabilities, coefficient alphas, and coefficients of
stability) that exist in ratings of each dimension of job
performance. A third purpose is to provide reliability dis-
tributions that could be used in future meta-analytic cu-
mulations involving ratings of performance.
Method
Database
We searched the literature for articles that reported reliability
coefficients either for job performance dimensions or for overall
job performance. Only studies that were based on actual job
performance were included. Interviewer ratings, assessment
center ratings, and ratings of performance in simulated exer-
cises were excluded. We searched all issues starting from the first
issue of each journal through January 1994 of the following 15
journals: Journal of Applied Psychology, Personnel Psychology,
Academy of Management Journal, Human Relations, Journal
of Business and Psychology, Journal of Management, Organiza-
tional Behavior and Human Decision Processes, Accident Anal-
1
Coefficient alpha computed on ratings from a single rater is
an estimate of the rate-rerate reliability with the same rater. As
such, it is a form of intrarater reliability. However, it should be
noted that a different coefficient alpha can be used to index in-
terrater reliability. This is possible if the variance-covariance
matrix across raters is used in the computations. In this study,
we did not examine coefficient alphas obtained by using data
across raters.
2
For a recent discussion of these and other reliabilities in in-
dustrial and organizational psychology research, see Schmidt
and Hunter (1996).
3
Frequency distributions of the reliabilities contributing to
the analyses reported in this article may be obtained by writing
to Chockalingam Viswesvaran.
 560 VISWESVARAN, ONES, AND SCHMIDT
ysis and Prevention, International Journal of Intercultural Re-
lations, Journal of Vocational Behavior, Journal of Applied Be-
havioral Analysis, Human Resources Management Research,
Journal of Occupational Psychology, Psychological Reports, and
Journal of Organizational Behavior.
Analyses
In cumulating results across studies, the same job perfor-
mance dimensions can be referred to with different labels. Any
grouping of the different labels as measuring the same criteria
has to be guided by theoretical considerations. That is, we need
to theoretically define the criteria first (Campbell et al., 1993).
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
The broader the definition, the more general and possibly more
This document is copyrighted by the American Psychological Association or one of its allied publishers.
useful the criteria are; on the other hand, the narrower the defi-
nition (up to a point), the more unambiguous the criteria be-
come. The delineation of the job performance domain to its
component dimensions was undertaken as part of a study ex-
amining whether a general job performance factor is responsi-
ble for the covariation among job performance dimensions
(Viswesvaran, 1993). Viswesvaran (1993) identified 10 job
performance dimensions that comprehensively represented the
entire job performance domain. In this study, all the job perfor-
mance measures used in the individual studies were listed and
then grouped into one of the conceptually similar categories by
the authors. That is, the definition of the job performance di-
mensions and classification of the job performance ratings into
these 10 dimensions preceded the coding of the reliability esti-
mates. We read all the articles making up our database and then
classified the reliabilities. In other words, not only did we take
into account the definitions but also the context (and all other
information) provided in the article in classifying the reliabili-
ties into the dimensions. Interrater agreement was 93%. Dis-
agreements were resolved through mutual discussion until con-
sensus was reached. Definitions for the 10 groups of ratings for
which analyses are reported here are provided in Table 1.
Given 10 dimensions of job performance and three types of
reliabilities (interrater, stability, and equivalence), there were
potentially 30 reliability distributions to be investigated. Be-
cause our interest was in examining the reliability of both su-
pervisory and peer ratings, there are potentially 60 distributions
to be meta-analyzed. Of these, some combinations have not
been assessed in the literature. The reliability values obtained
from the individual studies were coded into 1 of the 60
distributions.
Next, in cumulating the reliability of any particular criteria
across several studies, the length of the measuring instrument
(number of raters for interrater reliability estimates and num-
ber of items for coefficient alpha estimates) varied across the
studies. One option was to use the Spearman-Brown formula
to bring all estimates to a common length. We reduced all in-
terrater reliability estimates to that of one rater. In many organ-
izations, there will almost never be more than one supervisor
doing the rating, but there will almost never be an instrument
with only one item (i.e., performance dimension rated). As
such, we did not correct the coefficient alphas for the number
of items. Furthermore, most rating instruments had a range of
items where the Spearman-Brown adjustments did not make a
practical difference.
For the coefficient of stability, without knowing the func-
tional relationship between the estimates of stability and the
time interval between the measurements, corrections to bring
estimates of stability to the same interval are impossible. All we
can say, intuitively speaking, is that as the time interval in-
creases, the reliability estimate generally decreases. The func-
tion that captures this decrease is unknown. Jensen (1980),
based on fitting curves to empirical data, reported that the sta-
bility of IQ test scores is a function of the square root of the
ratio of the chronological ages at the two points of measure-
ment. Another possibility is to assume an asymptotic function
where reliability estimate falls as time interval increases (at in-
finite time interval, the reliability estimate will be zero, or an
asymptote at zero). This is similar to Rothstein (1990) who
presented empirical evidence that as the opportunity to observe
(indexed by number of years supervised or observed) increases,
the interrater reliability increases but reaches an asymptotic
maximum of .60. Lacking information on the functional rela-
tionship between reliability estimates and time intervals be-
tween measurements, no corrections were made to bring all es-
timates of coefficients of stability included in a meta-analysis to
the same interval.
Note that in our intrarater reliability analyses, we were care-
ful to include only the coefficients of stability that were based
on ratings from the same rater. Rate-rerate correlations from
different raters at two points in time are interrater reliability
coefficients and will be lower than the former estimates where
the same rater provides the ratings at the two points in time and
thus intrarater reliability is assessed (Cronbach, 1947).
A meta-analysis correcting only for sampling error was con-
ducted for each of the 60 distributions for which there were at
least four estimates to be cumulated. The sample size weighted
mean, observed standard deviation, and residual standard devia-
tion were computed for each distribution. We also computed the
unweighted mean and standard deviation. The computations of
the unweighted mean and standard deviation do not weight the
reliability estimates by sample size of each study contributing to
the analysis. Each reliability coefficient is equally weighted. The
sample size weighted mean gives the best estimate of the mean
reliability, whereas the unweighted mean ensures that our results
are not skewed by a few large sample estimates.
In addition, we also computed the mean and standard devia-
tion of the square root of the reliabilities. The mean of the square
root of the reliabilities differ slightly from the square root of the
mean of the reliabilities. Therefore, we also assessed the mean
and standard deviation of the square root of the reliabilities. Both
sample size weighted and unweighted (i.e., frequency weighted)
analyses were undertaken. Thus, for each of the 60 distributions,
the objective was to estimate the mean and standard deviation
of (a) sample size weighted reliability estimates, (b) reliability
estimates (unweighted or frequency weighted), (c) sample size
weighted square root of the reliabilities, and (d) square root of
the reliabilities (unweighted or frequency weighted).
The sampling error variance associated with the mean of the
reliability was estimated as the variance divided by the number
of estimates averaged (Callender & Osburn, 1988). The sam-
pling error of the mean was used to construct an 80% confidence
interval around the mean. Assuming normality, 80% of points
in the distribution falls within this interval. That is, the proba-
 562 VISWESVARAN, ONES, AND SCHMIDT

Table 2
Interrater Reliabilities of Supervisory Ratings of Job Performance

Dimension Afw SDW AC 80% CI cred

Overall job
performance 14,650 40 .52 .0950 .68 .1469 .72 .0605 .82 .0924 .50-.54 .0870 .41 -.63
Productivity 2,015 19 .57 .1540 .57 .1769 .75 .1079 .75 .1236 .S2-.62 .1392 .39-.7S
Quality 1,225 10 .63 .1191 .65 .1406 .79 .0756 .80 .0885 .S8-.68 .1058 A9-.ll
Leadership 2,171 20 .53 .0928 .55 .1124 .73 .0598 .74 .0742 .50-.56 .0617 .45-.61
Communication
competence 1,563 9 .45 .1282 .43 .1824 .66 .1071 .64 .1568 .40-.50 .1129 .31-.59
Administrative
competence 1,120 9 .58 .1040 .59 .1674 .76 .0659 .76 .1056 .S4-.62 .0851 .47-.69
Effort 2,714 24 .55 .1250 .56 .1601 .74 .0858 .74 .1113 .S2-.58 .1062 .41 -.69
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Interpersonal
This document is copyrighted by the American Psychological Association or one of its allied publishers.

competence 3,006 31 .47 .1664 .53 .1983 .68 .1332 .70 .1711 .43-.51 .1461 .2S-.66
Job knowledge 14,072 20 .53 .0508 .56 .1976 .73 .0392 .73 .2356 .S2-.54 .0429 .48-.5S
Compliance with or
acceptance of
authority 905 8 .56 .1276 .60 .1295 .74 .1548 .77 .0900 .50-.62 .1099 .42-.70

Note, k = number of reliabilities included in the meta-analysis; wt = sample size weighted; unwt = unweighted or frequency weighted; sqwt =
square root of the estimates, weighted; squnwt = square root of the estimates, unweighted; CI = confidence interval; cred = credibility interval;
res = residual.

eluded in the meta-analysis. Columns 4 and 5 provide the
sample size weighted mean and standard deviation of the
values meta-analyzed. The unweighted (or frequency
weighted) mean and standard deviation of the values
meta-analyzed are in Columns 6 and 7, respectively. The
sample size weighted mean and standard deviation of the
square root of the reliabilities are in Columns 8 and 9,
respectively. Finally, the unweighted (or frequency
weighted) mean and standard deviation of the square
root of the reliabilities are in Columns 10 and 11, respec-
tively. Column 12 provides the 80% confidence interval
that is based on the sample size weighted mean reliability
values. Different intervals (e.g., 95%) can be constructed
on the basis of the values reported in Columns 3, 4, and
5. Similarly, different intervals can be computed for (a)
unweighted (or frequency weighted) reliability values de-
rived from data reported in Columns 3, 6, and 7; (b)
sample size weighted square root of the reliability esti-

Table 3
Interrater Reliabilities of Peer Ratings and Coefficients of Stability for Supervisory Ratings of Job Performance
Performance 80%
dimension « k A/« SDM A/unwl 5Dunwt
AC.W1
5Aqwt
AC.unwt Oi'squnwt 80% CI 5»» cred

Peer ratings
Overall job
performance 2,389 9 .42 .1063 .44 .1615 .64 .0764 .66 .1141 .S7-.47 .0935 .30-.54
Productivity 205 4 .34 .1419 .30 .1775 .57 .1414 .52 .1743 .2S-.43 .0676 .2S-.43
Leadership 434 4 .38 .1142 .36 .1595 .61 .0950 .59 .1387 .31-.45 .0789 .28-.4S
Effort 348 7 .42 .2454 .48 .2452 .61 .2332 .66 .2243 .30-.54 .2152 .14-.70
Interpersonal
competence 635 9 .42 .1451 .50 .1613 .64 .1095 .70 .1196 .36-.4S .1063 .28-.S6
Job knowledge 249 4 .33 .1012 .33 .1268 .57 .0900 .57 .1151 .27-.39 .1000 .20-.46
Compliance with or
acceptance of
authority 220 5 .71 .0493 .73 .0740 .84 .0290 .86 .0437 .6S-.74 .0467 .6S-.77

Supervisory ratings
Overall job
performance 1,374 12 .81 .0895 .84 .0875 .95 .0685 .91 .0462 .7S-.84 .0835 .70-.92

Note, k = number of reliabilities included in the meta-analysis; wt = sample size weighted; unwt = unweighted or frequency weighted; sqwt =
square root of the estimates, weighted; squnwt = square root of the estimates, unweighted; CI = confidence interval; cred = credibility interval;
res = residual.
 RELIABILITY OF RATINGS 561

Table 1
Definition of Job Performance Rating

Dimension rated Definition

Overall job performance
Job performance or productivity
Quality
Leadership
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
Ratings on statements (or ranking of individuals on statements) referring to overall performance, overall
effectiveness, overall job performance, overall work reputation, or the sum of all individual
dimensions rated.
Ratings of the quantity or volume of work produced. Raters' rating or ranking individuals were based on
productivity or sales; examples include ratings of the number of accounts opened by bank tellers and
the number of transactions completed by sales clerks.
Measure of how well the job was done. Ratings of (or rankings of individuals on statements referring to)
the quality of tasks completed, lack of errors, accuracy to specifications, thoroughness, and amount of
wastage.
Measure of the ability to inspire, to bring out extra performance in others, to motivate others to scale
great heights, and professional stature; includes performance appraisal statments such as "gets

subordinates to work efficiently," "stimulates subordinates effectively," and "maintains authority

This document is copyrighted by the American Psychological Association or one of its allied publishers.

easily and comfortably."

Communication competence
Administrative competence
Effort
Interpersonal competence
Job knowledge
Compliance with or acceptance
of authority
Skill in gathering and transmitting information (both in oral and written format). The proficiency to
express, either in written or oral format, information views, opinions, and positions. This refers to the
ability to make oneself understood; includes performance appraisal statements such as "very good in
making reports," "reports are clear," "reports are unambiguous," and "reports need no further
clarification."
Proficiency in handling the coordination of different roles in an organization. This refers to proficiency
in organizing and scheduling work periods, administrative maintenance of records (note, however,
that clarity is under Communication competence above), ability to place and assign subordinates, and
knowledge of the job duties and responsibilities of others.
Amount of work an individual expends in striving to do a good job. Measure of initiative, attention to
duty, alertness, resourcefulness, enthusiasm about work, industriousness, earnestness at work,
persistence in seeking goals, dedication, personal involvement in the job, and effort and energy
expended on the job characterize this dimension of job performance.
Ability to work well with others. Ratings or rankings of individuals on cooperation with others,
customer relations, working with co-workers, and acceptance by others, as well as nominations for
"easy to get along with," are included in this dimension.
Measure of the knowledge required to get the job done. Includes ratings or rankings of individuals on
job knowledge, keeping up-to-date, as well as nominations of who knows the job best and
nominations of who keeps up-to-date.
A generally positive perspective about rules and regulations; includes obeying rules, conforming to
regulations in the work place, having a positive attitude toward supervision, conforming to
organizational norms and culture, without incessant complaining about organizational policies and
following instructions.

bility of obtaining a value higher than the upper bound of the
interval and the probability of obtaining a value lower than the
lower bound is. 10.
For both interrater reliability and coefficient of stability
(rate-rerate reliability with the same rater), in addition to the
confidence interval, the sampling error of the correlation was
computed, and credibility intervals were constructed. A resid-
ual standard deviation was computed as the square root of the
difference between observed and sampling error variance of the
correlation (i.e., the interrater reliability coefficient in the for-
mer case and the rate-rerate reliability coefficient in the latter
case). Note, however, that the sampling error formula for co-
efficient alpha is different from those of interrater reliability co-
efficients and coefficients of stability (i.e., correlation
coefficients). Given the mean and residual standard deviation
along with the normality assumption (assuming two-tailed
tests), we can compute the estimated reliability below which the
population reliability value is likely to fall with a 90% chance; M
+ 1.28(residual standard deviation). Though different (90%,
95%, etc.) credibility intervals (and upper bound values) can be
constructed, we report only on the 80% credibility interval for
the sample size weighted mean reliability estimate for the reli-
ability distributions. Interested readers can compute the other
credibility intervals (90%, 95%, etc.) on the basis of the mean
reliability and the residual standard deviation.
Results
Tables 2-4 summarize the results of the meta-analyses.
Interrater reliability estimates for supervisory ratings are
summarized in Table 2. Interrater reliability estimates
for peer ratings are in Table 3, and the estimates of co-
efficient of stability for supervisory ratings of overall job
performance are also in Table 3. Estimates of coefficient
alpha for supervisory and peer ratings are provided in
Table 4. Notice that all 10 dimensions are not present in
every table; we do not present the results of meta-analyses
that were based on less than four reliability estimates.
In each table, Column 1 indicates the job performance
dimension being meta-analyzed, Column 2 indicates the
total sample size (the total number of individuals rated
across studies included in that meta-analysis), and Col-
umn 3 provides the number of independent estimates in-
 RELIABILITY OF RATINGS 563

Table 4
Coefficient Alpha Reliabilities of Supervisory and Peer Ratings of Job Performance (Intrarater Reliabilities)
Dimension n k MM SDW M unwt
ivl SDmw M^ SAqw, Miqunwl SD^n 80% CI
Supervisory
Overall job
performance 17,899 89 .86 .1433 .84 .1510 .92 .0942 .91 .1089 .84-.S8
Productivity 2,697 17 .82 .1248 .85 .1110 .90 .0711 .92 .0630 .7S-.86
Quality 739 6 .81 .0752 .81 .0828 .90 .0413 .90 .0455 .77-.8S
Leadership 3,821 21 .77 .1239 .77 .1315 .87 .0735 .87 .0768 .74-.80
Communication
competence 943 8 .73 .1707 .69 .2091 .85 .1103 .82 .1359 .65-81
Administrative
competence 4,754 16 .79 .0544 .79 .0965 .89 .0305 .89 .0543 .77-81
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

Effort 3,112 20 .79 .1147 .75 .1392 .88 .0678 .86 .0835 .76-82
This document is copyrighted by the American Psychological Association or one of its allied publishers.

Interpersonal
competence 10,955 56 .77 .1691 .75 .1902 .87 .1185 .85 .1327 .74-80
Job knowledge 959 9 .79 .1077 .77 .1290 .89 .0645 .87 .0770 .74-84
Compliance with or
acceptance of
authority 3,438 15 .77 .1194 .76 .1858 .87 .0790 .86 .1383 .73-81
Peer
Overall job
performance 1,270 10 .85 .1193 .81 .1205 .92 .0664 .85 .0677 .80-90
Leadership 1,082 5 .61 .1931 .53 .2934 .76 .1892 .67 .3014 .50-72
Effort 1,205 7 .77 .2372 .72 .2526 .86 .1705 .83 .1852 .66-88
Interpersonal
competence 325 8 .61 .2067 .61 .2298 .77 .2054 .76 .1797 .52-70
Note, k = number of reliabilities included in the meta-analysis; wt = sample size weighted; unwt = unweighted or frequency weighted; sqwt
square root of the estimates, weighted; squnwt = square root of the estimates, unweighted; CI = confidence interval.

mates derived from data presented in Columns 3, 8, and
9; and (c) unweighted (or frequency weighted) square
root of reliabilities derived from information provided in
Columns 3, 10, and 11. For interrater reliability and co-
efficient of stability, the residual standard deviations of
the reliability distributions and the 80% credibility in-
tervals are reported in Columns 13 and 14, respectively.
The credibility interval refers to the entire distribution,
not the mean value. Also, it refers to population values
(the estimated distribution of population values), not
observed values, which were affected by sampling error.
In discussing the results, we first compared the super-
visory rating reliability of different dimensions of rated
performance for each type of reliability (e.g., interrater).
Then we focused on the same type of reliability (e.g.,
interrater), but it was based on peer ratings of the differ-
ent dimensions. Third, we compared the reliability for
peer and supervisory ratings. These three steps were re-
peated for each type of reliability: interrater, stability, and
coefficient alphas. A final section discusses assessment of
the relative influence of the different sources of error.
Interrater Reliability
From the results reported in Table 2, the mean in-
terrater reliability for supervisory ratings of overall job
performance was .52 (k = 40, N = 14,650). The 80% cred-
ibility interval ranged from .41 to .63. That is, it is esti-
mated that 90% of the values of interrater reliability of
supervisory ratings of overall job performance are below
.63. For supervisors, the mean sample size weighted mean
interrater reliability across nine specific job performance
dimensions (excluding overall job performance) was .53.
It appears that, for supervisors, interrater reliability of
overall job performance ratings is similar to the mean in-
terrater reliability across job performance dimensions.
This is noteworthy because most interrater reliabilities for
overall performance in our database were for sums of
items across different job performance dimensions. Con-
trary to expectations, higher intrarater reliability associ-
ated with longer rating forms (see also the interrater reli-
ability section below) does not appear to improve in-
terrater reliability in the job performance domain.
A second interesting point to note is that there is vari-
ation across the 10 dimensions in the mean interrater re-
liabilities for supervisory ratings. Although the credibil-
ity intervals for all the 10 dimensions do overlap, the 80%
confidence intervals indicate that for example, both com-
munication competence and interpersonal competence
are rated less reliably, on average, than productivity or
quality. Thus, the hypothesis of Wohlers and London
(1989) and Borman (1979) is partially supported.
 564 VISWESVARAN, ONES, AND SCHMIDT
Interrater reliability for peer ratings of 7 of the 10 di-
mensions are reported in Table 3 (there were less than
four estimates for the other three dirhensions). The esti-
mates ranged from .34 for ratings of productivity (SD =
.14) to .71 for ratings of compliance with authority (SD
= .05). For peers, the sample size weighted mean in-
terrater reliability across six specific dimensions of job
performance (i.e., excluding overall job performance)
was .42. For ratings of overall job performance, interrater
reliability for peer ratings was also .42 (SD = . 11). The
80% credibility intervals for interrater reliability of peer
ratings of overall job performance ranged from .30 to .54.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
That is, 90% of the actual (population) values are esti-
This document is copyrighted by the American Psychological Association or one of its allied publishers.
mated to be less than .54, and 90% of the values were
estimated to be greater than .30. Similar to the results for
supervisors, interrater reliability of overall job perfor-
mance ratings is the same as the sample size weighted
mean interrater reliability across individual job perfor-
mance dimensions. Even though a large portion of the
peer interrater reliabilities for overall performance in our
database were computed in studies by summing items
across different job performance dimensions, higher in-
trarater reliability associated with longer rating forms
(also see Coefficient Alphas: Measures oflntrarater Reli-
ability below) does not appear to lead to higher peer in-
terrater reliability. This mirrors the case for supervisors.
A comparison of the results reported in Tables 2 and 3
seems to indicate that there was generally more agreement
between two supervisors than there was between two peers.
However, caution is needed in inferring such a conclusion.
First, the interrater reliability estimates for peer ratings were
based on small number of studies (so are some dimensions
of supervisory ratings). Second, there is considerable over-
lap in credibility intervals between interrater reliability es-
timates of peer and supervisory ratings. Finally, two of the
studies reporting interrater reliabilities of peer ratings
(Borman, 1974; Hausman & Strupp, 1955) reported very
low values. When these two studies were eliminated from
the database as outliers, peers and supervisors had compa-
rable levels of interrater agreement. However, similar to the
overall results of this meta-analysis, we should note that a
recent large sample primary study has also reported lower
interrater reliability estimates for peers compared with su-
pervisors (Scullen, Mount, & Sytysma, 1995). Further-
more, in practice, given that peer ratings are based on the
average ratings of several peers, the averaged multiple peer
ratings may be more reliable than the ratings from a single
supervisor. The Spearman-Brown prophecy formula can
be used to determine the number of peer raters required.
Coefficient of Stability
Compared with the number of studies reporting in-
terrater reliabilities or coefficient alphas, very few studies
reported coefficients of stability. This is consistent with
the general trend of more cross-sectional than longitudi-
nal studies among published journal articles. In fact, we
were able to assess the coefficient of stability only for su-
pervisory ratings of overall job performance. There were
12 reliabilities across 1,374 individuals contributing to
this analysis. For supervisory ratings of overall job per-
formance, the sample size weighted mean coefficient of
stability was .81 (SD = .09). This analysis included only
estimates where the same rater was used at the two points
in time.
Coefficient Alphas: Measures oflntrarater
Reliability
Intrarater reliabilities assessed by coefficient alphas
were also substantial. For supervisory ratings, overall job
performance was most reliably rated (.86). The least re-
liably rated dimension was communication competence
(.73). Although the alpha estimates for all dimensions
as well as for overall ratings were higher than .70, it is
important to note that these estimates are inclusive of,
among other things, a halo component. Another observa-
tion is that for supervisory ratings of overall job perfor-
mance, the coefficient of stability reported above and the
coefficient alpha were similar in size (.81 and .86,
respectively). This finding supports the inference that the
variance of transient errors (variance because of rater
mental states or moods that vary over days) is small.
These figures suggest that this source of measurement er-
ror variance in overall job performance ratings is only 5%
of the total variance (.86-.81 = .05).
In Table 4, it can be seen that peer ratings of overall job
performance had a mean alpha of .85 (k = 10, N =
1,270). The intrarater reliability associated with peer rat-
ings of leadership, effort, and interpersonal competence
were above .60. Similar to interrater reliability, compar-
ing alphas for peer and supervisory ratings should be ten-
tative. When comparing peer and supervisory ratings, it
appears that the intrarater reliability is lower for peer
than for supervisory ratings of specific job performance
dimensions, but not for overall performance.
An interesting point to note for both peer and supervi-
sory ratings is that the alphas were higher for the overall
job performance ratings than for any of the dimensional
ratings. For supervisors, the coefficient alpha for overall
job performance was .86, whereas the mean sample size
weighted alpha across the specific job performance di-
mensions was .78. For peers, the coefficient alpha for
overall job performance was .85, whereas the mean sam-
ple size weighted alpha across the specific job perfor-
mance dimensions was .68. There are two potential ex-
planations for this result. First, this could have been due
to greater length of the instrument used for measure-
 RELIABILITY OF RATINGS
ment. In a large number of the studies we coded, overall
job performance was measured by summing the various
dimensions of job performance into a composite. How-
ever, we should point out that the nature of the relation-
ship between the number of items and reliability can be
best described as convex. The reliability increases rapidly
initially as the number of items increases, but after some
point the increase in the reliability is very small. Most of
the scales meta-analyzed in this article had enough items
that further increases in length and the application of the
Spearman-Brown formula did not make an appreciable
difference. Note that this could indirectly explain our
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
earlier finding that both supervisor and peer interrater
This document is copyrighted by the American Psychological Association or one of its allied publishers.
reliabilities for specific dimensions of job performance
are similar to the interrater reliabilities for overall job
performance. The second potential explanation for
higher alphas for ratings of overall job performance stems
from the broadness of the construct of overall job perfor-
mance compared with any of the constructs represented
by the individual dimensions of job performance. More-
over, there is some evidence (at least in the personality
domain) that broader constructs are more reliably rated
than narrower constructs (Ones & Viswesvaran, in
press). That is, this finding could reflect that broader
traits or constructs are more reliably rated than narrowly
defined traits (Ones & Viswesvaran, in press). Unfortu-
nately, this meta-analytic investigation cannot determine
which of the two potential explanations is correct.
Comparison of Different Types of Reliability
Estimates
Conceptually, given that (a) the reliability coefficient is
the ratio of true to observed variance and (b) observed
variance is true plus error variance, all types of reliability
estimates have the same denominator. Coefficient alpha
(using a single rater) has variance specific to the rater
and variance because of transient error in the numerator.
Coefficient of stability or rate-rerate with the same rater
has variance specific to the rater in the numerator
(assuming true performance did not change in the rate-
rerate interval), but not transient variance. Thus, the
difference between coefficient alpha and coefficient of
stability with the same rater gives an estimate of the tran-
sient error in that job performance dimension, as noted
earlier. Interrater reliability does not have variance spe-
cific to the rater or transient error variance in the numer-
ator. Therefore, the difference between interrater and co-
efficient of stability provides an estimate of the variance
from rater idiosyncrasy.
For both peer and supervisory ratings and for all di-
mensions and overall ratings, interrater reliability esti-
mates are substantially lower than intrarater reliability
estimates (coefficients of stability and coefficient alphas).
565
For example, consider supervisory ratings of overall job
performance: the mean interrater reliability estimate is
.52, the mean coefficient of stability is .81 (on the basis of
ratings from same rater at the two points in time), and
the mean coefficient alpha is .86. Approximately 29% of
the variance (.81-.52 = .29) in supervisory ratings of
overall job performance appears to be due to rater idio-
syncrasy, whereas 5%, or .86-.81 X 100, of the variance
is estimated to be from transient error, assuming true job
performance is stable. Similar analyses can be done for
other dimensions on the basis of data reported in Tables
2-4 to compare the magnitude of the source of error in
ratings of different dimensions of job performance. In-
trarater reliability for supervisory ratings of job perfor-
mance dimensions are between .70 and .90. However, the
mean interrater reliabilities range approximately be-
tween .50 and .65. The difference between intrarater and
interrater reliability estimates that we obtained indicate
that 20% to 30% of the variance in job performance di-
mension ratings of the average rater is specific to the rater.
Using coefficient alpha instead of interrater reliability of
job performance ratings to correct observed validities
(say in validating interviews) will underestimate the va-
lidity. Lacking empirically derived reliability distribu-
tions, like those yielded by this study, previous meta-ana-
lysts may have combined the correct interrater and incor-
rect intrarater reliabilities. However, future meta-
analyses involving job performance ratings should use
the appropriate reliability coefficients (Schmidt &
Hunter, 1996) to obtain more precise estimates of corre-
lations that could be used for theory testing.
Discussion
Job performance measures play a crucial role in re-
search and practice. Ratings (both peer and supervisory)
are an important method of job performance measure-
ment in organizations. Many decisions are made on the
basis of ratings. As such, the reliability of ratings is an
important concern in organizational science. Depending
on the objective of the researcher, different reliability es-
timates need to be assessed. In personnel selection, the
use of intrarater reliabilities to correct criterion-related
validity coefficients for unreliability in job performance
ratings may result in substantial downward biases in esti-
mates of actual operational validity. This bias arises
mostly from including rater specific error variance
(variance due to rater idiosyncrasies) as true job perfor-
mance variance in computing intrarater reliability. On
the other hand, what is needed to assess actual job per-
formance and its dimensions is an answer to the question:
Would the same ratings be obtained if a different but
equally knowledgeable judge rated the same employees.
This calls for an assessment of interrater reliability. This
 566 VISWESVARAN, ONES, AND SCHMIDT
is why interrater reliability is the appropriate reliability
in making corrections for criterion unreliability in vali-
dation research, not coefficient alpha or rate-rerate reli-
ability with the same rater.
This article quantitatively summarizes the available
evidence in the literature for use by researchers and prac-
titioners. A question for future research is whether in-
terrater reliability ratings of overall job performance can
be increased by obtaining dimensional ratings before ob-
taining the overall ratings.4 (Note that a similar potential
does not exist for intrarater reliabilities.) It is possible
that when overall performance is rated after dimension
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
ratings are made, interrater reliabilities for overall ratings
This document is copyrighted by the American Psychological Association or one of its allied publishers.
are higher because all raters have a more similar frame-
of-reference compared to when the overall performance
rating is made on its own or when overall ratings precede
dimensional ratings. Furthermore, the issue here is com-
plicated by the fact that in many studies overall job per-
formance ratings are obtained by summing the dimen-
sional ratings, whereas in others overall ratings are ob-
tained on a single item (or a few items) before or after
dimensional ratings are provided. To the extent frame-of-
reference effects were operating, the standard deviation
of the mean interrater reliability for overall ratings
should be higher than the standard deviation of mean di-
mensional ratings. That is, some studies would have ob-
tained overall performance ratings prior to dimensional
ratings, and others would have obtained overall ratings
after dimensional ratings. If the frame-of-reference hy-
pothesis were correct, in a meta-analytic investigation
this would have been detected as greater variance in the
interrater reliability of overall job performance ratings.
Of course, the interrater reliability of dimensional ratings
would not have this source of variance. Hence, the stan-
dard deviation of the interrater reliability for overall rat-
ings would be high compared with the standard deviation
of dimensional ratings. Our results indicate that this is
not the case. However, given that the studies contributing
to our overall job performance analyses were a mixture of
a sum of dimensional ratings and items directly assessing
overall job performance, we cannot reach any definite
conclusions regarding the frame-of-reference effects. In
any event, this is an interesting hypothesis for future
research.
In cumulating results across studies, a concern exists
whether moderating influences are obscured. The low
values of the standard deviations (compared with the
means) mitigate this concern to some extent. Further-
more, Churchill and Peter (1984) and Peterson (1994)
examined as many as 13 moderators of reliability esti-
mates (e.g., whether the reliabilities were obtained for re-
search or administrative purposes). No substantial rela-
tionships were found between any hypothesized modera-
tor and magnitude of reliability estimates. A potentially
important moderating influence may be whether the rat-
ings were obtained for research or administrative
purposes. McDaniel, Whetzel, Schmidt, and Maurer
(1994) found that the purpose of the performance ratings
(administrative vs. research) moderated the validities of
employment interviews. In this study, we examined three
moderators of job performance rating reliabilities: type
of reliability (interrater vs. intrarater), source of rating
(peer vs. supervisors), and job performance-dimension
rated. We were not able to examine the moderating in-
fluences of administrative versus research-based ratings.
This was primarily because, given the number of studies,
analysis of any other moderator in a fully hierarchical
design (Hunter & Schmidt, 1990) would have resulted in
too few studies for a robust meta-analysis. The concern
for sufficient data to detect moderators, coupled with the
fact that previous meta-analyses (e.g., Peterson, 1994)
that included alternate moderators did not find support
for those alternate moderators, led us to focus only on
these three moderators (type of reliability, source of rat-
ing, and rating content). However, future research should
examine the interaction of these three moderators with
other potential moderators, such as the purpose for
which the ratings were obtained (administrative vs.
research).
The results reported here can be used to construct re-
liability artifact distributions to be used in meta-analyses
(Hunter & Schmidt, 1990) when correcting for unreli-
ability in the criterion ratings. For example, the report by
a National Academy of Sciences (NAS) panel (Hartigan
& Wigdor, 1989) evaluating the utility gains from validity
generalization (Hunter, 1983) maintained that the mean
interrater reliability estimate of .60 used by Hunter
(1983) was too small and that the interrater reliability of
supervisory ratings of overall job performance is better
estimated as .80. The results reported here indicate that
the average interrater reliability of supervisory ratings of
job performance (cumulated across all studies available
in the literature) is .52. FurthermoVe, this value is similar
to that obtained by Rothstein (1990), although we
should point out that a recent large-scale primary study
(N = 2,249) obtained a lower value of .45 (Scullen et al.,
1995). On the basis of our findings, we estimate that the
probability of interrater reliability of supervisory ratings
of overall job performance being as high as .80 (as
claimed by the NAS panel) is only .0026. These findings
indicate that the reliability estimate used by Hunter
(1983) is, if anything, probably an overestimate of the
reliability of supervisory ratings of overall job perfor-
mance. Thus, it appears that Schmidt, Ones, and Hunter
(1992) were correct in concluding that the NAS panel
underestimated the validity of the General Aptitude Test
4
We thank an anonymous reviewer for suggesting this.
 RELIABILITY OF RATINGS
Battery (GATE). The estimated validity of other opera-
tional tests may be similarly rescrutinized.
An anonymous reviewer presented two concerns as
fundamental questions that need to be addressed. First,
the reviewer raised the question whether reliability cor-
rections should be undertaken when one does not have
estimates from the same study in which the validity was
estimated. Second, if the answer was affirmative to the
first question, another question arises as to whether one
should use the mean reliabilities reported in this article
or some conservative value (e.g., the 80% upper bound
values reported in this article).
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
There are two reasons for answering the first question
This document is copyrighted by the American Psychological Association or one of its allied publishers.
in the affirmative. First, any bias introduced in the esti-
mated true validity from using reliability estimates re-
ported in this article will be much less than the downward
bias in validity estimates if no corrections were under-
taken. That is, when reliability estimates from the sample
are not available, the alternative is to make no correc-
tions. Second, the meta-analytically obtained reliability
estimates reported here may be more accurate than the
sample-based estimates a primary researcher could ob-
tain, given the major effect of sampling error on reliabil-
ity estimates in single studies (which typically have small
sample sizes). Using the meta-analytically obtained esti-
mates reported here instead of the sample-based esti-
mates may result in greater accuracy. The numerous sim-
ulation studies indicating the robustness of artifact dis-
tribution-based meta-analyses (cf. Hunter & Schmidt,
1990) support the conclusion that bias is lower when
meta-analytically obtained means are used to correct for
bias than if either (a) sample-based estimates are used in
the corrections or (b) no corrections are made.
The answer to the second question raised by the re-
viewer can also be framed in terms of bias in the esti-
mated correlations. Using conservative values for reli-
ability results in more bias than the use of the mean val-
ues. Many researchers maintain that being conservative
is good science, but conservative estimates are by defini-
tion biased estimates. We believe it is more appropriate
to aim for unbiased estimates because the research goal
is to maximize the accuracy of the final estimates.
Future meta-analytic research is needed to examine
the reliability of criteria obtained from other sources
such as customer ratings, self-ratings, and subordinate
ratings. In a large-scale primary study (N= 2,273), Scul-
len et al. (1995) reported that the inter rater reliability of
subordinate ratings is similar to those obtained for peers
(ranging between .31 and .36) for various dimensions of
job performance. We see the efforts of Scullen et al.
(1995) as a valuable first step in reaching generalizable
conclusions about the reliability of subordinate ratings.
Future research is also needed to examine the process
mechanisms (e.g., Campbell, 1990; DeNisi, Cafferty, &
567
Meglino, 1984) by which the criterion data are gathered
and thus improve the reliability of the obtained ratings.
There are several unique contributions of the present
study. Particularly, we want to clearly delineate how our
study contributes over and beyond the Rothstein (1990)
study, which was the largest scale primary study reported
examining the interrater reliability of supervisory rat-
ings. First, Rothstein (1990) focused only on interrater
reliabilities. Here, we investigated interrater and intrara-
ter reliabilities, cumulated interrater reliabilities, co-
efficient alphas, and test-retest reliabilities. Second,
Rothstein (1990) focused on overall job performance.
Rothstein (1990) did not examine the reliabilities of di-
mensions of the job performance construct. Given the
theoretical arguments and rating processes hypothesized
that posit different reliabilities for different dimensions,
we examined the reliability of different dimensions of job
performance as well as the reliability of overall job per-
formance. Third, whereas the Rothstein (1990) study
was based on a large sample, it was nevertheless a single
primary study confined to one research corporation that
markets the Supervisory Profile Record (see Rothstein,
1990). Finally, Rothstein (1990) focused on reliabilities
of supervisory ratings only. We analyzed both supervi-
sory and peer ratings, and we examined whether the reli-
abilities of peer and supervisory ratings are similar across
job performance dimensions.
However, in contrast to our study, Rothstein (1990)
was able to examine the effects of length of exposure on
interrater reliability with her primary data. We were not
able to test this effect, as most studies did not specify how
long the raters were exposed to the ratees. (Of course,
that was not the focus in many studies making up our
database.) Future meta-analytic research should attempt
to generalize the Rothstein (1990) findings with regard
to length of exposure to other rating instruments.
The results of this article offer psychometric insights
into the psychological and substantive characteristics of
job performance measures. The construction of gener-
alizable theories of job performance starts with an ex-
amination of the reliable measurement of job perfor-
mance dimensions. Given that ratings (supervisory and
peer) are used most frequently in the measurement of
this central construct, it is crucial that researchers and
managers be concerned about the reliability of these
measurements. For research involving the construct of
job performance, accurate construct measurement is
predicated on reliable job performance measurement.
For practice, accurate administrative decisions depend
on the reliable measurement of job performance. It is
our hope that the results presented here can be used to
understand and improve job performance measurement
in organizations.
 568 VISWESVARAN, ONES, AND SCHMIDT
References
The asterisk (*) indicates studies that were included in
the meta-analysis.
*Albrecht, P. A., Glaser, E. M., & Marks, J. (1964). Validation
of a multiple-assessment procedure for managerial person-
nel. Journal of Applied Psychology, 48, 351-360.
*Anderson, H. E., Jr., Roush, S. L., & McClary, J. E. (1973).
Relationships among ratings, production, efficiency, and the
general aptitude test battery scales in an industrial setting.
Journal of Applied Psychology, 58, 77-82.
*Arvey, R. D., Landon, T. E., Nutting, S. M., & Maxwell,
S. E. (1992). Development of physical ability tests for police
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
officers: A construct validation approach. Journal of Applied
Psychology, 77, 996-1009.
*Ashford, S. J., & Tsui, A. S. (1991). Self-regulation for mana-
gerial effectiveness: The role of active feedback seeking. Acad-
emy of Management Journal, 34, 251-280.
Austin, J. T, & Villanova, P. (1992). The criterion problem:
1917-1992. Journal of Applied Psychology, 77, 836-874.
*Baird, L. S. (1977). Self and superior ratings of performance:
As related to self-esteem and satisfaction with supervision.
Academy of Management Journal, 20, 291-300.
Bandura, A. (1977). Social learning theory. Englewood Cliffs,
NJ: Prentice-Hall.
*Barrick, M. R., Mount, M. K., & Strauss, J. P. (1993). Con-
scientiousness and performance of sales representatives: Test
of the mediating effects of goal setting. Journal of Applied
Psychology, 78, 715-722.
*Bass, A. R., & Turner, J. N. (1973). Ethnic group differences
in relationships among criteria of job performance. Journal
of Applied Psychology, 57, 101-109.
*Becker, T. E., & Vance, R. J. (1993). Construct validity of
three types of organizational citizenship behavior: An illus-
tration of the direct product model with refinements. Journal
of Management, 19, 663-682.
*Bernardin, H. J. (1987). Effect of reciprocal leniency on the
relation between consideration scores from the leader behav-
ior description questionnaire and performance ratings. Psy-
chological Reports, 60, 479-487.
Bernardin, H. J., & Beatty, R. W. (1984). Performance ap-
praisal: Assessing human behavior at work. Boston: Kent.
*Bhagat, R. S., & Allie, S. M. (1989). Organizational stress,
personal life style, and symptoms of life strains: An examina-
tion of the moderating role of sense of competence. Journal
of Vocational Behavior, 35, 231-253.
*Blank, W., Weitzel, J. R., & Green, S. G. (1990). A test of the
situational leadership theory. Personnel Psychology, 43, 579-
597.
*Blanz, F., & Ghiselli, E. E. (1972). The mixed standard scale:
A new rating system. Personnel Psychology, 25, 185-199.
*Blau, G. (1986). The relationship of management level to
effort level, direction of effort, and managerial performance.
Journal of Vocational Behavior, 29, 226-239.
*Blau, G. (1988). An investigation of the apprenticeship organ-
izational socialization strategy. Journal of Vocational Behav-
ior, 32, 176-195.
*Blau, G. (1990). Exploring the mediating mechanisms affect-
ing the relationship of recruitment source to employee per-
formance. Journal of Vocational Behavior, 37, 303-320.
*Bledsoe, J. C. (1981). Factors related to academic and job per-
formance of graduates of practical nursing programs. Psy-
chological Reports, 49, 367-371.
Blum, M. L., & Naylor, J. C. (1968). Industrial psychology: Its
theoretical and social foundations. New York: Harper & Row.
*Borman, W. C. (1974). The rating of individuals in organiza-
tions: An alternate approach. Organizational Behavior and
Human Performance, 12, 105-124.
Borman, W. C. (1979). Format and training effects on rating
accuracy and rater errors. Journal of Applied Psychology, 64,
410-412.
Borman, W. C., Hough, L. M., & Dunnette, M. D. (1976). Per-
formance ratings: An investigation of reliability, accuracy,
and relationship between individual differences and rater er-
ror. Minneapolis, MN: Personnel Decisions.
*Breaugh, J. A. (1981 a). Predicting absenteeism from prior ab-
senteeism and work attitudes. Journal of Applied Psychology,
66, 555-560.
*Breaugh, J. A. (1981b). Relationships between recruiting
sources and employee performance, absenteeism, and work
attitudes. Academy of Management Journal, 24, 142-147.
Brogden, H. E. (1946). An approach to the problem of differ-
ential prediction. Psychometrika, 11, 139-154.
*Buckner, D. N. (1959). The predictability of ratings as a func-
tion of interrater agreement. Journal of Applied Psychology,
43, 60-64.
*Buel, W. D., & Bachner, V. M. (1961). The assessment of cre-
ativity in a research setting. Journal of Applied Psychology,
45, 353-358.
*Bushe, G. R., & Gibbs, B. W. (1990). Predicting organization
development consulting competence from the Myers-Briggs
type indicator and state of ego development. Journal of Ap-
plied Behavioral Science, 26, 337-357.
*Butler, M. C., & Ehrlich, S. B. (1991). Positional influences on
job satisfaction and job performance: A multivariate, predic-
tive approach. Psychological Reports, 69, 855-865.
Callender, J. C., & Osburn, H. G. (1988). Unbiased estimation
of the sampling variance of correlations. Journal of Applied
Psychology, 73, 312-315.
*Campbell, C. H., Ford, P., Rumsey, M. G., Pulakos, E. D.,
Borman, W. C., Felker, D. B., De Vera, M. V, & Riegelhaupt,
B. J. (1990). Development of multiple job performance mea-
sures in a representative sample of jobs. Personnel Psychol-
ogy, 43, 277-300.
Campbell, J. P. (1990). Modeling the performance prediction
problem in industrial and organizational psychology. In M.
Dunnette&L. M. Hough (Eds.), Handbook ofindustrial or-
ganizational psychology (2nd ed.. Vol. 1, pp. 687-732). Palo
Alto, CA: Consulting Psychologists Press.
*Campbell, J. P., Dunnette, M. D., Arvey, R. D., & Hellervik,
L. V. (1973). The development and evaluation of behavior-
ally based rating scales. Journal of Applied Psychology, 57,
15-22.
Campbell, J. P., Gasser, M. B., & Oswald, F. L. (1996). The
substantive nature of job performance variability. In K. R.
Murphy (Ed.), Individual differences and behavior in organi-
zations (pp. 258-299). San Francisco: Jossey-Bass.
 RELIABILITY OF RATINGS
Campbell, J. P., McCloy, R. A., Oppler, S. H., & Sager, C. E.
(1993). A theory of performance. In N. Schmitt & W. C.
Borman (Eds.), Personnel selection in organizations (pp. 35-
70). San Francisco: Jossey-Bass.
Cascio, W. F. (1991). Applied psychology in personnel manage-
ment (4th ed.). Englewood Cliffs, NJ: Prentice-Hall.
*Cascio, W. F., & Valenzi, E. R. (1977). Behaviorally anchored
rating scales: Effects of education and job experience of raters
andratees. Journal of Applied Psychology, 62, 278-282.
*Cascio, W. E, & Valenzi, E. R. (1978). Relations among cri-
teria of police performance. Journal of Applied Psychology,
63, 22-28.
*Cheloha, R. S., & Farr, J. L. (1980). Absenteeism, job involve-
ment, and job satisfaction in an organizational setting.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
Journal of Applied Psychology, 65, 467-473.
Christensen, L. (1974). The influence of trait, sex, and infor-
mation accuracy of personality assessment. Journal of Per-
sonality Assessment, 38, 130-135.
Churchill, G. A., Jr., & Peter, J. P. (1984). Research design
effects on the reliability of rating scales: A meta-analysis.
Journal of Marketing Research, 21, 360-375.
*Cleveland, J. N., & Landy, F. J. (1981). The influence of rater
and ratee age on two performance judgments. Personnel Psy-
chology, 34, 19-29.
Cleveland, J. N., Murphy, K. R., & Williams, R. E. (1989).
Multiple uses of performance appraisal: Prevalence and cor-
relates. Journal of Applied Psychology, 74, 130-135.
*Cleveland, J. N., & Shore, L. M. (1992). Self- and supervisory
perspectives on age and work attitudes and performance.
Journal of Applied Psychology, 77, 469-484.
*Colarelli, S. M., Dean, R. A., & Konstans, C. (1987). Com-
parative effects of personal and situational influences on job
outcomes of new professionals. Journal of Applied Psychol-
ogy, 72, 558-566.
"Cooper, R. (1966). Leader's task relevance and subordinate
behaviour in industrial work groups. Human Relations, 19,
57-84.
*Cooper, R., & Payne, R. (1967). Extraversion and some as-
pects of work behavior. Personnel Psychology, 20, 45-57.
*Cortina, J. M., Doherty, M. L., Schmitt, N., Kaufman, G., &
Smith, R. G. (1992). The "Big Five" personality factors in
the IPI and MMPI: Predictors of police performance. Person-
nel Psychology, 45, 119-140.
*Cotton, J., & Stoltz, R. E. (1960). The general applicability of
a scale for rating research productivity. Journal of Applied
Psychology, 44, 276-277.
Cronbach, L. J. (1947). Test reliability: Its meaning and deter-
mination. Psychometrika, 12, 1-16.
Cronbach, L. J. (1951). Coefficient alpha and the internal
structure of tests. Psychometrika, 16, 297-334.
*David, F. R., Pearce, J. A., II, & Randolph, W. A. (1989).
Linking technology and structure to enhance group perfor-
mance. Journal of Applied Psychology, 74, 233-241.
*Day, D. W., & Silverman, S. B. (1989). Personality and job
performance: Evidence of incremental validity. Personnel
Psychology, 42, 25-36.
*Deadrick, D. L., & Madigan, R. M. (1990). Dynamic criteria
revisited: A longitudinal study of performance stability and
predictive validity. Personnel Psychology, 43,117-744.
569
DeNisi, A. S., Cafferty, T. P., & Meglino, B. M. (1984). A cog-
nitive view of the performance appraisal process: A model
and research propositions. Organizational Behavior and Hu-
man Performance, 33, 360-396.
*Dicken, C. F, & Black, J. D. (1965). Predictive validity of
psychometric evaluations of supervisors. Journal of Applied
Psychology, 49, 34-47.
"Dickinson, T. L., & Tice, T. E. (1973). A multitrait-
multimethod analysis of scales developed by retranslation.
Organizational Behavior and Human Performance, 9, 421-
438.
*Distefano, M. K., Jr., Pryer, M. W., & Erffmeyer, R. C. (1983).
Application of content validity methods to the development
of a job-related performance rating criterion. Personnel Psy-
chology, 56,621-631.
*Dreher, G. F. (1981). Predicting the salary satisfaction of ex-
empt employees. Personnel Psychology, 34, 579-589.
*Dunegan, K. J., Duchon, D., & Uhl-Bien, M. (1992). Exam-
ining the link between leader-member exchange and subor-
dinate performance: The role of task analyzability and vari-
ety as moderators. Journal of Management, 18, 59-76.
Dunnette, M. D. (1963). A note on the criterion. Journal of
Applied Psychology, 47, 251-253.
*Edwards, P. K. (1979). Attachment to work and absence be-
havior. Human Relations, 32, 1065-1080.
*Ekpo-Ufot, A. (1979). Self-perceived task-relevant abilities,
rated job performance, and complaining behavior of junior
employees in a government ministry. Journal of Applied Psy-
chology, 64, 429-434.
*Farh, J., Podsakoff, P. M., & Organ, D. W. (1990). Accounting
for organizational citizenship behavior: Leader fairness and
task scope versus satisfaction. Journal of Management, 16,
705-721.
*Farh, J., Werbel, J. D, & Bedeian, A. G. (1988). An empirical
investigation of self-appraisal-based performance evalua-
tion. Personnel Psychology, 41, 141-156.
*Farr, J.-L., O'Leary, B. S., & Bartlett, C. J. (1971). Ethnic
group membership as a moderator of the prediction of job
performance. Personnel Psychology, 24, 609-636.
*Flanders, J. K. (1918). Mental tests of a group of employed
men showing correlations with estimates furnished by em-
ployer. Journal of Applied Psychology, 2, 197-206.
*Gardner, D. G., Dunham, R. B., Cummings, L. L., & Pierce,
J. L. (1989). Focus of attention at work: Construct definition
and empirical validation. Journal of Occupational Psychol-
ogy, 62, 61-77.
*Gerloff, E. A., Muir, N. K., & Bodensteiner, W. D. (1991).
Three components of perceived environmental uncertainty:
An exploratory analysis of the effects of aggregation. Journal
of Management, 17, 749-768.
*Ghiselli, E. E. (1942). The use of the Strong vocational inter-
est blank and the Pressy senior classification test in the selec-
tion of casualty insurance agents. Journal of Applied Psychol-
ogy, 26, 793-799.
*Gough, H. G., Bradley, P., & McDonald, J. S. (1991). Perfor-
mance of residents in anesthesiology as related to measures
of personality and interests. Psychological Reports, 68, 979-
994.
*Graen, G., Dansereau, F, Jr., & Minami, T. (1972). An em-
 570 VISWESVARAN, ONES, AND SCHMIDT
pirical test of the man-in-the-middle hypothesis among exec-
utives in a hierarchical organization employing a unit-set
analysis. Organizational Behavior and Human Performance,
8, 262-285.
*Graen, G., Novak, M. A., & Sommerkamp, P. (1982). The
effects of leader-member exchange and job design on pro-
ductivity and satisfaction: Testing a dual attachment model.
Organizational Behavior and Human Performance, 30, 109-
131.
*Green, S. B., & Stutzman, T. (1986). An evaluation of meth-
ods to select respondents to structured job-analysis question-
naires. Personnel Psychology, 39, 543-564.
*Greenhaus, J. H., Bedeian, A. G., & Mossholder, K. W.
(1987). Work experiences, job performance, and feelings of
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
personal and family well-being. Journal of Vocational Behav-
ior, 31,200-215.
*Griffin, R. W. (1991). Effects of work redesign on employee
perceptions, attitudes, and behaviors: A long-term investiga-
tion. Academy of Management Journal, 34,425-435.
*Guion, R. M. (1965). Synthetic validity in a small company:
A demonstration. Personnel Psychology, 18, 49-63.
"•Gunderson, E. K. E., & Nelson, P. D. (1966). Criterion mea-
sures for extremely isolated groups. Personnel Psychology, 19,
67-80.
*Gunderson, E. K. E., & Ryman, D. H. (1971). Convergent and
discriminant validities of performance evaluations in ex-
tremely isolated groups. Personnel Psychology, 24,115-724.
"Hackman, J. R., & Lawler, E. E., III. (1971). Employee reac-
tions to j ob characteristics [ Monograph ]. Journal of Applied
Psychology, 55, 259-286.
"Hackman, J. R., & Porter, L. W. (1968). Expectancy theory
predictions of work effectiveness. Organizational Behavior
and Human Performance, 3,417-426.
Hartigan, J. A., & Wigdor, A. K. (Eds.). (1989). Fairness in
employee testing: Validity generalization, minority issues,
and the General Aptitude Test Battery. Washington, DC: Na-
tional Academy Press.
*Hatcher, L., Ross, T. L., & Collins, D. (1989). Prosocial be-
havior, job complexity, and suggestion contribution under
gainsharing plans. Journal of Applied Behavioral Science, 25,
231-248.
*Hater, J. J., & Bass, B. M. (1988). Superiors' evaluations and
subordinates' perceptions of transformational and transac-
tional leadership. Journal of Applied Psychology, 73, 695-
702.
*Hausman, H. J., & Strupp, H. H. (1955). Non-technical fac-
tors in supervisors' ratings of job performance. Personnel
Psychology, 5,201-217.
"Heneman, H. G., III. (1974). Comparisons of self- and supe-
rior ratings of managerial performance. Journal of Applied
Psychology, 59, 638-642.
*Heron, A. (1954). Satisfaction and satisfactoriness: Comple-
mentary aspects of occupational adjustment. Occupational
Psychology, 28, 140-153.
*Hilton, A. C., Bolin, S. R, Parker, J. W, Jr., Taylor, E. K., &
Walker, W. B. (1955), The validity of personnel assessments
by professional psychologists. Journal of Applied Psychology,
39, 287-293.
"Hoffman, C. C., Nathan, B. R., & Holden, L. M. (1991). A
comparison of validation criteria: Objective versus subjective
performance measures and self- versus supervisor ratings.
Personnel Psychology, 44, 601-619.
"Hogan, J., Hogan, R., & Busch, C. M. (1984). How to measure
service orientation. Journal of Applied Psychology, 69, 167-
173.
*Hough, L. M. (1984). Development and evaluation of the "ac-
complishment record" method of selecting and promoting
professionals. Journal of Applied Psychology, 69, 135-146.
*Huck, J. R., & Bray, D. W. (1976). Management assessment
center evaluations and subsequent job performance of white
and black females. Personnel Psychology, 29, 13-30.
"•Hughes, G. L., & Prien, E. P. (1986). An evaluation of al-
ternate scoring methods for the mixed standard scale. Person-
nel Psychology, 39, 839-847.
Hunter, J. E. (1983). Test validation for 12,000 jobs: An appli-
cation of job classification and validity generalization to Gen-
eral Aptitude Test Battery (U.S. Employment Service Test
Research Report No. 45). Washington, DC: U.S. Depart-
ment of Labor.
Hunter, J. E., & Schmidt, F. L. (1990). Methods ofmeta-analy-
sis: Correcting for error and bias in research findings. New-
bury Park, CA: Sage.
*Ivancevich, J. M. (1980). A longitudinal study of behavioral
expectation scales: Attitudes and performance. Journal of
Applied Psychology, 65, 139-146.
"Ivancevich, J. M. (1983). Contrast effects in performance
evaluation and reward practices. Academy of Management
Journal, 26, 465-476.
*Ivancevich, J. M. (1985). Predicting absenteeism from prior
absence and work attitudes. Academy of Management Jour-
nal, 28, 219-228.
"Ivancevich, J. M., & McMahon, J. T. (1977). Black-white
differences in a goal-setting program. Organizational Behav-
ior and Human Performance, 20, 287-300.
"Ivancevich, J. M., & McMahon, T. J. (1982). The effects of
goal setting, external feedback, and self-generated feedback
on outcome variables: A field experiment. Academy of Man-
agement Journal, 25, 359-372.
"•Ivancevich, J. M., & Smith, S. V. (1981). Goal setting in-
terview skills training: Simulated and on-the-job analyses.
Journal of Applied Psychology, 66, 697-705.
"•Ivancevich, J. M., & Smith, S. V. (1982). Job difficulty as in-
terpreted by incumbents: A study of nurses and engineers.
Human Relations, 35, 391 -412.
"Jamal, M. (1984). Job stress and job performance contro-
versy: An empirical assessment. Organizational Behavior and
Human Performance, 33, 1-21.
"•James, L. R., & Ellison, R. L. (1973). Creation composites for
scientific creativity. Personnel Psychology, 26, 147-161.
Jensen, A. R. (1980). Bias in mental testing. New York: Free
Press.
"Johnson, J. A., & Hogan, R. (1981). Vocational interests, per-
sonality and effective police performance. Personnel Psychol-
ogy, 34, 49-53.
"•Jones, J. W., & Terris, W. (1983). Predicting employees' theft
in home improvement centers. Psychological Reports, 52,
187-201.
 RELIABILITY OF RATINGS
*Jordan, J. L. (1989). Effects of race on interrater reliability of
peer ratings. Psychological Reports, 64, 1221-1222.
"Jurgensen, C. E. (1950). Intercorrelations in merit rating
traits. Journal of Applied Psychology, 34,240-243.
*Keller, R. T. (1984). The role of performance and absenteeism
in the prediction of turnover. Academy of Management Jour-
nal, 27, 176-183.
*King, L. M., Hunter, J. E., & Schmidt, F. L. (1980). Halo in a
multidimensional forced-choice performance evaluation
scale. Journal of Applied Psychology, 65, 507-516.
*Klaas, B. S. (1989). Managerial decision making about em-
ployee grievances: The impact of the grievant's work history.
Personnel Psychology, 42, 53-68.
*Klaas, B. S., & DeNisi, A. S. (1989). Managerial reactions to
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
employee dissent: The impact of grievance activity on perfor-
mance ratings. Academy of Management Journal, 32, 705-
717.
*Klimoski, R. J., & Hayes, N. J. (1980). Leader behavior and
subordinate motivation. Personnel Psychology, 33, 543-555.
*Knauft, E. B. (1949). A selection battery for bake shop man-
agers. Journal of Applied Psychology, 33, 304-315.
*Kubany, A. J. (1957). Use of sociometric peer nominations in
medical education research. Journal of Applied Psychology,
41, 389-394.
*Landy, F. J., & Guion, R. M. (1970). Development of scales
for the measurement of work motivation. Organizational Be-
havior and Human Performance, 5, 93-103.
"Latham, G. P., Fay, C. H., & Saari, L. M. (1979). The devel-
opment of behavioral observation scales for appraising the
performance of foremen. Personnel Psychology, 32, 299-311.
*Latham, G. P., & Wexley, K. N. (1977). Behavioral observa-
tion scales for performance appraisal purposes. Personnel
Psychology, 30, 255-268.
*Lawshe, C. H., & McGinley, A. D., Jr. (1951). Job perfor-
mance criteria studies: I. The job performance of proofread-
ers. Journal of Applied Psychology, 35, 316-320.
*Lee, R., Malone, M., & Greco, S. (1981). Multitrait-
multimethod-multirater analysis of performance ratings for
law enforcement personnel. Journal of Applied Psychology,
66, 625-632.
*Love, K. G. (1981). Comparison of peer assessment methods:
Reliability, validity, friendship bias, and user reaction.
Journal of Applied Psychology, 66, 451-457.
*Love, K. G., & O'Hara, K. (1987). Predicting job perfor-
mance of youth trainees under a job training partnership act
program (JTPA): Criterion validation of a behavior-based
measure of work maturity. Personnel Psychology, 40, 323-
340.
*MacKenzie, S. B., Podsakoff, P. M., & Fetter, R. (1991). Or-
ganizational citizenship behavior and objective productivity
as determinants of managerial evaluations of salespersons'
performance. Organizational Behavior and Human Decision
Processes, 50, 123-150.
*Matteson, M. T., Ivancevich, J. M., & Smith, S. V. (1984).
Relation of type A behavior to performance and satisfaction
among sales personnel. Journal of Vocational Behavior, 25,
203-214.
*Mayfield, E. C. (1970). Management selection: Buddy nomi-
nations revisited. Personnel Psychology, 23, 377-391.
571
*McCarrey, M. W., & Edwards, S. A. (1973). Organizational
climate conditions for effective research scientist role perfor-
mance. Organizational Behavior and Human Performance,
9, 439-459.
*McCauley, C. D., Lombardo, M. M., & Usher, C. J. (1989).
Diagnosing management development needs: An instrument
based on how managers develop. Journal of Management, 15,
389-403.
McCloy, R. A., Campbell, J. P., & Cudeck, R. (1994). A con-
firmatory test of a model of performance determinants.
Journal of Applied Psychology, 79,493-505.
McDaniel, M. A., Whetzel, D. L., Schmidt, F. L., & Maurer,
S. D. (1994). The validity of employment interviews: A com-
prehensive review and meta-analysis. Journal of Applied Psy-
chology, 79, 599-616.
*McEvoy, G. M., & Beatty, R. W. (1989). Assessment centers
and subordinate appraisals of managers: A seven-year exam-
ination of predictive validity. Personnel Psychology, 42, 37-
52.
*Meglino, B. M., Ravlin, E. C., & Adkins, C. L. (1989). A work
values approach to corporate culture: A field test of the value
congruence process and its relationship to individual out-
comes. Journal of Applied Psychology, 74,424-432.
*Meredith, G. M. (1990). Dossier evaluation in screening can-
didates for excellence in teaching awards. Psychological Re-
ports, 67, 879-882.
*Meyer, J. P., Paunonen, S. V., Gellatly, I. R., Coffin, R. D., &
Jackson, D. N. (1989). Organizational commitment and job
performance: It's the nature of the commitment that counts.
Journal of Applied Psychology, 74, 152-156.
*Miner, J. B. (1970). Executive and personnel interviews as
predictors of consulting success. Personnel Psychology, 23,
521-538.
*Miner, J. B. (1970). Psychological evaluations as predictors of
consulting success. Personnel Psychology, 23, 393-405.
*Mitchell, T. R., & Albright, D. W. (1972). Expectancy theory
predictions of the satisfaction, effort, performance, and re-
tention of naval aviation officers. Organizational Behavior
and Human Decision Processes, 8, 1-20.
*Morgan, R. B. (1993). Self- and co-worker perceptions of eth-
ics and their relationships to leadership and salary. Academy
of Management Journal, 36, 200-214.
*Morse, J. J., & Wagner, F. R. (1978). Measuring the process of
managerial effectiveness. Academy of Management Journal,
21, 23-35.
*Mossholder, K. W., Bedeian, A. G., Norris, D. R., Giles, W. F.,
& Feild, H. S. (1988). Job performance and turnover deci-
sions: Two field studies. Journal of Management, 14, 403-
414.
*Motowidlo, S. J. (1982). Relationship between self-rated per-
formance and pay satisfaction among sales representatives.
Journal of Applied Psychology, 67, 209-213.
*Mount, M. K. (1984). Psychometric properties of subordinate
ratings of managerial performance. Personnel Psychology, 37,
687-702.
*Nathan, B. R., Morman, A. M., Jr., & Milliman, J. (1991).
Interpersonal relations as a context for the effects of appraisal
interviews on performance and satisfaction: A longitudinal
study. Academy of 'Management Journal, 34, 352-369.
 572 VISWESVARAN, ONES, AND SCHMIDT
*Nealey, S. M., & Owen, T. W. (1970). A multitrait-
multimethod analysis of predictors and criteria of nursing
performance. Organizational Behavior and Human Perfor-
mance, 5, 348-365.
*Niehoff, B. P., & Moorman, R. H. (1993). Justice as a media-
tor of the relationship between methods of monitoring and
organizational citizenship behavior. Academy of Manage-
ment Journal, 36, 527-556.
*Noe, R. A., & Schmitt, N. (1986). The influence of trainee
attitudes on training effectiveness: Test of a model. Personnel
Psychology, 39, 497-523.
*Norris, D. R., & Niebuhr, R. E. (1984). Organization tenure
as a moderator of the job satisfaction-job performance rela-
tionship. Journal of Vocational Behavior, 24, 169-178.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
Nunnally, J. C. (1978). Psychometric theory. New York:
McGraw Hill.
""O'Connor, E. J., Peters, L. H., Pooyan, A., Weekley, J., Frank,
B., & Erenkrantz, B. (1984). Situational constraint effects on
performance, affective reactions, and turnover: A field repli-
cation and extension. Journal of Applied Psychology, 69,
663-672.
*Oldham, G. R. (1976). The motivational strategies used by
supervisors: Relationships to effectiveness indicators. Orga-
nizational Behavior and Human Performance, 15, 66-86.
Ones, D. S., & Viswesvaran, C. (in press). Bandwidth-fidelity
dilemma in personality measurement for personnel selection.
Journal of Organizational Behavior.
*Organ, D. W., & Konovsky, M. (1989). Cognitive versus
affective determinants of organizational citizenship behavior.
Journal of Applied Psychology, 74, 157-164.
*Otten, M. W., & Kahn, M. (1975). Effectiveness of crisis cen-
ter volunteers and the personal orientation inventory. Psycho-
logical Reports, 37, 1107-1111.
*Parker, J. W., Taylor, E. K., Barrett, R. S., & Martens, L.
(1959). Rating scale content: III. Relationship between su-
pervisory- and self-ratings. Personnel Psychology, 12, 49-63.
*Parsons, C. K., Herold, D. M., & Leatherwood, M. L. (1985).
Turnover during initial employment: A longitudinal study of
the role of causal attributions. Journal of Applied Psychology,
70, 337-341.
*Penley, L. E., & Hawkins, B. L. (1980). Organizational com-
munication, performance, and job satisfaction as a function
of ethnicity and sex. Journal of Vocational Behavior, 16, 368-
384.
Peterson, R. A. (1994). A meta-analysis of Cronbach's coeffi-
cient alpha. Journal of Consumer Research, 21, 381-391.
*Podsakoff, P. M., Niehoff, B. P., MacKenzie, S. B., & Williams,
M. L. (1993). Do substitutes for leadership really substitute
for leadership? An empirical examination of Kerr and Jer-
mier's situational leadership model. Organizational Behavior
and Human Decision Processes, 54, 1-44.
*Podsakoff, P. M., Todor, W. D., & Skov, R. (1982). Effects of
leader contingent and noncontingent reward and punishment
behaviors on subordinate performance and satisfaction.
Academy of Management Journal, 25, 810-821.
*Prien, E. P., & Kult, M. (1968). Analysis of performance cri-
teria and comparison of a priori and empirically-derived keys
for a forced-choice scoring. Personnel Psychology, 21, 505-
513.
*Prien, E. P., & Liske, R. E. (1962). Assessments of higher level
personnel: III. Rating criteria: A comparative analysis of su-
pervisor ratings and incumbent self-ratings of job perfor-
mance. Personnel Psychology, 15, 187-194.
*Puffer, S. M. (1987). Prosocial behavior, noncompliant behav-
ior, and work performance among commission salespeople.
Journal of Applied Psychology, 72, 615-621.
*Pulakos, E. D., Borman, W. C., & Hough, L. M. (1988). Test
validation for scientific understanding: Two demonstrations
of an approach to studying predictor-criterion linkages. Per-
sonnel Psychology, 41, 703-716.
*Pulakos, E. D., & Wexley, K. N. (1983). The relationship
among perceptual similarity, sex, and performance ratings in
manager-subordinate dyads. Academy of Management Jour-
nal, 26, 129-139.
*Pym, D. L. A., & Auld, H. D. (1965). The self-rating as a
measure of employee satisfactoriness. Occupational Psychol-
ogy, 39, 103-113.
*Rabinowitz, S., & Stumpf, S. A. (1987). Facets of role conflict,
role-specific performance, and organizational level within the
academic career. Journal of Vocational Behavior, 30, 72-83.
*Ronan, W. W. (1963). A factor analysis of eleven job perfor-
mance measures. Personnel Psychology, 16, 255-267.
*Rosinger, G., Myers, L. B., Levy, G. W., Loar, M., Mohrman,
S. A., & Stock, J. R. (1982). Development of a behaviorally
based performance appraisal system. Personnel Psychology,
35, 75-88.
*Ross, P. F., & Dunfield, N. M. (1964). Selecting salesmen for
an oil company. Personnel Psychology, 17, 75-84.
*Rosse, J. G. (1987). Job-related ability and turnover. Journal
of Business and Psychology, 1, 326-336.
*Rosse, J. G., & Kraut, A. I. (1983). Reconsidering the vertical
dyad linkage model of leadership. Journal of Occupational
Psychology, 56,63-71.
*Rosse, J. G., Miller, H. E., & Barnes, L. K. (1991). Combining
personality and cognitive ability predictors for hiring service-
oriented employees. Journal of Business and Psychology, 5,
431-445.
*Rothstein, H. R. (1990). Interrater reliability of job perfor-
mance ratings: Growth to asymptote level with increasing op-
portunity to observe. Journal of Applied Psychology, 75, 322-
327.
*Rothstein, H. R., Schmidt, F. L., Erwin, F. W., Owens, W. A.,
& Sparks, C. P. (1990). Biographical data in employment
selection: Can validities be made generalizable? Journal of
Applied Psychology, 75, 175-184.
*Rousseau, D. M. (1978). Relationship of work to nonwork.
Journal of Applied Psychology, 63, 513-517.
*Rush, C. H. Jr. (1953). A factorial study of sales criteria. Per-
sonnel Psychology, 6, 9-24.
*Russell, C. J. (1990). Selecting top corporate leaders: An ex-
ample of biographical information. Journal of Management,
16, 73-86.
*Russell, C. J., Mattson, J., Devlin, S. E., & Atwater, D. (1990).
Predictive validity of biodata items generated from retrospec-
tive life experience essays. Journal of Applied Psychology, 75,
569-580.
*Sackett, P. R., Zedeck, S., & Fogli, L. (1988). Relations be-
 RELIABILITY OF RATINGS
tween measures of typical and maximum job performance.
Journal of Applied Psychology, 73, 482-486.
Salancik, G. R., & Pfeffer, J. (1978). A social information pro-
cessing approach to job attitudes and task design. Admin-
istrative Science Quarterly, 23, 224-253.
*Schaubroeck, J., Ganster, D. C, Sime, W. E., & Ditman, D.
(1993). A field experiment testing supervisory role clarifi-
cation. Personnel Psychology, 46, 1-25.
*Schippmann, J. S., & Prien, E. P. (1986). Psychometric evalu-
ation of an integrated assessment procedure. Psychological
Reports, 59, 111-122.
Schmidt, F. L., & Hunter, J. E. (1992). Development of a causal
model of processes determining job performance. Current
Directions in Psychological Science, 1, 89-92.
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
Schmidt, F. L., & Hunter, J. E. (1996). Measurement error in
psychological research: Lessons from 26 research scenarios.
Psychological Methods, 1, 199-223.
Schmidt, F. L., Ones, D. S., & Hunter, J. E. (1992). Personnel
selection. In M. R. Rosenzweig & L. W. Porter (Eds.), Annual
review of psychology (pp. 627-670). Palo Alto, CA: Annual
Reviews.
*Schuerger, J. M., Kochevar, K. F., & Reinwald, J. E. (1982).
Male and female correction officers: Personality and rated
performance. Psychological Reports, 51, 223-228.
Scullen, S. E., Mount, M. K., & Sytysma, M. R. (1995). Com-
parison of self, peer, direct report and boss ratings of manag-
ers'performance. Unpublished manuscript.
*Seybolt, J. W., & Pavett, C. M. (1979). The prediction of effort
and performance among hospital professionals: Moderating
effects of feedback on expectancy theory formulations.
Journal of Occupational Psychology, 52, 91-105.
*Siegel, A. I., Schultz, D. G., Fischl, M. A., & Lanterman,
R. S. (1968). Absolute scaling of job performance. Journal
of Applied Psychology, 52, 313-318.
*Siegel, L. (1982). Paired comparison evaluations of manage-
rial effectiveness by peers and supervisors. Personnel Psychol-
ogy, 35, 843-852.
*Slocum, J. W., Jr., & Cron, W. L. (1985). Job attitudes and
performance during three career stages. Journal of Voca-
tional Behavior, 26, 126-145.
*Smircich, L., & Chesser, R. J. (1981). Superiors' and subordi-
nates' perceptions of performance: Beyond disagreement.
Academy of Management Journal, 24, 198-205.
*Sneath, F. A., White, G. C., & Randell, G. A. (1966). Validat-
ing a workshop reporting procedure. Occupational Psychol-
ogy, 40, 15-29.
*Soar, R. S. (1956). Personal history as a predictor of success in
service station management. Journal of Applied Psychology,
40, 383-385.
*South, J. C. (1974). Early career performance of engineers—
Its composition and measurement. Personnel Psychology, 27,
225-243.
*Spector, P. E., Dwyer, D. J., & Jex, S. M. (1988). Relation of
job stressors to affective, health, and performance outcomes:
A comparison of multiple data sources. Journal of Applied
Psychology, 73, 11-19.
*Spencer, D. G., & Steers, R. M. (1981). Performance as a
moderator of the job satisfaction-turnover relationship.
Journal of Applied Psychology, 66, 511-514.
573
*Spitzer, M. E., & McNamara, W. J. (1964). A managerial se-
lection study. Personnel Psychology, 17, 19-40.
*Sprecher, T. B. (1959). A study of engineers' criteria for cre-
ativity. Journal of Applied Psychology, 43, 141-148.
*Springer, D. (1953). Ratings of candidates for promotion by
co-workers and supervisors. Journal of Applied Psychology,
37,347-351.
*Steel, R. P., & Mento, A. J. (1986). Impact of situational con-
straints on subjective and objective criteria of managerial job
performance. Organizational Behavior and Human Decision
Processes, 37, 254-265.
*Steel, R. P., Mento, A. J., & Hendrix, W. H. (1987). Con-
straining forces and the work performance of finance com-
pany cashiers. Journal of Management, 13, 473-482.
*Steel, R. P., & Ovalle, N. K. (1984). Self-appraisal based upon
supervisory feedback. Personnel Psychology, 37, 667-685.
*Steel, R. P., Shane, G. S., & Kennedy, K. A. (1990). Effects of
social-system factors on absenteeism, turnover, and job per-
formance. Journal of Business and Psychology, 4, 423-430.
•Stout, S. K., Slocum, J. W., Jr., & Cron, W. L. (1987). Career
transitions of superiors and subordinates. Journal of Voca-
tional Behavior, 30, 124-137.
Stuit, D. B., & Wilson, J. T. (1946). The effect of an increasingly
well defined criterion on the prediction of success at naval
training school (tactical radar). Journal of Applied Psychol-
ogy, 30, 614-623.
*Stumpf, S. A. (1981). Career roles, psychological success, and
job attitudes. Journal of Vocational Behavior, 19, 98-112.
*Stumpf, S. A., & Rabinowitz, S. (1981). Career stage as a
moderator of performance relationships with facets of job
satisfaction and role perceptions. Journal of Vocational Be-
havior, 18,202-218.
*Sulkin, H. A., & Pranis, R. W. (1967). Comparison of griev-
ants with non-grievants in a heavy machinery company. Per-
sonnel Psychology, 20, 111-119.
*Swaroff, P. G., Barclay, L. A., & Bass, A. R. (1985). Recruiting
sources: Another look. Journal of Applied Psychology, 70,
720-728.
*Szilagyi, A. D. (1980). Causal inferences between leader re-
ward behaviour and subordinate performance, absenteeism,
and work satisfaction. Journal of Occupational Psychology,
53, 195-204.
"Taylor, E. K.., Schneider, D. E., & Symons, N. A. (1953). A
short forced-choice evaluation form for salesmen. Personnel
Psychology, 6, 393-401.
Taylor, R. L., & Wilsted, W. D. (1974). Capturing judgment
policies: A field study of performance appraisal. Academy of
Management Journal, 17, 440-449.
Taylor, R. L., & Wilsted, W. D. (1976). Capturing judgment
policies in performance rating. Industrial Relations, 15,216-
224.
Taylor, S. M., & Schmidt, D. W. (1983). A process-oriented
investigation of recruitment source effectiveness. Personnel
Psychology, 36, 343-354.
Tenopyr, M. L. (1969). The comparative validity of selected
leadership scales relative to success in production manage-
ment. Personnel Psychology, 22, 77-85.
Thompson, D. E., & Thompson, T. A. (1985). Task-based per-
 574 VISWESVARAN, ONES, AND SCHMIDT
formance appraisal for blue-collar jobs: Evaluation of race
and sex effects. Journal of Applied Psychology, 70, 747-753.
"Thomson, H. A. (1970). Comparison of predictor and crite-
rion judgments of managerial performance using the
multitrait-multimethod approach. Journal of Applied Psy-
chology, 54, 496-502.
Toops, H. A. (1944). The criterion. Educational and Psycho-
logical Measurement, 4, 271-297.
Tsui, A. S., & Ohlott, P. (1988). Multiple assessment of man-
agerial effectiveness: Interrater agreement and consensus in
effectiveness models. Personnel Psychology, 41, 779-803.
Tucker, M. F., Cline, V. B., & Schmitt, J. R. (1967). Prediction
of creativity and other performance measures from biograph-
ical information among pharmaceutical scientists. Journal of
This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
This document is copyrighted by the American Psychological Association or one of its allied publishers.
Applied Psychology, 51, 131-138.
Turner, W. W. (1960). Dimensions of foreman performance:
A factor analysis of criterion measures. Journal of Applied
Psychology, 44, 216-223.
"Validity information exchange. (1954). No. 7-045. Personnel
Psychology, 7, 279.
"Validity information exchange. (1954). No. 7-089. Personnel
Psychology, 7, 565-566.
"Validity information exchange. (1956). No. 9-32. Personnel
Psychology, 9, 375-377.
"Validity information exchange. (1958). No. 11-10. Personnel
Psychology.il, 121-123.
"Validity information exchange. (1958). No. 11-27. Personnel
Psychology, 11, 583-584.
"Validity information exchange. (1958). No. 11-30. Personnel
Psychology, 11, 587-590.
"Validity information exchange. (1960). No. 13-03. Personnel
Psychology, 13, 449-450.
"Validity information exchange. (1963). No. 16-04. Personnel
Psychology, 16, 181-183.
"Validity information exchange. (1963). No. 16-05. Personnel
Psychology, 16, 283-288.
"Vecchio, R. P. (1987). Situational leadership theory: An ex-
amination of a prescriptive theory. Journal of Applied Psy-
chology, 72,444-451.
"Vecchio, R. P., & Gobdel, B. C. (1984). The vertical dyad link-
age model of leadership: Problems and prospects. Organiza-
tional Behavior and Human Performance, 34, 5-20.
"Villanova, P., & Bernardin, J. H. (1990). Work behavior cor-
relates of interviewer job compatibility. Journal of Business
and Psychology, 5, 179-195.
Viswesvaran, C. (1993). Modeling job performance: Is there a
generalfactor? Unpublished doctoral dissertation, University
of Iowa, Iowa City.
Viswesvaran, C., & Ones, D. S. (1995). Theory testing: Com-
bining psychometric meta-analysis and structural equations
modeling. Personnel Psychology, 48, 865-887.
"Waldman, D. A., Yammarino, F. J., & Avolio, B. J. (1990).
A multiple level investigation of personnel ratings. Personnel
Psychology, ¥5,811-835.
"Wanous, J. P., Stumpf, S. A., & Bedrosian, H. (1979). Job
survival of new employees. Personnel Psychology, 32, 651-
662.
"Wayne, S. J., & Ferris, G. R. (1990). Influence tactics, affect,
and exchange quality in supervisor-subordinate interactions:
A laboratory experiment and field study. Journal of Applied
Psychology, 75, 487-499.
"Wernimont, P. F, & Kirchner, W. K. (1972). Practical prob-
lems in the revalidation of tests. Occupational Psychology, 46,
25-30.
"Wexley, K. N., Alexander, R. A., Greenawalt, J. P., & Couch,
M. A. (1980). Attitudinal congruence and similarity as re-
lated to interpersonal evaluations in manager-subordinate
dyads. Academy of Management Journal, 23, 320-330.
"Wexley, K. N., & Pulakos, E. D. (1982). Sex effects on perfor-
mance ratings in manager-subordinate dyads: A field study.
Journal of Applied Psychology, 67,433-439.
"Wexley, K. N., & Youtz, M. A. (1985). Rater beliefs about
others: Their effects on rating errors and rater accuracy.
Journal of Occupational Psychology, 58, 265-275.
"Williams, C. R., Labig, C. E., Jr., & Stone, T. H. (1993). Re-
cruitment sources and posthire outcomes for job applicants
and new hires: A test of two hypotheses. Journal of Applied
Psychology, 78, 163-172.
"Williams, L. J., & Anderson, S. E. (1991). Job satisfaction and
organizational commitment as predictors of organizational
citizenship and in-role behaviors. Journal of Management,
77,601-617.
"Williams, W. E., & Seiler, D. A. (1973). Relationship between
measures of effort and job performance. Journal of Applied
Psychology, 57, 49-54.
Wohlers, A. J., & London, M. (1989). Ratings of managerial
characteristics: Evaluation difficulty, co-worker agreement,
and self-awareness. Personnel Psychology, 42, 235-261.
"Woodmansee, J. J. (1978). Validation of the nurturance scale
of the Edwards Personal Preference Schedule. Psychological
Reports, 42, 495-498.
"Worbois, G. M. (1975). Validation of externally developed as-
sessment procedures for identification of supervisory poten-
tial. Personnel Psychology, 28, 77-91.
"Yammarino, F. J., & Dubinsky, A. J. (1990). Salesperson per-
formance and managerially controllable factors: An investi-
gation of individual and work group effects. Journal of Man-
agement, 16, 87-106.
"Yukl, G. A., & Latham, G. P. (1978). Interrelationships
among employee participation, individual differences, goal
difficulty, goal acceptance, goal instrumentality, and perfor-
mance. Personnel Psychology, 31, 305-323.
"Zedeck, S., & Baker, H. T. (1972). Nursing performance as
measured by behavioral expectation scales: A multitrait-
multirater analysis. Organizational Behavior and Human De-
cision Processes, 7, 457-466.
Received October 10, 1995
Revision received March 29, 1996
Accepted April 22, 1996 •