Reliability and Validity of

Research Instruments
An overview
Measurement error
Error variance--the extent of variability in
test scores that is attributable to error rather
than a true measure of behavior.

Observed Score=true score + error variance
(actual score obtained) (stable score) (chance/random error)
(systematic error)
Validity
The accuracy of the measure in reflecting
the concept it is supposed to measure.
Reliability
Stability and consistency of the measuring
instrument.
A measure can be reliable without being
valid, but it cannot be valid without being
reliable.

Instrument validity
The definition of instrument validity is the
extent to which an instrument measures
what it is supposed to. Validity is
established by correlating the scores with a
similar instrument. Also, expert review
establishes validity.

Validity
The extent to which, and how well, a
measure measures a concept.
face
content
construct
concurrent
predictive
criterion-related

Face validity
Just on its face the instrument appears to be
a good measure of the concept. intuitive,
arrived at through inspection
e.g. Concept=pain level
Measure=verbal rating scale rate your pain
from 1 to 10.
Face validity is sometimes considered a subtype
of content validity.
Question--is there any time when face validity is not desirable?
Content validity
is the extent to which a measuring
instrument provides adequate coverage of
the topic under study.
If the instrument contains a representative
sample of the universe, the content validity
is good.
It can also be determined by using a panel of
persons who shall judge how well the
measuring instrument meets the standards,
but there is no numerical way to express it.
A CVI (content validity index) of .80 or
more is desirable.
Construct validity
Sensitivity of the instrument to pick up
minor variations in the concept being
measured.
Can an instrument to measure anxiety pick up different levels
of anxiety or just its presence or absence? Measure two
groups known to differ on the construct.
Ways of arriving at construct validity
Hypothesis testing method
Convergent and divergent
Multitrait-multimatrix method
Contrasted groups approach
factor analysis approach
Concurrent validity
Correspondence of one measure of a
phenomenon with another of the same
construct.(administered at the same time)
Two tools are used to measure the same concept and
then a correlational analysis is performed. The
tool which is already demonstrated to be valid is
the gold standard with which the other measure
must correlate.
Concurrent validity- Does your attitude survey give
scores that agree with other things that go along with
attitude? For example, if someone scores low, indicating
that they ahve a negative attitude, are low attitude scores
concurrent with (happen at the same time as) negative
remarks from that person? High bolld pressure? If you
administer your attutude survey to someone who is
cheerful and smiling a lot, but they rate low, indicating a
negative attitude, your survey may not have concurrent
validity.
Predictive validity
The ability of one measure to predict
another future measure of the same concept.
If IQ predicts SAT, and SAT predicts QPA, then shouldnt IQ predict QPA (we
could skip SATs for admission decisions)
If scores on a parenthood readiness scale indicate levels of integrity, trust,
intimacy and identity couldnt this test be used to predict successful
achievement of the devleopmental tasks of adulthood?

The researcher is usually looking for a more efficient way to measure a
concept.
Predictive validity- Can your attitude survey
predict? For example, if someone scores high,
indicating that they have a positive attitude, can
high attitude scores also be predictive of job
promotion?

Criterion related validity
The ability of a measure to measure a criterion
(usually set by the researcher).
If the criterion set for professionalism is nursing is
belonging to nursing organizations and reading
nursing journals, then couldnt we just count
memberships and subscriptions to come up with a
professionalism score.
Can you think of a simple criterion to measure
leadership?
Concurrent and predictive validity are often listed as
forms of criterion related validity.
Reliability
Reliability has to do with accuracy and
precision of a measurement procedure.
Homogeneity, equivalence and stability of a
measure over time and subjects. The
instrument yields the same results over
repeated measures and subjects.
Expressed as a correlation coefficient (degree of agreement
between times and subjects) 0 to +1.
Reliability coefficient expresses the relationship between
error variance, true variance and the observed score.
The higher the reliability coefficient, the lower the error
variance. Hence, the higher the coefficient the more
reliable the tool! .70 or higher acceptable.
Reliability can be improved in the following two ways:-
a) By standardizing the conditions under which the
measurement takes place i.e. we must ensure that external
sources of variation such as boredom, fatigue etc. are
minimized to the extent possible. That will improve
stability aspect.
b) By carefully designed directions for measurement
with no variation from group to group, by using trained
and motivated persons to conduct the research and also by
broadening the sample of items used. This will improve
equivalence aspect.


Stability
The same results are obtained over repeated
administration of the instrument.
Test-restest reliability
parallel, equivalent or alternate forms