Validity, Reliability and

Generalizability
 Validity
HOW MIGHT YOU BE WRONG?
Internal Validity
External Validity
Construct Validity
 Validity
Are your findings really about what they
appear to be about or did something else
cause them?
 Internal Validity
Did your treatment actually cause the
outcomes?
Did anything else potentially cause the
outcome instead of your treatment?
 INTERNAL VALIDITY

Internal validity addresses the "true" causes of the

outcomes that you observed in your study. Strong
internal validity means that you not only have
reliable measures of your independent and
dependent variables BUT a strong justification that
you are able to rule out extraneous variables, or
alternative, often unanticipated, causes for your
findings. Thus strong internal validity refers to the
unambiguous assignment of causes to effects.
Internal validity is about causal control.
 EXTERNAL VALIDITY

External validity addresses the ability

to generalize your study to other
people and other situations. To have
strong external validity (ideally), you
need a probability sample of subjects
or respondents drawn using "chance
methods" from a clearly defined
population
 External Validity
When you have strong external validity,
you can generalize to other people and
situations with confidence.
– External validity refers to the population we
use and the setting. If I do a study with fifth
graders, then the results can be applied to a
similar set of fifth graders.
 Construct Validity
Does your test measure what you think it
really measures. What are you measuring
or observing?
Keep it simple and account for it.
 CONSTRUCT VALIDITY

Construct validity is about the correspondence

between your concepts (constructs) and the actual
measurements that you use. A measure with high
concept validity accurately reflects the abstract concept
that you are trying to study. Since we can only know
about our concepts through the concrete measures that
we use, you can see that construct validity is extremely
important. It also becomes clear why it is so important to
have very clear conceptual definitions of our variables.
 Construct Validity

Construct validity is often established

through the use of a multi-trait, multi-method
matrix. At least two constructs are measured.
Each construct is measured at least two different
ways, and the type of measures is repeated
across constructs. For example, each construct
first might be measured using a questionnaire,
then each construct would be measured using a
similar set of behavioral observation categories.
 RELIABILITY
In order to make any kind of causal
assessments in your research situation,
you must first have reliable measures,
i.e., stable and/or repeatable measures. If
the random error variation in your
measurements is so large that there is
almost no stability in your measures, you
can't explain anything!
 Reliability
Researcher Bias-How might you have
effected the study? Account for it!
 Reliability
Reliability is required to make statements about
validity. However, reliable measures could be
biased and hence "untrue" measures of a
phenomenon) or confounded with other factors
such as acquiescence response set. Picture a
scale that always weighs five pounds too light.
The results are reliable, but inaccurate or
biased. Or, picture an intelligence test on which
women or people of color always score lower
(even if this doesn't occur on other tests). Again,
the measure may be reliable but biased.
 Reliability
Identifying and controlling threats to
reliability and validity in qualitative
research is critical. Asking the question of
whether another researcher, going into the
field and asking the same interview
questions would get the same answers is
the overarching premise of reliability.
 Generalizability
Can you apply your findings across similar
populations? If so how and what
boundaries exist?
 Generalizability (Be Honest)
The study is generalizable to other school heads
from international schools in Asia. However, one
limit to this study was that the study will be
conducted with schools only in Asia. Therefore,
the findings will not apply universally. A second
limit to the study is that the entire study was
funded by the researcher, with limited funds for
air travel and miscellaneous expenses.
Therefore, schools were selected within a four
hour flight range of the researcher’s home town.
Telephone interviews were deemed less reliable
as they did allow for interpretation of body
language and visual clues.