Validity

Reliability
Control

Establishing Cause and Effect

Temporal Precedence

Covariation of Cause and Effect

Cause has to happen before the effect

Show there is a relationship between X and Y

No Plausible Alternative Explanations

Any other cause that can bring about the effect

Establishing Cause and Effect

Validity and Reliability

Measure-Related Concerns

Construct Validity

Are you measuring what you think youre measuring?

Reliability

Is your measure consistent in its evaluation of the

same individuals?

Types of Reliability
Inter-rater reliability

Assesses the degree to which different

raters/observers give consistent
estimates of the same phenomenon

% agreement

Correlation between observers scores

5

Types of Reliability
Test-retest

Correlation between two observations (i.e.,

set of scores) on the same test administered
to the same (or similar) sample on two
different occasions

Assumes no change in construct being

measured

Time between observations is crucial

6

Types of Reliability
Parallel-forms

Correlation between two observations (i.e.,

set of scores) on parallel forms of a test
administered to the same sample

Requires generation of many items that

measure the same construct

Two forms of test can be used

independently

Assumes randomly divided halves are

equivalent

Types of Reliability
Internal consistency

Single test administered to a sample on one

occasion

Assesses the consistency of the results for

different items for the same construct within
the measure

Average inter-item correlation

Average item-total correlation

Split-half reliability

Cronbachs Alpha

Relationship between reliability and validity

Validity and Reliability

Method-Related Concerns

Internal Validity

Is the independent variable the only possible

explanation of the results shown?

External Validity

How far can the study generalize?

10

Validity

Internal Validity

Is the independent variable the only possible

explanation of the results shown?
Protecting internal validity allows us to eliminate
potential alternative explanations for the
outcome of an experiment

Extraneous variables

Influence the DV without your intent or

consent!
11

Experimental Control

Extraneous Variables

Nuisance variables

Make the effects of the IV more difficult to

determine by affecting variation within groups

Participant characteristics
Unintended influences of experimental situation

12

Experimental Control

Extraneous Variables

Confounding variables
Vary systematically with the IV
Influence the difference between groups
The differences observed between groups could be
attributed to the IV or the confound

13

Experimental Control

Extraneous Variables

Experimenters

Physiological differences

Participants
Demand characteristics
Good participant effect
Response biases

14

Internal Validity

Exercise control to protect internal validity

before the experiment is conducted

Random Assignment
Elimination
Constancy
Counterbalancing
Single- and Double-blind Experiments
Altering Response Sets
15

Basic Control Techniques

Randomization

Individuals have an equal chance of being

assigned to any group in the experiment

Allows us to assume groups are roughly equivalent

before administering the IV

16

Basic Control Techniques

Elimination

Extraneous variables are completely removed

from the experiment

17

Basic Control Techniques

Constancy

An extraneous variable is reduced to a single

value experienced by all participants
Standard control technique
Hold environment, temperature, lighting constant

18

Basic Control Techniques

Balancing and Counterbalancing

Balancing

Achieves group equality by distributing extraneous

variables to all groups

Counterbalancing

Controls order effects by presenting treatments in a

different sequence

Order or sequence effects

Carryover effects
Differential carryover
19

Basic Control Techniques

Within-Subject Counterbalancing
Within-Group Counterbalancing

Each treatment must be presented to each participant

an equal number of times
Each treatment must occur an equal number of times at
each testing session
Each treatment must precede and follow each of the
other treatments an equal number of times

20

The Cola Wars!

Participant
Participant
Participant
Participant
Participant
Participant

1
2
3
4
5
6

P
P
P
C
C
C

C
C
C
P
P
P
21

The Cola Wars!

Participant
Participant
Participant
Participant
Participant
Participant

1
2
3
4
5
6

P
P
C
C
DP
DP

C
DP
P
DP
P
C

DP
C
DP
P
C
P
22

Basic Control Techniques

Single- and Double-Blind Experiments

The experimenter (single-blind) or both the

experimenter and participants (double-blind)
are unaware of the participants condition

23

Basic Control Techniques

Altering response sets

Rewrite survey items so that some negative

responses indicate agreement and some
positive responses indicate disagreement
Check for socially desirable responses in your
survey and/or potentially undesirable cues

24

Internal Validity
Exercise control to protect internal validity
before the experiment is conducted
Assess the internal validity of the
experiment afterward

25

Threats to Internal Validity

History

Significant events occur between measurements

26

Threats to Internal Validity

Maturation

Participants change over time in the experiment

27

Threats to Internal Validity

Testing (Pretest Sensitization)

Repeated tests cause changes in DV;

practice effects

28

Threats to Internal Validity

Instrumentation

Changes in measurement due to changes in the

measurement device (human or machine)

29

Threats to Internal Validity

Statistical regression

Extreme scores regress toward the mean

Difficult to maintain extreme scores over
repeated measures

30

Threats to Internal Validity

Selection

Selecting participants into groups that were not

equivalent before the experiment

31

Threats to Internal Validity

Attrition or Mortality

Experimental dropouts

32

Threats to Internal Validity

Diffusion of treatment

Participants in one group communicate

information from the IV to the other group

33

Threats to Internal Validity

Interactions with Selection

Maturation
History
Instrumentation

34

How Important is Internal Validity?

It is the most important property of any

experiment

An experiment without internal validity cannot

have external validity
With no internal validity, you can have
no confidence in your results

35

External Validity
How far can you generalize your results
beyond your experiment?
3 types:

Population
Environmental
Temporal

36

Threats to External Validity

Testing/Treatment Interaction

Pretest may make reaction to the IV different

from those not tested

Selection/Treatment Interaction

Effect is found only for a specific group of

participants

37

Threats to External Validity

Reactive Arrangements

A la Hawthorne effects
Demand characteristics

Participant Characteristics

Specific animal species

Experimental participant populations:
College students
White male Americans

38

39

Is External Validity Necessary?

May want to find out if something can happen

We may be predicting from the real world to the
lab
If we can show something happens in the labs
unnatural setting, we may have more confidence
in the phenomenon
We may study something without a real-world
analogy

40