Generalizability & Field Research

Readings on Sakai: Baumeister, Vohs, & Funder, 2007; Mitchell, 2012

External Validity and Generalizability

External validity: The degree to which results are likely to generalize to other samples, situations, etc. (generalizability) Can we generalize findings from data collected in the lab to everyday life? Establishing external validity or generalizability requires collecting data outside of the lab (e.g., field research)

Is External Validity Always Essential?

External validity is not always essential for establishing that associations exist Basic vs. applied science Theory testing mode vs. Generalizing mode

Generalizing results across participants

Most participants in top psychology journals are WEIRD

WEIRD: Western, Educated, Industrialized, Rich, and Democratic The weirdest people in the world? (Henrich, Heine, & Norenzayan, 2010)

80% of participants in top psychology journals and WEIRD undergraduate students 96% of psychology samples come from countries that make up only 12% of the worlds population Can we assume that results from WEIRD allow us to draw conclusions about human nature?

Generalizing results across cultures

What psychological phenomena might be universal?

Emotional experiences? Self-enhancement?

We should not assume that a specific psychological phenomenon is universal Cross-cultural research is essential for establishing universality Challenges in cross-cultural research:

Recruitment Methods and measures designed in WEIRD populations may not be equally valid or sensible in other cultures

Generalizing across settings

Does behaviour observed in the lab generalize to real world behaviour? Ecological Validity - the extent to which laboratory paradigms are similar to natural experiences

Ecological validity is not absolutely essential for a study to have high external validity but it might help

Mundane realism vs. Experimental realism We might also collect data in naturalistic settings (outside of the lab)

Field Research and Naturalistic Methods

Naturalistic Methods

Broadly, naturalistic methods measure/describe behavior in the context of natural, daily life Maximizes external validity

Examining behavior where it naturally occurs But, minimal control and internal validity Observational research Experience sampling methods

Examples:

Field Experiments

Research conducted in a naturalistic setting (outside of the lab)

Very high external validity because data are collected in the context of real life However, internal validity can be lower due to a lack of control over extraneous variables

Because of challenges to internal validity, field experiments are often paired with a lab study

Psychology as the Science of Self-Reports and Finger Movements: What happened to actual behavior?
(Baumeister, Vohs, & Funder, 2007)

Psychology is the science of behaviour, yet psychologists rarely measure behaviour directly Many domains in psychology rely on:

Self-report questionnaires Performance on cognitive tasks (e.g., implicit measures) 11 articles with 38 studies The only measure of behaviour involved asking participants to select one of two stimulus persons

Example: January, 2006 issue of JPSP:

Classic studies in social-personality psychology included behavioural observation

Bystander intervention (Darley & Latan, 1968) Milgrams obediance studies Cognitive dissonance manipulations (Festinger & Carlsmith, 1959)

Does it matter if we measure behaviour?

Introspection can be in accurate (Nisbett & Wilson, 1977, Telling more than we can know) Self-reported hypothetical behaviour often differs from actual behaviour! (e.g., West & Brown, 1975)

Would you donate money to a passerby asking for money for medical care? Actual donations were much lower than people predicted

Affective forecasting (e.g., Wilson & Gilbert, 2003) Hypothetical vs. live interaction (Eastwick, Hunt, & Neff, 2013) Personality and behaviour But, behavioural observation can be difficult!

Example: Studying familiarity-attraction with different paradigms

Norton, Frost & Ariely (2007) used a trait evaluation paradigm to study the effect of familiarity on attraction

Participants receive a list of 4, 6, 8, or 10 randomly selected trait adjectives describing a hypothetical person How much do you think you would like this person? Number of traits & liking, r(76) = -.23, p = .05 Live interaction does not involve sharing randomly selected traits! People seek out commonalities Interaction might be responsive and enjoyable

Is this paradigm ecologically or externally valid?

Example: Studying familiarity-attraction with different paradigms (cont.)

Reis, Maniaci, Caprariello, Eastwick, & Finkel (2011) used a live interaction paradigm to see if this effect changed with a more ecologically valid design Study 2: A sample of 220 participants were paired with a stranger and chatted either 1, 2, 4, 6, or 8 times via instant messaging over the same number of days
4
3.8 Attraction 3.6 3.4 F-linear(1,101) = 5.15, p<.03

3.2
3 2.8 1 2 4 6 Number of Chats 8

Example: Studying familiarity-attraction with different paradigms (cont.)

40 30

Percentage of dyads who 20 contacted each other after the study

10
0 1 2 4 6 8 Number of Chats
2 (1) = 2.83, p=.05

Meta-Analysis and Convergence Between Lab and Field Results

Aggregating Data: Meta-Analysis

meta-analysis a procedure used to examine every study that has been conducted on a particular topic to assess the relationship between whatever variables are the focus of the analysis.

By looking at effect sizes across many studies, a general estimate of the strength of the relationship between the variables can be calculated

Social and Medical Research relies on MetaAnalysis more than Physics, Chemistry, Etc.

Older sciences have a larger base of existing research Social and medical sciences are often less integrative In psychology specifically:

People are variable! We cannot control for all sources of error variance Research with human subjects must follow ethical guidelines In predicting human behaviour, many effects are small and many studies tend to be underpowered

Steps to Conducting a Meta-Analysis

1.

2.

Identify your research question Collect studies

inclusion criteriameasures, methods, controls, etc.) Multiple databases Identifying nonsignificant and unpublished results

3.

Extract data from studies

Effect size Code characteristics of samples

4.
5.

Analyze data; Examine aggregated effects Interpret effects across studies

Does behaviour observed in the lab generalize to real world behaviour?

Mitchell (2012) combined results from 82 metaanalyses, with 217 comparisons of effects in lab vs. field studies Effect sizes in the lab vs. field are highly correlated (r = 64; r = .71 excluding 1 outlier) suggesting that lab studies in psychology tend to be externally valid!

Does behaviour observed in the lab generalize to real world behaviour? (cont.)

The aggregate results support external validity but they also hide considerable variability:

30 of 215 effects (14%) from lab studies changed direction when examined in the field! Lab and field studies are more consistent for industrial/organizational psychology (r = .89) than for social psychology (r = .53, with 26% changing direction!)

Homework Assignment (for discussion in next weeks class)

Pretend that you MUST obtain a statistically significant result in your group project at any cost. Try to change your analyses to find a statistically significant result. For instance, you could:

Exclude participants for any reason Add control variables Change scores that look unusual

The significant result does not need to be relevant to your hypotheses Could you write a paper that makes sense of this significant result?