Analyzing Complex Designs

Chapter 12

Factorial Designs
Consider multiple factors simultaneously
Main effects (individual effects of each factor) Interactions (combined or joint effects)

Problem with Multiple Tests

It is inappropriate to use t-tests to analyze

differences between condition means with more than two conditions

Each comparison includes risk of a Type I error

(e.g., 5% if H0 is true and a = .05) Risk of Type I error accumulates and is called the experimentwise alpha level.
Increasing the number of hypothesis tests

increases the total probability of a Type I error

Preventing Type I Error

Bonferroni adjustment divide the desired

alpha-level (.05) by the number of statistical tests that will be conducted. The overall likelihood of making a Type I error across all tests is .05.
However, by decreasing the alpha-level for each

particular test, the probability of making a Type II error (failing to detect real effects) increases.

Analysis of Variance (ANOVA)

ANOVA is used to analyze data from designs that

involve more than two conditions and, thus, more than two means. An ANOVA analyzes the differences between all condition means simultaneously. ANOVA holds the alpha level at .05 regardless of the number of means being tested (even with post hoc tests). Uses the F-testthe ratio of the variance among conditions (between groups) to the variance within conditions (within groups). A larger F-ratio means that the treatment effect is explaining more variance

Variance Components
TOTAL VARIANCE

BETWEEN-TREATMENTS VARIANCE:
Treatment effect (systematic differences caused by treatment condition) Error (random, unsystematic differences)
Individual differences Experimental (measurement) error

WITHIN-TREATMENTS VARIANCE:
Error (random, unsystematic differences)
Individual differences Experimental (measurement) error

(No systematic differences related to treatment groups can occur within each group)
(We call this the error term)

Test statistic for ANOVA (F-ratio)

The t-statistic focused on mean differences The F-ratio is based on the ratio of variance components

between tr eatments variance F within treatments variance

SO
treatment effect unsystematic error F unsystematic error

F-ratio calculation
ANOVA partitions the total sum of squares (i.e.,

the total variability) into its parts:

Sum of squares within-groups reflects error

variance Sum of squares between-groups is systematic variance that reflects differences between the experimental groups
Use SS and df to calculate MS (Mean Square

i.e., variance)

MS between F MS within

ANOVA for Factorial Designs

Sums of squares between-groups (SSbetween) can be broken down to test for different main effects and interactions. In a two-way factorial design (A X B), the total variance is composed of four parts
1.
2. 3. 4.

Main effect of A Main effect of B A x B interaction Error variance

Post-Hoc (Follow-Up) Tests

If an F-test is significant, we know that at least

one group mean differs from another. However, we do not know which of the means differ. Post hoc tests are needed to determine which means differ significantly.
Tukey HSD
Scheffe test . many more

Follow-up to an Interaction

If the F-test shows that an interaction is significant, tests of simple main effects are conducted. A simple main effect is an effect of one independent variable at a particular level of another independent variable. For a two-way Simple main effect of B at OR 1. Simple main effect of A at A1interaction of A X B: B1
Simple main effect of B at A2
2.

1. 2.

Simple main effect of A at B2

Test simple main effects of one factor at each level of the other

Multivariate Analysis of Variance (MANOVA)

MANOVA tests differences between the means of two or more conditions on two or more dependent variables MANOVA is used:

when the dependent variables are conceptually related to one another to avoid increasing Type I error by conducting multiple ANOVAs on several dependent variables

How MANOVA Works

MANOVA creates a canonical variable, a

weighted sum of the dependent variables A multivariate version of the F-test determines whether participants scores on the canonical variable differ among the experimental conditions If interested in the individual dependent variables, a significant multivariate F-test allows you to look at the individual ANOVAs.

Regression and ANOVA

You can conduct analyses similar to a t-test or

ANOVA using regression Create dummy coded variables (e.g.,. 0 or 1) for condition Create interaction terms for a factorial ANOVA Enter all predictors simultaneously