ANOVA

Analysis of Variance

Ankitha DK
10SJCCMIB006
 What is ANOVA?

 it is the analysis of variance which

enables us to test for the significance of
the differences among more than two
samples.
 Where is ANOVA tests used?

 Comparing the output/efficiency among different

brands/situations.
 Combination of factors which result in an output.
 Why is ANOVA used?

 To avoid Type I error

 What is type I error?

 the error of rejecting a null hypothesis when it is

actually true
 Assumptions in ANOVA tests
 Normally distributed outcome
 Equal variances between the groups
 Groups are independent

 each group is approximately normal

 check this by looking at histograms and/or normal quartile
plots, or use assumptions
 can handle some non normality, but not severe outliers
 standard deviations of each group are approximately equal
 rule of thumb: ratio of largest to smallest sample St. dev.
must be less than 2:1
 Checks in ANOVA
NORMALITY CHECK
 We should check for normality using:
• assumptions about population
• histograms for each group
• normal quartile plot for each group

 With such small data sets, there really isn’t a really

good way to check normality from data, but we make
the common assumption that physical measurements
of people tend to be normally distributed.
 Standard Deviation Check

Variable treatment N Mean Median St.Dev

days A 8 7.250 7.000 1.669
B 8 8.875 9.000 1.458
P 9 10.111 10.000 1.764

Compare largest and smallest standard deviations:

• largest: 1.764
• smallest: 1.458
• 1.458 x 2 = 2.916 > 1.764

Note: variance ratio of 4:1 is equivalent.

 Notation for ANOVA
• n = number of individuals all together
• I = number of groups
x
• = mean for entire data set is

Group i has
• ni = # of individuals in group i
• xij = value for individual j in group i
• x=i mean for group i
• si = standard deviation for group i
 How ANOVA works (outline)
ANOVA measures two sources of variation in the data and
compares their relative sizes

• variation BETWEEN groups

• for each data value look at the difference between its
group mean and the overall mean

 xi  x  2

• variation WITHIN groups

• for each data value we look at the difference between
that value and the mean of its group

x ij  xi 
2
 The ANOVA F-statistic is a ratio of the Between Group
Variaton divided by the Within Group Variation:

Between MSG
F 
Within MSE
A large F is evidence against H0, since it indicates that there is
more difference between groups than within groups.
One Way Analysis of Variance

 The analysis of variance is a procedure that tests to

determine whether differences exits between two or
more population means.

 To do this, the technique analyzes the sample

VARIANCES.
One Way Analysis of Variance

 Example
An apple juice manufacturer is planning to develop a new
product -a liquid concentrate.
 The marketing manager has to decide how to market the new
product.
 Three strategies are considered
 Emphasize convenience of using the product.
 Emphasize the quality of the product.
 Emphasize the product’s low price.
One Way Analysis of Variance

 Example continued
 An experiment was conducted as follows:
 In three cities an advertisement campaign was launched .
 In each city only one of the three characteristics (convenience,
quality, and price) was emphasized.
 The weekly sales were recorded for twenty weeks following
the beginning of the campaigns.
One Way Analysis of Variance
Convnce Quality Price
529 804 672
658 630 531
793 774 443
514 717 596
663 679 602
719 604 502
711 620 659
606 697 689
461 706 675
529 615 512
498 492 691
663 719 733
604 787 698
Weekl 495 699 776
485 572 561
y 557 523 572
sales 353 584 469
557 634 581
542 580 679
614 624 532
One Way Analysis of Variance

 Solution
 The data are interval( ie weekly! )
 The problem objective is to compare sales in three cities.
 We hypothesize that the three population means are equal
Defining the Hypotheses

• Solution
H0: m1 = m2= m3
H1: At least two means differ

To build the statistic needed to test the

hypotheses use the following notation:
 Terminology

 In the context of this problem…

Response variable – weekly sales
Responses – actual sale values
Experimental unit – weeks in the three cities when we
record sales figures.
Factor – the criterion by which we classify the populations
(the treatments). In this problems the factor is the
marketing strategy.
Factor levels – the population (treatment) names. In this
problem factor levels are the marketing strategies.
The rationale of the test statistic
Two types of variability are employed
when testing for the equality of the
population means
Graphical demonstration:
Employing two types of
variability
 30

25
x 3  20
x 3  20
20 20
19
x 2  15
16 x 2  15
15
14
x1  10 12
11 x1  10
10 10
9 9

A small variability within The1 sample means are the same as before,
the samples makes it easier but the larger within-sample variability
TreatmentTreatment
1 Treatment
2 3 Treatment 1 Treatment 2Treatment 3
to draw a conclusion about the makes it harder to draw a conclusion
population means. about the population means.
The rationale behind the test statistic – I

 If the null hypothesis is true, we would expect all the

sample means to be close to one another (and as a
result, close to the grand mean).
 If the alternative hypothesis is true, at least some of
the sample means would differ.
 Thus, we measure variability between sample means.
 Variability between sample means

• The variability between the sample means

is measured as the sum of squared
distances between each mean and the
grand mean.

This sum is called the

Sum of Squares for Treatments
SST
In our example treatments are
represented by the different
advertising strategies.
Sum of squares for treatments (SST)

k
SST   n j ( x j  x) 2

j 1

There are k treatments

The size of sample j The mean of sample j

Note: When the sample means are close to

one another, their distance from the grand
mean is small, leading to a small SST. Thus,
large SST indicates large variation between
sample means, which supports H1.
 Sum of squares for treatments (SST)

 Solution – continued
Calculate SST

x1  577.55 x 2  653.00 x 3  608 .65

k
SST   n j ( x j  x) 2
j 1
The grand mean is calculated by = 20(577.55 - 613.07)2 +
n1 x1  n2 x 2  ...  nk x k + 20(653.00 - 613.07)2 +
X + 20(608.65 - 613.07)2 =
n1  n2  ...  nk
= 57,512.23
Sum of squares for treatments (SST)

Is SST = 57,512.23 large enough

to reject H0 in favor of H1?
The rationale behind test statistic – II

 Large variability within the samples weakens the

“ability” of the sample means to represent their
corresponding population means.
 Therefore, even though sample means may markedly
differ from one another, SST must be judged relative to
the “within samples variability”.
 Within samples variability

 The variability within samples is measured by adding all

the squared distances between observations and their
sample means.

This sum is called the

Sum of Squares for Error
SSE

In our example this is the

sum of all squared differences
between sales in city j and the
sample mean of city j (over all
the three cities).
Sum of squares for errors (SSE)
 Solution – continued
Calculate SSE

s12  10,775.00 s 22  7,238,11 s32  8,670.24

k nj
SSE  
j 1 i 1
(xij  x j )=2 (n1 - 1)s12 + (n2 -1)s22 + (n3 -1)s32

= (20 -1)10,774.44 + (20 -1)7,238.61+ (20-

1)8,670.24
= 506,983.50
Sum of squares for errors (SSE)

Is SST = 57,512.23 large enough

relative to SSE = 506,983.50 to
reject the null hypothesis that
specifies that all the means are
equal?
The mean sum of squares

To perform the test we need to

calculate the mean squares as
follows:
Calculation of MST - Calculation of MSE
Mean Square for Mean Square for Error
Treatments
SST SSE
MST  MSE 
k 1 nk
57,512.23 509,983.50
 
3 1 60  3
 28,756.12  8,894.45
 Calculation of the test statistic

MST
F
MSE
28,756.12

8,894.45
Required Conditions:  3.23
1. The populations tested
are normally distributed.
2. The variances of all the with the following degrees of freedom:
populations tested are v1=k -1 and v2=n-k
equal.
The F test rejection region

And finally the hypothesis test:

H0: m1 = m2 = …=mk
H1: At least two means differ

Test statistic:
MST
F
R.R: F>Fa,k-1,n-k
MSE
The F test
MST
F
MSE
28,756.12
Ho: m1 = m2= m3 
8,894.17
H1: At least two means differ
 3.23
Test statistic F= MST/ MSE= 3.23

R.R. : F  Fk 1nk  F0.05,31,60 3  3.15

Since 3.23 > 3.15, there is sufficient evidence
to reject Ho in favor of H1, and argue that at
least one
of the mean sales is different than the others.
The F test p- value

 fx Statistical FDIST(3.23,2,57) = .0467

0.1
0.08
p Value = P(F>3.23) = .0467
0.06
0.04
0.02
0
-0.02 0 1 2 3 4
 Excel single factor ANOVA
Xm15-01.xls

Anova: Single Factor

SUMMARY
Groups Count Sum Average Variance
Convenience 20 11551 577.55 10775.00
Quality 20 13060 653.00 7238.11
Price 20 12173 608.65 8670.24

ANOVA
Source of Variation SS df MS F P-value F crit
Between Groups 57512 2 28756 3.23 0.0468 3.16
Within Groups 506984 57 8894

Total 564496 59

SS(Total) = SST + SSE

 So How big is F?
Since F is
Mean Square Between / Mean Square Within

= MSG / MSE

A large value of F indicates relatively more

difference between groups than within groups
(evidence against H0)

To get the P-value, we compare to F(I-1,n-I)-distribution

• I-1 degrees of freedom in numerator (# groups -1)
• n - I degrees of freedom in denominator (rest of df)
 Connections between SST, MST, and standard
deviation
If ignore the groups for a moment and just compute the standard deviation of
the entire data set, we see

s 2

  x
ij x  2


SST
 MST
n 1 DFT
So SST = (n -1) s2, and MST = s2. That is, SST and MST measure the TOTAL
variation in the data set.
 Connections between SSE, MSE, and standard
deviation

Remember:
si
2

  x ij  xi 
2


SS[ Within Group i ]
ni  1 df i
So SS[Within Group i] = (si2) (dfi )

This means that we can compute SSE from the standard deviations and sizes (df)
of each group:

SSE  SS[Within ]   SS[Within Group i ]

  s (ni  1)   s (df i )
2
i
2
i
 Pooled estimate for st. dev
One of the ANOVA assumptions is that all groups have the same standard
deviation. We can estimate this with a weighted average:

( n  1) s 2
 ( n  1) s 2
 ...  ( n  1) s 2
s 2p  1 1 2 2 I I

nI
( df ) s 2
 ( df ) s 2
 ...  ( df ) s 2
s 2p  1 1 2 2 I I
df1  df 2  ...  df I
so MSE is the pooled
SSE estimate of variance
s 
2
p  MSE
DFE
 In Summary
SST   ( xij  x)  s ( DFT )
2 2

obs

SSE   ( xij  xi )  s
2 2
i (df i )
obs groups

SSG   ( xi  x) 
2
 n ( x  x)
i i
2

obs groups

SS MSG
SSE SSG  SST ; MS  ; F
DF MSE
 R2 Statistic
R2 gives the percent of variance due to between
group variation

SS [ Between] SSG
R 2

SS [Total ] SST

We will see R2 again when we study regression.

 Multiple Comparisons
Once ANOVA indicates that the groups do not all
have the same means, we can compare them two
by two using the 2-sample t test

• We need to adjust our p-value threshold because we are

doing multiple tests with the same data.

• There are several methods for doing this.

• If we really just want to test the difference between one

pair of treatments, we should set the study up that way.
 Tuckey’s Pairwise Comparisons
Tukey's pairwise comparisons
95% confidence
Family error rate = 0.0500
Individual error rate = 0.0199
Use alpha = 0.0199 for
Critical value = 3.55
each test.
Intervals for (column level mean) - (row level mean)

A B
These give 98.01%
B -3.685
0.435
CI’s for each pairwise
difference.
P -4.863 -3.238
-0.859 0.766 Only P vs A is significant
(both values have same sign)
98% CI for A-P is (-0.86,-4.86)
 Tukey’s Method in R
Tukey multiple comparisons of means
95% family-wise confidence level

diff lwr upr

B-A 1.6250 -0.43650 3.6865
P-A 2.8611 0.85769 4.8645
P-B 1.2361 -0.76731 3.2395