Lab 4: Alpha and Standard Error of Measurement

Reliability

Reliability refers to consistency Types of reliability estimates

Test-retest reliability Interobserver (or interrater) reliability Internal consistency

Alpha measures the consistency of responses across items

Calculating Alpha of the Extraversion Scale by Hand

k: number of items Extraversion scale: k = 10 rij: average inter-item correlation Extraversion scale: rij = .46

How large would alpha be if the scale had 16 items?

Calculating Alpha of the Extraversion Scale for the Time 2 IPIP

SPSS: Analyze Scale Reliability Analysis Put the following items in the items box:

e1_t2, e2_r_t2, e3_t2, e4_r_t2, e5_t2, e6_r_t2, e7_t2, e8_r_t2, e9_t2, & e10_r_t2 Make sure you use the reverse scored items!

Click on Statistics

Inter-Item: Correlations Summaries: Correlations Then hit Continue

Then click OK

Questions to Answer from Output

1. What was the average inter-item correlation? 2. How many items are on the scale? 3. What was the standardized alpha?

Calculating Alphas for the other four scales (A, C, N, and O)

Agreeableness: a1_r_t2, a2_t2, a3_r_t2, a4_t2, a5_r_t2, a6_t2, a7_r_t2, a8_t2, a9_t2, & a10_t2 Conscientiousness: c1_t2, c2_r_t2, c3_t2, c4_r_t2, c5_t2, c6_r_t2, c7_t2, c8_r_t2, c9_t2, & c10_t2 Neuroticism: n1_t2, n2_r_t2, n3_t2, n4_r_t2, n5_t2, n6_t2, n7_t2, n8_t2, n9_t2, & n10_t2 Openness: o1_t2, o2_r_t2, o3_t2, o4_r_t2, o5_t2, o6_r_t2, o7_t2, o8_t2, o9_t2, & o10_t2

What happens if you forget to reverse score the items?

Calculate alpha for the Extraversion scale again, but now use the following items:

e1_t2, e2_t2, e3_t2, e4_t2, e5_t2, e6_t2, e7_t2, e8_t2, e9_t2, & e10_t2

What is the new alpha? How can you tell that you did something wrong?

What happens if you calculate alpha of two scales at the same time?

Calculate alpha for the following eight items:

e1_t2, e2_r_t2, e3_t2, e5_t2 (Extraversion items) c1_t2, c3_t2, c5_t2, c7_t2 (Conscientiousness items)

Can you interpret your result? What might clue you into your mistake? Remember: Alpha does not indicate unidimensionality!

Standard Error of Measurement

Standard deviation of the error around any individuals true score If a student were to take the same test repeatedly, with no change in his level of knowledge and preparation, it is possible that some of the resulting scores would be slightly higher or slightly lower than the score that precisely reflects the students actual level of knowledge and ability. The difference between a students actual score and his highest or lowest hypothetical score is known as the standard error of measurement.

Standard Error of Measurement

False Negative

A small percentage of students may score lower than anticipated on a test

False Positive

A small percentage of students may score higher than otherwise would have been expected

Standard Error of Measurement

Using a reliability coefficient and the tests standard deviation, we can calculate this value:

SEM sx 1 rxx

Where:

sx = the standard deviation for the test rxx = the reliability coefficient (alpha) for the test

Standard Error of Measurement

Standard Error of Measurement for Agreeableness Scale To obtain rxx: Repeat steps from earlier in Lab

Analyze Scale Reliability Analysis Enter all Agreeableness items for Time 2
Inter-Item: Correlations Summaries: Correlations Then hit Continue

Click on Statistics

SEM sx 1 rxx

Then click OK

Standard Error of Measurement

To obtain standard deviation:

Analyze Descriptive Descriptives Enter a_t2 (non-standardized) Check standard deviation in options Click Ok

Hence:

SEM= .55 1-.86 = .20

SEM Confidence Interval

96% Confidence:

Z = 2.0

68% Confidence:

Z = 1.0
CI Observed Score Z Confidence SEM

96% Band = Score (2 *SEM)

CI = 4 (2 * .20) = 3.6 - 4.4

68% Band = Score (1 *SEM)

CI = 4 (.20) = 3.8 4.2

Homework Questions 1 & 2

1. Report the standardized alphas for all five IPIP scales at Time 2. Also report the average inter-item correlation and number of items. (2 point) 2. Imagine that the average inter-item correlation for a scale is .40. Compute the standardized alphas for tests with the following numbers of items: 5, 10, 15, 20, 25. (3 points)

Homework Questions 3 & 4

3. Report the SEM for all 5 IPIP scales at Time 2 with interpretation. (3 points) 4. Report the confidence interval for the mean conscientiousness at time 2 at the 96% CI with interpretation. (2pts.)