Types of Reliability

Reliability of Consistency of What?

Observers or raters
Tests over time
Different versions of the same test
A test at one point in time

Inter-Rater or Inter-Observer Reliability

Object or
phenomenon

Inter-Rater or Inter-Observer Reliability

Object or
phenomenon

Observer 1

Inter-Rater or Inter-Observer Reliability

Object or
phenomenon

Observer 1

Observer 2

Inter-Rater or Inter-Observer Reliability

Object or
phenomenon

?
=
Observer 1

Observer 2

Inter-Rater or Inter-Observer Reliability

Are different observers consistent?

Can establish this outside of your study
in a pilot study.
Can look at percent of agreement
(especially with category ratings).
Can use correlation (with continuous
ratings).

Test-Retest Reliability

Time 1

Time 2

Test-Retest Reliability

Test

Time 1

Test

Time 2

Test-Retest Reliability

Stability over time

Test

Time 1

Test

Time 2

Test-Retest Reliability

Measure instrument at two times for

multiple persons.
Compute correlation between the two
measures.
Assumes there is no change in the
underlying trait between time 1 and time
2.

Parallel-Forms Reliability

Time 1

Time 2

Parallel-Forms Reliability

Form A

Form B

Time 1

Time 2

Parallel-Forms Reliability

Stability across forms

Form A

Form B

Time 1

Time 2

Parallel-Forms Reliability

Administer both forms to the same

people.
Get correlation between the two forms.
Usually done in educational contexts
where you need alternative forms
because of the frequency of retesting
and where you can sample from lots of
equivalent questions.

Internal Consistency Reliability

Average inter-item correlation

Internal Consistency Reliability

Average Inter-Item Correlation

Test

Internal Consistency Reliability

Item 1
Item 2

Item 3
Test
Item 4

Item 5
Item 6

Average Inter-Item correlation

Internal Consistency Reliability

Item 1

Average inter-item correlation

Item 2

Item 3
Test
Item 4

Item 5
Item 6

I1
I1
I2
I3
I4
I5
I6

1.00
.89
.91
.88
.84
.88

I2

I3

I4

I5

I6

1.00
.92
.93
.86
.91

1.00
.95
.92
.95

1.00
.85
.87

1.00
.85

1.00

Internal Consistency Reliability

Item 1

Average inter-item correlation

Item 2

Item 3
Test
Item 4

Item 5

I1
I1
I2
I3
I4
I5
I6

1.00
.89
.91
.88
.84
.88

I2

I3

I4

I5

I6

1.00
.92
.93
.86
.91

1.00
.95
.92
.95

1.00
.85
.87

1.00
.85

1.00

Item 6

.90

Internal Consistency
Reliability
Average item-total correlation

Internal Consistency
Reliability
Average item-total correlation

Test

Internal Consistency
Reliability
Average item-total correlation
Item 1

Item 2
Item 3
Test
Item 4
Item 5
Item 6

Internal Consistency
Reliability
Average item-total correlation
Item 1

Item 2
Item 3
Test
Item 4
Item 5
Item 6

I1
I1 1.00
I2 .89
I3 .91
I4 .88
I5 .84
I6 .88
Total .84

I2

I3

I4

I5

I6

1.00
.92
.93
.86
.91
.88

1.00
.95
.92
.95
.86

1.00
.85
.87
.87

1.00
.85
.83

1.00
.82

1.00

Internal Consistency
Reliability
Average item-total correlation
Item 1

Item 2
Item 3
Test
Item 4

I1
I1 1.00
I2 .89
I3 .91
I4 .88
I5 .84
I6 .88
Total .84

I2

I3

I4

I5

I6

1.00
.92
.93
.86
.91
.88

1.00
.95
.92
.95
.86

1.00
.85
.87
.87

1.00
.85
.83

1.00
.82

Item 5
Item 6

.85

1.00

Internal Consistency
Reliability
Split-half correlations

Internal Consistency
Reliability
Split-half correlations

Test

Internal Consistency
Reliability
Split-half correlations
Item 1

Item 2
Item 3
Test
Item 4
Item 5
Item 6

Internal Consistency
Reliability
Split-half correlations
Item 1
Item 2
Item 3
Test
Item 4
Item 5
Item 6

Item 1

Item 3

Item 4

Internal Consistency
Reliability
Split-half correlations
Item 1
Item 2

Item 1

Item 3

Item 4

Item 2

Item 5

Item 6

Item 3
Test
Item 4
Item 5
Item 6

Internal Consistency
Reliability
Split-half correlations
Item 1
Item 2

Item 1

Item 3

Item 4

Item 3
Test

.87

Item 4
Item 5
Item 6

Item 2

Item 5

Item 6

Internal Consistency
Reliability
Cronbachs alpha ()

Internal Consistency
Reliability
Cronbachs alpha ()

Test

Internal Consistency
Reliability
Cronbachs alpha ()
Item 1

Item 2
Item 3
Test
Item 4
Item 5
Item 6

Internal Consistency
Reliability
Cronbachs alpha ()
Item 1
item 1

item 2

Test
Item 4
Item 5
Item 6

item 4

item 1

.87

Item 2
Item 3

item 3

item 5

item 3

item 4

item 1

.87
item 6

item 2

item 5

item 3

item 4

.87
item 6

item 2

item 5

item 6

Internal Consistency
Reliability
Cronbachs alpha ()
Item 1
item 1

item 2

Test
Item 4
Item 5
Item 6

item 4

item 1

.87

Item 2
Item 3

item 3

SH1
SH2
SH3
SH4
SH5
...
SHn

item 5

item 3

item 4

item 1

.87
item 6

.87
.85
.91
.83
.86
.85

item 2

item 5

item 3

item 4

.87
item 6

item 2

item 5

item 6

Internal Consistency
Reliability
Cronbachs alpha ()
Item 1
item 1

item 3

item 2

Test
Item 4
Item 5
Item 6

item 1

.87

Item 2
Item 3

item 4

item 5

SH1
SH2
SH3
SH4
SH5
...
SHn

item 3

item 4

item 1

.87
item 6

.87
.85
.91
.83
.86
.85

= .85

item 2

item 5

item 3

item 4

.87
item 6

item 2

item 5

item 6

Internal Consistency
Reliability
Cronbachs alpha ()
Item 1
item 1

item 3

item 2

Test
Item 4
Item 5
Item 6

item 1

.87

Item 2
Item 3

item 4

item 5

SH1
SH2
SH3
SH4
SH5
...
SHn

item 3

item 4

item 1

.87
item 6

.87
.85
.91
.83
.86
.85

= .85

item 2

item 5

item 3

item 4

.87
item 6

item 2

item 5

item 6

Like the average

of all possible
split-half
correlations

Internal Consistency Reliability - Summary

Average inter-item correlation

Average item-total correlation
Split-half reliability
Cronbachs alpha ()