ENVIRONMENTAL REMOTE SENSING

(EVT571)

Lecture 9-2:
Classification Accuracy
Assessment
Dr. Arnis Asmat
Email: rnis_annis@salam.uitm.edu.my
arnisasmat@gmail.com

Introduction
Land-use/land-cover maps derived from remotely sensed
data are of little use if classification accuracy is not assessed.

Many studies ignore accuracy, or report an overall number

(e.g. 75%), which ignores locational accuracy.
Accuracy assessment involves comparison between:
A map derived from remotely sensed data; and
Reference test information.

The comparison is often made through an error matrix.

It is imperative that the error matrix be filled with unbiased
information.

The relationship between these two sets of information (a)

and (b) is commonly summarized in error matrix.

An error matrix is a square array of numbers laid out in rows

and column that expresses the number of sample units (i.e.,
pixels, clusters of pixels or polygon) assigned to a particular
category as verified in the field.
The columns normally represent the reference data while rows
indicate the classification generated from remotely sensed
data.

Classification Accuracy
4

How do we tell if classification is any good?

Classification error matrix (aka confusion matrix or contingency table)

Need truth data sample pixels of known classes

How many pixels of KNOWN class X are incorrectly classified as anything other
than X (errors of omission)?
So-called Type 2 error, or false negative
Divide correctly classified pixels in each class of truth data by COLUMN
totals (Producers Accuracy)
How many pixels are incorrectly classified as class X when they should be some
other known class (errors of commission)?
So-called Type 1 error, or false positive
Divide correctly classified pixels in each class by ROW totals (Users
Accuracy)

Classification Accuracy
5

Errors of
comission for
class U

Errors of
omission for
class U

 Can use original training data to test BUT.

this only tells us how well the classifier can
classify the training areas

Ideally, use an independent set

of samples to give a better
'overall' accuracy estimate
6

Accuracy
Accuracy assessment
assessment
Comparison of a classification with ground-truth data to evaluate
how well the classification represents the real world.
Overall map The total number of correctly classified pixels that were
accuracy
sampled from image divided the total number of image
pixels sampled from the accuracy assessment procedure.
Users
accuracy

Probability that the land cover class an image pixel was

assigned to actually represent that category on the
ground.

Producers
accuracy

The probability that a reference pixel has been correctly

classified.

Individual
class
accuracy

Accuracy levels reported for an example an individual

land cover class (for example, the number of pixels that
was correctly classified when compared with the
reference data)

Example of classification accuracy report

Cont

Convert pixel to ha

Cont

Convert percent pixels to number of pixel

Training vs. Test Reference

Do not use training data to assess accuracy!

Instead, reference test pixels should be located.

These are sites for which land cover class is known but not
used in training.

They are sometimes collected before the classification, but

more commonly after.
The constraint of time is an important factor.

Sampling Size and Scheme

It is important to gather reference test pixels in sufficient
numbers to assess accuracy correctly.
Minimum of 50 samples for
each land cover class in the
error matrix.

Rule of thumb:

If the study area is large (>1

million acres) or there are a
large number of classes
(>12), increase the number
to 75-100 per class.

Cont
It is often appropriate to:
reduce the number of samples
needed in some classes (e.g.
water) and
increase the number in other,
more variable classes.
The goal is to balance the statistical need to
get a representative sample with:

time

cost

practical limitations
associated with the project

Cont
1. Most accuracy assessment statistics assume random
sampling.

2. But random sampling can under-sample small but

important classes.

3. Thus, stratified random sampling is usually employed.

4. This technique samples a minimum number of reference
pixels per land cover class.

Cont
5. After classification, the computer selects sites (geographic
x,y location) within each land cover class at random.

6. A site visit is then made (using GPS) to each to determine

reference land cover class.

7. Of course, this is not always possible aerial photos and

maps are often used as reference.

Analysis of spectral signatures

The uniqueness of spectral signatures necessary for deriving a
meaningful classification.

Spectral signatures
should:

be representative
of the entire
spectral space

have minimal
overlap with other
signatures

Uniqueness measured as separability

Spectral Separability
Spectral separability can be analysed using JeffriesMatusita Distance (J-M Distance)
Measured values were between 0 and 2, where:

0 indicated a complete
overlap between the
signatures of two classes

2 indicated a
complete separation

SIGNATURE SEPARABILITY REPORT FOR EIGHT LAND USE/COVER CLASSES

Table A: Signature separability summary
___________________________________________________________________________

Separability Measure
Jeffries-Matusita Distance
Signature Segments
38, 39, 40, 41, 42, 43, 44, and 45
Signature Channels
1, 2, 3, 4, 5, and 7
Average Separability
1.93
Minimum Separability
1.41
Maximum Separability
2.00
Signature Pair with Minimum Separability: (Mixed crops, Grasslands)
___________________________________________________________________________
Table B. Separability matrix for land use/cover classes
Category
Water Forest
Oil palm
Mixed
crops
Forest
2.00
Oil palm
2.00
1.68
Mixed crops
2.00
1.98
1.58
Grasslands
2.00
2.00
1.93
1.41
Rubber
2.00
1.88
1.89
1.85
Cleared
2.00
2.00
2.00
2.00
areas
Urban
1.95
2.00
2.00
2.00

Grasslands

Rubber

1.88
2.00

2.00

2.00

2.00

Cleared
areas

1.93

a. 1989

b. 1993

c. 1999

a. 1989

a. 1989

4 Kilometres

b. 1993

c. 1999

Water bodies
Forest
Oil palm
Mixed crops
Grasslands
Rubber
Cleared areas
Urban

b. 1993

4 Kilometres

Evaluation of Error Matrices

Once test reference data have been properly sampled, they
are compared pixel by pixel to the classification map.
Agreement or disagreement is summarized in the error
matrix.
Error matrix information is then evaluated using:

simple descriptive
statistics

and/or

multivariate
analytical statistics.

Evaluation of Error Matrices

Overall accuracy =

total correct
total number of pixels in the error matrix

Example of Overall Accuracy

From Jensen (1996)

Evaluation of Error Matrices

Producers accuracy =

total correct in a given class

total number of pixels in that class from reference data
Indicates the probability of a reference pixel being
classified correctly.
Called producers error because it tells the
producer of the map how well a certain class can be
classified.

Example of Producers Accuracy

Error of omission = error of exclusion

From Jensen (1996)

Evaluation of Error Matrices

Users accuracy =
total correct in a given class
total number of pixels included in that class
Indicates the probability that a pixel classified on
the map actually represents that category on the
ground.
Called users error because it tells the user of the
map how accurate the classification is.

Example of Users Accuracy

Error of commission = error of inclusion

From Jensen (1996)

Evaluation of Error Matrices

KAPPA analysis is a multivariate technique for accuracy
assessment.

It yields a Khat statistic that is a measure of agreement or

accuracy:
r
r
N xii xi x i
i 1
Khat i 1
r
2
N ( xi x i )
i 1

where r = the number of rows in the error matrix; xii = the number of
observations in row i and column i, xi+ and x+i are marginal totals for row
i and column i, and N is the total number of observations.

Evaluation of Error Matrices

1.

Khat is used to determine if the results

presented in the error matrix are
significantly better than random class
assignment.

2.

It is also used to compare similar

classifications to determine if they are
significantly different.

3.

When reporting accuracy, include both

overall accuracy and Kappa!