RAVI SHANKAR

Architect Urban Planner

ACCURACY ASSESSMENT OF POST - EARTHQUAKE
BUILDING DAMAGE CLASSIFICATION IN HAITI

Master of Disaster Management (MDMa)

Masters Thesis 200910

RAVI SHANKAR - Architect - Planner

Supervisor: Asst. Prof. Peter Kjr Jensen
Co-Supervisor: Michael Calopietro
Institute of International Health, Immunology & Microbiology
Faculty of Health Sciences, University of Copenhagen
 Acknowledgements
I am grateful to my supervisor, Asst. Prof. Peter Kjr Jensen, a man who waves a magic wand. It
was a great pleasure working with my Co- Supervisor, Michael Calopietro, who made it possible
for me to write my thesis in absentia, away from Copenhagen.

It is an honour to work with the United Nations Operational Satellite Applications Programme
(UNOSAT) for my thesis and would like to thank our director, Francesco Pisano, and my
managers, Dr. Einar Bjorgo and Oliver Senegas, for their extended guidance. My friend Michael
Jendryke needs a special appreciation for his constant support and valuable input. I am indebted
to my colleagues Luca DellOro, Joshua Lyons, Wendi Pedersen, Fana Woldegebreal, Karl Nei-
man, and Paola, Fred and Oliver Vandamme for their constant help as well. This thesis would not
have been possible without assistance from Centre National de LInformation Go-Spatiale- Haiti
(CNIGS) in field data collection, which was one of the most difficult parts of the whole exercise.

I would also like to thank Dr. Norman Kerle (ITC, Netherlands) and Dr. Keiko Saito (Cambridge
Architectural Research Limited) for their valuable insights. I would like to show my gratitude to
to

My parents, my wife, and my son for their love and support & European Commission-Joint Research Centre (EC-JRC) and World Bank (WB) for sharing their
to the Haitians who are waiting patiently. data. Finally, it is a pleasure to thank my University Department staff, classmates, and friends
from Danida Fellowship Centre, and all others who made this thesis possible.
 Abstract
Quantitative measurements of building damage are extremely important after an earthquake. An
integrated remote sensing and GIS-based rapid mapping approach can provide timely informa-
tion on building damages to be used by the international humanitarian community in planning aid
response. The unprecedented scale of damages wreaked by the 2010 earthquake in Haiti, com-
pounded by the difficult social and political conditions there, precluded widespread field damage
assessment. Therefore, most of the preliminary damage assessments were carried out through
remote sensing images. Remote sensing imagery techniques are indispensable where difficult
ground conditions prevail, but there are limitations as well. More than 2000 maps were published
by different agencies within 75 days after the Haiti disaster. During this period, the variance in
the spatial data produced by these different agencies could have compromised relief efforts.
Naturally, then, there was a sense of urgency in assessing the accuracy of the information and
the methods used to obtain it.

This thesis investigates the accuracy of photo interpretation, one of the methods used to conduct
the virtual building damage assessment by the three main agencies involved: UNOSAT, World
Bank, and European Commission-Joint Research Centre. An error(confusion) matrix was used
to evaluate the accuracy of these assessments. The analysis was carried out to check overall ac-
curacy , accuracy per communes, accuracy per damage classes, and accuracy per each agency. 5
Finally, the assessment errors were investigated in detail in order to identify their causes, which
helped to quantify the limitations of using the remote sensing images. The results show that
the overall accuracy was around 60% and the range varied between 36-82% among different
communes. Overall accuracy was higher with fewer numbers of damage classes. Post-mortem
analysis of errors suggests that there is a strong possibility the accuracy levels can be improved.

source: google earth

Abbreviations and keywords
Aerial photography Aerial photography is the taking of photographs of the ground from an elevated position. The term
usually refers to images in which the camera is not supported by a ground-based structure.

CNIGS Centre Nationale de lInformation Geo Spatiale

EC-JRC European Commission - Joint Research Centre

Damage assessment An assessment of the total or partial destruction of physical assets, both physical units and
replacement cost

ECLAC Economic Commission for Latin America and the Caribbean

GEER Geo-Engineering Extreme Events Reconnaissance

GIS Geographic Information System (GIS), or geographical information system, is any system
that captures, stores, analyzes, manages, and presents data that are linked to location. In
the simplest terms, GIS is the merging of cartography and database technology.

Loss assessment An analysis of the changes in economic flows that occur after a disaster and over time, valued at
current prices.

Housing damage assessment A damage assessment that analyzes the impact of the disaster on residential communities, living
quarters, and land used or housing

Rapid assessment An assessment conducted soon after a major event, usually within the first two weeks. It may
7
be preceded by an initial assessment, and it may be multi-sector or sector-specific. It provides
immediate information on needs, possible intervention types, and resource requirements.

RS Remote sensing (RS) is the small- or large-scale acquisition of information of an object or

phenomenon through the use of either recording or real-time sensing device(s) that are
wireless, or not in physical or intimate contact with the object (such as by way of aircraft, spacecraft,
satellite, buoy, or ship).

UNOSAT UNOSAT is the UN Institute for Training and Research (UNITAR) Operational Satellite
Applications Programme, implemented in cooperation with the European Organization for
Nuclear Research(CERN).

WB World Bank
Table of contents List of figures
1 INTRODUCTION Figure 1 Map of Haiti.
1.1 Background .................................................................................................................. 11 Figure 2 Area of populations affected by the 2010 earthquake.
1.2 The Earthquake ............................................................................................................ 12 Figure 3 Relief, Response, Recovery and Reconstruction phases as described by World Bank.
1.3 The Critical Period......................................................................................................... 13 Figure 4 Sequence of the Post-Disaster Needs Assessments.
1.4 Post-Disaster Needs Assessment................................................................................. 14 Figure 5 Published maps after 2010 Haiti earthquake.
1.5 Role of GIS in Disaster Management ........................................................................... 15 Figure 6 Various methods of damage assessments.
1.6 Disaster Management and Remote Sensing ................................................................ 15 Figure 7 Damage catalogue as used by various agencies for damage assessments in Haiti.
1.7 Use of RS Images after Haiti Earthquake ..................................................................... 16 Figure 8 The pre- and post-images with faulty building damage assessments.
1.8 Methods of Damage Assessments................................................................................ 17 Figure 9 Showing various damage classes (destroyed to no visible damage) as adopted by UNOSAT.
1.9 Damage Catalogue........................................................................................................ 18 Figure 10 Percentage composition of the assessment among different agencies.
1.10 Statement of Problem.................................................................................................. 19 Figure 11 Methodologies of accuracy assessments.
2 OBJECTIVES Figure 12 Showing random samples collected for field validation.
2.1 Main Objectives............................................................................................................. 21 Figure 13 Showing the theory of error matrix
2.2 Hypothesis..................................................................................................................... 21 Figure 14 Showing the methodology adopted for accuracy assessment.
2.3 Specific Objectives ....................................................................................................... 21 Figure 15 Showing comparison on four and two class for various agencies.
3 METHODS Figure 16 Showing kappa statistics for various communes.
3.1 Introduction ................................................................................................................... 23 Figure 17 Showing kappa statistics for various agencies
3.2 Sample Selection .......................................................................................................... 24 Figure 18 Showing accuracy levels comparison between four and two class.
3.3 Data Collection ............................................................................................................. 25 Figure 19 Showing logic behind post mortem of failed class.
3.4 Quality Assurance.......................................................................................................... 25
3.5 Methods for Accuracy Assessments .............................................................................. 26
3.6 Creation of Database and Analysis Approach ...............................................................
4 FINDINGS
27
List of tables 9

4.1 Overall Findings............................................................................................................. 31

4.2 Variations by Communes............................................................................................... 33 Table 1 Buildings Assessed in Various Communes.
4.3 Four-Class and Two-Class Variations ........................................................................... 34 Table 2 Field Samples Collected by CNIGS from Various Communes.
4.4 Postmortem of Failed Class .......................................................................................... 35 Table 3 Overall findings .
5 DISCUSSION Table 4 Range on Kappa Statistics Agreement
5.1 Methodology.................................................................................................................. 37
5.2 Overall Communes ....................................................................................................... 38
6 CONCLUSION & RECOMMENDATIONS 41
7 BIBLIOGRAPHY 43
8 ANNEXURE 45
 1.1 Background

Haiti is located in the Caribbean Sea on the Island of Hispaniola, which it shares with the Domini-
can Republic (Fig. 1). Haiti occupies the western third of the island and has 8.4 million inhabitants
and a population density of 300 people per km2 (Canadian International Development Agency,
2005). Haiti is the poorest country in the Western Hemisphere with an estimated 80% of its people
living under the poverty line; 54% live in abject poverty (CIA, 2010). When it comes to natural dis-
asters, Haiti seems to have a bulls-eye on it. There were eight major natural disasters within last 15
years. Floods during 2008 hurricane killed more than 2500 people.

Haiti is a twentieth-century Greek tragedy of Malthusian prophecy run wild, and little time remains
to reverse the course toward destruction of what once was the lush tropical beauty of the Pearl of
the Antilles (Ernest H, 1996). High population density, chaotic urbanization, a weak political struc-
ture, and poorly developed disaster-management systems are some of the factors contributing
to the vulnerability of its people.

11

1 INTRODUCTION

Figure 1. - Map of Haiti

1.2 The Earthquake
The earthquake that struck the Republic of Haiti on January 12, 2010, is one of the most signifi-
cant earthquakes in recorded history. Destruction and death on a huge scale unfolded across
the densely populated Haitian capital, Port-au-Prince, and into the shanties and villas of the sur-
rounding mountains. As of February 16, 2010, the death toll reported by the government of Haiti
exceeded 217,000 with an additional 300,000 injuries. More than 5 million people live in the area
directly affected by the earthquake (Fig. 2), and 1.2 million people are now living in temporary
shelters (UN, 2010).
Figure 2.- Area of populations affected by the 2010 earthquake.
Most of the buildings are masonry either brick or construction blocks which tend to perform badly
in an earthquake and with sub standard construction techniques worsened the situation. Damage
to infrastructure was extensive and ministry of education estimated that more than 15,000 primary
schools and 1500 secondary schools were severely damaged. Most of Port-au-Princes municipal
government buildings were destroyed or heavily damaged in the earthquake, including the City
Hall, which was described by the Washington Post as, a skeletal hulk of concrete and stucco,
sagging grotesquely to the left(Manuel Roig-Franzia, 2010).
1.3 The Critical Period
After the earthquake, the government of Haiti led the emergency relief operation with support
from the international community. Preliminary physical damage assessments on buildings were
out within days after the earthquake from various agencies. In its handbook on disaster man-
agement, recovery, and reconstruction, the World Bank describes the initial 25 days as the most
crucial period for disaster relief and damage loss assessment (Fig. 3; ECLAC & World Bank, 2003).
Major response guidelines from the World Bank suggest that the following activities take place
during the initial phase,
Disaster Response
Disaster Relief
Disaster and loss assessments
The decisions on reconstruction in Haiti were based on these timely assessments along with the
updated field results.
13
Figure 3. Relief, Response, Recovery and Reconstruction phases as described by World Bank.
1.4 Post-Disaster Needs Assessment
The Post-Disaster Needs Assessment and Recovery Framework (PDNA) is an interagency mecha-
nism set up to facilitate a comprehensive, coordinated, and efficient international approach to
damage, loss, and needs assessment and recovery planning. The Recovery Framework outlined
by the PDNA facilitates integrity of damages data and helps to ensure that financing needs and
the recovery pathway are matters of mutual agreement. PDNA is designed to address one or
more of the following issues (Darcy and Hofmann, 2003):
Whether to intervene.
The nature and scale of the intervention.
Prioritization and allocation of resources.
Programme design and planning.
PDNA is not intended to replace more in-depth assessments, which may still be required outside
of the PDNA for refining the understanding of needs, generating response options, and develop-
ing recovery projects within a given sector. In Haiti, the PDNA is a government-led exercise, with
integrated support from the United Nations, the European Commission, the World Bank, and other
national and international partners. A PDNA prepares a single consolidated report containing
crucial information on the disaster: its physical impacts; the economic value of the damages
and losses; the human impacts as experienced by the affected populations; and the early and
long-term recovery needs and priorities. The type and scope of the disaster affects the develop-
ment vision. Based on this assessment, various responses are proposed, analyzed, and selected
for inclusion in the Recovery Framework (Fig. 4).
Figure 4. Sequence of the Post-Disaster Needs Assessments.
1.5 Role of GIS in Disaster Management
Geographic Information System (GIS) is a computerized method of handling the spatial data,
which includes information system and remote sensing data. In simple terms, GIS is a merging of
the cartography and database technology. GIS is significant to all stages of the disaster manage-
ment cycle. It provides for efficient and rapid management of data, as well as data manipulation
and analysis, thereby effecting better decision making. GIS can be an essential support system
to estimate the loss after a disaster. In the present context of Haiti, humanitarian organizations
fully exploited the role of GIS in disaster management. Haiti represents a milestone in the use
of GIS in the post-disaster cycle, as in agencies relied upon it more than ever before to collect,
process,and analyze the spatial information gathered.
1.6 Disaster Management and Remote Sensing
Remote sensing (RS) is the small- or large-scale acquisition of information of an object or phe-
nomenon using either recording or real-time sensing devices that are wireless, or otherwise not in
physical or intimate contact with the object (such as by way of aircraft, spacecraft, satellite, buoy,
or ship) (Lillesand et al., 2003). The critical period after the disaster is the response phase, and
the field data collected during this period, if it is possible to do so, is very valuable. However, it is
a difficult task to collect data during this period from the ground. But remote sensing technology
can accommodate this lag in collecting field data.
Using remote sensing technology, the damage concentration in the urban areas can be located
in a shorter time compared with the conventional ground survey (Rao, 2000). An unprecedented
international satellite imagery campaign with the participation of all leading Earth observation
operators was undertaken after the destructive Haiti earthquake. Annexure 1 shows the various
remote sensing accusations that took place after the earthquake. Later, high-resolution aerial im-
ages of the capital city of Port-au-Prince of up to 15 cm spatial resolution were taken for more pre-
cise assessments of building stability. Never before have such high- resolution spatial images and
aerial photography been so accessible. Multi-source data were collected from various missions:
WB-Image CAT-RIT Remote Sensing Mission (LiDAR optical data, 15 cm & 2 pt/m), Google (optical
data, 15 cm), NOAA (optical data, 25 cm), and Pictometry (Haiti_Reconnaissance.pdf, 2010).
15
 1.7 Use of RS Images after Haiti Earthquake 1.8 Methods of Damage Assessments

There were 2000+ maps* published within the first 75 days of the Haiti earthquake. More than
50 agencies were directly involved in the preparation and publication of the maps. Most of the
maps detailed earthquake coverage, affected populations, and damage assessments, and
some offered hazard analysis and cluster- or sector related data. There were around 10 agencies
involved in the building damage assessments. With the Post-Disaster Needs Assessment requir-
ing building damage extents for Haiti, these assessments happened to be a valuable exercise.
Participating agencies used different approaches in assessing the extent of building damage.
Predominant among them was the photo interpretation method. Figure 6 shows various methods
of damage assessment made possible by remote sensing images. Photo interpretation method
is theoretically one of the simplest methods of building damage assessment using RS images.

The UN Institute for Training and Research (UNITAR), the Operational Satellite Applications Pro-
gramme (UNOSAT), the European Commission -Joint Research Centre (EC-JRC), the National
Geospatial Information Centre (CNIGS) representing the Haiti Government, the World Bank (WB),
and the Global Facility for Disaster Reduction and Recovery (GFDRR), supported by its consult-
ant Image CAT, have carried out a detailed, building-by-building assessment of the damage.
This analysis relied heavily on the use of space remote sensing techniques. The joint agreement
among UNOSAT, WB, and EC-JRC to publish a common database for all of Haiti must have been
a welcome relief for the humanitarian community.

Figure 7. Relief, Response, Recovery and Reconstruction phases as described by World Bank.

Figure 5. Published maps after 2010 Haiti earthquake.
The analyst uses GIS software where the pre- and post- satellite/aerial images are loaded, and a
new database is created to record the assessments. The analyst visually identifies the extent of
damage by comparing the pre- and post- images and records it into the database. Though the
method sounds simple, it is time consuming and labour intensive. And there are drawbacks to
the technology, as described by UNOSAT: Satellite imagery generally provides an overhead view-
point. Lateral damage and especially damage to the internal structures of buildings, is not de-
tectable from the analysis of satellite imagery (2010). In the manual photo interpretation method,
the quality of the assessment might depend on the expertise of image analysts working away from
17
Haiti, who are better equipped for the task if they are knowledgeable about Haitis construction
techniques, building materials, and housing typology...

Figure 6. Intensity of Damage assessment as published by UNOSAT.

* Authors estimate based on the relief web website (www. reliefweb.int)

1.9 Damage Catalogue 1.10 Statement of Problem

The important aspect of building damage assessment is the expression of the damage levels.
The damage catalogue is the scale which portraits the intensity of the damage, with categories
usually ranging from no visible damage to destroyed. There are several international field dam-
age assessment scales (e.g.European Macroseismic Scale-98[EMS-98], ACT ) that quantify the
extent of damages after an earthquake. It is very clear from the published maps of Haiti that the
participating agencies used different damage catalogues for assessment. UNOSAT and EC-JRC
used four classes in its damage assessments whereas WB used just two classes for the damage
assessment (Fig. 8).
With fast spreading application of GIS using remotely sensed data and ever increasing evaluation
of the damages using these technologies, it is essential that the accuracy of these products be
assessed. Accuracy assessments evaluate either a single method or various methods used to col-
lect spatial data in the aftermath of the disaster. The accuracy assessment allows the analyst to
quantitatively compare different methods. . The cross comparisons should identify the processes
that yield the most accurate results, information critical for decision making in future disasters.
The ultimate objective should be to make accuracy assessment part of the metadata in the later
updates of published maps. Most of the maps produced after the disaster as rapid mapping or
damage assessment had not been subject to extensive accuracy assessments. It appears there
was often a casual attitude toward these maps, in part because they had been prepared so hast-
ily in the first place. In some instances, it seemed the appearance of the map was valued over
its accuracy.

The sample validation in field by GEER (Geo - Ellen Tathje, Jeff Bachhuber, et, all, GEOTECHNICAL
engineering Extreme Events Reconnaissance) ENGINEERING RECONNAISSANCE OF THE 2010 HAITI
stated However, it must also be accepted with EARTHQUAKE, 22 Feb 2010.
some limitations as these types of surveys do not
always capture the magnitude of destruction on
Figure 8. Damage catalogue as used by various agencies for damage assessments in Haiti. the ground. As an isolated example, consider
the two steel frame warehouses ........ Both of
these buildings were heavily damaged and will
need to be torn down, yet they were classified
as No Visible Damage by the UNOSAT survey.

19

Figure 10. The pre- and post-images with faulty building damage assessments.

Figure 9. Showing various damage classes (destroyed to no visible damage) as adopted by UNOSAT.

A- UNOSAT , B- EC-JRC/ISFEREA, C- DLR,GERMAN SPACE AGENCY, D- SERTIT E- ITHACA & E-Geos

 2.1 Main Objectives

The aim is to evaluate the accuracy of photo interpretation methods of remote/virtual building
damage assessment in the aftermath of the Haiti earthquake. The study will focus on the assess-
ments of the three main coordinating agencies: UNOSAT, EC-JRC, and WB.

2.2 Hypothesis

There is no difference in accuracy of preliminary damage assessments among different com-

munes and agencies for the Haiti earthquake.

2.3 Hypothesis

. To evaluate the overall accuracy of building damage assessments for Haiti.

. To compare the accuracy of building damage assessments by commune.

. To compare the accuracy levels among the various agencies.

. To compare the accuracy levels of the four and two-class building damage classification.

. To perform a post-mortem analysis of the errors of the building damage assessment.

21

2 OBJECTIVES
 3.1 Introduction

There were around 400,000 building points in Haiti which had been under the direct influence of
the Jan 12th earthquake. The study area encompasses communes of the Artibonite region. In
these, Port-au-Prince, Cite-Soleil and Carrefour are the most densely populated areas with the
maximum population spread. It is also the area of the greatest economic development in Haiti. In
total, 290,672 building points were assessed by these three organizations. UNOSAT covered 83%
of the total assessment with EC-JRC responsible for about 12 % and WB for the remaining 5 % (Fig.
9). World Bank assessed only the Destroyed and Severely damaged classes whereas UNOSAT and
EC-JRC assayed four classes, Destroyed, Severely Damaged, Moderate Damage, and No Visible
Damage.

NAME OF COMMUNE NO OF BUILDINGS ASSESSED ORGANIZATION

CROIX-DES-BOUQUES 3562 WORLD BANK

CITE-SOEIL 6709 WORLD BANK

CARREFOUR 69384 UNOSAT

LEOGANE 23790 EC-JRC

DELMAS 64394 UNOSAT

TABARRE 3919 WORLD BANK

PETIN- VILLE 10587 UNOSAT / WORLD BANK

GRESSIER 1347 WORLD BANK

PORT-AU-PRINCE 92432 UNOSAT

JACMEL 13133 EC-JRC

PETIT-GOAVE 409 WORLD BANK

GRAND-GOAVE 273 WORLD BANK

TOTAL 290672

Table 1 Building Assessed in Various Communes.

EC-JRC 23
36284

3 METHODS 12%

WB
13686
5%

Figure 11. Percentage composition of the assessments among

the different agencies.
UNOSAT
240702
83%
3.2 Sample Selection 3.3 Data Collection

The earthquake affected areas of Haiti was divided into 200X200 m cells. To be considered for a In total, around 7000 building points were collected in the field by CNIGS, who were entrusted
sample selection, each cell had to satisfy certain criteria. Cells were sorted based on the follow- with that work. Samples from the communes of Jacmel, Petit Goane, and Grand Goane were
ing three steps: not collected for various reasons, including damages, accessibility, and distance from the base
station. Even though little pre-processing was undertaken in the site office at Haiti, this was one of
1.Each cell was assigned either high or low density as per existing population. the most time consuming components of the whole project.
2. Each cell was also assigned high or low damage intensity based on damage assessment.
NAME OF COMMUNE NO OF FIELD SAMPLES NO OF BUILDINGS
3. A land cover map was relied upon for uniform representation within each land use. ASSESSED VIRTUALLY
CROIX-DES-BOUQUES 343 3562
Analysis using these criteria identified around 2150 cells from which samples were to be cho- CITE-SOEIL 243 6709
sen. A stratified sampling method was then used to further refine the sample-cell collection. To
CARREFOUR 1305 69384
determine the sample size to be selected from the 2150 retained cells, confidence level and
LEOGANE 97 23790
confidence interval were defined as prior criteria and calculations were performed accordingly.
A 95% confidence level and a 10.5 confidence interval were selected. Hence, the computed DELMAS 735 64394

sample size was equal to 86 cells, representing 4% of the population. Finally, the 86 cells selected TABARRE 648 3919

randomly among the 2150 previously stratified cells in a number proportional to the stratums size. PETIN- VILLE 464 10587
A field questionnaire was prepared on the basis of European Macroseismic Scale-98, and every GRESSIER 50 1347
building from the chosen sample was graded accordingly.
PORT-AU-PRINCE 2607 92432

CNIGS FIELD DATA 6493

UNOSAT/ EC-JRC FIELD DATA 442

Table 2 Field samples Collected by CNIGS from Various communes.

3.4 Quality Assurance

Field samples were scrutinised during the preprocessing phase for quality assurance. The preproc-
essing for error checking included:

1. Sample points outside the building boundary or on the road

2. More than one sample point within the same building or more than one virtual assess-
ment point for a given building
25
3. Site photos giving extreme class variation in comparison to field assessment.

After the initial preprocessing of the sample for possible errors, the size of the building points was
reduced to 6492. This covers roughly 2% of virtual assessed building samples.

Figure 12. Showing random samples collected for field validation.

These also include the samples collected by UNOSAT/EC-JRC during the initial field visit to Haiti.
3.5 Methods for Accuracy Assessments 3.6 Creation of Database and Analysis Approach

To evaluate the accuracy levels of the remote damage assessments an error( confusion) matrix The following steps were carried out for data analysis.
is used. It compares category-by-category, based on damage class/catalogue, the relationship
between the produced and known reference data of field truth. Discrete Multivariate analysis Step 1: The joint database was created by UNOSAT combing the individual data sets of
technique is used following the error matrix with the assumption that these categories are de- UNOSAT, EC-JRC, and WB.
pendent and do not require any transformation. A Khat value (K^) will be calculated for each
agency, which is a measure of how well the classification agrees with the field data. Khat can Step 2: The data received from the field was subjected to preprocessing as explained
be tested using the normal curve deviate to determine if the two error matrixes are significantly before. After removing the field samples which did not meet the set criteria,
different. a new field database was created.

Step 3: The joint database was then merged with the field database. This created an
intermediate data base.

Step 4: The intermediate database was further divided into shape files as per communes
for 4-class and 2-vclass building classification.

Step 5: Accuracy assessment script was run using ESRI ARCGIS 9.3.1 software followed
by Kappa statistics script.

Step 6: The results are displayed and postmortem of the failed class is then carried out.

27

Figure 13. Showing the theory of error matrix

 29

Figure 14. Showing the methodology adopted for accuracy assessment.

 4.1 Overall Findings

Out of the 5806 points organized after preprocessing the field data, there were only 4351 virtu-
ally assessed points matching with the field points. This means that more than 1455 points were
not assessed during the virtual building assessment. Thus, for Haiti, the virtual/remote building
damage assessment was performed for only 75% of the total buildings that were affected within
the earthquake impact zone. A number of factors may have contributed to this oversight. There
were significant problems defining the various classes of buildings and indeed what constitut-
ed a building, especially given the makeshift character of many of the settlements. A number
of conditionals were exacted upon the labelling causing further confusion. Also, there was a
troubling lack of pre-earthquake data establishing building parcel boundaries. One could ar-
gue that this is generally the case with most of the disasters that occur in developing or under-
developed countries. Had building parcel boundaries been available, damages could have
been associated with the building parcel rather than with the building itself. Some of the screen
shots illustrate that individual analysts had clearly failed to mark the buildings in most instances.

4 class Post mortem of errors

NAME OF COMMUNE Accuracy Users Producers 2 class Kappa Errors Doubt Damages
levels accuracy accuracy Accuracy statistics by ful cases could not
% % levels % % analyst % or class be
confu- detected
sion % through
RS
imagery %

CROIX-DESBOUQUES 35 11.5 31 63 10.5 25 29 11

CITE-SOEIL 44 25 38 72 20.29 21 20 15

CARREFOUR 67 37 38 79 21.28 12 9 12

LEOGANE 46 40 38 78 25.40 25 18 9

DELMAS 82 31 40 94 20.93 6 8 4

TABARRE 76 32 36 92 26.46 11 6 7

PETINVILLE 61 24 25 94 - 14 12 13

GRESSIER 56 - 35 76 12.73 25 13 8
31
PORT-AUPRINCE 56 39 40 84 25.29 20 14 9

4 FINDINGS OVERALL COMMUNES 61 36 39 86 23.06 - - -

Table 3. Overall findings .

Of 290,672 buildings virtually assessed, we had a field sample of 1.5 %, or 4329 buildings. The
4.2 Variations by Communes
accuracy of the joint database was found to be 61%. The Moderate Damage class was the most
poorly performed class in the entire assessment, with the Kappa statistics around 23%. The accu-
For Port-au-Prince, there were around 92,423 buildings which were virtually assessed. Of the to-
racy level increased by 25% when the error matrix was applied with just two classes. Furthermore,
tal field samples collected for Port-au-Prince, 2073 virtual assessments matched with the cor-
the postmortem of the errors showed that around 15-20% of errors could have been avoided by
responding field sample, and the overall accuracy was found to be 56%. Figure 17 charts the
the analyst and that around 10% obtained because of practical impossibilities with aerial images
performance accuracy of the user and the producer, which dips sharply in the Moderate class.
as applied to building damages. Further analysis on the accuracy levels for each organization
Kappa statistics was around 25.29%. The accuracy level went up to 28% when the error matrix
showed that UNOSAT performed better in the overall accuracy, followed by World Bank first and
was applied with just two classes. Furthermore, the postmortem of the errors showed that around
then EC-JRC. For the Kappa statistical analysis, EC-JRC performed slightly better the other two
20% of them could have been avoided by the analyst and around 9% occurred due to practical
organizations.
impossibilities with the aerial images.

Figure 16. Showing kappa statistics for various communes.

33
Figure 15. Showing comparison on four and two class for various agencies.

Figure 17. Showing kappa statistics for various agencies

4.3 Four-Class and Two-Class Variations 4.4 Post-mortem of Failed Class
It would be interesting to see how the accuracy levels behaved if the entire damage assessment The question confronting us is how the results of the accuracy assessment can lead to a construc-
was carried out with just two classes, Destroyed and Not Destroyed. It was evident from the start tive discussion of ways to improve the reliability of the numbers..It is important to discover why it
that the four-class approach was an ambitious and time consuming undertaking. If the accuracy failed and what are the chances of improving it.
is better for two classes than four classes, then it would be wise to start with two damage classes From the previous sections we know that there are limitations with the aerial images and that the
immediately after the disaster and then later improvise with more classes. Figure 18 is a hypotheti- outcome of the photo interpretation method is heavily dependent on analyst performance. Fig-
cal scenario to study the accuracy assessment with two classes; slight adjustments were made in ure 19 shows the three most likely factors producing the failed results. They are:
combining the EMS-98 scale.
1.Building damages which could not be detected through remote sensing imagery, the
main limitation of using the aerial images.
2.Errors which could have been avoided by the analyst.
FOUR CLASS % TWO CLASS %
3.Doubtful cases or class confusion.
PETION VILLE
94 If we could assess postmortem the failed classes in a more refined manner, researchers would
100

ALL COMMUNES 90
CROIX DES BOUQUES have a better way of gauging the potential accuracy of aerial imagery assessment for disasters
86 80 63 of similar scale in the future. The failed assessment points need to be analysed individually with
70
61 the pre-and post-images. This by itself is a time consuming exercise if the accuracy levels are very
60
61 low.
50

94 40
35
DELIMAS 30 72
CIRE SOLIEL
82 20
44
10

76 46
92 78
TABRRE LEOGANE

67 56
56

89 84
CARREFOUR PORT-AU-PRINCE
74
GRESSIR

35
Figure 18. Showing accuracy levels comparison between four and two class.

Figure 19. Showing logic behind post mortem of failed class.

 5.1 Methodology

Binomial distribution was used for the sample size calculation, and the majority of the research-
ers are using the same for error matrix or the normal approximation of the binomial distribution to
compute required sample size. This seems to be a standard practice. According to Congalton,
a general rule of thumb is a minimum of 50 samples for each category in error matrix; our sam-
ples meet that requirement. But it was unfortunate that we could not have samples collected for
Jacmel, Petit Goave, and Gran Goave communes where EC-JRC was responsible for the assess-
ment.

Another interesting aspect of the sample is the use of land cover maps with land use data as one
of the input parameters. The land use map is a gross approximation based on land cover. The
use of the virtual building damage assessment as an input criteria could be debatable depend-
ing on whether it will influence the sample selection. But considering the use of this at the grid
level to ensure the sample is spread evenly on the study area is not a bad idea in and of itself
The knowledge and expertise of the field sample collectors in using the EMS-98 scale is question-
able even though UNOSAT and EC-JRC had briefed them in detail about the scale. Had structural
engineers or other experts collected the field sample, they would have had different results. But
we have to admit that CNIGS were the best available choice given their local knowledge of the
situation.

The accuracy of the GIS and the other manual errors associated with the field sample collection
is an issue to be discussed. From the field data, around 300 samples were discarded during pre-
processing either because they were marked on the streets or away from the building footprints or
because there were two samples with varying class over the same building footprint. Even though
this exercise was tedious, it was necessary for quality assurance on the field sample and, in addi-
tion, personally rechecking the samples improves the confidence level of the data. Among other
practical issues tied to the CNIGS side of the data collection are a lack of access to certain grids
due to nonexistent roads and relatively high slopes, difficulty differentiating one building from
another even on a map 1/1000, and problems sending photos due to low speed Internet con-
nections.

As we saw in the introduction, more than 10 agencies were involved in the virtual damage as- 37
sessment for Haiti, and when I looked closely at their maps, I realized that there was no similarity in

5 DISCUSSION defining the damage in the Legends. It might be impossible for a person in the field to compare
these maps and to achieve common understanding of the damages based on the damage
classes. There could be a need for closer inspection here to determine whether it is possible to
develop a standardized catalogue in rapid mapping of virtual damage assessment situations.
In conjunction with this, we saw that it is more difficult to coordinate viable postmortem analyses
with common field data produced by different agencies relying on different interpretations of
damage classes.
5.2 Overall Communes

The error matrix has become widely accepted as the standard method for reporting the accuracy As I mentioned in the introduction, without spatial data on the property boundaries and land use
of the GIS data derived from the remote sensing data. The analysis in the previous section very data, it is a complicated situation to define a building. I will provide some specific examples.
clearly indicates accuracy levels fluctuating between 35% and 82%. This wide range of band in In the case of a building apparently not affected but in the midst of a destroyed area, analysts
the accuracy makes it difficult to draw direct conclusions from the results. The overall accuracy struggled to choose between the moderate and severely destroyed classes. There were also
from the error matrix for all the communes put together for all four classes was around 60% . This situations where buildings were marked as destroyed when in fact they were under construction.
rate was much improved by 25 % with just two classes. This is a valuable lesson as far as the dam- Closer attention to the pre- and post- images was required to ascertain the status of the building.
age classes are concerned. Directly after the disaster, it would be wiser to confine the study to just Depending on the length of time between the pre-scenario and the post-scenario, there was a
two classes and then later expand it to more classes if required. chance the house had been completed during this gap, thus creating a wrong entry as under
construction. In some instances, the presence of elements like internally displaced camps in the
streets or road blocks gave a false impression that the buildings around it had been affected by
the earthquake. But we realized that people were moving out of their houses into the streets fear-
KAPPA VALUES INTERPRETATION
ing aftershocks, which was another reason the analyst might make a wrong decision on the as-
<0 No agreement
sessments. Shantytowns or urban slums posed a significant challenge to the analyst owing to the
0.0 0.20 Slight agreement dense nature of the buildings and their poor quality of construction with unconventional building
0.21 0.40 Fair agreement materials.
0.41 0.60 Moderate agreement

0.61 0.80 Substantial agreement

0.81 1.00 Almost perfect agreement

Table 4. Range on Kappa Statistics Agreement

We know that Kappa statistics is a measure of agreement rather than the percentage of occur-
rence. Landis and Koch have provided instructions for interpreting K values (Table 3). This is not a
proven frequency but more of a guideline. The average of the Kappa statistics for all the com-
munes put together is around 21%, just barely falling into the category of fair agreement. This
result raises questions about the strength of the photo interpretation method since almost all the
agency values are around 20%. The perspective of the individual analyst assigned to damage
assessment, myself included, is important to any analysis of the reasons for the errors. My experi-
ence indicates that many of the problems derived from ambiguous building definitions. The pro- 39
cess would have gone more smoothly if building definitions had been agreed upon before the
start of the exercise.
 CONCLUSION & RECOMMENDATIONS

This study offers some valuable lessons for future damages assessment analysis. The data reveal
the limitations of aerial images and especially photo interpretation method in building damage
assessments. The reliability of virtual damage assessment through photo interpretation is put into
question and leads us to ponder the best method going forward. On a more detailed level, the
study elaborated the specific errors to which may be attributed the low percentage of accuracy.

The four-class damage categorization could have been avoided and the whole damage as-
sessment instead progressed from the start with two classes. This would have allowed the analyst
more time, and many of the class confusion errors could have been avoided as well. If the initial
digitalization of the building points (without error assessment) had been out sourced completely
via crowd sourcing or with academic institutes who were willing to participate, overall results could
have been improved, as happened in the case of the Delmas area of Port-au-Prince. The field
data collection could have been better coordinated to avoid repetition of the preprocessing of
the field data both by CNIGS and UNOSAT. (For example, building points were moved by CNIGS
over the building footprints after the field survey, but they were not snapped over the virtual as-
sessment points; this work was carried out by the author.) For UNOSAT, the labour shifts might have
been better organized, and a detailed training catalogue would have been welcomed by the
analysts.

Time constraints and the unavoidable confusion attendant upon a disaster of this magnitude
precluded analysts from exploiting the full potential of such a study. While much can be gleaned
from the analysis, circumstances did not allow other desired avenues of investigation: more ex-
tensive review of the existing literature; samples collected from the Jacmel, Petit-Goave, and
Grand Goave communes; a comparison of accuracy levels of satellite imagery of damages and
aerial imagery; refined procedures for sample collection and sample size distribution.

My research points to the importance of standardizing the building damage catalogue. This could
possibly require a detailed research and intervention in the rapid mapping community. Research
41
in this field should also look to alternative methods of damage assessments using high resolution

6 CONCLUSION & RECOMMENDATIONS imagery like object-based image analysis.

 1. Canadian International Development Agency. (2005). Haiti: facts at a glance. <http://www.

acdi-cida.gc.ca/CIDAWEB/webcountry.nsf/VLUDocEn/sHaiti-Factsataglance>.

2. CIA (2010). The Central Intelligence Agency World Factbook. <https://www.cia.gov/library/

publications/the-world-factbook/index.html>.

3. Congalton R. G. 1991. A review of assessing the accuracy of classifications of remotely sensed

Data. Remote Sensing of Environment.

4. Darcy, J. & Hofmann, C. (2003). Needs assessment and decision-making in the humanitarian

sector. London: Humanitarian Policy Group, Overseas Development Institute.

5. Eberhard, M. O., Baldridge, S., Marshall, J., Mooney, W., & Rix, G. J. (2010). USGS_EERI_

HAITI_V1.1.pdf (application/pdf Object). Web. 00/00/10. <http://www.eqclearinghouse.

org/20100112-haiti/wpcontent/uploads/2010/02/USGS_EERI_HAITI_V1.1.pdf.>

6. Ernest, H. Preeg (1992). The Haitian Dilemma. International Studies.

7. Geospatial Data Availability for Haiti: An Aid in the Development of GIS-Based Natural

Resource Assessments for Conservation Planning.

8. Haiti_Reconnaissance.pdf. (2010). http://www.eqclearinghouse.org/20100112- haiti/wp-con-

tent/uploads/2010/02/Haiti_Reconnaissance.pdf.

9. Handbook for Estimating the Socio-economic and Environmental Effects of Disasters. (2003).

ECLAC & World Bank http://www.gdrc.org/uem/disasters/disenvi/VOLUME%20I.pdf. Wiley. ISBN

0-471-15227-7.

10. Lillesand, T.M, Kiefer, R.W., and Chipman, J. W. (2003). Remote sensing and image

interpretation (5th ed.).

43

7 BIBLIOGRAPHY 11. Manuel Roig-Franzia(2010). Washington post, retrieved from http://www.washingtonpost.com/

wp-yn/content/article/2010/01/19/AR2010011904614.html

12. Rao, (2000) Disaster management

13. UN (2010). United Nations Office for the Coordination of Humanitarian Affairs. Haiti Earthquake

Situation Report #21, February 16, 2010.

14. UNOSAT.(2010).Atlas of building damage assessment, http://unosat.web.cern.ch/unosat/

shared/Haiti-HQ-2010/maps/PDNA/PDNA_atlas_intro.pdf.
 Annexure 1

Satellite tasking after the earthquake.

Satellite Tasking Satellite Sensor Time (UTC) Area Type

1 SPOT 5 14&19 Jan Port-au-Prince

2 BJ-1 SensonML (red band /
infrared band) 13-Jan Port-au-Prince
3 ALOS AVNIR-2 13-Jan
PALSAR 16-Jan
AVNIR-2 23-Jan
PRISM 23-Jan
PAN 23-Jan
4 DigitalGlobe multiple dates
WV-01 13&14 Jan multiple
collection dates Mapping entire region B/W
WV-02 14 Jan multiple
collection dates Port-au-Prince Multi
spectral
5 QuickBird multiple collection
dates Port-au-Prince
6 Ikonos 14&15&17 Jan
7 GeoEye-1 13&16&18 Jan
8 Cosmo-Skymed 15-Jan
9 Formosat-2 13&14&15&16&17
Jan Port-au-Prince
10 Radarsat-2 14&15 Jan Port-au-Prince
11 HJ-1-A/B CCD, Hyper-spectral
camera, Infrared
camera 14-Jan
12 RapidEye 13&14&15& 17
45
Jan full country

7 ANNEXURE
13 EO-1 15-Jan
coverage
Port-au-Prince ALI
14 EROS-B 17-Jan Port-au-Prince

Source: http://www.un-spider.org/haiti

Annexure 2
EMS Scale grade
Classification of Damage to masonry Buildings
Negligible to sligbt damage
(no Structual damage) Hair-
line cracks in very ferw walls
Grade 1 fall of smallpleces of laaster
only. Fall of loose stones from
upper parts of buildings in
very frw class
Moderate damage
(slight structural damage.
moderate non- structural
damage) Cracks in manu
Grade 2 walls fall of fairly large pieces
of plaster. parts of chlmmys
fall down
Substantial to heavy
damage (moderate
structural damage, heavy
Grade 3 non- structural damage) </
Large and exensive cracks
in most walls: panfiles of
slates slip off. Chimneys are
broken at the roof line failure
of individual non-structural
elements.
Very Heavy damage (heavy
structural damage, very
Grade 4 heavy non-structural
damage) Serlous failure of
walls partial structural failure.
Grade 5 Destructions (very heavy
structural damage) Total or
near total collapse,
Annexure 3
Users and Procedures accuracy.
OVERALL COMMUNES
Classification of Damage to masonry Buildings
Negligible to sligbt damage
(no Structual damage) Fine
cracks in plaster over frame
members and in partitions. Grade 1
LEOGANE
Moderate damage
(slight structural damage.
moderate non- structural
damage) Hair-line cracks
in columns and beams Grade 2
mortar falls from the joints of
suspended wall panels
cracks in partition walls fall
of pieces of brittle cladding
and plaster
GRESSIER
Substantial to heavy
damage (moderate
structural damage, heavy
non-Cracks in cotumns with Grade 3
detachment of pleces of
contrete cracls in beams
Very heavy damage CARREFOUR
(heavy structural damage,
very heavynon-strucural Grade 4
damage) Severe damage
to the joints of the building
skeleton with destruction
of concrele and protution
of reingorcing rods: partial
collapse, tilting of columns
47
Destruction (very heavy Grade 5
structureal damage) Total or
near total collapse.
PETION VILLE
 Annexure 4
DELMAS Four class and two class accuracy.

TABARRE

CITE-SOLIEL

CROIX -DES- BOUQUES

49

PORT-AU-PRINCE
Institute of International Health, Immunology & Microbiology
Faculty of Health Sciences, University of Copenhagen