accuracy assessment of post - earthquake building damage classification in haiti

7/22/2019 Accuracy Assessment of Post - Earthquake Building Damage Classification in Haiti

1/27


2/27

ACCURACY ASSESSMENT OF POST - EARTHQUAKE

BUILDING DAMAGE CLASSIFICATION IN HAITI

Master of Disaster Management (MDMa)Masters Thesis 200910

RAVI SHANKAR - Architect - Planner

Supervisor: Asst. Prof. Peter Kjr JensenCo-Supervisor: Michael Calopietro

Institute of International Health, Immunology & Microbiology

Faculty of Health Sciences, University of Copenhagen


3/27

My parents, my wife, and my son for their love and support &

to the Haitians who are waiting patiently.

I am grateful to my supervisor, Asst. Prof. Peter Kjr Jensen, a man who waves a magic wand. It

was a great pleasure working with my Co- Supervisor, Michael Calopietro, who made it possible

for me to write my thesis in absentia, away from Copenhagen.

It is an honour to work with the United Nations Operational Satellite Applications Programme

(UNOSAT) for my thesis and would like to thank our director, Francesco Pisano, and my

managers, Dr. Einar Bjorgo and Oliver Senegas, for their extended guidance. My friend Michael

Jendryke needs a special appreciation for his constant support and valuable input. I am indebted

to my colleagues Luca DellOro, Joshua Lyons, Wendi Pedersen, Fana Woldegebreal, Karl Nei-

man, and Paola, Fred and Oliver Vandamme for their constant help as well. This thesis would not

have been possible without assistance from Centre National de LInformation Go-Spatiale- Haiti

(CNIGS) in eld data collection, which was one of the most difcult parts of the whole exercise.

I would also like to thank Dr. Norman Kerle (ITC, Netherlands) and Dr. Keiko Saito (Cambridge

Architectural Resea rch Limited) for their valuable insi ghts. I would like to show my gratitude to

European Commission-Joint Research Centre (EC-JRC) and World Bank (WB) for sharing their

data. Finally, it is a pleasure to thank my University Department staff, classmates, and friends

from Danida Fellowship Centre, and all others who made this thesis possible.

Acknowledgements

to


4/27

Quantitative measurements of building damage are extremely important after an earthquake. An

integrated remote sensing and GIS-based rapid mapping approach can provide timely informa-

tion on building damages to be used by the international humanitarian community in planning aid

response. The unprecedented scale of damages wreaked by the 2010 earthquake in Haiti, com-

pounded by the difcult social and political conditions there, precluded widespread eld damage

assessment. Therefore, most of the preliminary damage assessments were carried out throughremote sensing images. Remote sensing imagery techniques are indispensable where difcult

ground conditions prevail, but there are limitations as well. More than 2000 maps were published

by different agencies within 75 days after the Haiti disaster. During this period, the variance in

the spatial data produced by these different agencies could have compromised relief efforts.

Naturally, then, there was a sense of urgency in assessing the accuracy of the information and

the methods used to obtain it.

This thesis investigates the accuracy of photo interpretation, one of the methods used to conduct

the virtual building damage assessment by the three main agencies involved: UNOSAT, World

Bank, and European Commission-Joint Research Centre. An error(confusion) matrix was used

to evaluate the accuracy of these assessments. The analysis was carried out to check overall ac-

curacy , accuracy per communes, accuracy per damage classes, and accuracy per each agency.

Finally, the assessment errors were investigated in detail in order to identify their causes, which

helped to quantify the limitations of using the remote sensing images. The results show thatthe overall accuracy was around 60% and the range varied between 36-82% among different

communes. Overall accuracy was higher with fewer numbers of damage classes. Post-mortem

analysis of errors suggests that there is a strong possibility the accuracy levels can be improved.

Abstract

5

source: google earth


5/27

Aerial photography Aerial photography is the taking of photographs of the ground from an elevated position. The term

usually refers to images in which the camera is not supported by a ground-based structure.

CNIGS Centre Nationale de lInformation Geo Spatiale

EC-JRC European Commission - Joint Research Centre

Damage assessment An assessment of the total or partial destruction of physical assets, both physical units and

replacement cost

ECLAC Economic Commission for Latin America and the Caribbean

GEER Geo-Engineering Extreme Events Reconnaissance

GIS Geographic Information System (GIS), or geographical information system, is any system

that captures, stores, analyzes, manages, and presents data that are linked to location. In

the simplest terms, GIS is the merging of cartography and database technology.

Loss assessment An analysis of the changes in economic ows that occur after a disaster and over time, valued at

current prices.

Housing damage assessment A damage assessment that analyzes the impact of the disaster on residential communities, living

quarters, and land used or housing

Rapid assessment An assessment conducted soon after a major event, usually within the rst two weeks. It may

be preceded by an initial assessment, and it may be multi-sector or sector-specic. It provides

immediate information on needs, possible intervention types, and resource requirements.

RS Remote sensing (RS) is the small- or large-scale acquisition of information of an object or

phenomenon through the use of either recording or real-time sensing device(s) that are

wireless, or not in physical or intimate contact with the object (such as by way of aircraft, spacecraft,

satellite, buoy, or ship).

UNOSAT UNOSAT is the UN Institute for Training and Research (UNITAR) Operational Satellite

Applications Programme, implemented in cooperation with the European Organization for

Nuclear Research(CERN).

WB World Bank

Abbreviations and keywords

7


6/27

1 INTRODUCTION

1.1 Background ..................... ...................... ..................... ...................... ...................... ...... 11

1.2 The Earthquake ..................... ...................... ..................... ...................... ...................... 12

1.3 The Critical Period................. ...................... ...................... ...................... ...................... 13

1.4 Post-Disaster Needs Assessment............. ...................... ...................... ...................... .. 14

1.5 Role of GIS in Disaster Management .................... ...................... ...................... ........... 15

1.6 Disaster Management and Remote Sensing ..................... ...................... ..................... 15

1.7 Use of RS Images after Haiti Earthquake .................... ...................... ...................... ..... 16

1.8 Methods of Damage Assessments........... ...................... ...................... ...................... ... 17

1.9 Damage Catalogue................ ...................... ...................... ...................... ...................... 18

1.10 Statement of Problem.................. ...................... ...................... ...................... .............. 19

2 OBJECTIVES

2.1 Main Objectives........... ...................... ...................... ...................... ...................... .......... 21

2.2 Hypothesis............. ...................... ...................... ..................... ...................... ................. 21

2.3 Specic Objectives ..................... ...................... ...................... ...................... ................ 21

3 METHODS

3.1 Introduction .................... ...................... ...................... ...................... ...................... ....... 23

3.2 Sample Selection ...................... ...................... ...................... ...................... .................. 24

3.3 Data Collection .................... ...................... ...................... ...................... ...................... . 25

3.4 Quality Assurance.............. ...................... ...................... ...................... ...................... .... 25

3.5 Methods for Accuracy Assessments ...................... ...................... ...................... ............ 26

3.6 Creation of Database and Analysis Approach ................... ...................... ...................... 27

4 FINDINGS

4.1 Overall Findings............ ...................... ..................... ...................... ...................... .......... 31

4.2 Variations by Communes................. ...................... ...................... ...................... ............ 33

4.3 Four-Class and Two-Class Variations ...................... ...................... ...................... ......... 34

4.4 Postmortem of Failed Class ..................... ...................... ...................... ...................... ... 35

5 DISCUSSION

5.1 Methodology.................... ...................... ...................... ...................... ...................... ...... 37

5.2 Overall Communes ...................... ...................... ...................... ...................... ............... 38

6 CONCLUSION & RECOMMENDATIONS 41

7 BIBLIOGRAPHY 43

8 ANNEXURE 45

Figure 1 Map of Haiti.

Figure 2 Area of populations affected by the 2010 earthquake.

Figure 3 Relief, Response, Recovery and Reconstruction phases as described by World Bank.

Figure 4 Sequence of the Post-Disaster Needs Assessments.

Figure 5 Published maps after 2010 Haiti earthquake.

Figure 6 Various methods of damage assessments.

Figure 7 Damage catalogue as used by various agencies for damage assessments in Haiti.

Figure 8 The pre- and post-images with faulty building damage assessments.

Figure 9 Showing various damage classes (destroyed to no visible damage) as adopted by UNOSAT.

Figure 10 Percentage composition of the assessment among different agencies.

Figure 11 Methodologies of accuracy assessments.

Figure 12 Showing random samples collected for eld validation.

Figure 13 Showing the theory of error matrix

Figure 14 Showing the methodology adopted for accuracy assessment.

Figure 15 Showing comparison on four and two class for various agencies.

Figure 16 Showing kappa statistics for various communes.

Figure 17 Showing kappa statistics for various agencies

Figure 18 Showing accuracy levels comparison between four and two class.

Figure 19 Showing logic behind post mortem of failed class.

Table 1 Buildings Assessed in Various Communes.

Table 2 Field Samples Collected by CNIGS from Various Communes.

Table 3 Overall ndings .

Table 4 Range on Kappa Statistics Agreement

Table of contents List of figures

List of tables 9


7/27

1.1 Background

Haiti is located in the Caribbean Sea on the Island of Hispaniola, which it shares with the Domini-

can Republic (Fig. 1). Haiti occupies the western third of the island and has 8.4 million inhabitants

and a population density of 300 people per km2 (Canadian International Development Agency,

2005). Haiti is the poorest country in the Western Hemisphere with an estimated 80% of its people

living under the poverty line; 54% live in abject poverty (CIA, 2010). When it comes to natural dis-

asters, Haiti seems to have a bulls-eye on it. There were eight major natural disasters within last 15

years. Floods during 2008 hurrican e killed more than 2500 people.

Haiti is a twentieth-century Greek tragedy of Malthusian prophecy run wild, and little time remains

to reverse the course toward destruction of what once was the lush tropical beauty of the Pearl of

the Antilles (Ernest H, 1996). High population density, chaotic urbanizatio n, a weak political struc-

ture, and poorly developed disaster-management systems are some of the factors contributing

to the vulnerability of its people.

Figure 1. - Map of Haiti

11

INTRODUCTION

1


8/27

1.2 The Earthquake

The earthquake that struck the Republic of Haiti on January 12, 2010, is one of the most signifi-

cant earthquakes in recorded history. Destruction and death on a huge scale unfolded across

the densely populated Haitian capital, Port-au-Prince, and into the shanties and villas of the sur-

rounding mountains. As of February 16, 2010, the death toll reported by the government of Haiti

exceeded 217,000 with an additional 300,000 injuries. More than 5 million people live in the area

directly affected by the earthquake (Fig. 2), and 1.2 million people are now living in temporary

shelters (UN, 2010).

Most of the buildings are masonry either brick or construction blocks which tend to perform badly

in an earthquake and with sub standard construction techniques worsened the situation. Damage

to infrastructure was extensive and ministry of education estimated that more than 15,000 primary

schools and 1500 secondary schools were severely damaged. Most of Port-au-Princes municipal

government buildings were destroyed or heavily damaged in the earthquake, including the City

Hall, which was described by the Washington Post as, a skeletal hulk of concrete and stucco,

sagging grotesquely to the left(Manuel Roig-Franzia, 2010).

1.3 The Critical Period

After t he ea rthquake, the government of H aiti led t he emerge ncy reli ef operat ion with su pport

from the international community. Preliminary physical damage assessments on buildings were

out within days after the earthquake from various agencies. In its handbook on disaster man-

agement, recovery, and reconstruction, the World Bank describes the initial 25 days as the most

crucial period for disaster relief and damage loss assessment (Fig. 3; ECLAC & World Bank, 2003).

Major response guidelines from the World Bank suggest that the following activities take place

during the initial phase,

Disaster Response

Disaster Relief

Disaster and loss assessments

The decisions on reconstruction in Haiti were based on these timely assessments along with the

updated field results.

Figure 2.- Area of populations affected by the 2010 earthquake.

Figure 3. Relief, Response, Recovery and Reconstruction phases as described by World Bank.

13


9/27

1.4 Post-Disaster Needs Assessment

The Post-Disaster Needs Assessment and Recovery Framework (PDNA) is an interagency mecha-

nism set up to facilitate a comprehensive, coordinated, and efficient international approach to

damage, loss, and needs assessment and recovery planning. The Recovery Framework outlined

by the PDNA facilitates integrity of damages data and helps to ensure that financing needs and

the recovery pathway are matters of mutual agreement. PDNA is designed to address one or

more of the following issues (Darcy and Hofmann, 2003):

Whether to intervene.

The nature and scale of the intervention.

Prioritization and allocation of resources.

Programme design and planning.

PDNA is not intended to replace more in-depth assessments, which may still be required outside

of the PDNA for refining the understanding of needs, generating response options, and develop-

ing recovery projects within a given sector. In Haiti, the PDNA is a government-led exercise, with

integrated support from the United Nations, the European Commission, the World Bank, and other

national and international partners. A PDNA prepares a single consolidated report containing

crucial information on the disaster: its physical impacts; the economic value of the damages

and losses; the human impacts as experienced by the affected populations; and the early and

long-term recovery needs and priorities. The type and scope of the disaster affects the develop-

ment vision. Based on this assessment, various responses are proposed, analyzed, and selected

for inclusion in the Recovery Framework (Fig. 4).

1.5 Role of GIS in Disaster Management

Geographic Information System (GIS) is a computerized method of handling the spatial data,

which includes in formation system and remot e sensing data. In simple terms, GIS is a mergi ng of

the cartography and database technology. GIS is significant to all stages of the disaster manage-

ment cycle. It provides for efficient and rapid management of data, as well as data manipulation

and analysis, thereby effecting better decision making. GIS can be an essential support system

to estimate the loss after a disaster. In the present context of Haiti, humanitarian organizations

fully exploited the role of GIS in disaster management. Haiti represents a milestone in the use

of GIS in the post-disaster cycle, as in agencies relied upon it more than ever before to collect,

process,and analyze the spatial information gathered.

1.6 Disaster Management and Remote Sensing

Remote sensing (RS) is the small- or large-scale acquisition of information of an object or phe-

nomenon using either recording or real-time sensing devices that are wireless, or otherwise not in

physical or intimate contact with the object (such as by way of aircraft, spacecraft, satellite, buoy,

or ship) (Lillesand et al., 2003). The critical period after the disaster is the response phase, and

the field data collected during this period, if it is possible to do so, is very valuable. However, it is

a difficult task to collect data during this period from the ground. But remote sensing technology

can accommodate this lag in collecting field data.

Using remote sensing technology, the damage concentration in the urban areas can be located

in a shorter time compared with the conventional ground survey (Rao, 2000). An unprecedented

international satellite imagery campaign with the participation of all leading Earth observation

operators was undertaken after the destructive Haiti earthquake. Annexure 1 shows the various

remote sensing accusations that took place after the earthquake. Later, high-resolution aerial im-

ages of the capital city of Port-au-Prince of up to 15 cm spatial resolution were taken for more pre-

cise assessments of building stability. Never before have such high- resolution spatial images and

aerial photography been so accessible. Multi-source data were collected from various missions:

WB-Image CAT-RIT Remote Sensing Mission (LiDAR optical data, 15 cm & 2 pt/m), Google (optical

data, 15 cm), NOAA (optical data, 25 cm), and Pictometry (Haiti_Reconnaissance.pdf, 2010).

Figure 4. Sequence of the Post-Disaster Needs Assessments.

15


10/27

1.7 Use of RS Images after Haiti Earthquake

There were 2000+ maps* published within the first 75 days of the Haiti earthquake. More than

50 agencies were directly involved in the preparation and publication of the maps. Most of the

maps detailed earthquake coverage, affected populations, and damage assessments, and

some offered hazard analysis and cluster- or sector related data. There were around 10 agencies

involved in the building damage assessments. With the Post-Disaster Needs Assessment requir-

ing building damage extents for Haiti, these assessments happened to be a valuable exercise.

The UN Institute for Training and Research (UNITAR), the Operational Satellite Applications Pro-

gramme (UNOSAT), the European Commission -Joint Research Centre (EC-JRC), the National

Geospatial Information Centre (CNIGS) representing the Haiti Government, the World Bank (WB),

and the Global Facility for Disaster Reduction and Recovery (GFDRR), supported by its consult-

ant Image CAT, have carried out a detailed, building-by-building assessment of the damage.

This analysis relied heavily on the use of space remote sensing techniques. The joint agreement

among UNOSAT, WB, and EC-JRC to publish a common database for all of Haiti must have been

a welcome relief for the humanitarian community.

1.8 Methods of Damage Assessments

Participating agencies used different approaches in assessing the extent of building damage.

Predominant among them was the photo interpretation method. Figure 6 shows various methods

of damage assessment made possible by remote sensing images. Photo interpretation method

is theoretically one of the simplest methods of building damage assessment using RS images.

The analyst uses GIS software where the pre- and post- satellite/aerial images are loaded, and a

new database is created to record the assessments. The analyst visually identifies the extent of

damage by comparing the pre- and post- images and records it into the database. Though the

method sounds simple, it is time consuming and labour intensive. And there are drawbacks to

the technology, as described by UNOSAT: Satellite imagery generally provides an overhead view-

point. Lateral damage and especially damage to the internal structures of buildings, is not de-

tectable from the analysis of satellite imagery (2010). In the manual photo interpretation method,

the quality of the assessment might depend on the expertise of image analysts working away from

Haiti, who are better equipped for the task if they are knowledgeable about Haitis construction

techniques, building materials, and housing typology...

Figure 5. Published maps after 2010 Haiti earthquake.

Figure 6. Intensity of Damage assessment as published by UNOSAT.

* Authors estimate based on the relief web website (www. reliefweb.int)

Figure 7. Relief, Response, Recovery and Reconstruction phases as described by World Bank.

17


11/27

1.9 Damage Catalogue

The important aspect of building damage assessment is the expression of the damage levels.

The damage catalogue is the scale which portraits the intensity of the damage, with categories

usually ranging from no visible damage to destroyed. There are several international field dam-

age assessment scales (e.g.European Macroseismic Scale-98[EMS-98], ACT ) that quantify the

extent of damages after an earthquake. It is very clear from the published maps of Haiti that the

participating agencies used different damage catalogues for assessment. UNOSAT and EC-JRC

used four classes in its damage assessments whereas WB used just two classes for the damage

assessment (Fig. 8).

Figure 8. Damage catalogue as used by various agencies for damage assessments in Haiti.

Figure 9. Showing various damage classes (destroyed to no visible damage) as adopted by UNOSAT.

A- UNOSAT , B- EC-JRC/ISFEREA, C- DLR,GERMAN SPACE AGENCY, D- SERTIT E- ITHACA & E-Geos

Figure 10. The pre- and post-images with faulty building damage assessments.

1.10 Statement of Problem

With fast spreading application o f GIS using remotely sensed data and ever increasing evaluat ion

of the damages using these technologies, it is essential that the accuracy of these products be

assessed. Accuracy assessments evaluate either a single method or various methods used to col-

lect spatial data in the aftermath of the disaster. The accuracy assessment allows the analyst to

quantitatively compare different methods. . The cross comparisons should identify the processes

that yield the most accurate results, information critical for decision making in future disasters.

The ultimate objective should be to make accuracy assessment part of the metadata in the later

updates of published maps. Most of the maps produced after the disaster as rapid mapping or

damage assessment had not been subject to extensive accuracy assessments. It appears there

was often a casual attitude toward these maps, in part because they had been prepared so hast-

ily in the first place. In some instances, it seemed the appearance of the map was valued over

its accuracy.

19

The sample validation in field by GEER (Geo -

engineering Extreme Events Reconnaissance)

stated However, it must also be accepted with

some limitations as these types of surveys do not

always capture the magnitude of destruction on

the ground. As an isolated example, consider

the two steel frame warehouses ........ Both of

these buildings were heavily damaged and will

need to be torn down, yet they were classified

as No Visible Damage by the UNOSAT survey.

Ellen Tathje, Jeff Bachhuber, et, all, GEOTECHNICAL

ENGINEERING RECONNAISSANCE OF THE 2010 HAITI

EARTHQUAKE, 22 Feb 2010.


12/27

OBJECTIVES

2

2.1 Main Objectives

The aim is to evaluate the accuracy of photo interpretation methods of remote/virtual building

damage assessment in the aftermath of the Haiti earthquake. The study will focus on the assess-

ments of the three main coordinating agencies: UNOSAT, EC-JRC, and WB.

2.2 Hypothesis

There is no difference in accuracy of preliminary damage assessments among different com-

munes and agencies for the Haiti earthquake.

2.3 Hypothesis

. To evaluate the overall accuracy of building damage assessments for Haiti.

. To compare the accuracy of building damage assessments by commune.

. To compare the accuracy levels among the various agencies.

. To compare the accuracy levels of the four and two-class building damage classification.. To perform a post-mortem analysis of the errors of the building damage assessment.

21


13/27

METHODS

3

3.1 Introduction

There were around 400,000 building points in Haiti which had been under the direct influence of

the Jan 12th earthquake. The study area encompasses communes of the Artibonite region. In

these, Port-au-Prince, Cite-Soleil and Carrefour are the most densely populated areas with the

maximum population spread. It is also the area of the greatest economic development in Haiti. In

total, 290,672 building points were assessed by these three organizations. UNOSAT covered 83%

of the total assessment with EC-JRC responsible for about 12 % and WB for the remaining 5 % (Fig.

9). World Bank assessed only the Destroyed and Severely damaged classes whereas UNOSAT and

EC-JRC assayed four classes, Destroyed, Severely Damaged, Moderate Damage, and No Visible

Damage.

NAME OF COMMUNE NO OF BUILDINGS ASSESSED ORGANIZATION

CROIX-DES-BOUQUES 3562 WORLD BANK

CITE-SOEIL 6709 WORLD BANK

CARREFOUR 69384 UNOSAT

LEOGANE 23790 EC-JRC

DELMAS 64394 UNOSAT

TABARRE 3919 WORLD BANK

PETIN- VILLE 10587 UNOSAT / WORLD BANK

GRESSIER 1347 WORLD BANK

PORT-AU-PRINCE 92432 UNOSAT

JACMEL 13133 EC-JRC

PETIT-GOAVE 409 WORLD BANK

GRAND-GOAVE 273 WORLD BANK

TOTAL 290672

Table 1 Building Assessed in Various Communes.

Figure 11. Percentage composition of the assessments among

the different agencies.

EC-JRC

36284

12%

WB136865%

UNOSAT

240702

83%

23


14/27

3.2 Sample Selection

The earthquake affected areas of Haiti was divided into 200X200 m cells. To be considered for a

sample selection, each cell had to satisfy certain criteria. Cells were sorted based on the follow-

ing three steps:

1.Each cell was assigned either high or low density as per existing population.

2. Each cell was also assigned high or low damage intensity based on damage assessment.

3. A land cover map was relied upon for uniform representation within each land use.

Analysis using these criteria identified around 2150 cells from whic h samples were t o be cho-

sen. A stratified sampling method was then used to further refine the sample-cell collection. To

determine the sample size to be selected from the 2150 retained cells, confidence level and

confidence interval were defined as prior criteria and calculations were performed accordingly.

A 95% confidence level a nd a 10.5 confidence interval were select ed. Hence , the computed

sample size was equal to 86 cells, representing 4% of the population. Finally, the 86 cells selected

randomly among the 2150 previously stratified cells in a number proportional to the stratums size.

A field quest ionnaire was prepared on the basis of European Macroseismic Scale-98, and every

building from the chosen sample was graded accordingly.

3.3 Data Collection

In total, around 7000 building points were collected in the field by CNIGS, who were entrusted

with t hat work. Samples from the co mmunes of Jacmel, Petit Goane, and Grand Goane were

not collected for various reasons, including damages, accessibility, and distance from the base

station. Even though little pre-processing was undertaken in the site office at Haiti, this was one of

the most time consuming components of the whole project.

These also include the samples collected by UNOSAT/EC-JRC during the initial eld visit to Haiti.

Table 2 Field samples Collected by CNIGS from Various communes.

25

NAME OF COMMUNE NO OF FIELD SAMPLES NO OF BUILDINGS

ASSESSED VIRTUALLY

CROIX-DES-BOUQUES 343 3562

CITE-SOEIL 243 6709

CARREFOUR 1305 69384

LEOGANE 97 23790

DELMAS 735 64394

TABARRE 648 3919

PETIN- VILLE 464 10587

GRESSIER 50 1347

PORT-AU-PRINCE 2607 92432

CNIGS FIELD DATA 6493

UNOSAT/ EC-JRC FIELD DATA 442

3.4 Quality Assurance

Field samples were scrutinised during the preprocessing phase for quality assurance. The preproc-

essing for error checking included:

1. Sample points outside the building boundary or on the road

2. More than one sample point within the same building or more than one virtual assess-

ment point for a given building

3. Site photos giving extreme class variation in comparison to field assessment.

After the in itial preprocessing of the sample for possible errors, the size of the bui lding points wasreduced to 6492. This covers roughly 2% of virtual assessed building samples.

Figure 12. Showing random samples collected for eld validation.


15/27

Figure 13. Showing the theory of error matrix

3.5 Methods for Accuracy Assessments

To evaluate the accuracy levels of the remote damage assessments an error( confusion) matrix

is used. It compares category-by-category, based on damage class/catalogue, the relationship

between the produced and known reference data of field truth. Discrete Multivariate analysis

technique is used following the error matrix with the assumption that these categories are de-

pendent and do not require any transformation. A Khat value (K^) will be calculated for each

agency, which is a measure of how well the classification agrees with the field data. Khat can

be tested using the normal curve deviate to determine if the two error matrixes are significantly

different.

3.6 Creation of Database and Analysis Approach

The following steps were carried out for data analysis.

Step 1: The joint database was created by UNOSAT combing the individual data sets of

UNOSAT, EC-JRC, and WB.

Step 2: The data received from the field was subjected to preprocessing as explained

before. After removing the field samples which did not meet the set criteria,

a new field database was created.

Step 3: The joint database was then merged with the field database. This created an

intermediate data base.

Step 4: The intermediate database was further divided into shape files as per communes

for 4-class and 2-vclass building classification.

Step 5: Accuracy assessment script was run using ESRI ARCGIS 9.3.1 software followed

by Kappa statistics script.

Step 6: The results are displayed and postmortem of the failed class is then carried out.

27


16/27

Figure 14. Showing the methodology adopted for accuracy assessment.

29


17/27

FINDINGS

4

4.1 Overall Findings

Out of the 5806 points organized after preprocessing the field data, there were only 4351 virtu-

ally assessed points matching with the field points. This means that more than 1455 points were

not assessed during the virtual building assessment. Thus, for Haiti, the virtual/remote building

damage assessment was performed for only 75% of the total buildings that were affected within

the earthquake impact zone. A number of factors may have contributed to this oversight. Therewere si gnificant problems defining the various classes of buildings and indeed what constitut-

ed a building, especially given the makeshift character of many of the settlements. A number

of conditionals were exacted upon the labelling causing further confusion. Also, there was a

troubling lack of pre-earthquake data establishing building parcel boundaries. One could ar-

gue that this is generally the case with most of the disasters that occur in developing or under-

developed countries. Had building parcel boundaries been available, damages could have

been associated with the building parcel rather than with the building itself. Some of the screen

shots illustrate that individual analysts had clearly failed to mark the buildings in most instances.

31

CROIX-DESBOUQUES 35 11.5 31 63 10.5 25 29 11

CITE-SOEIL 44 25 38 72 20.29 21 20 15

CARREFOUR 67 37 38 79 21.28 12 9 12

LEOGANE 46 40 38 78 25.40 25 18 9

DELMAS 82 31 40 94 20.93 6 8 4

TABARRE 76 32 36 92 26.46 11 6 7

PETINVILLE 61 24 25 94 - 14 12 13

GRESSIER 56 - 35 76 12.73 25 13 8

PORT-AUPRINCE 56 39 40 84 25.29 20 14 9

OVERALL COMMUNES 61 36 39 86 23.06 - - -

NAME OF COMMUNE

4 class Post mortem of errors

Accuracylevels

Usersaccuracy

%

Producersaccuracy

%

2 classAccuracy

levels %

Kappastatistics

%

Errorsby

analyst %

Doubtful cases

or classconfu-sion %

Damagescould not

bedetectedthrough

RSimagery %

Table 3. Overall ndings .


18/27

4.2 Variations by Communes

For Port-au-Prince, there were around 92,423 buildings which were virtually assessed. Of the to-

tal field samples collected for Port-au-Prince, 2073 virtual assessments matched with the cor-

responding field sample, and the overall accuracy was found to be 56%. Figure 17 charts the

performance accuracy of the user and the producer, which dips sharply in the Moderate class.

Kappa statistics was around 25.29%. The accuracy level went up to 28% when the error matrix

was applied with just two classe s. Furthermore, the postmortem of th e errors showed th at around20% of them could have been avoided by the analyst and around 9% occurred due to practical

impossibilities with the aerial images.

Figure 16. Showing kappa statistics for various communes.

33

Of 290,672 buildings virtually assessed, we had a field sample of 1.5 %, or 4329 buildings. The

accuracy of the joint database was found to be 61%. The Moderate Damage class was the most

poorly performed class in the entire assessment, with the Kappa statistics around 23%. The accu-

racy level increased by 25% when the error matrix was applied with just two classes. Furthermore,

the postmortem of the errors showed that around 15-20% of errors could have been avoided by

the analyst and that around 10% obtained because of practical impossibilities with aerial images

as applied to building damages. Further analysis on the accuracy levels for each organization

showed that UNOSAT performed better in the overall accuracy, followed by World Bank first and

then EC-JRC. For the Kappa statistical analysis, EC-JRC performed slightly better the other two

organizations.

Figure 17. Showing kappa statistics for various agencies

Figure 15. Showing comparison on four and two class for various agencies.


19/27

4.3 Four-Class and Two-Class Variations

It would be interesting to see how the accuracy levels behaved if the entire damage assessment

was carried out with just two classes, Destroyed and Not Destroyed. It was evident from the start

that the four-class approach was an ambitious and time consuming undertaking. If the accuracy

is better for two classes than four classes, then it would be wise to start with two damage classes

immediately after the disaster and then later improvise with more classes. Figure 18 is a hypotheti-

cal scenario to study the accuracy assessment with two classes; slight adjustments were made in

combining the EMS-98 scale.

4.4 Post-mortem of Failed Class

The question confronting us is how the results of the accuracy assessment can lead to a construc-

tive discussion of ways to improve the reliability of the numbers..It is important to discover why it

failed and what are the chances of improving it.

From the previous sections we know that there are limitations with the aerial images and that the

outcome of the photo interpretation method is heavily dependent on analyst performance. Fig-

ure 19 shows the three most likely factors producing the failed results. They are:

1.Building damages which could not be detected through remote sensing imagery, the

main limitation of using the aerial images.

2.Errors which could have been avoided by the analyst.

3.Doubtful cases or class confusion.

If we could assess postmortem the failed classes in a more refined manner, researchers would

have a better way of gauging the potential accuracy of aerial imagery assessment for disasters

of similar scale in the future. The failed assessment points need to be analysed individually with

the pre-and post-images. This by itself is a time consuming exercise if the accuracy levels are very

low.

FOUR CLASS % TWO CLASS %

PETION VILLE

ALL COMMUNES CROIX DES BOUQUES

DELIMASCIRE SOLIEL

TABRRE LEOGANE

CARREFOUR

GRESSIR

PORT-AU-PRINCE

56

5667

76

82

61

61

35

44

46

100

90

80

70

60

50

40

30

20

10

94

92

89

74

84

78

72

63

94

86

35Figure 18. Showing accuracy levels comparison between four and two class.

Figure 19. Showing logic behind post mortem of failed class.


20/27

DISCUSSION5

5.1 Methodology

Binomial distribution was used for the sample size calculation, and the majority of the research-

ers are using the same for error matrix or the normal approximation of the binomial distribution to

compute required sample size. This seems to be a standard practice. According to Congalton,

a general rule of thumb is a minimum of 50 samples for each category in error matrix; our sam-

ples meet that requirement. But it was unfortunate that we could not have samples collected for

Jacmel, Petit Goave, and Gran Goave communes where EC-JRC was responsible for the asse ss-

ment.

Another interesting as pect of the sample is the use of lan d cover maps with land use data as one

of the input parameters. The land use map is a gross approximation based on land cover. The

use of the virtual building damage assessment as an input criteria could be debatable depend-

ing on whether it will influence the sample selection. But considering the use of this at the grid

level to ensure the sample is spread evenly on the study area is not a bad idea in and of itself

The knowledge and expertise of the field sample collectors in using the EMS-98 scale is question-

able even though UNOSAT and EC-JRC had briefed them in detail about the scale. Had structural

engineers or other experts collected the field sample, they would have had different results. But

we have to admit that CNIGS were the best availabl e choice gi ven their local knowledge of the

situation.

The accuracy of the GIS and the other manual errors associated with the field sample collection

is an issue to be discussed. From the field data, around 300 samples were discarded during pre-

processing either because they were marked on the streets or away from the building footprints or

because there were two samples with varying class over the same building footprint. Even though

this exercise was tedious, it was necessary for quality assurance on the field sample and, in addi-

tion, personally rechecking the samples improves the confidence level of the data. Among other

practical issues tied to the CNIGS side of the data collection are a lack of access to certain grids

due to nonexistent roads and relatively high slopes, difficulty differentiating one building from

another even on a map 1/1000, and problems sending photos due to low speed Internet con-

nections.

As we saw in the in troduction, more than 10 agencies were involv ed in the v irtual da mage as-

sessment for Haiti, and when I looked closely at their maps, I realized that there was no similarity in

defining the damage in the Legends. It might be impossible for a person in the field to compare

these maps and to achieve common understanding of the damages based on the damage

classes. There could be a need for closer inspection here to determine whether it is possible to

develop a standardized catalogue in rapid mapping of virtual damage assessment situations.

In conjunction with this, we saw that it is more difficult to coordinate viable postmortem analyses

with common field data produced by different agencies relying on different interpretations of

damage classes.

37


21/27

5.2 Overall Communes

The error matrix has become widely accepted as the standard method for reporting the accuracy

of the GIS data derived from the remote sensing data. The analysis in the previous section very

clearly indicates accuracy levels fluctuating between 35% and 82%. This wide range of band in

the accuracy makes it difficult to draw direct conclusions from the results. The overall accuracy

from the error matrix for all the communes put together for all four classes was aroun d 60% . This

rate was much improved by 25 % with just two classes. This is a valuable lesson as far as the dam-

age classes are concerned. Directly after the disaster, it would be wiser to confine the study to just

two classes and then later expand it to more classes if required.

We know t hat Kappa statistics is a measure of agreement rather than the percent age of occu r-

rence. Landis and Koch have provided instructions for interpreting K values (Table 3). This is not a

proven frequency but more of a guideline. The average of the Kappa statistics for all the com-

munes put together is around 21%, just barely falling into the category of fair agreement. This

result raises questions about the strength of the photo interpretation method since almost all the

agency values are around 20%. The perspective of the individual analyst assigned to damage

assessment, myself included, is important to any analysis of the reasons for the errors. My experi-

ence indicates that many of the problems derived from ambiguous building definitions. The pro-

cess would have gone more smoothly if building definitions had been agreed upon before the

start of the exercise.

KAPPA VALUES INTERPRETATION

< 0 No agreement

0.0 0.20 Slight agreement

0.21 0.40 Fair agreement

0.41 0.60 Moderate agreement

0.61 0.80 Substantial agreement

0.81 1.00 Almost perfect agreement

As I mentioned in the introd uction, without spatial data o n the property boundaries and land use

data, it is a complicated situation to define a building. I will provide some specific examples.

In the case of a building apparently not affected but in the midst of a destroyed area, analysts

struggled to choose between the moderate and severely destroyed classes. There were also

situations where buildings were marked as destroyed when in fact they were under construction.

Closer attention to the pre- and post- images was required to ascertain the status of the building.

Depending on the length of time between the pre-scenario and the post-scenario, there was a

chance the house had been completed during this gap, thus creating a wrong entry as under

construction. In some instances, the presence of elements like internally displaced camps in the

streets or road blocks gave a false impression that the buildings around it had been affected by

the earthquake. But we realized that people were moving out of their houses into the streets fear-

ing aftershocks, which was another reason the analyst might make a wrong decision on the as-

sessments. Shantytowns or urban slums posed a significant challenge to the analyst owing to the

dense nature of the buildings and their poor quality of construction with unconventional building

materials.

Table 4. Range on Kappa Statistics Agreement

39


22/27

CONCLUSION & RECOMMENDATIONS6

CONCLUSION & RECOMMENDATIONS

This study offers some valuable lessons for future damages assessment analysis. The data reveal

the limitations of aerial images and especially photo interpretation method in building damage

assessments. The reliability of virtual damage assessment through photo interpretation is put intoquestion and leads us to ponder the best method going forward. On a more detailed level, the

study elaborated the specific errors to which may be attributed the low percentage of accuracy.

The four-class damage categorization could have been avoided and the whole damage as-

sessment instead progressed from the start with two classes. This would have allowed the analyst

more time, and many of the class confusion errors could have been avoided as well. If the initial

digitalization of the building points (without error assessment) had been out sourced completely

via crowd sourcing or with academic institutes who were willing to participat e, overall results could

have been improved, as happened in the case of the Delmas area of Port-au-Prince. The field

data collection could have been better coordinated to avoid repetition of the preprocessing of

the field data both by CNIGS and UNOSAT. (For example, building points were moved by CNIGS

over the building footprints after the field survey, but they were not snapped over the virtual as-

sessment points; this work was carried out by the author.) For UNOSAT, the labour shifts might have

been better organized, and a detailed training catalogue would have been welcomed by theanalysts.

Time constraints and the unavoidable confusion attendant upon a disaster of this magnitude

precluded analysts from exploiting the full potential of such a study. While much can be gleaned

from the analysis, circumstances did not allow other desired avenues of investigation: more ex-

tensive review of the existing literature; samples collected from the Jacmel, Petit-Goave, and

Grand Goave communes; a comparison of accuracy levels of satellite imagery of damages and

aerial imagery; refined procedures for sample collection and sample size distribution.

My research points to the importance of standardizing the building damage catalogue. This could

possibly require a detailed research and intervention in the rapid mapping community. Research

in this field should also look to alternative methods of damage assessments using high resolution

imagery like object-based image analysis.

41


23/27

BIBLIOGRAPHY7

1. Canadian International Development Agency. (2005). Haiti: facts at a glance. .

2. CIA (2010). The Central Intelligence Agency World Factbook. .

3. Congalton R. G. 1991. A review of assessing the accuracy of classifications of remotely sensed

Data. Remote Sensing of Environment.

4. Darcy, J. & Hofmann, C. (2003). Needs assessment and decision-making in the humanitarian

sector. London: Humanitarian Policy Group, Overseas Development Institute.

5. Eberhard, M. O., Baldridge, S., Marshall, J., Mooney, W., & Rix, G. J. (2010). USGS_EERI_

HAITI_V1.1.pdf (application/pdf Object). Web. 00/00/10.

6. Ernest, H. Preeg (1992). The Haitian Dilemma. International Studies.

7. Geospatial Data Availability for Haiti: An Aid in the Development of GIS-Based Natural

Resource Assessments for Conservation Planning.

8. Haiti_Reconnais sance.pdf. (2010). http://www.eqclearinghouse .org/20100112- haiti/wp-con-

tent/uploads/2010/02/Haiti_Reconnaissance.pdf.

9. Handbook for Estimating the Socio-economic and Environmental Effects of Disasters. (2003).

ECLAC & World Bank http://www.gdrc.org/uem/disasters/disenvi/VOLUME%20I.pdf. Wiley. ISBN

0-471-15227-7.

10. Lillesand, T.M, Kiefer, R.W., and Chipman, J. W. (2003). R emote sensing and image

interpretation (5th ed.).

11. Manuel Roig-Franzia(2010). Washington post, retrieved from http://www.washingtonpost.com/

wp-yn/content/articl e/2010/01/19/AR2010011904614.html

12. Rao, (2000) Disaster management

13. UN (2010). United Nations Office for the Coordination of Humanitarian Affairs. Haiti Earthquake

Situation Report #21, February 16, 2010.

14. UNOSAT.(2010).Atlas of building damage asse ssment, http://unosat .web.cern.ch/unosat/

shared/Haiti-HQ-2010/maps/PDNA/PDNA_atlas_intro.pdf.

43


24/27

ANNEXURE7

Annexure 1

Satellite Tasking Satellite Sensor Time (UTC) Area Type

1 SPOT 5 14&19 Jan Port-au-Prin ce

2 BJ-1 SensonML (red band /

infrared band) 13-Jan Port-au-Prin ce

3 ALOS AVNIR-2 13-Jan

PALSAR 16-Jan

AVNIR-2 23-Jan

PRISM 23-Jan

PAN 23-Jan

4 DigitalGlobe multiple dates

WV-01 13&14 Jan multi ple

collection dates Mapping entire region B/W

WV-02 14 Jan multiple

collection dates Port-au-Prin ce Multi

spectral

5 QuickBird multiple collection

dates Port-au-Prince

6 Ikonos 14&15&17 Jan

7 GeoEye-1 13&16&18 Jan

8 Cosmo-Skymed 15-Jan

9 Formosat-2 13&14&15&16 &17

Jan Port-au-Prince

10 Radarsat-2 14&15 Jan Port-au-Prin ce

11 HJ-1-A/B CCD, Hyper-spectra l

camera, Infrared

camera 14-Jan

12 RapidEye 13&14&15& 17

Jan full country

coverage

13 EO-1 15-Jan Port-au-Prince ALI

14 EROS-B 17-Jan Port-au-Prin ce

45

Source: http://www.un-spider.org/haiti

Satellite tasking after the earthquake.


25/27

Annexure 2

Negligible to sligbt damage(no Structual damage) Hair-line cracks in very ferw wallsfall of smallpleces of laasteronly. Fall of loose stones fromupper parts of buildings invery frw class

Moderate damage(slight structural damage.moderate non- structuraldamage) Cracks in manuwalls fall of fairly large piecesof plaster. parts of chlmmysfall down

Substantial to heavydamage (moderate

structural damage, heavynon- structural damage)


26/27

CROIX -DES- BOUQUES

TABARRE

CITE-SOLIEL

PORT-AU-PRINCE

DELMAS

Annexure 4

49

Four class and two class accuracy.


27/27

Institute of International Health, Immunology & Microbiology

Faculty of Health Sciences, University of Copenhagen

accuracy assessment of post - earthquake building damage classification in haiti

Documents