Copyright, 1998-2013 © Qiming Zhou

GEOG3610 Remote Sensing and Image Interpretation

Accuracy Assessment and GPSAccuracy Assessment and GPS

2

Global positioning systemSpatial accuracy assessmentRepresentational accuracy assessmentTemporal accuracy assessmentError propagation and modelling

Accuracy assessment and GPS

3

Global positioning system

Accurate positioning is fundamental for accuracy assessment in remote sensing.

NAVSTAR (USDOD): Global Positioning System (GPS) fully operational in 1994 24 orbiting satellites (21+3) positioned in 6 evenly spaced orbital planes standard position service (SPS) and precise

positioning service (PPS) Other positioning satellites have also been launched by

different nations, e.g. Galileo project – EU Bei-dou - China

4

Use of GPS

Use of a Global Positioning System (GPS)

5

GPS positioning

Satellite 1

Satellite 2

Satellite 3

6

Spatial accuracy assessment

All accuracy measures are relative.The term ‘absolute accuracy’ is often used

to identify how well the position matches certain predetermined map accuracy standards.

The standard deviation of the observations (or root mean standard error – RMS error) gives an indication of the spread of the observations.

Resolution - the minimum possible observable difference between adjacent measurements.

7

RMS error

mean-1 +1 +2 std-2

r.m.s.e

freq

uen

cy

N

xxRMSE

n

ii

1

2

8

Ground control points

Locational accuracy is assessed by ground control point (GCP) table, often generated from image processing software.

Distribution of GCPs is very important for the accuracy of geometric correction.

Selection of resampling algorithm may also play an important role.

9

ID Error Residual X Residual Y Map X Map Y Image X Image Y

1 1.02 -0.78 0.65 641750 2464550 2161.38 6608.63

2 0.47 -0.41 0.22 701120 2568220 4141.88 3152.13

3 0.97 0.96 -0.16 663000 2495300 2871.88 5582.63

4 1.37 -0.35 -1.32 702800 2612500 4197.88 1674.88

5 0.92 -0.04 0.92 701050 2586500 4139.88 2543.63

6 0.57 -0.56 0.12 676500 2617600 3320.38 1506.88

7 1.57 -0.87 1.31 587100 2567600 338.13 3176.13

8 0.58 0.29 0.50 634530 2567120 1921.38 3190.38

9 0.83 -0.64 0.53 676050 2471450 3305.63 6377.88

10 2.05 1.35 -1.54 246370 2539650 2317.44 4103.56

11 0.87 0.54 0.68 681140 256350 3476.38 3248.63

12 1.13 -0.53 -1.00 608850 2499350 1064.13 5447.88

13 1.07 0.68 -0.83 626300 2511700 1647.38 5036.13

14 0.52 -0.20 0.48 615800 2565750 1296.13 3236.38

15 0.80 0.56 -0.57 614100 2590400 1240.13 2413.88

RMSE 1.07 0.66 0.83

A typical GCP table

10

Representational accuracy assessment

The representational accuracy involves the attribute accuracy.

Normally this involves the test of classification.

At nominal or ordinal levels the test is generally to justify either the classification is right or wrong.

Testing is conducted a posteriori using techniques such as error matrix (or confusion tables).

A reference data set must be used for confusion table analysis.

11

About reference data

A reference data set must be independent from the classification to be tested.

A reference data set must be reasonably distributed in geographical space.

A reference data set must be representative for every class in the classification.

There must be sufficient samples for each class to generate sufficient significance in statistics.

Ground investigation is fundamental for reference data acquisition.

12

Field investigation

The only way for geographers to get first-hand information is the field investigations.

This applies to both human and physical geography.

Even with today’s technology (e.g. remote sensing) field investigation is still fundamental.

13

Locate yourself in the field

Object 1

Object 2

Object 3

Your location

Use of compass and maps

Bearing

N

14

Find your position using map

15

Find your position using GPS

16

Sampling

Qualitative sampling Landuse types Soil and vegetation classes Objects

Quantitative sampling Measurements: e.g. vegetation cover

(%), biomass (kg/ha) e.g. Soil organic matter contents

17

Qualitative sampling

Record position on map and in geographical coordinates using GPS

Identify and observe land cover types: e.g. farmland, forest, bare soil and rocks, etc.

Interpret air photograph or digital imagery

18

Field information recording

Route map, date, time, weather conditions location on the map or aerial photographs,

site ID numbers and map coordinates site descriptions. What do you see? portrait of interested phenomena samples taken and their ID numbers photographs

19

Record land cover types

20

Field airphoto interpretation

21

We use vegetation sampling as example Common methodology used in ecological

studies applies Purpose:

Quantifying vegetation parameters such as cover (%) and biomass (kg/ha)

Methods: Cover estimation: quadrant sampling, visual

estimation, point/line intercept, 3-D plant model and ground photo

Biomass estimation: cut and weigh, estimation using 3-D plant model or cover

Quantitative sampling

22

Measuring trees

23

Line intercept sampling

24

3-D plant model

h

ab

25

Ground imaging

26

Remotely piloted aircraft

27

Confusion table

Confusion table, also called error matrix, is used to assess a single image classification results (i.e. one-time classification).

This needs two independent data sets for test – samples on images (classes on map, or classification results) and reference (classes on ground, or training data). Important: the reference must be independent

from the classification results, i.e. it must not be used for training the classifier.

28

Class on ground

Class on map Forest Pasture Arable Bushland Total sampled sites

Forest 93 8 15 - 116

Pasture 6 65 23 1 95

Arable 11 34 503 32 580

Bushland 5 - 21 72 98

Total sampled sites 115 107 562 105 889

Total accuracy = 733/889 = 82%; 68% (65/95) of pasture was correctly classified; 32% (34/107) of pasture was incorrectly classed as arable; while 24% (23/95) of pasture on the map was actually arable on the ground.

Confusion table interpretation

29

Accuracies and errors

User’s accuracy: how many cases shown on the result are correctly classified?

Producer’s accuracy: how many known cases have been correctly classified?

Commission error: how many samples of other classes were wrongly committed into this class?

Omission error: how many known cases were omitted from this class?

Class on ground

Class on map Forest Pasture Arable Bushland Total sampled sites

Forest 93 8 15 - 116

Pasture 6 65 23 1 95

Arable 11 34 503 32 580

Bushland 5 - 21 72 98

Total sampled sites 115 107 562 105 889

For pasture: the user’s accuracy = 65/95 = 68%,the producer’s accuracy = 65/107 = 61%.

For pasture: the commission error = (6+23+1)/95 = 32%,the omission error = (8+34+0)/107 = 39%.

30

Cohen’s Kappa

A measure considers significantly unequal sample sizes and likely probabilities of expected values for each class:

qN

qd

where N = total number of

samples;d = total number of cases in diagonal cells;

N

aa

q

n

i

n

jij

n

jji

1 1,

1,

31

Kappa computation

The perfect score is 1.0 (i.e. 100% correct).

678.067.404889

67.404733

qN

qd

67.404889

10598562580107951151161 1,

1,

N

aa

q

n

i

n

jij

n

jji

In our case:

733725036593 d

Class on ground

Class on map Forest Pasture Arable Bushland Total sampled sites

Forest 93 8 15 - 116

Pasture 6 65 23 1 95

Arable 11 34 503 32 580

Bushland 5 - 21 72 98

Total sampled sites 115 107 562 105 889

32

Temporal accuracy assessment

The temporal accuracy assessment requires the same tests on spatial and attribute accuracies.

One radical method is to undertake confusion table test for single image classification and then use an error propagation model for overall assessment.

However one further assess the analysis accuracy by considering the characteristics in time dimension.

Change trajectory assessment Trajectory rationality analysis

33

Confusion table for trajectory classification

AA BB CC AB AC BA BC CA CB

AA

BB

CC

AB

AC

BA

BC

CA

CB

Cla

ssif

ied

dat

a

Reference data

The confusion table for trajectory classification. The classified change classes are assessed by the known change classes. The remaining operations are the same as normal classification.

34

Change rationality

From To Forest Farmland Construction Built-up

Forest

Farmland

Construction

Built-up

Possible Unlikely Unchanged

Two-date change rational

35

Error propagation

Error made in each stage of processing will be carried to the next stage.

The errors made in each stage may magnify the total error, or they may cancel each other out.

Estimating the cumulative (or propagated) error over multi-stage processing is often through error propagation modelling.

36

Spatial error propagation

Consider: Framework of a map from a control survey:

R.M.S.E. = ±0.005 mm at map scale Plotting of control (±0.10 mm) Detail survey (±0.25 mm) Compilation (±0.30 mm) Human input in drawing (±0.20 mm) Conventional reprographics techniques (±0.30

mm) Digitisation/conversion (±0.20 mm)

37

Compute propagated spatial error

n

iiset

esmresmr1

2........

In this case:

577.020.030.020.030.025.010.0005.0.... 2222222 set

esmr

where i denotes individual step

This would be equivalent to 11.54 m on the ground for data represented at 1:20,000 scale. This translates to a statement that 95% of the points in this data set would be positionally accurate to within approximately ±23 m (r.m.s.e. x 2) of their true location.

38

Attribute error propagation

Consider:Accuracy of the referencing landuse

map = 0.91Image classification accuracy = 0.82Post classification processing = 0.97

39

Compute propagated attribute error

iset min

In this case:

where i denotes individual stepi = 1, …, n

This is a ‘friendly’ model for error propagation involving multiple-step image classification, influenced only by the worst classification result.

82.097.0,82.0,91.0min set

40

Summary

Remote sensing data processing will not complete without accuracy assessment.

The errors occur in remote sensing data processing including spatial and representational errors.

RMSE is the common parameter to assess spatial accuracy.

Confusion table is often used to assess representational accuracy.

In a multiple step data processing procedure, propagated errors should be assessed.