GEOGG142 GMESCalibration & validation of EO products

Dr. Mat Disneymdisney@geog.ucl.ac.ukPearson Building room 113020 7679 0592www.geog.ucl.ac.uk/~mdisney

2

Outline· Calibration & validation

· Example: MODIS LAI and NPP products· Meaning of parameters??· Time, space, measurements?· Scaling?

Good place to start: CEOS Working Group on Cal/Val, Land Product Validation sub-group: http://lpvs.gsfc.nasa.gov/ and see eg Biophysical & references therein: http://lpvs.gsfc.nasa.gov/LAI_references.html and good practice guidelines: http://lpvs.gsfc.nasa.gov/PDF/CEOS_LAI_PROTOCOL_Aug2014_v2.0.1.pdf

3

Calibration & validation?• Calibration:

– Process of adjusting empirical relationship between empirical estimates of biophysical parameter estimated from 2 (or more) sources

– e.g. ground-based and EO-derived LAI, or NDVI and LAI, or NDVI and fAPAR etc. etc.

– Local calibration to estimate fitting parameters (slope, intercept for a linear relationship) and uncertainty

– Limitations???

4

Calibration & validation?• Calibration:

– process of converting an instrument reading to a physically meaningful measurement

– Particularly radiometric calibration i.e. from DN to radiance measurement

– OR– Process of adjusting empirical relationship between

estimates of biophysical parameter estimated from 2 (or more) sources e.g. ground and EO-derived

5

Calibration & validation?• Validation:

– experiments designed to verify instrument measurements using independent measurements

– Caveat: EO ‘validation’ often means testing one model-derived estimate against another• EO LAI, NPP etc. all require models• Field estimates of LAI also require models

– i.e. NOT validation in true sense at all– See later: when is LAI not LAI

• BUT: cal/val both essential to scientific remote sensing

6

Aside: focus on validation here but ….

· Eg LAI CEOS WGCV: recommends CALIBRATION comparison between EO and ground-based

· NEEDS: reference estimates traceable to in situ measurements· 3 sources available:

1. LAI measurements over individual Elementary Sampling Units (ESUs)2. Spatially extensive LAI reference maps based on data driven

relationships calibrated using ESU LAI3. Spatially extensive LAI reference maps based on functional

relationships calibrated using ESU LAI.

· And what is an ESU? See later

7

Validation example: MODIS NPP· Productivity recap: Net Primary Productivity

(NPP)· annual net carbon exchange· quantifies actual plant growth

· Conversion to biomass (woody, foliar, root)– i.e. not just C02 fixation (GPP)– NPP = GPP – Ra (plant respiration)

• MODIS product example used here– MOD17 GPP/NPP ATBD

• ntsg.umt.edu/MOD17• http://neo.sci.gsfc.nasa.gov/Search.html

– Turner et al (2005)

8

Productivity recap

• GPP/NPP from MODIS• Requirements?• MOD17 ATBD• Running et al. (2004)• Turner et al. (2005)• Zhao et al. (2005)• Heinsch et a. (2006)

9

MODIS GPP/NPP + QC??

http://secure.ntsg.umt.edu/projects/index.php/ID/ca2901a0/fuseaction/projects.detail.htm

10

MOD17 validation approach· Need to address time (days to years) and

space (local to global)· Permanent network of ground validation sites

· Quantify seasonal and interannual dynamics of ecosystem activity (cover time domain)

· EO to quantify heterogeneity of biosphere· Quantify land cover, land cover change dynamics

· Models to:· Quantify, understand unmeasured ecosystem· Provide predictive capability (in time AND space)

11

How on earth…..????

• …can we “validate” an EO-derived estimate of something that depends on soil, climate, land cover etc.?

• Given that it requires various models to go from a satellite observation (radiance), to reflectance, to LAI/FAPAR, to PSN, to GPP to NPP

• At 500m-1km pixels. Globally.• And how do you even “measure” NPP on the

ground??

12

So, how might we validate?

• Need to consider scale• Relate measurements at the

small scale to 1km pixels??• Flux tower approach• Eg BIGFOOT approach,

FLUXNET etc.• Measurements and

validation at many scales• Models to bridge time/space

scales – (but how good are models…?)

Fig from MOD17 ATBD

13

Ecosystem measurements: FLUXNET

http://daac.ornl.gov/FLUXNET/

Fig from MOD17 ATBD

14

Ecosystem measurements: FLUXNET 2009

http://daac.ornl.gov/FLUXNET/

http://earthobservatory.nasa.gov/Features/Fluxnet/

15

Ecosystem measurements: FLUXNET 2009-11

http://daac.ornl.gov/FLUXNET/http://www.fluxnet.ornl.gov/fluxnet/graphics.cfmhttp://earthobservatory.nasa.gov/Features/Fluxnet/

16

Ecosystem measurements: FLUXNET

http://daac.ornl.gov/FLUXNET/

17

Ecosystem measurements: FLUXNET by biome

http://daac.ornl.gov/FLUXNET/

Some distribution of biome types, but clearly biased in locationEven considering only limited biomes

18

BigFoot approach to validating MODIS NPP· E.g. Turner et al. (2005), 6 sites spanning range of

vegetation and climate· Crops, forest, tundra, grassland

· 5 x 5 km site at each plot (25 MODIS pixels)· Flux tower & 100 (25x25m) sample plots within each area,

seasonally measured for LAI and above-ground (A)NPP (from harvested leaf and wood material)

· Land cover from high res EO· Use measured data at sample plots to calculate NPP, GPP· Spatially distribute across site using (vegetation-calibrated)

BiomeBGC model· Requires daily met data, land cover, LAI

· Gives measured estimate from ground AND flux tower

19

BigFoot v flux tower GPP

Turner et al. (2005)

20

BigFoot v MODIS GPP

Turner et al. (2005)

Not such good agreement as for flux tower (not surprisingly)

21

Comparison of MODIS NPP with flux data

Turner et al. (2005)

Differences due to Ra (autotrophic i.e. plant respiration)?PAR, VPD differences between those from DAO and actual?(VPD = deficit between the amount of moisture in the air and how much moisture the air can hold when it is saturated)

22

DAO PAR, VPD?

Turner et al. (2005)

Clearly some sites better agreement than othersPAR generally good (relatively easy to measure)VPD less so e.g. SEVI (desert grassland site) VPDOther issues?

23

MODIS-estimated v BigFoot FPAR

Turner et al. (2005)

How do you measure FPAR even on the ground??Requires models to interpret measurements of radiation

24

MODIS-estimated v BigFoot LUE (light use efficiency)

Turner et al. (2005)

LUE inferred from flux dataAgain, hard to even measure this on the ground…..

25

Zhao et al. (2005)

Heinsch et al. (2006)

26

Process/SVAT (soil-veg-atm-transport) models

Fig from MOD17 ATBD

27

From Running et al. (2004) MOD17 ATBDBiome-BGC model predicts the states and fluxes of water, carbon, andnitrogen in the system including vegetation, litter, soil, and the near-surface atmosphere i.e. daily PSN

Process models: how do we test/validate?

28

Process models: how do we test/validate?

Fig from MOD17 ATBDhttp://www.ntsg.umt.edu/models/bgc/

29Canadell et al. 2000

Data-ModelFusion

[Using multiplestreams of datasets withparameter optimization]

C stock and flux measurementsInventory analysesProcess-based informationClimate dataRemote sensing informationCO2 column from space

Inverse modelingProcess-based modelingRetrospective and forward analyses

30

Multi-level model/data validation

• MOD17 ATBD: Synergy of various carbon measurement programs

Fig from MOD17 ATBD

31

How do we decide on ground-based sampling strategy, scale?· CEOS WGCV LPV: Elementary Sampling Units (ESUs)

· “ …. a contiguous spatial region over which the expected value of LAI can be estimated through in situ measurement … corresponds to finest spatial scale of LAI estimates used for reference LAI maps.”

· ESU size: “….at least as large as one measurement footprint of the in situ instrument and typically includes a number of instrument measurements.”

· ESU size: “ …varies with surface condition, instrument field of view, illumination conditions (when transmission based measurements are used) and spatial sampling design.”

· ESU size: “….should be sufficient to allow repeat visit with minimum uncertainty due to changes in illumination or geolocation.”

32

ESUs: CEOS WGCV LAI validation protocol

33

Summary· Calibration

· Needed to allow comparison of data & products from multiple sensors & algorithms over time

· AND/OR to · Can be done on-board, or via sensor intercomparison etc.

· Validation example: NPP· Far removed from EO measurement & spatially, temporally variable· Requires: observation networks over time and space and

measurement of met. & biophysical data· Models to interpolate spatially from ground-based, site-scale

measurements· Testing and intercomparison of models· Ideally: optimal combinations of models + data across scales (e.g.

via data assimilation)

34

References:NPP• Running et al. (2004) A Continuous Satellite-Derived Measure of Global Terrestrial Primary Production,

Bioscience 54(6), 547-560.• Ganguly et al. (2008a, b) Generating vegetation leaf area index earth system data record from multiple

Sensors, RSE, 112, 4318-4332 (Part II) and 4333-4343 (Part I)• Turner et al. (2005) Site-level evaluation of satellite-based global terrestrial gross primary production

and net primary production monitoring, Glob Change Biol, 11, 666-684.• Zhao et al. (2005) Improvements of the MODIS terrestrial net and gross primary production data sets,

RSE, 95, 164-176.• Heinsch et al. (2006) Evaluation of Remote Sensing Based Terrestrial Productivity From MODIS Using

Regional Tower Eddy Flux Network Observations, IEEE TGRS, 44(7), 1908-1925.

General validation• http://lpvs.gsfc.nasa.gov/LAI_references.html• Morisette et al. (2002) A framework for the validation of MODIS Land products, RSE, 83, 77-96.• Disney et al. (2004) IJRS, 25(23), 5297-5317

35

Other cal/val links· NPP: http://daac.ornl.gov/NPP/npp_home.html· Cal/val programs

· CEOS-WFGCV (Committee on EO Working Group on Cal/Val)

· http://calvalportal.ceos.org/CalValPortal/welcome.do· http://lpvs.gsfc.nasa.gov/· http://landval.gsfc.nasa.gov/· SAFARI2000: http://daac.ornl.gov/S2K/safari.html· VALERI: http://w3.avignon.inra.fr/valeri/· NCAVEO: http://www.ncaveo.ac.uk/· JAXA:

http://www.eorc.jaxa.jp/ALOS/en/calval/calval_index.htm· Etc etc etc

36

Example: MODIS core val sites

http://landval.gsfc.nasa.gov/coresite_gen.htmlJustice et al. (1998) http://eospso.gsfc.nasa.gov/eos_observ/5_6_98/p55.htmlPrivette et al. (2002) and RSE 83, 1-2, 1-359