IOCS 2017 – multi-water algorithms and algorithm performance assessment 1

Multi-water algorithms and algorithm performance assessment

IOCS 2017 - Breakout Workshop #7

Co-chairs: Ewa Kwiatkowska (EUMETSAT), Bridget Seegers (NASA/USRA),

Carsten Brockmann (Brockmann Consult), Tim Moore (Uni. New Hampshire), Blake Schaffer (US-EPA), Susanne Craig (Dalhousie University)

2 IOCS 2017 – multi-water algorithms and algorithm performance assessment

14:00 - 14:05 Session introduction E. Kwiatkowska (EUMETSAT) and B. Seegers (NASA) Part I: Multi-Water Algorithms and Algorithm Performance Assessment 14:05 - 14:15 User requirements for blended multi-water products Stewart Bernard (CSIR) 14:17 – 14:27 Overview of atmospheric correction methods for open-ocean, coastal and inland water transitions Menghua Wang (NOAA) 14:29 – 14:39 Overview bio-optical algorithms for open-ocean, coastal and inland water transitions Daniel Odermatt (Odermatt & Brockmann GmbH) 14:41 – 14:51 Assessing algorithm performance and blending in the context of optical water classes Thomas Jackson (PML) 14:53 – 15:03 Needs and approaches to algorithm assessment Rick Stumpf (NOAA) BREAK Part II: Moderated Discussion Multi-water and Algorithm Assessment Moderators: co-chairs 15:20 – 15:50 Multi-water algorithms 15:50 – 16:20 Algorithm assessment 16:30 – 16:45 Formulation of Actions and Recommendations

Workshop agenda

I’d like a nice simple ocean colour chlorophyll product

please, one that works everywhere?

0

50000

100000

150000

200000

Ocean Color SST

Substantial and growing operational and scientific user community requiring ocean colour data…

Num

ber o

f pu

blic

atio

ns

End Users

From Stewart Bernard (CSIR)

Chlorophyll distribution in North Sea and Skagerrak region as retrieved by the MERIS Algal 2, MERIS Algal 1, MODIS/Aqua Chl

Much variability in “equivalent” products between sensors and algorithms From Stewart Bernard (CSIR)

5 IOCS 2017 – multi-water algorithms and algorithm performance assessment

From Menghua Wang (NOAA)

Overview of Bio-optical Algorithms Band ratio algorithms, Algorithm blending, Vertical non-uniformities, Neural Network algorithms

Hieronymi, M., Müller, D., and Doerffer, R. (2017). The OLCI Neural Network Swarm (ONNS): A Bio-Geo-Optical Algorithm for Open Ocean and Coastal Waters. Front. Mar. Sci. 4, 140.

From Daniel Odermatt (Odermatt & Brockmann)

Hieronymi et al. (2017): OLCI NN Swarm

Water class based algorithm assessment and blending approaches are now being trialled for new applications e.g smaller water bodies such as lakes.

Apply algorithms by water class From Thomas Jackson

multi-water algorithms discussion bits • There is substantial and growing operational and scientific user

community • Non-specialist users require simple, ready-to-go products • Need products that can deliver accurate transitions between open

ocean, coastal waters, and inland waters • SeaDAS provides the framework to explore a variety of approaches • Atmospheric correction is a big problem in coastal/ inland waters,

unless it’s not being used at all • Adjacency effect and shallow bottom influences must be remembered • Ocean colour uptake can be improved if expert decisions on best

algorithm selection are incorporated upfront into the products

Develop an atmospheric correction prototype processor for coastal and inland waters Develop a prototype processor that will deliver accurate transitions between open ocean, coastal waters and inland waters • Set-up as a funded OCR-VC working group with specific deliverables • Enable active collaboration of atmospheric correction experts • Push science forward, explore new ideas, utilize sensor capabilities (this

is NOT an algorithm shoot-out) • Base new development on an existing framework / sand-box • Develop for missions of participating OCR-VC agencies (e.g. OLCI, S2,

VIIRS) • Initial ambitious timeframe: 2-3 years

Multi-water algorithms recommendations

9 IOCS 2017 – multi-water algorithms and algorithm performance assessment

1

2

Which is the better model? Model D Model C

R-squared 0.75 0.75 Slope 1.0 1.3 RMSE 2.5 3.0 bias 0 0.9

D MAD=2.1

C MAD=1.4

Graphics are essential And Mean square error stats are not robust

Algorithm Performance Assessment

Algorithm assessment From Rick Stumpf

Metric can determine ‘best’ algorithm

Smyth

Best Slope

Best r2

Best bias

From Thomas Jackson

Summary:

Assess algorithm performance through multi-metric approach rather than relying on a single measure of performance.

The multi-metric assessment is applicable (and required) for both atmospheric correction and in-water algorithms.

Consider what is important for the end users (number of retrievals, unbiased results, minimal error, IOPs?).

Techniques such as bootstrapping can allow insight into the robustness of results and the sensitivity to sample selection. Questions? docs/news : http://www.esa-oceancolour-cci.org

data : https://www.oceancolour.org

From Thomas Jackson

13 IOCS 2017 – multi-water algorithms and algorithm performance assessment

Discussion Bits from Algorithm Assessment

• Uncertainties should be explored as algorithms are developed. It is interesting and insightful to keep one’s eye on the error propagation as you move through the steps of algorithm development.

• Collaborate with statisticians

• Spatial and spectral patterns in combination at inter and intra pixel variable to gain insight. Spatially map variance.

• Spatial and temporal performance.

14 IOCS 2017 – multi-water algorithms and algorithm performance assessment

Develop community recommendations on the standardization of statistical metrics to assess algorithm performance in water and on satellite data. -Define a core set of statistical metrics, with agreement on their meaning - Establish guidelines on their conditions and appropriateness for use, which may be research question dependent • Approach: Identify appropriate literature to develop consensus

and gaps to guide publishing of new papers that build on that collection of work

Algorithm performance recommendations

15 IOCS 2017 – multi-water algorithms and algorithm performance assessment

Develop a strategy to inform the community of best practices for performance assessment of algorithms. • Approach: Create a website providing best practices. Website should host 3 sections:

1) The most up to date algorithm assessment literature 2) A library of code/packages to ease the implementation of more robust algorithm assessment. And guidance about which metrics are valid and appropriate skill metrics, which is research question dependent. 3) Provide a common data set to run algorithm to give first look at how it compares to the community set of algorithms.

• Community statistical training opportunities • Timeframe: 1 year and updated yearly

Algorithm performance Recommendations