Observing System Evaluation Task Team

Peter Oke1 and Gilles Larnicol21CSIRO – Centre for Australian Weather and Climate Research

2Collecte Localisation Satellites

Talk Outline

• GOV OSE TT mission• How the TT works …• Membership proposal• Relationship to other GOV TTs• Relationships to broader community• Past activities• Future plans >>> action items from 2nd GODAE OSE WS

OSE TT mission

1. Demonstrate the value of existing observing systems– support observational agencies / patrons by demonstrating the

importance of each observation platform to GODAE

2. Contribute to the ongoing design and assessment of the GOOS– where are the developing gaps / problems– what additional observations would be beneficial

… through coordinated community activities and capability building

How the OSE TT works?• Sharing of experiences, expertise, etc, through workshops:

– 1st GODAE-OOPC OSE/OSSE WS (Nov 2007)– 2nd GOV OSE WS (June 2009)

• Reactive activities – responding to requests:– e.g., demonstrating importance of Jason-1 continuation– evidence to support decisions in changes to GOOS components (e.g., Jason-1/2

switch to inter-leaved orbits … was an opportunity missed)

• Proactive activities – anticipating future requirements:– Building capability for more timely reactive activities (2nd GODAE OSE WS) and

coordinating community activities:• Argo design (i.e., 3x3 degree global optimal; benefit of deep profiling;

importance of marginal seas)• design of altimeter missions/constellations

– importance of high quality reference mission– delivery of Cryosat-2 ocean product– wide vs single swath

• benefits of SSS

In practice … the co-chairs will coordinate, and facilitate OSE activities; and collate “evidence” and deliver to GOV co-chairs for engagement with patrons.

… proposals to ET-OOFS for the statement of guidance.

Membership proposal

Co-chairs

Core members from observational agencies

Core members from each GODAE group

Invited experts

Action ALL >>> please provide co-chairs with recommendations for core membersAction co-chairs >>> formalise membership through invitation

Relationships to other GOV TTs

Inter-comparison and validation

TT

Coastal and Shelf Seas TT

OSE TT

1. Demonstrate the value of existing observing systems>>> partly achieved through routine demonstration of forecast &

analysis skill, and how this changes when the observing system changes (e.g., altimeter safe-hold, reduced Argo capacity, etc)

Capability building and activities

Common metrics & diagnostics

Relationships to broader community• OOPC, Clivar, …

• GOOS, GCOS, …

• OSTST, Argo, GHRSST, …

• should GOV OSE TT become a GOV-OOPC-ETOOFS - SCOR working group?

… our intention is to be inclusive

We recognise that:

• GODAE systems critically depend on the availability of ocean observations … but most observing systems are not maintained for GODAE activities – they are primarily maintained for climate monitoring.

• To maintain/acquire relevance and influence, GODAE must proactively contribute to GOOS planning/discussions.

• GODAE must contribute positively – and not contradict, or compete with our more climate-centric colleagues.

Past Activities

GODAE Symposium contributionsOke, P. R., M. Balmaseda, J. Cummings, E. Dombrowsky, Y. Fujii, S. Guinehut, G.

Larnicol, P.-Y. Le Traon, M. Martin, 2008: Observing System Evaluation, Proceedings of the GODAE Final Symposium, Nice, France

Oke, P. R., M. Balmaseda, M. Benkiran, J. A. Cummings, E. Dombrowsky, Y. Fujii, S. Guinehut, G. Larnicol, P.-Y. Le Traon, and M. J. Martin, 2009: Observing System Evaluations using GODAE systems, Oceanography, accepted.

Ocean Obs 09 contributionsWhite paper - Ocean Observing System EvaluationG. Larnicol, P. R. Oke, M. A. Balmaseda, M. Benkiran, J. A. Cummings, E. Dombrowsky,

Y. Fujii, S. Guinehut, P.-Y. Le Traon, M. J. Martin… Oceanography paper + OceanObs09 White paper = GOAE symposium paper

Poster - Ocean Observing System Evaluation: Towards Routine Monitoring under GODAE OceanView

P. R. Oke, G. Larnicol… summaries the outcomes of 2nd GODAE OSE WS

Conventional methods: OSEs

Altimeter

SST

XBT

Moorings

ForecastForecastor or

BGFBGF

AnalysisAnalysisoror

ForecastForecast Altimeter

SST

T/S

XBT

Moorings

Evaluation/Validation

Assimilated observations

Assimilated and with-held observations

• Assimilate real observations

• Systematically with-hold observation types

• Conventional method for demonstrating benefit/value of existing observation platforms

… but very expensive and not always easy to interpret

T/S

Conventional methods: OSSEs

ForecastForecastor or

BGFBGF

AnalysisAnalysisoror

ForecastForecast

Evaluation/Validation

Assimilated “observations”

Simulatedobservations

• Assimilate pretend “observations”

… from a model

• Systematically include different observation types

… including future observation types

• Conventional method for demonstrating the potential benefit/value of future/possible observation platforms

Number of altimeter missions: DUACS system

Pascual A., Y. Faugère, G. Larnicol and P.-Y. Le Traon (2006) Improved description of the ocean mesoscale variability by combining four satellite altimeters, Geophys. Res. Lett., 33, doi:10.1029/2005 GL024633.

RMS of SLA signal using data from 4

missions

Difference between RMS of SLA signal using data from 4 missions and 2

missions

Approximately half the SLA signal is lost when data from only 2 missions is used.

Errors of up to 10 cm.

1/3o DUACS system

October 2002 -August 2003

during the T/P interleaved

mission

… other studies have demonstrated that 4 altimeters in NRT is approximately equivalent to 2 altimeters in DT

NO ALTIM NO Argo NO SSTNO DATA ALL DATA

Complimentary data types: mesoscale prediction

Oke, P. R., and A. Schiller (2007): Impact of Argo, SST and altimeter data on an eddy-resolving ocean reanalysis. Geophys. Res. Lett., 34, L19601, doi:10.1029/2007GL031549.

No SST

No ALTIM

• 1/10o Bluelink system

• 6-month long OSEs starting December 2005

Assessing the value of future observation types: SMOS

Tranchant, B., C.E. Testut, L. Renault, N. Ferry, F. Birol, P. Brasseur (2008) Expected impact of the future SMOS and Acquarius ocean surface salinity missions in the Mercator Ocean operational systems: new perspectives to monitor ocean circulation. Remote Sens. Env., 112, 1476-1487.

SMOS/Aquarius data should be very beneficial

… but depends on errors

1/3o Mercator System:

North Atlantic

SEEK filter

1-year OSSEs

2003

SMOS errors are spatially varying @ ~0.2-2.5 psu depending on brightness temp, SST & wind. SMOS w/ 2x expected errorsSMOS w/ 2x expected errors

SMOS w/ 1/2x expected errorsSMOS w/ 1/2x expected errors SMOS w/ expected errorsSMOS w/ expected errorsControl RunControl Run

Future Plans

2nd GODAE Observing System Evaluation Workshop:Towards Routine Monitoring

4-5 June 2009CLS/Mercator-Ocean, Toulouse, France

http://www.godae.org/OSSE-OSE-Second-workshop.html

Attendees included representation from:-Bluelink, Mercator, UK Met, TOPAZ, JMA-MRI, NRL-HYCOM, CLS-JCOMM-OPS (Argo, in situ programs), NOAA (altim), CNES (altim)

Presentations covered:-Various OSE & OSSE experiments (e.g., TAO vs Argo vs TAO+Argo; impact of SSS; short-range to seasonal applications and coastal applications)-Existing GODAE system metrics/diagnostics & evaluation activities-NWP experience (THORPEX) from special guest (Florence Rabier)-Proposed action items

Workshop Objectives

Key question: How can we use GODAE systems for routine monitoring of the GOOS?

Key objective: Establish a plan for coordinated routine monitoring of the GOOS using GODAE systems –

- what will we do?

- how will we do it?

- when will we do it? how regularly?

Workshop Objectives

• Scope:– physical, biological, ocean-atmosphere ?– global, regional, coastal ?– what is required for each observing system from

GODAE ?

• Appropriate experiments:– Conventional OSEs/OSSEs– analysis / forecast sensitivity– assimilation diagnostics (obs-space)– other …

• Method of delivery:– Central host vs distributed web pages– ET-OOFS – JCOMM – WMO ?

Proposed Action Items

1. Short-term actions re: Jason-1 continuation

2. Facilitating community OSSEs

3. Community analysis of assimilation diagnostics

4. Capability building for routine analysis sensitivity

1. Short-term actions re: Jason-1 continuation

Motivation: We need to provide evidence of the need to continue Jasn-1 operations … required by July 2009

Actions:• Jim Cummings send demonstration of benefits of Jason-

1 for wave height forecasts• Gary Brassington and Daniel Lea send demonstrations

of the benefits of Jason-1 for GODAE forecasts

… send to PO & GL (we’ll collate and forward to AS & ED)• Deadline – 15 July 2009

2. Facilitating community OSSEs

Motivation: timely response to potential developments in possible new observing system platforms

• We agreed to:– generate/identify a single nature run to be used for OSSE

activities– agree to collate a list of key questions that we need to ask– communicate what OSSEs we can find resources for (e.g.,

through students, PostDocs etc)

2. Facilitating community OSSEs

Specific immediate action items• ALL WS participants >>> Send ideas for possible nature run

to PO & GL• ALL WS participants >>> Send key OSSE questions to PO

& GL• PO & GL >>> seek input on OSSE questions from obs

agencies and facilitate access to community nature run– e.g.,

• Do we need a reference altim mission?• Design of Jason-4• Cryosat 2• Drifting mission with addition error• When is obs error for altim, sst, sss etc too large to be useful?• Is a 3x3 Argo array optimal?• What are the benefits of SSS?• Impact of Argo in EEZs

• Deadline for input: 15 July 2009

3. Community analysis of assimilation diagnostics

Motivation: facilitate timely demonstration of the impact of “observing system events” (e.g., safe hold of altimeter mission; addition of new observations; …)

• Each group already calculates and archives common diagnostics in observation space:

– for each observation type - SLA, SST, T/S(z):• wo, Hwb, Hwa, including x, y, z, and t

– WS participants agreed*** to make these fields available so that they can routinely be synthesised for different domains (e.g., each WBC region, each tropical basin, etc.) and disseminating via the GODAE website

• Each group agreed*** to provide analysis and 0-5 d forecasts of:– U, V, SLA, SST, and MLD on native grid (all 2d fields)

… to quantify forecast errors and how these change with “events”… to enable qualitative animations with drifter trajectories overlaid … to estimate eddy statistics (paths, intensity, etc)

… perhaps even just for the 1st and 3rd Monday of each month

***Some groups need permission>>> PO & GL will generate a metrics document, based on the metrics already

established by the inter-comparisons/validation TT

4. Capability building for routine analysis sensitivity

Motivation: routine evaluation of impact of each observation to feed back to observation agencies

• Analysis self-sensitivity is possible for all groups– requires an additional analysis, using perturbed observations– and an inexpensive calculation, in observation space to compute

the trace, and sub-trace of different variables, regions etc.

Each group agreed to develop this capability in research mode?

Each group agreed to perform this analysis for a demonstration case … and work towards routine performance

>>> PO to circulate relevant reverences and sample code

Method of delivery

Bluelink

Mercator NRL

JMA TOPAZ

UK Met

Example page only

Points for comment

• Feedback of WS action items1. Short-term actions re: Jason-1 continuation

2. Facilitating community OSSEs

3. Community analysis of assimilation diagnostics

4. Capability building for routine analysis sensitivity

• GOV OSE TT GOV-OOPC-ETOOFS TT?– does the extra level of management/reporting return any

practical benefit?– thoughts on proposition for joining/leading a SCOR OSE WG?

Action items on GOV ST >>> please provide OSE TT co-chairs with recommendations for core members

Analysis self-sensitivities

Step 1: Perform an analysis

Step 2: perturb observations (d*) according to their expected error (R)

Step 3: Perform another analysis

Step 4: Compute self-sensitivities:

di is the original ith obs; di* is the perturbed ith obs;

Hai is the original analysis, interpolated to the obs;

Hai* is the new analysis, using d*, interpolated to the obs;

HKii is an estimate of the self-sensitivity.

HKii = (di*-di) (Hai*-Hai) / Rii

… repeat steps 2-4 to check robustness of results.

All elements of HK can be analysed, or the partial trace of HK can be averaged for different regions, variables, etc.

Analysis self-sensitivities

Preliminary estimates of the Information Content (IC; %), degrees of freedom of signal (DFS) and the number of assimilated super-observations (# Obs) for the Bluelink reanalysis system in the region 90-180oE, 60oS-equator, computed for 1 January 2006. The scale for the IC is to the left and the scale for the DFS and

# Obs is to the right.

Analysis sensitivity of Argo T over top 200 m depth

• 5-different realisations of the self-sensitivity (HKii)

• Averaged over top 200 m depth• Shows some dependence on the

perturbations to observations

Analysis sensitivity of Argo T over top 200 m depth

If HKii – std err >0.5 High

If HKii – std err >0 & < 0.5 Med

If HKii – std err <0 Low (?)

Average Standard error Influence

Analysis sensitivity of Argo T over top 200 m depth

Analysis sensitivity of Argo S over top 200 m depth

Analysis sensitivity of Argo S over top 200 m depth

If HKii – std err >0.5 HighIf HKii – std err >0 & < 0.5 MedIf HKii – std err <0 Low (?)

Average Standard error Influence

Analysis sensitivity of Argo S over top 200 m depth

Method of delivery

Bluelink

Mercator NRL

JMA TOPAZ

UK Met

Example page only

… from Juliette and Gerald’s presentations:• A key issue in altimetry is that missions with degraded quality are

inevitable. It is expensive to maintain missions, so the operating agencies need to know when a degraded mission becomes useless. Q/ At what point is an altimeter not worth assimilating? When the errors get to what level? re: spatially correlated errors, not random errors.

• Conclusions that 3-4 altimeters are required are insufficient. We need to generate evidence that certain satellite constellations are needed for GODAE.

• Through routine monitoring we need to illustrate the impact of major events.– routine, online, NRT skill assessments

• Possibly consider annual OSSEs based on possible future missions: which option is better.

• We may need to have a OSSE experiments design ready for response. e.g., have a nature run completed, and a