near real time assessment of operational oceanography ...godae-data/symposium/gov...multimodel...
TRANSCRIPT
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Near real time assessment of operational oceanography products:
advances in the GOV community, overview of Class 4 intercomparison
and multimodel assessment approaches
Greg Smith Environment Canada
F. Hernandez Mercator-Ocean
M. Martin, A. Sellar UK Metoffice
With thanks to the many IV-TT contributors!
www.godae-oceanview.org
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
• Pursues activities developed during GODAE
• Develop a framework for comparing outputs of the
various operational ocean forecasting systems (OOFS)
– leads to improvements to the systems and to the quality of products
from those systems
– provides a framework for scientific discussions on forecasting system
performance assessment, on ocean analysis and validation from
numerical simulations
– offers a demonstration of the work in GODAE OceanView and has the
potential to increase the visibility to the external community
The Intercomparison and Validation Task Team:
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
The Intercomparison and Validation Task Team
Coordinates and promotes the development of scientific validation and intercomparison of operational oceanography systems – Mainly for short term forecast assessment, but benefit for/from
reanalysis assessment, and medium range forecast assessment (eg, seasonal forecast),
– include the definition of metrics to assess the quality of analyses and forecasts (e.g. forecast skills) both for physical and biogeochemical parameters and the setting up of specific global and regional intercomparison experiments
– metrics related to specific applications also considered
– links with the OSE-TT
– links with the JCOMM ET-OOFS team for operational implementation
– cooperation with CLIVAR/GSOP for climate issues.
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
• Main outcomes from the IV-TT: – Class 4 metrics, forecast skill monitored in near real time
among 6 operational centres
– Class 1 metrics, ensemble and multi-model approach
– Participation to the GSOP Intercomparison project (CLIVAR/GOV initiative, presented by M. Balmaseda)
• Advances in real time monitoring and validation methodology implemented in operational oceanography centres: highlights with some examples
• How operational validation is provided to users in term of Product Quality: some illustration and examples
• Conclusions
Outline of this talk
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Class 4 metrics
• Create a real-time system for generating model-equivalents to a common set of observations
• Major effort made to ensure consistency in methods and comparisons
• Initial comparison: – Jan-Jun 2013, 6 institutions participated
• Systems evaluated using: – SST from drifting buoys – Argo temperature and salinity profiles – Sea level anomaly
• Statistics compiled for analyses as well as as a function of lead time
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
USGODAE – In-situ drifters Courtesy Andy Ryan, UK Metoffice
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
ARGO – Salinity profiles
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Taylor diagram of SST around Australia
• Taylor diagram provides another perspective
See poster from Divakaran for details
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Class4 intercomparison
• Radar plots of RMS error computed for the period of 1st January to 30th September 2013, for SLA, SST and in the layer 0-500m for temperature and salinity
• Systems involved : PSY3 (Mercator OCEAN), FOAM (UK Met Office), GIOPS (CMC), RTOFS(NOAA/NCEP), OceanMAPS (Bureau Of Meteorology)
• For salinity PSY3 shows the best performance
• For temperature, except OceanMAPS (lower resolution except in Australian region), four other systems are very close
• FOAM SST is the most accurate system with GIOPS approaching in terms of forecast
See poster from C. Regnier for details
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Mapping of RMS error
• We are working with the July-August-September 2013 quarter with the five systems
• The ocean is divided in regular bins of 2°x2°
• In the left we show the system for which the RMS error is the lowest
• In the right the corresponding RMS value is displayed for each bin
• Maps exhibit a regional distribution of the error
• This kind of map is a way to compare system with a given criterion
• Need to cross several criteria for robust conclusion
See poster from C. Regnier for details
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Class 1 Multi-model and ensemble validations
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Multimodel ensemble SST validation
• Demonstrates value of multimodel ensemble
• Results sensitive to method used to calculate ensemble mean:
– simple
– weighted
– kmeans
3. Time series of RMS and bias for all members and ensembles for period 8 June 2013 to 8 July 2013
See poster from T. Spindler for details
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
CLASS1: drifter trajectory forecast
• How can we best use OOFS to determine the position of the AF447 crash?
• Use 15m drifter trajectories to estimate error from 6 systems
• Demonstrates added value of multimodel ensemble
7-day fcst 3-day 1-day
1-day forecast: in 50% of the forecasts, the distance between the drifter and the end point trajectory is less than 20km
~20 km
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Evaluating oil-spill dispersion
forecasting in the Northern
Aegean Sea
See poster by S. Sofianos et al. for details
Highlights the importance of ensemble methods!
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Sea ice forecast verification using analyses is unreliable due to large analysis uncertainty
• Patchy coverage, foot-print issues, data reliability
• Representation of leads
Only evaluate points where the analysis changes by more than 10%
Pro: Only includes points where we have confidence in ice analysis
• Focus on ice edge in particular
Con: Excludes areas of incorrect model changes
• E.g: coastal polynyas, false alarms along the ice edge
Sea ice verification: Analysis tendencies
7day GIOPS forecast
error for 2011-03-30
Van Woert et al. (2004)
See poster from G. Smith for details
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Sea ice verification : Analysis tendencies
• Over a full annual cycle, this method provides a reasonably reliable idea of forecast skill
• But what about leads, coastal polynyas and false alarms in the marginal ice zone?
GIOPS 7 day forecasts for 2011
GIOPS Analysis persistence
Van Woert et al. (2004)
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Contingency table analysis
• Comparison with IMS Analyses:
– Interactive Multisensor Snow and Ice Mapping System (NOAA-NIC)
– Daily Northern Hemisphere ice analyses on 4km grid (ice/water)
• Evaluation Methodology:
– Calculate contingency table values using 0.4 ice concentration cutoff
IMS Ice IMS No ice
Forecast
Ice
Hit ice False
Alarm
Forecast
No ice
Miss Hit water
• Proportion Correction Ice:
PCI = Hit ice / (Hit ice + Miss) [0,1]
• Proportion Correction Water:
PCW = Hit water / (Hit water + False Alarm) [0,1]
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Contingency table verification
• Proportion Correct Ice
– Ability to forecast ice formation and advection
– PCI= Hits ice / (Hits + Misses)
– Shows skillful forecasts along most ice edges
• Proportion Correct Water
– PCW= Hits water / (Hits water + False Alarms)
– Shows important false alarm rate missing from analysis tendency verification
ΔPCI ( Fcst – Pers)
ΔPCW ( Fcst – Pers)
GIOPS 7 day forecasts for 2011
Ice? YIMS NIMS
YFcst Hit ice False
Alarm
NFcst Miss Hit
water
Skill
Error
See poster from G. Smith for details
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
BAL MFC: Developing ice edge metrics for the
Baltic Sea
GODAE OceanView Symposium – November 2013
obs2mod
distance
mod2obs
distance
Observed ice concentration
obs ice edge
model ice edge min distance for
each edge point
N
i
iDN
RMSdist1
21modnnN obs
mod_mod,21mod,2_,2mod1,2mod ...... nobsobsobsnobsobsi ddddD
Metric must be calculated sub-regionally to avoid long overland distances
Ice edge RMS distance
Find minimum distance from modeled ice edge for each observed ice edge cell and vice versa
Take the RMS of these distances
Straightforward interpretation for the users
Courtesy of P. Lagemaa
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Conclusions
• Direct benefit from IV-TT activity:
– several OOFS have now implemented routine Class-4 and Class-1 delivery, allowing enhanced monitoring for these teams, and definition of new validation approaches
– Operational oceanography validation experts organized similar to that in the atmospheric community
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Conclusions
• Direct benefit from IV-TT activity:
– several OOFS have now implemented routine Class-4 and Class-1 delivery, allowing enhanced monitoring for these teams, and definition of new validation approaches
– Operational oceanography validation experts organized similar to that in the atmospheric community
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Conclusions (con’t)
• General overview:
– Validation techniques are implemented in most OOFS, global or regional
– New metrics are defined to satisfy specific user needs and down stream applications
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Thanks!
What is the role of GOV in improving Polar Predictions?
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Sea Level Forecast Skill at Mercator Océan
Improvement of the forecast / persistence
Global:
¼° old
¼° new
Zoom:
1/12° old
1/12° new
• Implemented routinely, based on Class-4 metrics
• Statistics published every 3 months • Used here to compare former and new
operatioanl system
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Communicate Product Quality to users:
• Regular reporting:
– QuOVaDis quarterly report by Mercator Océan
– MyOcean2 Quality Information Docucments, complemented by quarterly Class-4 reporting
– …..
• Real time reporting
– Website
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
« NARVAL »: regional validation webserver for the
MyOcean2 IBI area
n 2 NARVAL modes
n On-line “Real-time” mode: Val procedures launched after daily cycle
n Delayed mode Component: Val procedures on specific metrics and statistics covering longer periods (i.e. monthly, seasonal and annual)
n NARVAL updated to generate automatically Statistics for MyO Quarterly Validation Reports.
n Intercompares with same metrics available MyO fcst in same area. eg, 3 in the western Med sea (MFS, Mercator, IBI)
Courtesy of Puertos del Estado and MyOcean2.
MyOcean2
V3 Acceptance & V4 Product Design Workshops – Madrid 28,29th January 2013
Partial snapshots of NARVAL Real-Time & Delayed-Modes web pages
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
Inter-comparison partners
Institution System Ocean
Model
Resolution
UK Met Office FOAM NEMO ¼ degree
Bureau of Meteorology OceanMAPS mom4p1 1 degree (1/10 around Australia)
NOAA/NWS/NCEP/EMC/MMAB RTOFS HYCOM 1/12 degree
Mercator-Ocean PSY3 NEMO ¼ degree
Canadian Meteorological Centre CONCEPTS-GIOPS NEMO ¼ degree
REMO HYCOM ATLe0.25 ¼ degree (Atlantic only)
•Results available for all partners for January-June 2013
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
ARGO – Temperature profiles
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
ARGO – Temperature profiles
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
ARGO – Salinity profiles
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
CLASS 1: drifter trajectory forecast
Scientific question: How to make the best trajectory forecast in the context of operational oceanography ?
6 products used: • GLORYS • PSY2 • PSY3 • NCOM • HYCOM • CLS
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
CLASS 1: drifter trajectory forecast
•
•
•
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
CLASS1: drifter trajectory forecast
7-day fcst 3-day 1-day
1-day forecast: in 50% of the forecasts, the distance between the drifter and the end point trajectory is less than 20km
~20 km
GODAE OceanView Symposium, Hilton Baltimore, 4-6 November 2013
CLASS1: drifter trajectory forecast
Main conclusions: • Trajectory forecasts is a nice way to evaluate OOFS skill • The added value of an ensemble of forecasts is shown • The ensemble filters out the error
Smoothing increases
Erro
r R
MS
(km
)