verification activity in sub-seasonal forecasting at jma
DESCRIPTION
Verification Activity in Sub-seasonal Forecasting at JMA. Climate Prediction Division / JMA Yuhei Takaya WWRP/THORPEX/WCRP Sub-seasonal to Seasonal Implementation Planning Meeting 2-3 December 2011, Geneva Switzerland - PowerPoint PPT PresentationTRANSCRIPT
Verification Activity in Sub-seasonal Forecasting at JMA
1
Climate Prediction Division / JMAYuhei Takaya
WWRP/THORPEX/WCRP Sub-seasonal to Seasonal Implementation Planning Meeting
2-3 December 2011, Geneva Switzerland
Thanks to N. Adachi
1. Verification of real-time operational forecastMonitoring of the performance of real-time operational forecasts on a routine basis.
2. Verification of hindcastAssessing the performance of forecast systems using hindcasts (re-forecasts) for every implementation of systems.
Verification of Sub-seasonal Forecast
Climate Prediction Division, JMA2
• 5 members from 3 initial dates in all months during last ~ 30 years (currently 1979-2009)
Setups of Hindcast
Climate Prediction Division, JMA4
5 members
10th 20th The last day of the month
• Verification of JMA monthly EPS is based on the SVS-LRF.
• Scores include deterministic scores (i.e., RMSE, ACC)as well as probabilistic scores (i.e., ROC, BSS).
• Reference data are JRA-25 re-analysis, GPCPprecipitation.
Verification Scores
Climate Prediction Division, JMA5
ROC scores for 2-29 days mean precipitation (I.C.: Nov. 30 1979-2009)
0.8
0.5
0.6
• Our sub-seasonal/seasonal forecasts are provided together with skill assessment based on hindcasts.Tokyo Climate Center: http://ds.data.jma.go.jp/tcc/tcc/products/model/
Provision of Skill Assessment
Climate Prediction Division, JMA6 http://ds.data.jma.go.jp/tcc/tcc/products/model/map/1mE/map1/pztmap.php
The number of ensemble members (5 members) is not enough to get precise assessments of the performance.
Supplementary experiments with a larger number of ensemble members for limited initial dates.
This would be done in collaborative work of development of integrated EPS.
Current verification focus on forecast skill rather than representation of processes.
More process-oriented verification would be needed. Representation from a viewpoint of climate modeling.
(e.g., energy budget at the surface and TOA, MJO, etc… )
Actions Needed for Hindcast Verification
Climate Prediction Division, JMA7
Score Dependence on Periods of Climatology
9-0.2
0
0.2
0.4
0.6
0.8
1
-0.2 0 0.2 0.4 0.6 0.8 1
81-1
0 平
年値
79-04 平年値
3-4週目
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
81-1
0平
年値
79-04 平年値
Anomaly correlation of Z500 in NHaveraged over 28 days
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
81-1
0 平
年値
79-04 平年値
2週目
0.6
0.7
0.8
0.9
1
0.6 0.7 0.8 0.9 1
81-1
0 平
年値
79-04 平年値
1週目
1981
-201
0 cl
imat
olog
y
Anomaly correlation of GPH500 for 1979-2004, 1981-2010 climatology (operational forecasts during 2009/03/26-2011/04/21)
1979-1994 climatology
28 days mean Week-1
Week-3&4Week-2
Climate Prediction Division, JMA
Score Dependence on Periods of Climatology
10-0.2
0
0.2
0.4
0.6
0.8
1
-0.2 0 0.2 0.4 0.6 0.8 1
81-1
0 平
年値
79-04 平年値
3-4週目
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
81-1
0平
年値
79-04 平年値
Anomaly correlation of Z500 in NHaveraged over 28 days
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
81-1
0 平
年値
79-04 平年値
2週目
0.6
0.7
0.8
0.9
1
0.6 0.7 0.8 0.9 1
81-1
0 平
年値
79-04 平年値
1週目
1981
-201
0 cl
imat
olog
y
Anomaly correlation of GPH500 for 1979-2004, 1981-2010 climatology (operational forecasts dufing 2009/03/26-2011/04/21)
1979-1994 climatology
28 days mean Week-1
Week-3&4Week-2
Lesson :
ACC scores are affected by change of climatology periods.
Lesson :
ACC scores are affected by change of climatology periods.
Interannual Variability of Scores
Climate Prediction Division, JMA11
T106L40 (Top 0.4hPa)BGM(NH)/26-member
BGM(NH+TRO)land analysis
improved BGM(TRO)improved Cu conv. Scheme
improved radiation Scheme
SSM/I
improved Cu conv. scheme
Sc scheme
TL159L40 / 50-memberimproved radiation scheme
COBE-SST (for B.C.)
TL159L60 (Top 0.1hPa)improved Cu conv. schemeimproved gr. wave scheme
Interannual Variability of Scores
Climate Prediction Division, JMA12
T106L40 (Top 0.4hPa)BGM(NH)/26-member
BGM(NH+TRO)land analysis
improved BGM(TRO)improved Cu conv. Scheme
improved radiation Scheme
SSM/I
improved Cu conv. scheme
Sc scheme
TL159L40 / 50-memberimproved radiation scheme
COBE-SST (for B.C.)
TL159L60 (Top 0.1hPa)improved Cu conv. schemeimproved gr. wave scheme
Lessons from this exercise:
Forecast skill would be affected by the inherent predictability of the climate variability.
Sufficient length (samples) of forecast data is needed to verify the performance of sub-seasonal forecast systems. It is better to use both hindcasts and operational forecasts to assess the performance of forecast systems.
Lessons from this exercise:
Forecast skill would be affected by the inherent predictability of the climate variability.
Sufficient length (samples) of forecast data is needed to verify the performance of sub-seasonal forecast systems. It is better to use both hindcasts and operational forecasts to assess the performance of forecast systems.
• Ongoing work• Integrated verification environment for an experiment
system of integrated EPS (weekly-monthly) . • Re-assessment of operational forecast scores with the
consistent JRA-25 analysis. (Previously forecast skill has been routinely verified against the deterministic analysis at the time.)
• Future work…• More appropriate setups in order to verify EPS
performance. (5 members hindcasts are obviously NOT enough!)
• Verification of precipitation with in-situ observations.
Planned and Ongoing Activity in Verification
Climate Prediction Division, JMA13