© crown copyright met office wafc cat verification objective verification of grib cat forecasts dr...
TRANSCRIPT
© Crown copyright Met Office
WAFC CAT verificationObjective verification of GRIB CAT forecasts Dr Philip G Gill, WAFS Workshop on the use and visualisation of gridded SIGWX forecasts, Paris, 14 September 2009
© Crown copyright Met Office
Contents
This presentation covers the following areas
• Introduction
• SIGWX forecast comparison
• Aircraft data
• Verification methodology
• Verification results
• Summary
• Further improvements
© Crown copyright Met Office
Introduction
• What – Objective verification of gridded binary (GRIB) and significant weather (SIGWX) Clear Air Turbulence (CAT) forecasts
• Where – Global verification
• When – November 2008 to May 2009
• Why – To demonstrate the quality of the new GRIB forecasts using objective verification.
• How – Verification against aircraft observations from the Global Aircraft Data Set (GADS)
© Crown copyright Met Office
Manual SIGWX BUFR and GRIB CAT forecasts
• Manual SIGWX chart
• Example field from new GRIB forecast
© Crown copyright Met Office
Comparison of SIGWX BUFR charts
UK US UK&US
© Crown copyright Met Office
SIGWX BUFR CAT forecast comparison
• One month UK-US comparison (January 2009)
• Average coverage of globe UK ~6%, US ~3%
• Percentage overlap of all forecasts between UK and US ~20%
Areas forecast by both UK and US
Areas forecast by UK but not USAreas forecast by US but not UK
© Crown copyright Met Office
GRIB forecast comparisonproduced by HKO
© Crown copyright Met Office
Global Aircraft Data Set
• Archive of aircraft data set up by Joel Tenenbaum (State University of New York)
• British Airways fleet of Boeing 747-400 aircraft
• Global coverage, but flights mainly over northern hemisphere
• Automated aircraft observations every 4 seconds
• Indicator of turbulence derived from vertical acceleration, aircraft mass, altitude and airspeed called the derived equivalent vertical gust (DEVG).
© Crown copyright Met Office
GADS Data coverage
10-day sample of GADS data
© Crown copyright Met Office
Verification methodology
© Crown copyright Met Office
Forecast assessment
• Turbulent/non turbulent event defined on 10min aircraft track ~120km - approx grid size
• Forecast turbulent event – CAT potential >= Threshold
• Observed turbulent event –
Slight or greater turb (DEVG>=2m/s)
Moderate or greater turb (DEVG>=4.5m/s)
• Construct 2x2 contingency tables for each threshold
• Sum entries in contingency tables over the verification period
• Produce a Relative Operating Characteristic (ROC) curve by plotting the Hit rate against False alarm rate for each threshold.
Turbulence observed
No turbulence observed
Turbulence forecast
Hit False alarm
No turbulence forecast
Miss Correct rejection
2x2 contingency table
© Crown copyright Met Office
Results UK GRIB and SIGWXmoderate or greater turbulence
ROC curve UK WAFC GRIB v BUFR SIGWX CAT moderate or greater turbulence (DEVG>=4.5m/s) Jan 2009 - May 2009 verified against global GADS data
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
rat
e
UK T+24 GRIB
UK T+24 BUFR
~500,000 events
~300 turbulent events (devg>=4.5m/s)
© Crown copyright Met Office
Variation with forecast rangeROC curve January 2009 UK GRIB against Global GADS data
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
rat
e
T+6
T+12
T+18
T+24
~100 000 events
Devg>=2m/s
© Crown copyright Met Office
Latitudinal variation UK GRIBmoderate or greater turbulence
ROC curve UK WAFC GRIB v BUFR SIGWX CAT moderate or greater turbulence (DEVG>4.5m/s) Jan 2009 - May 2009 verified against GADS data 50N-90N
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Flalse alarm rate
Hit
rat
e
UK T+24 GRIB
UK T+24 BUFR
ROC curve UK WAFC GRIB v BUFR SIGWX CAT moderate or greater turbulence (DEVG>=4.5m/s) Jan 2009 - May 2009 verified against GADS data 20N-50N
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
rat
e
UK T+24 GRIB
UK T+24 BUFR
ROC curve UK WAFC GRIB v BUFR SIGWX CAT moderate or greater turbulence (DEVG>=4.5m/s) Jan 2009 - May 2009 verified against GADS data 20S-20N
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
rat
e
UK T+24 GRIB
UK T+24 BUFR
ROC curve UK WAFC GRIB v BUFR SIGWX CAT moderate or greater turbulence (DEVG>=4.5m/s) Jan 2009 - May 2009 verified against GADS data 50S-20S
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
ra
te
UK T+24 GRIB
UK T+24 BUFR
20S to 20N~60,000 events
~100 turb events
90N to 50N~200,000 events
~50 turb events
20N to 50N~200,000 events
~200 turb events
50S to 20S~12,000 events
~10 turb events
© Crown copyright Met Office
Latitudinal variation US GRIBmoderate or greater turbulence
20S to 20N~30,000 events
~50 turb events
90N to 50N~280,000 events
~200 turb events
20N to 50N~100,000 events
~100 turb events
50S to 20S~6,000 events
~3 turb events
ROC curve US WAFC GRIB v BUFR SIGWX CAT moderate or greater turbulence (DEVG>=4.5m/s) Feb 2009 - Apr 2009 verified against GADS data 50S-20S
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
rat
e
US T+24 GRIB
US T+24 BUFR
ROC curve US WAFC GRIB v BUFR SIGWX CAT moderate or greater turbulence (DEVG>=4.5m/s) Jan 2009 - May 2009 verified against GADS data 20S-20N
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
rat
e
UK T+24 GRIB
UK T+24 BUFR
ROC curve US WAFC GRIB v BUFR SIGWX CAT moderate or greater turbulence (DEVG>=4.5m/s) Jan 2009 - May 2009 verified against GADS data 20N-50N
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
ra
te
US T+24 GRIB
US T+24 BUFR
ROC curve US WAFC GRIB v BUFR SIGWX CAT moderate or greater turbulence (DEVG>4.5m/s) Feb 2009 - Apr 2009 verified against GADS data 50N-90N
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Flalse alarm rate
Hit
ra
te
US T+24 GRIB
US T+24 BUFR
© Crown copyright Met Office
UK and US Nov 2008 and Jan 2009 ROC curve
~200 000 events
Devg>=2m/s
ROC curve global CAT T+24 forecasts verified against GADS data Nov 2008 and Jan 2009
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
rat
e
EGRR GRIB T+24
KKCI GRIB T+24
EGRR SIGWX BUFR T+24
KKCI SIGWX BUFR T+24
© Crown copyright Met Office
UK GRIB and SIGWX
~300 000 events
Devg>=2m/s
SIGWX automated object - UK automated SIGWX chart production system based on GRIB data
ROC Curve UK WAFC, UK SIGWX and CAT object verification against global GADS data
November 2008-January 2009
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
False alarm rate
Hit
rat
e
GRIB T+24
SIGWX BUFR T+24
SIGWX SWAM T+24 moderateCAT
SIGWX SWAM T+24 severe CAT
© Crown copyright Met Office
Summary of results
• Both UK and US GRIB products show more skill than the manual SIGWX products.
• Global UK and US GRIB CAT forecasts score similarly
• Slight difference in scores as forecast range increases
• Some differences in scores at individual latitude bands –best performance between 20N and 90N.
• UK CAT coverage on manual BUFR SIGWX charts greater than the US
© Crown copyright Met Office
Further improvements
• Automate verification process
• Improve consistency of forecasts by analysing verification data and altering production systems.
• Use verification to test future model upgrades and re-tune algorithms
© Crown copyright Met Office
Questions and answers