![Page 1: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/1.jpg)
WWRP/WGNE Joint Working Group on Forecast Verification Research
Beth Ebert, JWGFVR co-chair
WGNR meeting, Geneva, 8-10 Feb 2011
![Page 2: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/2.jpg)
2
Aims
• Verification component of WWRP
• Develop and promote new verification methods
• Training on verification methodologies
• Ensure forecast verification is relevant to users
• Encourage sharing of observational data
• Promote importance of verification as a vital part of experiments
• Promote collaboration among verification scientists, model developers and forecast providers
• Collaborate with WGNE, WCRP, CBS
![Page 3: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/3.jpg)
3
Working group members
Beth Ebert (BOM, Australia)Laurie Wilson (CMC, Canada)• Barb Brown (NCAR, USA)• Barbara Casati (Ouranos, Canada)• Caio Coelho (CPTEC, Brazil)• Anna Ghelli (ECMWF, UK)• Martin Göber (DWD, Germany)• Simon Mason (IRI, USA)• Marion Mittermaier (Met Office, UK)• Pertti Nurmi (FMI, Finland)• Joel Stein (Météo-France)• Yuejian Zhu (NCEP, USA)
![Page 4: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/4.jpg)
4
Sydney 2000 FDP
-0.1
-0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0 10 20 30 40 50 60
Forecast (min)
Ski
ll MA
E
Bia
s (m
m)
sprog
![Page 5: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/5.jpg)
5
Sydney 2000 FDP
TITAN EXTRAP
![Page 6: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/6.jpg)
6
Sydney 2000 FDP
MAEBias
Boundary position error
![Page 7: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/7.jpg)
7
Beijing 2008 FDP
Real Time Forecast Verification (RTFV) system
Fast qualitative and quantitative feedback on forecast system performance in real time
– Verification products generated whenever new observations arrive
Ability to inter-compare forecast systems
3 levels of complexity– Visual (quick look)
– Statistics (quantitative)
– Diagnostic (more information)
![Page 8: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/8.jpg)
8
Training
In person Online
![Page 9: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/9.jpg)
9
Domain and variables
Outer Beijing
Beijing
Urban Beijing
Reflectivity, cell tracks,T-storm strike probability
Precipitation, probability of precipitation
Wind gusts, lightning
![Page 10: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/10.jpg)
10
Precipitation accumulation
Observations
Forecasts
![Page 11: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/11.jpg)
11
Standard verification
![Page 12: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/12.jpg)
12
Diagnostic verification (non-real time)
Power spectral density
Contiguous rain area (CRA)
Intensity-scale
Neighborhood methods
Forecast quality metric (FQM)
![Page 13: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/13.jpg)
13
Survey of BMB forecasters and FDP experts
• Real time verification considered very useful• Forecasters preferred scatter plots and quantile-quantile
plots• Experts wanted capability to drill down to more specific
information• Web interface not user-friendly enough
![Page 14: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/14.jpg)
14
B08FDP lessons for real time verification
• Format and standardization of nowcasts products was critical to making a robust verification system
• Difficult to compare "like" products that were produced with slightly different aims (e.g., QPF for warning vs hydrological applications)
• Verification system design could be improved
– Better display of forecasts with observations
– User control of verification parameters for exploring results
![Page 15: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/15.jpg)
15
SNOW-V10
• Verification plan
– User-oriented verification for Olympic period of all forecasts, tuned to decision points of VANOC
– Verification of parallel model forecasts for Jan to August 2010
– Nowcast and regional model verification
• Rich dataset for user-oriented verification and research
• Processing started
![Page 16: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/16.jpg)
16
SNOW-V10 observations
![Page 17: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/17.jpg)
17
Variable Cat 1 Cat 2 Cat 3 Cat 4 Cat 5 Cat 6 Cat 7 Cat 8 Cat 9
Temperature (°C)
-25 < -25≤ T<-20 -20≤ T<-4C -4≤ T<-2 -2≤ T< 0 0≤ T< +2 +2 ≤ T< +4 ≥ +4
RH (%) < 30% 30≤ RH<
65% 65≤ RH<
90% 90≤ RH<
94% 94≤ RH<
98%≥ 98%
Winds (m/s) < 3 3 ≤ w < 4 4 ≤ w < 5 5 ≤ w < 7 7 ≤ w < 11 11 ≤ w < 13 13 ≤ w < 15 15 ≤ w < 17 ≥ 17
Wind Gust (m/s)
< 3 3 ≤ w < 4 4 ≤ w < 5 5 ≤ w < 7 7 ≤ w < 11 11 ≤ w < 13 13 ≤ w < 15 15 ≤ w < 17 ≥ 17
Wind Direction
d ≥ 339 & d < 24º (N)
24 ≤ d < 69º (NE)
69 ≤ d < 114º (E)
114 ≤ d < 159º (SE)
159 ≤ d < 204º (S)
204 ≤ d < 249º (SW)
249 ≤ d < 294º (W)
294 ≤ d < 339º (NW)
Visibility (m) v < 30 30 ≤ v < 50 50 ≤ v < 200200 ≤ v <
300300 ≤ v <
500≥ 500 - -
Ceiling (m) c < 50 50 ≤ c< 120 120 ≤ c< 300 300 ≤ c< 750 750 ≤ c<
3000 c ≥ 3000 - - -
Precip Rate (mm/hr)
r = 0 (None)0 < r ≤ 0.2
(Trace)0.2 < r ≤ 2.5
(Light) 2.5 < r ≤ 7.5 (Moderate)
r > 7.5 (Heavy)
- - - -
Precip Type No Precip Liquid Freezing FrozenMixed
(w/Liquid)Unknown - - -
Table 5 (2nd Revised Suggestion for SNOW-V10 Verification)
Suggested categories for SNOW-V10 analysis
![Page 18: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/18.jpg)
18
Forecast < 30 30 ≤ x < 50 50 ≤ x < 200 200 ≤ x < 300 300 ≤ x < 500 > 500 Total< 30 0 0 0 0 0 2 2
30 ≤ x < 50 0 0 1 0 1 3 550 ≤ x < 200 0 0 36 10 12 53 111
200 ≤ x < 300 0 0 15 11 9 32 67300 ≤ x < 500 0 0 5 0 10 35 50
> 500 0 10 928 278 191 3824 5231Total 0 10 985 299 223 3949 5466
lam1k Min. Visibility (m) [FB param.] at VOL HSS=0.046Observed
Forecast < 30 30 ≤ x < 50 50 ≤ x < 200 200 ≤ x < 300 300 ≤ x < 500 > 500 Total< 30 0 0 0 0 0 0 0
30 ≤ x < 50 0 0 0 0 0 0 050 ≤ x < 200 0 0 52 20 22 43 137
200 ≤ x < 300 0 0 76 18 19 103 216300 ≤ x < 500 0 1 26 15 12 60 114
> 500 0 9 831 246 170 3743 4999Total 0 10 985 299 223 3949 5466
lam1k Min. Visibility (m) at VOL HSS=0.095Observed
Visibility verification
![Page 19: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/19.jpg)
19
Forecast 339° ≤ x < 24° 24° ≤ x < 69° 69° ≤ x < 114° 114° ≤ x < 159° 159° ≤ x < 204° 204° ≤ x < 249° 249° ≤ x < 294° 294° ≤ x < 339° Total339° ≤ x < 24° 15 9 21 4 1 6 10 15 81
24° ≤ x < 69° 39 66 178 40 15 24 30 30 42269° ≤ x < 114° 96 250 1155 205 48 56 45 71 1926
114° ≤ x < 159° 21 34 249 255 68 25 11 10 673159° ≤ x < 204° 7 14 121 166 144 20 11 4 487204° ≤ x < 249° 5 11 53 53 58 73 56 10 319249° ≤ x < 294° 52 26 102 29 18 74 232 162 695294° ≤ x < 339° 89 45 78 40 13 38 188 454 945Total 324 455 1957 792 365 316 583 756 5548
lam1k Wind Direction at YVR HSS=0.296Observed
Forecast < 3 3 ≤ x < 4 4 ≤ x < 5 5 ≤ x < 7 7 ≤ x < 11 11 ≤ x < 13 13 ≤ x < 15 15 ≤ x < 17 > 17 Total< 3 1498 407 201 226 28 0 0 0 0 2360
3 ≤ x < 4 382 278 200 234 48 1 0 0 0 11434 ≤ x < 5 202 205 203 288 82 4 0 0 0 9845 ≤ x < 7 69 74 91 328 269 7 1 0 0 839
7 ≤ x < 11 6 2 8 58 90 16 28 9 5 22211 ≤ x < 13 0 0 0 0 0 0 0 0 0 013 ≤ x < 15 0 0 0 0 0 0 0 0 0 015 ≤ x < 17 0 0 0 0 0 0 0 0 0 0
> 17 0 0 0 0 0 0 0 0 0 0Total 2157 966 703 1134 517 28 29 9 5 5548
lam1k Wind Speed (mps) at YVR HSS=0.234Observed
Wind verification
![Page 20: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/20.jpg)
20
Sochi 2014
Standard verification
Possible verification innovations:• Road weather forecasts
• Real-time verification
• Timing of events – onset, duration, cessation
• Verification in the presence of observation uncertainty
• Neighborhood verification of high-resolution NWP, including in time-height plane
• Spatial verification of ensembles
• User-oriented probability forecast verification
![Page 21: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/21.jpg)
21
PublicationsPublications
Recommendations for verifying deterministic and probabilistic quantitative precipitation forecasts
Available on WWRP web site
Coming soon:
Recommendations for verifying cloud forecasts
Later this year (?):
Recommendations for verifying tropical cyclone forecasts
![Page 22: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/22.jpg)
22
Publications
January 2008 special issue of Meteorological Applications on forecast verification
–Verification review paper authored by JWGFVR members
–18 contributed papers from participants in 2007 International Verification Methods workshop
![Page 23: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/23.jpg)
23
Publications
DVD from 2009 Helsinki Verification Tutorial
Video and powerpoint lectures– Verification basics– Continuous verification – Categorical verification– Warnings verification– Probabilistic verification – Spatial verification– Statistical inference
Available from WMO
![Page 24: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/24.jpg)
24
Spatial Verification Method Intercomparison Project
• International comparison of many new spatial verification methods
• Methods applied by researchers to same datasets (precipitation; perturbed cases; idealized cases)
• Subjective forecast evaluations
• Workshops: 2007, 2008, 2009
• Weather and Forecasting special collection
• 13 papers on specific methods, 2 overview papers
http://www.rap.ucar.edu/projects/icp
![Page 25: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/25.jpg)
25
Spatial Verification Method Intercomparison Project
![Page 26: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/26.jpg)
26
Facial verification?
![Page 27: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/27.jpg)
27
Spatial Verification Method Intercomparison Project
• Future variables – "Messy" precipitation– Wind– Cloud– Timing errors
• Future datasets– MAP D-PHASE– SRNWP / European data– Nowcast dataset(s)??
• A model for verification test bed
![Page 28: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/28.jpg)
28
Outreach
• Strong focus of the WG
• EUMETCAL training modules completed (Nurmi, Wilson)
• Verification web page
• Sharing of tools
• Tutorials (traveling and on-site)– ECMWF (Jan 2007)– South Africa (Sept 08)– Helsinki (June 2009)
http://www.cawcr.gov.au/projects/verification/
![Page 29: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/29.jpg)
29
International Verification Methods Workshops 4th Workshop – Helsinki 2009
Tutorial
• 26 students from 24 countries
• 3 days
• Lectures, hands-on (took tools home)
• Group projects - presented at workshop
Workshop
• ~100 participants
• Topics: – User-oriented verification
– Verification tools & systems
– Coping with obs uncertainty
– Weather warning verification
– Spatial & scale-sensitive methods
– Ensembles
– Evaluation of seasonal and climate predictions
![Page 30: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/30.jpg)
30
Workshop: New verification research
Spatial methods applied to:
http://www.space.fmi.fi/Verification2009/
Wind fields Ensemble forecasts
![Page 31: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/31.jpg)
31
Workshop: New verification research
http://www.space.fmi.fi/Verification2009/
ExtremesDiagnostics
False alarm ratio
Hit
rate
![Page 32: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/32.jpg)
32
![Page 33: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/33.jpg)
33
5th International Verification Methods Workshop
• Melbourne, December 2011
• 3-day tutorial + 3-day scientific workshop
• Additional tutorial foci
– Verifying seasonal predictions
– Brief intro to operational verification systems
• Capacity building for FDPs/RDPs, SWFDP, etc.
![Page 34: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/34.jpg)
34
New focus areas for JWGFVR research
• Seamless verification – crossing space/time scales
• Ensemble predictions
• Warnings / extreme events, especially timing
• Aviation
• Multivariate verification – joint distributions
![Page 35: WWRP/WGNE Joint Working Group on Forecast Verification Research](https://reader036.vdocuments.mx/reader036/viewer/2022062408/568139f0550346895da1aa04/html5/thumbnails/35.jpg)
35
Future collaboration with WGNR
• SNOW-V10
• Sochi 2014
• WENS?
• SWFDP (Africa, SW Pacific, SE Asia)
• Lake Victoria FDP
Intend to establish collaborations with SERA on(a) verification of tropical cyclone forecasts and other
high impact weather warnings(b) Lake Victoria FDP