![Page 1: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/1.jpg)
Barbara Brown1, Tara Jensen1, Michelle Harrold1, Tressa Fowler1, Randy Bullock1, Eric Gilleland1, and Brian Etherton2
1NCAR/RAL - Joint Numerical Testbed Program2NOAA/ESRL - Global Systems Division
AndDevelopmental Testbed Center
Verification Methods for High Resolution Ensemble Forecasts
Warn-On-Forecast Workshop: 8-9 February 2012
Evaluation and Diagnostic
![Page 2: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/2.jpg)
Large variability in space and time Difficult to identify meaningful
impactsExtreme, high impact, weather
eventsSmall regions of importance
Difficult to identify impacts of forecast “improvements” across whole domain
Verification scores Desire for a single score... But CSI
alone does not give a lot of information about performance or improvements
Relationships and dependencies among scores
Double penalty for displaced high-res forecasts
Challenges for objective evaluation of convective scale short-term prediction...
•ObservationsUncertainties in qualityNeed to be on time and spatial scaled that support evaluation
![Page 3: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/3.jpg)
Traditional Verification Scores
![Page 4: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/4.jpg)
CSI is a nonlinear function of POD and FARCSI depends on base rate (event frequency) and Bias
Ex: Relationships among scores
FAR
POD
CSI
Very different combinations of FAR and POD lead to the same CSI value
1CSI 1 1 1POD 1 FAR
PODBias1 FAR
![Page 5: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/5.jpg)
Freq Bias
Results from HMT-West 2010-2011 seasonCourtesy of Ed Tollerud, DTC/HMT Collab
9km - Ensemble Mean – 6h Precip
6HR>0.1 in.
6HR>0.5 in.
6HR>1.0 in.
6HR>2.0 in.
Performance DiagramAll on same plotPOD1-FAR
(aka Success Ratio)CSIFreq Bias
Best
Success Ratio (1-FAR)
Freq
Bia
s
CSI0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Here we see:Decreasing skill with higher thresholds even with multiple metricsRoebber (WAF, 2009)
Wilson (presentation, 2008)
Dots representDifferent leads
![Page 6: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/6.jpg)
Looking at variability in space and time
![Page 7: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/7.jpg)
14 May 2009 Init: 00 UTC MODE Objects Thresh: 30dBZ
No RadarAssim.
Objects ForecastField
ObservedField
RadarAssim.
SolidFCSTOBJ
LineOBS OBJ
![Page 8: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/8.jpg)
What if you had this: Instead of this:
Objects are 30% too big (MODE area ratio=1.3) FBIAS = 1.3 for grid
Shifted west 40 km(MODE centroid distance = 10 gs) POD = 0.35
Moving too slow(MODE-TD angle diff = 15%)
FAR = 0.72
Peak Rain 1/2” too much(MODE diff in 90th percentile of intensities = 0.5”)
CSI = 0.16
Comparing objects can give more descriptive assessment of the forecast
![Page 9: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/9.jpg)
QPE field Probability Field
Traditional MetricsBrier Score: 0.07Area Under ROC: 0.62
Spatial MetricsCentroid Distance:Obj1) 200 kmObj2) 88kmArea Ratio:Obj1) 0.69Obj2) 0.65
1
2 Median Of MaxInterest: 0.77
Obj PODY: 0.72Obj FAR: 0.32
Okay Forecast with
Displacement Error?
Or Bad Forecastbecause Too
SharpOr
Underdispersive?
QPE_06 >1 in. 50% Prob (APCP_06> 1 in.)
Perfe
ct Reli
abilit
y
OverFcst
UnderFcst
![Page 10: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/10.jpg)
Reflectivity > 30 dBZ MODE Objects (01-06hr)
Looks like there is definitely a timing error
What would a spaghetti plot of MODE objects look like?
Hatched - Observed OBJ Solid - Fcst OBJ
![Page 11: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/11.jpg)
Centroid of object
![Page 12: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/12.jpg)
MODE Time Domain - (MODE – TD)
West
East
Color gives W-Emovement
Tim
e in
crea
sing
![Page 13: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/13.jpg)
Preparing for implementation in Model Evaluation Tools (MET) in next release (Spring 2012)
Applied to model forecasts from 2010 HWT Spring Experiment
Example Attribute of MODE-TD
![Page 14: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/14.jpg)
Conv. Rad. = 10 grid squaresConv Thresh. = APCP_01 > 3 mm
HWT SE 2010 DataMay 17 – June 16, 2010 (18 forecasts/days)
Number of objects:Obs 143, ARPS 161ARW 172, NMM 238
E-W speed (m/s)
Duration (h)
90th percentile intensity
MODE-TD Attributes
![Page 15: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/15.jpg)
Continue with traditional scores (and methods of display) and not gain a true sense of why the ensembles are performing well and poorly- Or -
Adopt performance diagrams, reliability diagrams, rank histograms, and spatial methods for diagnostics and understanding
Take aways: To evaluate high resolution ensembles, you could…
![Page 16: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/16.jpg)
Model Evaluation Tools (MET) developed by the Developmental Testbed Center (DTC)
R statistics package, including R-spatial developed by Eric Gilleland at NCAR/RAL Joint Numerical Testbed (JNT) and R-verification
Where to get these methods?
Recommended websites:http://www.dtcenter.org/met/users
http://www.r-project.org/ http://www.cawcr.gov.au/projects/verification/
http://www.ral.ucar.edu/projects/icp/http://www.dtcenter.org/eval/hwt/2011
The DTC and JNT Program can also serve as a resource for Evaluation Techniques
![Page 17: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/17.jpg)
Extra MaterialToolsIntercomparisonsOutreachCollaboration
![Page 18: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/18.jpg)
www.dtcenter.org/met/usersMETv3.1AvailableNow
METv4.0With GRIB2 support and MODE-TD to be released in late spring 2012
![Page 19: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/19.jpg)
Object Oriented Method: MODEHow it works
OBSENS FCSTRadius=5Thresh>0.25”
Radius=5Thresh>0.25””
Merging
Matching
No false alarms
Misses
Merging
Matched Object 1
Matched Object 2
Unmatched Object
1. Smooth Field 2. Threshold 3. Restore Intensities
4. Merge within 5. Match between 6. Calculate Attributesand Categorical Stats
Example from HMT-West 2010-2011 seasonCourtesy of DTC/HMT Collaboration
![Page 20: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/20.jpg)
Volume / intersection / union / symmetric difference
Axis anglesSpatial orientationAverage speedCentriods – space and time
No way to combine calculation of centriod of space and time because of their differing units
Fuzzy logic handles them by weighting each independently
Metrics to be available in MODE-TD
![Page 21: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/21.jpg)
“Completely updated chapter on the Verification of Spatial Forecasts taking account of the wealth of new research in the area “
Authors: Brown, Ebert, Gilleland
Jolliffe and Stephenson, 2nd Ed
![Page 22: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/22.jpg)
Spatial verification methods in “R”
Implemented by Eric Gilleland
Includes all major spatial methods
R spatial verification package
Example: Neighborhood methods
![Page 23: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/23.jpg)
ICPInternational effort
focused on comparison of capabilities of the different spatial verification methods
Central U.S. precipitation forecasts and observations from HWT 2005
Many publications in WAF special collection
Preparing now for ICP2
Intercomparison project (ICP) and other collaborations
Web site: http://www.ral.ucar.edu/projects/icp/
Other collaborations• Collaborations with
WMO/WWRP Working Groups on Mesoscale and Nowcasting Research– Documents on verification of
mesoscale forecasts and cloud forecasts
• Workshops and tutorials
![Page 24: Verification Methods for High Resolution Ensemble Forecasts](https://reader034.vdocuments.mx/reader034/viewer/2022051821/56816223550346895dd25234/html5/thumbnails/24.jpg)
International collaborationGoals
Further testing of spatial and other methods from ICP1
Application to complex terrainVariables: Precipitation, WindForecasts (Model output), Obs analyses and
observations from 2007 COPS and MAP / D-PhaseVERA analyses (Include ensemble observation
analyses)2 or more sets of model output from MAP D-
PHASE
ICP2