testbeds and projects with ongoing ensemble research: hydrometeorology testbed (hmt)
DESCRIPTION
DET Collaboration with Other Projects: Past and Future Ed Tollerud 1 and Tara Jensen 2 1 NOAA/GSD, Boulder, CO 2 NCAR/RAL, Boulder, CO . Testbeds and Projects with Ongoing Ensemble Research: Hydrometeorology Testbed (HMT) Hazardous Weather Testbed (HWT) - PowerPoint PPT PresentationTRANSCRIPT
Testbeds and Projects with Ongoing Ensemble Research:
Hydrometeorology Testbed (HMT) Hazardous Weather Testbed (HWT) Hurricane Forecast Improvement Project
(HFIP)
DET Collaboration with Other Projects: Past and Future
Ed Tollerud 1 and Tara Jensen 2
1 NOAA/GSD, Boulder, CO 2 NCAR/RAL, Boulder, CO
Acknowledgements: Tim Schneider, Marty Ralph, Gary Wick, Rob Cifelli, Dave Reynolds, Steve Weiss, Barbara Brown, Jack Kain, Ming Xue, Fanyou Kong, Adam Clark, Patrick Marsh, Mike Coniglio, Russ Schneider, Evan Kucera, Tom Hamill, Tony Eckel, Louisa Nance, Mark DeMaria, James Franklin, Ed Rappaport, Mike Fiorino and many others
MotivationMany testbeds now have an ensemble componentRobust and objective evaluation of promising
techniques neededDetermine potential utility for operations
Model A
Model C
Model E
Model B
Ensemble D
Forecast
Exercise
Testbed Operations
General Format of Many Testbeds
Research
GoalsLeverage off work done in other
testbedsExtend understanding of their
resultsHelp in planning for next year or
seasonFacilitate faster Research to Ops
(R2O)
Effo
rt
Time
Fore
cast
Exe
rcis
e
DET Retrospective Runs
There is much work to be done in the off-season -DET might be able to help
Most effort expended around R2O ForecastExercise
Recent Activities
HWT Participants Evaluate Models
Traditional Scores for HWT ensemble members and baselines
Object-Oriented Evaluation Of HMT members
2010 DTC/HMT Ensemble Activities
High-resolution 9 member WRF regional ensemble for HMT-West Winter Exercises ◦First DTC ensemble evaluation◦Individual deterministic models◦Simple ensemble mean calculated and evaluated
30-day Summary Scoring Displays Baseline Comparisons (with GFS)
2010 HMT Ensemble NWP9 member ensemble
◦Grid: 9 km outer - 3 km inner◦LBC: GFS Ensemble Members
North American domain 80 km resolution
◦IC: LAPS
• January Summary, ETS
• 24h forecast, full domain
• Verified at 24h gage sites
• Segregated by threshold
• Dramatic performance hit at higher thresholds
• No real ensemble member advantages in statistics but large inter-quartile range
Real-time QPF verification for HMT-West
2011 HMT Ensemble NWPDET 1st Benchmark (NCEP) will
hopefully be run for experiment◦ North American Domain◦ 20 km resolution◦ IC “cycling” option
9 member ensemble◦ Grid: 9 km outer – 3 km inner (or potentially
finer resolution)◦ IC: LAPS and/or Cycling◦ LBC: DET 1st Benchmark (NCEP)
Future DTC/HMT Activities◦Use DET Ensemble Configuration
Module to set-up and run◦First use of MET v3.0
including Ensemble-Stat Tool◦Additional Baselines◦Ensemble-based
Spatial Verification◦Estimate Uncertainty in
Verification Statistics
What can HMT do for DET• HMT used for test domain for DET
Module 2 (IC/LBC) Benchmark• Workflow Procedures will backbone of
DET system• Input data options for initialization and
verification (eg., QPF)• Ensemble Products Algorithms for
inclusion in Module 5 (Products/Display)
• Exploration of Spatial Verification Applications
What can DET do for HMTHelp determine optimal
ensemble configurationEvaluate Ensemble Products
◦Visualization utilities (Displays)◦Probability Products
State-of-the-art probabilistic verification measures
HWT 2008 •Introduce Objective
Evaluation
HWT 2009
•Realtime system•Address scientific
question
HMT 2010
•1st Ensemble evaluation
•Satellite data into MET
HWT 2010
•Add Ensemble methods
•AWC/HPC present
HMT 2011
•Refine Ensemble methods
•Data Impact Studies
ModelEvaluatio
nTools (MET)
DTCEnsembleTestbed(DET)
etc…
2010 DTC/HWT Ensemble ActivitiesEvaluated 26 member CAPS Storm-Scale
Ensemble Forecast System during 2010 Spring Experiment◦ Baselines: HRRR, SREF, NAM◦ Ensemble Products
generated by CAPS– using 15 memberswith radarassimilation
◦ First plots of BrierScore and Area Under Curve
NSSL Q2 QPE
CAPS SimplePQPF
SREF SimplePQPF
CAPSPN PQPF
NAM QPF
SSEFPMQPF
PQPF QPF
•Evaluation of Simple and Derived Ensemble Products: Probability fields using Neighborhood Method Probability matching method for QPF
Future DTC/HWT Activities◦ DET participates in planning of CAPS ensemble
constituents potentially with DET Ensemble Configuration Module, IC/LBC, and Physics Modules
◦ Real-time verification performed using DET Verification Module (HWT-DTC Collaboration)
◦ HWT products evaluated in DET Module 5 (Products/Display)
◦ Feedback to model developers
RETOP
How DET may help HWTModule 1 - Work with CAPS to define optimal
ensemble member configurationModule 2 – Work with DTC GSI task to identify
GSI-based IC perturbation capabilities for CAPS ensemble
Module 3 – Evaluate individual members in 2010 dataset to provide suggestions for physics improvements
Module 4 – Implement bias correction technique that requires limited historical data
Module 5 – Evaluate ensemble products in 2010 dataset
Module 6 – Used in all evaluation
HFIP Ensemble WorkMany institutions involvedMany approaches being investigated
Global Ensembles◦ Single Models with IC/LBC
and physics perturbations (GFS, NOGAPS, and FIM)
Regional Ensembles◦ Multi-model ensemble made up of regional
models run by different organizations statistically post-processed (FSU)
◦ Ensembles initialized from EnKF (PSU)Consensus ensembles (NHC)
How DET might contribute to HFIP
• DET could be a testing ground for most promising ensemble techniques
• Retrospective testing to address specific questions:
• Module 2 – Modify near storm environment or storm position?
• Module 3 – Which cumulus schemes provide best spread?
• Module 4 – Which is best way to bias correct track and intensity?
• Module 5 – How do we best display of uncertainty in track and intensity?
• Module 6 – Does spatial (object oriented) verification of ensembles improve understanding of output?
SOME IDEAS FOR THEFUTURE…
MODULE 1:Ensemble
Configuration
MODULE 2:IC/LBC Perturbations
MODULE 3:Physics Perturbations MODULE 4:
Statistical Post-Processing
MODULE 5:Products/Display
MODULE 6:Verification
Mostly Retrospective--------------------------------------
1) ROBUST Test & Evalof promising techniques uncovered
during forecast exercises
Use Near Real-time & Retrospectively
----------------------------------------1) Immediate Feedback during
forecast exercises2) ROBUST Test & Eval
of promising techniques uncoveredduring forecast exercises
Concept of Leveraging DET Modules
Questions for Working Group:
Should DET consider capability for realtime (or near realtime) demonstrations?
Where do we place the “overlap area” between DET and other ensemble-related testbeds? What does that area include?
What are the most critical testbed collaboration issues to focus on in the next year or two?
What interactions with other testbeds beyond HMT and HWT should be explored in the next few years?
HWT 2012 •Pre-Experiment
Evaluation
HWT 2011
•Ensemble membership
•Evaluate New Products
HFIP 2011
•Ensemble Membership
•Statistical Post-Processing
HMT 2012
•Ensemble Methods for Spatial Verification
HMT 2011
•Refine Ensemble Verification methods
•Data Impact Studies
DTCEnsembleTestbed
etc…
ModelEvaluatio
nTools (MET)