ensemble-4dwx update: focus on calibration and verification

© 2009 UCAR. All rights reserved.ATEC-4DWX IPR, Aug 11-12 2009

National Security Applications Program Research Applications Laboratory

Ensemble-4DWX update: focus on calibration and verification

Tom Hopson, Josh Hacker, Jason Knievel, Yubao Liu, Gregory Roux, Wanli Wu

© 2009 UCAR. All rights reserved.


Reminder of status in April 2009

• Running WRF V3.0.1.1

• Had just finished running E-4DWX for ATC during Dec 2008 and Jan 2009, then returned configuration to DPG

• Configured and ran E-4DWX for supporting UT Division of Air Quality

• Improved MYJ and YSU PBL height diagnosis and PBL mixing

• Added new graphics and improve post-processing flexibility (installation for GMOD and plotting historical case archive) and computing parallelisms

• Fully automated calibration just being tested

• Calibrated variables: 2-m T

ATEC-4DWX IPR, Aug 11-12 2009



Progress since April 2009 (1 of 2)

• Improved scripting to manage the computational resources

• Increased stations from 26 to 34 (regional)

• Calibration hindcast increased from roughly 600 to over 1000 points– Increased hindcast size allows calibration of 2 additional lead-times (36hr, 42hr)

• Included new skill measure for ensemble skill-spread relationship

• Corrected over-dispersion of calibration process

• Calibrated variables: 2-m T, 2-m Td (new)




Progress since April 2009 (2 of 2)

• Examined results from CAD study at ATC (earlier talk)

• Using performance of ensemble members to optimize DPG’s single-model configuration

• Submitted proposal for brief paper in BAMS but was rejected. Have not decide whether to revise and resubmit

• Continuing to work on scientific paper for MWR

• Downtime of HPC (recent):May 13-20, Agami4 failureJune 16-18, SGI nodeJuly 11-19, power outage caused Agami4 failureJuly 29 - August 4, Agami4 failure




Calibration and verification

• Ensemble calibration to correct predicted distribution

• Calibration is needed for users capable of decision making with probabilistic guidance. Will be needed for foreseeable future

• Verification of different ensemble characteristics is easily completed when performing calibration




Calibration: review and recap

• Calibration (and verification) is now fully-automated

• Utilizes “persistence” if available

• 34 sites in and near DPG

• Full calibration for all sites ~ 1X per week for each weather variable

• Using lookup tables ~ every hour (8 hrs, was 1hr)


Pro

babi

lity

calibration

Temperature [K]

Pro

babi

lity

Temperature [K]

“spread” or “dispersion”

“bias”obs

obs

ForecastPDF

ForecastPDF



3-hr dewpoint time series

Before Calibration After Calibration


Station DPG S01



42-hr dewpoint time series

Before Calibration After Calibration


Station DPG S01

NCAR/RAL - National Security Applications Program10

obs

Blue is “raw” ensembleBlack is calibrated ensembleRed is the observed value

Notice: significant change in both “bias” and dispersion of final PDF

(also notice PDF asymmetries)


PDFs: raw vs. calibrated

NCAR/RAL - National Security Applications Program11ATEC-4DWX IPR, Aug 11-12 2009

Troubled rank histograms

1 2 3 4 5 6 7 8 9 10Ensemble #

1 2 3 4 5 6 7 8 9 10Ensemble #

Co

un

ts0

1020

30

Co

un

ts0

1020

30



3-hr dewpoint rank histograms


Station DPG S01




Station DPG S01

42-hr dewpoint rank histograms



RMSE of ensemble members

3hr Lead-time 42hr Lead-time


Station DPG S01



Significant calibration regressors

3hr Lead-time 42hr Lead-time


Station DPG S01



Future plans

Refine lookup table process to decrease computation time (required as calibration data size continues to increase and new weather variables are added)

Implement for a) wind speed, b) wind direction, c) precipitation,d) surface pressure

Diagnose most informative model set to use operationally

Develop scheme for model points without surface observations over whole model domain


ensemble-4dwx update: focus on calibration and verification

Documents