1

Object Based Cluster-Analysis and Verification of a Convection-Allowing Ensemble during the 2009 NOAA

Hazardous Weather Testbed Spring Experiment

Aaron Johnson and Xuguang Wang

School of Meteorology and Center for Analysis and Prediction of Storms

Fanyou Kong (CAPS), Ming Xue (SOM&CAPS), Kevin Thomas (CAPS), Keith Brewster (CAPS), Yunheng Wang (CAPS), Jidong Gao (NSSL)

Warn-on-Forecast and High Impact Weather Workshop9 February 2012

2

Motivation• Convection allowing forecasts result in realistic looking convective systems for storm mode

forecasts. (Coniglio et al. 2010).

• Object based verification is more consistent with subjective evaluations of high resolution precipitation forecasts than traditional metrics (e.g., Davis et al. 2006a; Johnson et al. 2011a).

– Physically descriptive diagnosis of errors – Deterministic, not just probabilistic, verification is needed for model development and ensemble design.– A different perspective on deterministic verification than traditional metrics (e.g., Kong et al. 2009)

• Object based cluster-analysis can show impact of perturbation sources on forecast diversity (Yussouf et al. 2004)

• Model dynamics have a dominant impact on the 2009 CAPS ensemble clustering (Johnson et al. 2011b)

– Object-based verification can help us understand the differences between ARW and NMM

• Optimal grid spacing to balance computational cost and forecast quality is still an open question (e.g., Schwartz et al. 2009)

– Is it worth going from 4 km to 1 km grid spacing?

3

Outline• Overview of ensemble and object-based methodology• Object-based cluster analysis• Forecast object realism evaluated with sample-climate average of

object attributes– Forecasts vs. observations– ARW vs. NMM– 1 km vs. 4 km

• Forecast accuracy evaluated with object-based MMI and a newly proposed OTS– Forecasts vs. observations– ARW vs. NMM– 1 km vs. 4 km

• Summary and Conclusions

20 members initialized 00 UTC, integrated 30 hours over near-CONUS domain on 26 days from 29 April through 5 June 2009, on 4 km grid without CP.

MemberIC LBC R MP PBL SW Rad. LSM

ARWCN CN NAMf Y Thompson MYJ Goddard NOAHARWC0 NAMa NAMf N Thompson MYJ Goddard NOAHARWN1 CN – em N1 em N1 Y Ferrier YSU Goddard NOAHARWN2 CN – nmm N1 nmm N1 Y Thompson MYJ Dudhia RUCARWN3 CN – etaKF N1 etaKF N1 Y Thompson YSU Dudhia NOAHARWN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Goddard NOAHARWP1 CN + em N1 em N1 Y WSM6 MYJ Dudhia NOAHARWP2 CN + nmm N1 nmm N1 Y WSM6 YSU Dudhia NOAHARWP3 CN + etaKF N1 etaKF N1 Y Ferrier MYJ Dudhia NOAHARWP4 CN + etaBMJ N1 etaBMJ N1 Y Thompson YSU Goddard RUC

NMMCN CN NAMf Y Ferrier MYJ GFDL NOAHNMMC0 NAMa NAMf N Ferrier MYJ GFDL NOAHNMMN2 CN – nmm N1 nmm N1 Y Ferrier YSU Dudhia NOAHNMMN3 CN – etaKF N1 etaKF N1 Y WSM6 YSU Dudhia NOAHNMMN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Dudhia RUCNMMP1 CN + em N1 em N1 Y WSM6 MYJ GFDL RUCNMMP2 CN + nmm N1 nmm N1 Y Thompson YSU GFDL RUCNMMP4 CN + etaBMJ N1 etaBMJ N1 Y Ferrier YSU Dudhia RUCARPSCN CN NAMf Y Lin TKE 2-layer NOAHARPSC0 NAMa NAMf N Lin TKE 2-layer NOAH

10 members are from WRF-ARW, 8 members from WRF-NMM, and 2 members from ARPS.

MemberIC LBC R MP PBL SW Rad. LSM

ARWCN CN NAMf Y Thompson MYJ Goddard NOAHARWC0 NAMa NAMf N Thompson MYJ Goddard NOAHARWN1 CN – em N1 em N1 Y Ferrier YSU Goddard NOAHARWN2 CN – nmm N1 nmm N1 Y Thompson MYJ Dudhia RUCARWN3 CN – etaKF N1 etaKF N1 Y Thompson YSU Dudhia NOAHARWN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Goddard NOAHARWP1 CN + em N1 em N1 Y WSM6 MYJ Dudhia NOAHARWP2 CN + nmm N1 nmm N1 Y WSM6 YSU Dudhia NOAHARWP3 CN + etaKF N1 etaKF N1 Y Ferrier MYJ Dudhia NOAHARWP4 CN + etaBMJ N1 etaBMJ N1 Y Thompson YSU Goddard RUC

NMMCN CN NAMf Y Ferrier MYJ GFDL NOAHNMMC0 NAMa NAMf N Ferrier MYJ GFDL NOAHNMMN2 CN – nmm N1 nmm N1 Y Ferrier YSU Dudhia NOAHNMMN3 CN – etaKF N1 etaKF N1 Y WSM6 YSU Dudhia NOAHNMMN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Dudhia RUCNMMP1 CN + em N1 em N1 Y WSM6 MYJ GFDL RUCNMMP2 CN + nmm N1 nmm N1 Y Thompson YSU GFDL RUCNMMP4 CN + etaBMJ N1 etaBMJ N1 Y Ferrier YSU Dudhia RUCARPSCN CN NAMf Y Lin TKE 2-layer NOAHARPSC0 NAMa NAMf N Lin TKE 2-layer NOAH

Initial background field from 00 UTC NCEP NAM analysis.Coarser (~35 km) resolution IC/LBC perturbations obtained from NCEP SREF forecasts

MemberIC LBC R MP PBL SW Rad. LSM

ARWCN CN NAMf Y Thompson MYJ Goddard NOAHARWC0 NAMa NAMf N Thompson MYJ Goddard NOAHARWN1 CN – em N1 em N1 Y Ferrier YSU Goddard NOAHARWN2 CN – nmm N1 nmm N1 Y Thompson MYJ Dudhia RUCARWN3 CN – etaKF N1 etaKF N1 Y Thompson YSU Dudhia NOAHARWN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Goddard NOAHARWP1 CN + em N1 em N1 Y WSM6 MYJ Dudhia NOAHARWP2 CN + nmm N1 nmm N1 Y WSM6 YSU Dudhia NOAHARWP3 CN + etaKF N1 etaKF N1 Y Ferrier MYJ Dudhia NOAHARWP4 CN + etaBMJ N1 etaBMJ N1 Y Thompson YSU Goddard RUC

NMMCN CN NAMf Y Ferrier MYJ GFDL NOAHNMMC0 NAMa NAMf N Ferrier MYJ GFDL NOAHNMMN2 CN – nmm N1 nmm N1 Y Ferrier YSU Dudhia NOAHNMMN3 CN – etaKF N1 etaKF N1 Y WSM6 YSU Dudhia NOAHNMMN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Dudhia RUCNMMP1 CN + em N1 em N1 Y WSM6 MYJ GFDL RUCNMMP2 CN + nmm N1 nmm N1 Y Thompson YSU GFDL RUCNMMP4 CN + etaBMJ N1 etaBMJ N1 Y Ferrier YSU Dudhia RUCARPSCN CN NAMf Y Lin TKE 2-layer NOAHARPSC0 NAMa NAMf N Lin TKE 2-layer NOAH

Assimilation of radar reflectivity and velocity using ARPS 3DVAR and cloud analysis for 17 members

MemberIC LBC R MP PBL SW Rad. LSM

ARWCN CN NAMf Y Thompson MYJ Goddard NOAHARWC0 NAMa NAMf N Thompson MYJ Goddard NOAHARWN1 CN – em N1 em N1 Y Ferrier YSU Goddard NOAHARWN2 CN – nmm N1 nmm N1 Y Thompson MYJ Dudhia RUCARWN3 CN – etaKF N1 etaKF N1 Y Thompson YSU Dudhia NOAHARWN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Goddard NOAHARWP1 CN + em N1 em N1 Y WSM6 MYJ Dudhia NOAHARWP2 CN + nmm N1 nmm N1 Y WSM6 YSU Dudhia NOAHARWP3 CN + etaKF N1 etaKF N1 Y Ferrier MYJ Dudhia NOAHARWP4 CN + etaBMJ N1 etaBMJ N1 Y Thompson YSU Goddard RUC

NMMCN CN NAMf Y Ferrier MYJ GFDL NOAHNMMC0 NAMa NAMf N Ferrier MYJ GFDL NOAHNMMN2 CN – nmm N1 nmm N1 Y Ferrier YSU Dudhia NOAHNMMN3 CN – etaKF N1 etaKF N1 Y WSM6 YSU Dudhia NOAHNMMN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Dudhia RUCNMMP1 CN + em N1 em N1 Y WSM6 MYJ GFDL RUCNMMP2 CN + nmm N1 nmm N1 Y Thompson YSU GFDL RUCNMMP4 CN + etaBMJ N1 etaBMJ N1 Y Ferrier YSU Dudhia RUCARPSCN CN NAMf Y Lin TKE 2-layer NOAHARPSC0 NAMa NAMf N Lin TKE 2-layer NOAH

• Perturbations to Microphysics, Planetary Boundary Layer, Shortwave Radiation and Land Surface Model physics schemes.

MemberIC LBC R MP PBL SW Rad. LSM

ARWCN CN NAMf Y Thompson MYJ Goddard NOAHARWC0 NAMa NAMf N Thompson MYJ Goddard NOAHARWN1 CN – em N1 em N1 Y Ferrier YSU Goddard NOAHARWN2 CN – nmm N1 nmm N1 Y Thompson MYJ Dudhia RUCARWN3 CN – etaKF N1 etaKF N1 Y Thompson YSU Dudhia NOAHARWN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Goddard NOAHARWP1 CN + em N1 em N1 Y WSM6 MYJ Dudhia NOAHARWP2 CN + nmm N1 nmm N1 Y WSM6 YSU Dudhia NOAHARWP3 CN + etaKF N1 etaKF N1 Y Ferrier MYJ Dudhia NOAHARWP4 CN + etaBMJ N1 etaBMJ N1 Y Thompson YSU Goddard RUC

NMMCN CN NAMf Y Ferrier MYJ GFDL NOAHNMMC0 NAMa NAMf N Ferrier MYJ GFDL NOAHNMMN2 CN – nmm N1 nmm N1 Y Ferrier YSU Dudhia NOAHNMMN3 CN – etaKF N1 etaKF N1 Y WSM6 YSU Dudhia NOAHNMMN4 CN – etaBMJ N1 etaBMJ N1 Y WSM6 MYJ Dudhia RUCNMMP1 CN + em N1 em N1 Y WSM6 MYJ GFDL RUCNMMP2 CN + nmm N1 nmm N1 Y Thompson YSU GFDL RUCNMMP4 CN + etaBMJ N1 etaBMJ N1 Y Ferrier YSU Dudhia RUCARPSCN CN NAMf Y Lin TKE 2-layer NOAHARPSC0 NAMa NAMf N Lin TKE 2-layer NOAH

4

5

Example of Objects by MODE

6

Object based scores• Similarity of two objects is quantified by Total Interest, I (0 < I < 1)

– Function of area ratio, aspect ratio difference, orientation angle difference, centroid distance

– Mean value of attributes is also used to evaluate overall realism of objects • Median of Maximum Interest (MMI; Davis et al. 2009)

– Compute the maximum possible Total Interest (I) for an object, when compared with all other objects at that time.

– Take the median of such maximum interests from all forecast and observed objects

• Object based Threat Score (OTS; Johnson et al. 2011a)– Weight the area of each object by its Total Interest when compared to the

corresponding object in the opposing field.– Sum over all pairs of corresponding objects and divide by total area of all

objects:1

1*( )

Pp p p

ij i jpi j

OTS w a aA A

ED= 1305 mm

1-OTS = 0.486

ED= 1595 mm

1-OTS = 0.381

OTS is subjectively more reasonable as a distance measure

7

Need for object-based approach

8

Advantage of object-based clustering

NED HCA is strongly sensitive to locations and amplitude

OTS HCA can form clusters based on storm modes

• Similar to how we interpret them subjectively

• Consistent with severe storm forecasting applications(Johnson et al. 2011ab, MWR)

9

Object based clustering of 3-h forecasts

• How sensitive are forecasts to different source of perturbations?– Quantify similarity with OTS– Members in same cluster are

systematically more similar than those in different clusters

• 3-hour forecasts are most sensitive to DA, model dynamics, and microphysics scheme

(Johnson et al. 2011ab, MWR)

10

• Little impact of DA

• Model dynamics still dominant

• Secondary clustering by PBL schemes rather than microphysics

Object based clustering of 24-h forecasts

11

Average attributes: forecast vs. observed

• Too many objects forecast after 1-h lead time.

• Average forecast object is smaller, more circular and farther east than average observed object.

12

Average attributes: ARW vs. NMM

• Objects from NMM model are on average more numerous, larger, more circular, more zonally oriented and, beginning at 18 UTC, farther south than ARW.

• Number of objects, mean aspect ratio and mean angle are most similar to Observations for ARW.

• Mean area is most similar to Observations for NMM.

13

Average attributes: 1 km vs. 4 km

• 1 km forecasts fewer, larger, less circular, and farther west on average than 4km.

• 1 km is generally closer to obs for number of objects, area and E-W location than 4km.

14

Accuracy: All members

• OTS maximum at 12-h lead time caused by better forecasts of large precipitation systems at 12 UTC. Diurnal cycle similar to traditional ETS.

• MMI maximum at 24-h lead time caused by realistic meso-scale placement of small precipitation systems at 00 UTC.

• Control member is generally more accurate than perturbed members.• NO DA members were worst especially at early lead times.• NMM worse than ARW and ARPS.

15

Accuracy: ARW vs. NMM• ARW group has significantly

higher OTS and higher frequency of containing the best OTS than the NMM group except short lead times

• Similar result for MMI but less pronounced

• Diagnostics found NMM configurations best at maintaining initial storms and ARW configurations best at forecasting future storms

16

Accuracy: 1 km vs. 4 km

• No significant difference in OTS between 1 km and 4 km members.• 1 km member has significantly lower MMI at 12, 24-h lead times.

– Lower MMI at 12-h lead time caused by missed observed objects, small objects in particular.

– Consistent with worse under-forecasting at 12 UTC seen earlier

17

Summary and ConclusionsClustering analysis:• Cluster analysis shows large impact of model dynamics on forecast clustering, even

after bias adjustment, and additional impact of microphysics at 3-h lead time, PBL scheme at 24-h lead time

Verification:Mean attributes• On average, forecast objects are too numerous, small, circular and east compared

to observation.• ARW vs. NMM: ARW better for number of objects, mean aspect ratio and mean

angle. NMM better for mean area. • 1km vs. 4km: After 1-h lead time, 1km better for number of objects, area and E-W

location than 4km.Accuracy• After 1-h lead time, ARW members are more accurate than NMM members. For

short lead time, NMM configurations seem to evolve assimilated storms better.• Generally similar accuracy at 1 and 4 km grid spacing.

18

4 instead of 16 km smoothing radius

• Still less objects, but not as few as obs

• Still larger area (better)

• Now, similar aspect ratio (less rounded than obs QPE)

• More similar location

19