Probabilistic QPFs for the Indian Monsoon

using Reforecasts

Tom Hamill

NOAA / ESRL

tom.hamill@noaa.gov

NOAA Earth SystemResearch Laboratory

Outline• Background

– Why reforecasting?– NOAA’s reforecast data set.– How skill can be overestimated using the conventional

method of applying metrics– Monsoon climatology– Logistic regression review

• Results– Stepwise elimination : logistic regression results– Brier skill scores and forecast reliability, before

and after calibration using logistic regression– Some examples of actual calibrated forecasts

Problem with current ensemble forecast systemsForecasts may be biased and/or deficient in spread,

so that probabilities are mis-estimated. “Calibration” (statistical correction) needed.

Heavy rain in anarea where none ofthe ensemble memberspredicted it.

http://www.spc.noaa.gov/exper/sref/

This article onreforecasting in the Bulletinof the AmericanMeteorological Society is a goodplace to start foran overview.

NOAA’s reforecast data set

• Model: T62L28 NCEP GFS, circa 1998

• Initial States: NCEP-NCAR Reanalysis II plus 7 +/- bred modes.

• Duration: 15-day integrations every day at 00Z from 19781101 to now. (http://www.cdc.noaa.gov/people/jeffrey.s.whitaker/refcst/week2).

• Data: Selected fields (winds, hgt, temp on 5 press levels, precip, t2m, u10m, v10m, pwat, prmsl, rh700, heating). NCEP/NCAR reanalysis verifying fields included (Web form to download at http://www.cdc.noaa.gov/reforecast). Data saved on 2.5-degree grid.

• Experimental precipitation forecast products: http://www.cdc.noaa.gov/reforecast/narr .

2.5°

1.0°

Reforecast data was archived on global domain. For this experimentwe saved forecast total precipitation,column precipitable water, and sea-levelpressure tendency)on coarse and fine grids, as shown,for May 15 - Oct 15, 1979-2007.

Overestimating skill: a review of the Brier Skill Score

Brier Score: Mean-squared error of probabilistic forecasts.

BSf=

1n

pkf −ok( )

2

k=1

n

∑ , ok =1.0 if kthobservation≥threshold0.0 if kthobservation< threshold

⎧⎨⎩

Brier Skill Score: Skill relative to some reference, like climatology.1.0 = perfect forecast, 0.0 = skill of reference.

BSS =BS

f−BS

ref

BSperfect

−BSref =

BSf−BS

ref

0.0 −BSref =1.0 −

BSf

BSref

Overestimating skill: another example

5-mm threshold

Location A: Pf = 0.05, Pclim = 0.05, Obs = 0

BSS =1.0 −BS

f

BSclim =1.0 −

.05 −0( )2

.05 −0( )2=0.0

Location B: Pf = 0.05, Pclim = 0.25, Obs = 0

BSS =1.0 −BS

f

BSclim =1.0 −

.05 −0( )2

.25 −0( )2=0.96

Locations A and B:

BSS =1.0 −BS

f

BSclim =1.0 −

.05 −0( )2 + .05 −0( )2

.25 −0( )2 + .05 −0( )2=0.923

why not 0.48?

An alternative BSSSay m overall samples, and k categories

where climatological event probabilities are similar in this category. ns(k) samples assigned to this category. Then form BSS from weighted average of skills in the categories.

BSS =ns k( )

mk= 1

nc

 1 -BS

fk( )

BSc l i m

k( )

Ê

Ë

ÁÁÁÁÁ

ˆ

¯

˜̃˜̃˜̃

(for more details on all of this, see Hamill and Juras, QJRMS, October C, 2006)

Monsoon precipitation climatology

Monsoon precipitation climatology

Monsoon precipitation climatology

Monsoon precipitation climatology

Logistic regression

• Predictors tested: √(ensemble-mean precip), precipitable water, SLP tendency

• Observed data: Indian precipitation analyses on 1-degree grid, 1979-2004

• Stepwise elimination to determine which predictors are useful.• Train with data +/- 10 days around date of interest. Cross

validated, so, for example, regression coefficients for 1979 were trained on 1980-2004 data.

• Test June 1 - October 1, 1979 - 2004.

P(Obs >Threshold) =1.0 −

1.01.0 + exp β0 + β1x1 +L + βnxn{ }

where x1, xn are model predictors, betas are fittedregression coefficients. Used NAG library routine.

Which predictors in logistic regression with stepwise elimination? Day 1

For every day of the monsoonseason, a stepwise linear regression was run to determinewhich predictors provided a reduction in error. As shown,a power-transformed ensemble-mean forecast precipitation was uniformly selected as an important predictor. Precipitablewater was occasionally selected,and sea-level pressure changewas virtually never selected. Based on these results, all subsequent logistic regressionanalyses will be based on usingonly one predictor, the power-transformed ensemble-meanprecipitation amount.

Which predictors in logistic regression with stepwise elimination? Day 3

The same conclusionis reached whenconsidering otherforecast leads.

Brier Skill Scores

Confidenceintervals areso small theydon’t show upon the plot.

Reliability, Ens. Relative Frequency, 1 and 5 mm

5, 95 percent confidenceintervals via block bootstrap.

solid lines: frequencydistribution of climatology

Reliability, Ens. Relative Frequency, 10 and 25 mm

Reliability, Logistic Regression, 1 and 5 mm

Reliability, Logistic Regression, 10 and 25 mm

Map of Logistic Regression BSS, Day 1

Note: This methodof calculating BSSlumps all samplesat a particular gridpoint together fordates between 1 Mayand 1 October. To theextent that theclimatological eventprobability varies overthis range of dates, theBSS may be somewhatinflated. Please readHamill and Juras,QJRMS, Oct (c) 2006.

Map of Logistic Regression BSS, Day 2

Map of Logistic Regression BSS, Day 3

Map of Logistic Regression BSS, Day 4

Map of Logistic Regression BSS, Day 5

Monthly variations of skill, 1 mm

Monthly variations of skill, 10 mm

More skill laterin the monsoonseason.

Logistic regression forecast example #1, 1-day lead

Logistic regression forecast example #1, 3-day lead

Logistic regression forecast example #2, 1-day lead

Logistic regression forecast example #2, 3-day lead

Logistic regression forecast example #3, 1-day lead

Logistic regression forecast example #3, 3-day lead

Logistic regression forecast example #4, 1-day lead

Logistic regression forecast example #4, 3-day lead

Conclusions

• Precipitation probabilities estimated directly from the ensemble are very unreliable and unskillful because of substantial model deficiencies (coarse resolution, sub-optimal model physics, methods of generating ensemble, limited ensemble size).

• With a large data set of past forecasts (“reforecasts”) using the same model that is run operationally (here, a 1998 version of NCEP’s GFS), the model forecasts can be post-processed to yield reliable and somewhat skillful probabilistic forecasts.

• The 1st-generation NOAA reforecast model is out of date, but there is growing interest worldwide in producing reforecast data sets with current models (e.g., ECMWF will produce limited reforecasts starting in 2008).

References(downloadable from www.cdc.noaa.gov/people/tom.hamill/cv.html) • Hamill, T. M., J. S. Whitaker, and X. Wei, 2004: Ensemble re-forecasting: improving medium-range forecast skill

using retrospective forecasts. Mon. Wea. Rev., 132, 1434-1447.

• Hamill, T. M., J. S. Whitaker, and S. L. Mullen, 2006: Reforecasts, an important dataset for improving weather predictions. Bull. Amer. Meteor. Soc., 87, 33-46.

• Hamill, T. M., and J. S. Whitaker, 2006: Probabilistic quantitative precipitation forecasts based on reforecast analogs: theory and application. Mon. Wea. Rev.,134, 3209-3229.

• Hamill, T. M., and J. Juras, 2006: Measuring forecast skill: is it real skill or is it the varying climatology? Quart. J. Royal Meteor. Soc., 132, 2905-2923.

• Wilks, D. S., and T. M. Hamill, 2007: Comparison of ensemble-MOS methods using GFS reforecasts. Mon. Wea. Rev., 135, 2379-2390.

• Hamill, T. M., and J. S. Whitaker, 2006: Ensemble calibration of 500 hPa geopotential height and 850 hPa and 2-meter temperatures using reforecasts. Mon. Wea. Rev., 135, 3273-3280

• Hagedorn, R, T. M. Hamill, and J. S. Whitaker, 2007: Probabilistic forecast calibration using ECMWF and GFS ensemble forecasts. Part I: 2-meter temperature. Mon. Wea. Rev., accepted.

• Hamill, T. M., R. Hagedorn, and J. S. Whitaker, 2007: Probabilistic forecast calibration using ECMWF and GFS ensemble forecasts. Part II: precipitation. Mon. Wea. Rev., accepted.