reliable probabilities through statistical post-processing ... · reliable probabilities through...

Reliable probabilities through statistical post-processingof ensemble predictions

Bert Van Schaeybroeck, Stephane Vannitsem

Royal Meteorological Institute of Belgium

s2d WorkshopToulouse, May 15, 2013

B. Van Schaeybroeck (RMIB) Reliable ensemble predictions s2d Workshop 1 / 16

1 Forecast Calibration

2 Constraints and Method

3 Applications:Lorenz 1996 model15-day EPS weather forecastAntarctic sea ice area

4 Conclusion & Outlook


Introduction

Models suffer from initial condition errors and model errors.Uncertainties are (partly) captures using ensembles.Bias correction of GCMs and RCMs has become commonpractice.Hindcasts available (CMIP5).


Introduction

Models suffer from initial condition errors and model errors.Uncertainties are (partly) captures using ensembles.Bias correction of GCMs and RCMs has become commonpractice.Hindcasts available (CMIP5).

How can we exploit ensemble hindcasts and go beyond correcting asystematic bias?


What can we correct?Bias correction at lead time τ :

µC(τ) = µO(τ).

Variance correction, or climatological reliability (CR):

σ2C(τ) = σ2O(τ).

Ensemble variance correction, or ensemble reliability (ER):

χ2

N=

1

N

N∑n=1

(XC,n −XO,n)2

σ2ε,n

= 1. (1)

Note that this does not imply “weak” ensemble reliability[Johnson and Bowler, 2009, Kharin and Zwiers, 2003]:

1

N

N∑n=1

σ2ε,n =1

N

N∑n=1

(XC,n −XO,n)2. (2)


How do we correct I

Given member m of uncorrected ensemble V m1

= V 1 + εm . Thecorrected forecast is:

Xm

C= α(τ) +

∑p

βp(τ)V p + γ(τ)εm , (3)

Here:

α corrects bias.β corrects ensemble-mean spread.Predictors may include time [Kharin et al., 2012].γ corrects ensemble spread since εm is the deviation from theensemble mean and

γ2(τ) = γ1(τ) + γ2(τ)σ−2ε

(4)


How do we correct II

Moreover:We assume the ensemble-mean error is distributed according todistribution P

(XO −XC

).

We maximize constrained log-likelihood wrt α, β, γ1,γ2,µ and λ:

lnL =

N∑n=1

ln[P(XO,n −XC,n

)]+ λ

(σ2C− σ2

O

)+ µ

(N − χ2

). (5)

Benefits:Simple member-by-member approach.No assumptions for ensemble distribution.Preservation of extremes and ensemble moments higher than two.Constraints can be added.


Other approaches

We compare our CR and ER approaches with:Bias-corrected forecast (α 6= 0, β1 = 1, γ1 = γ2 = 0)Method of Kharin & Zwiers (2003) and Johnson & Bowler (2009):CR + weak-ER (α 6= 0, β1 = 1, γ2 = 0).NGR approach [Gneiting et al., 2005], statistical approachassuming ensemble is normally-distributed and minimizes thecontinuous ranked probability score (CRPS):

CRPS =1

N

∑n

∫[fn(q)− xn(q)]2 dq,

0

0.2

0.4

0.6

0.8

1

Temperature

f=CDF

x

Tobs


Application: Lorenz ’96 modelIdealized case: Lorenz ’96 model

1 36 variables and 10.000 ensembles of 500 members.2 “Observation” run sampled from ensemble at τ = 0 and includes

small model error.3 No time dependence of climate.

-0.2

0

0.2

0.4

0.6

0.8

1

0 2 4 6 8 10

CRPSS

Uncorrected forecast

Bias-corrected forecast

Clim. rel. (CR) + weak ens. rel. (ER)

NGR forecastNew constrained approach

Clim. rel. (CR) forecast

τ LEAD TIME


Lorenz ’96 model: Ensemble reliability

0.1

1

10

100

1000

10000

0 2 4 6 8 10 12 14 16

χ2

NUncorrected forecast




τ LEAD TIME



Lorenz ’96 model: Climatic reliability& ensemble distribution

1

1.2

0 2 4 6 8 10 12 14 16 Ensemble skewness Ensemble kurtosis

0

0.04

0.08

0.12

0 2 4 6 8 10 12 14 2.5

3

3.5

4

4.5

5

5.5

0 2 4 6 8 10 12 14

Forecast variance / observational variance σ /σΟC

2 2

Uncorrected forecastBias-corrected forecast

Clim. rel. (CR) + weak ens. rel. (ER)NGR forecastNew constrained approach


τ LEAD TIME


Application: ECMWF EPS weather forecast1 Each week we train using 9 x 20 hindcasts with 5 members.2 360 verification days of operational 51-member EPS.3 Correction for difference in members.4 Comparison with 30 stations in Belgium.

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

0 50 100 150 200 250 300 350

τ LEAD TIME (hrs)





CRPSS 2m Temperature



Application: ECMWF EPS weather forecast1 Each week we train using 9 x 20 hindcasts with 5 members.2 360 verification days of operational 51-member EPS.3 Correction for difference in members.4 Comparison with 30 stations in Belgium.

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

0 50 100 150 200 250 300 350

τ LEAD TIME (hrs)



Clim. rel. (CR) forecastClim. rel. (CR) + weak ens. rel. (ER)


CRPSS 2m Temperature


EPS weather forecast: Climatic reliability

0.1

1

10

100

1000

0 50 100 150 200 250 300 350 400






χ2

N

τ LEAD TIME (hrs)


EPS weather forecast: Climatic reliability

0.1

1

10

100

1000

0 50 100 150 200 250 300 350 400






χ2

N

τ LEAD TIME


EPS weather forecast: 10m wind speed

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 50 100 150 200 250 300 350

τ LEAD TIME (hrs)



Clim. rel. (CR) forecastClim. rel. (CR) + weak ens. rel. (ER)


CRPSS 10m wind speed


Application: Antarctic sea ice area in LOVECLIMWork in progress on LOVECLIM model

Earth-system model of intermediate complexity.Low computational cost.Eight ten-year hindcasts starting 1980-1996 [Zunz et al., 2013].Initialization of temperature anomaly in SH.Antarctic sea-ice area compared with NSIDC data.

0 5 10 15 20 25 30 35

1013

1012

1011

1010

10 9

Antarctic sea ice areaMSE

MSE uncorrected forecastMSE bias-corrected forecastMSE climatologically-reliable forecast

τ LEAD TIME (months)


Summary & Outlook

ConclusionsWe present a simple post-processing method going beyond biascorrection and constrain:

climatological reliability,ensemble reliability.

Method better at all lead times than established methods thatcorrect ensemble spreads.Outlook: application on sea-ice area with LOVECLIM model +time-dependence due to change of climate [Vannitsem, 2011]


ReferencesGneiting, T., Raftery, A. E., Westveld, A., Goldman, T. (2005).Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation.Mon. Weather Rev. 133, 1098.

Johnson C., and N. Bowler (2009)On the reliability and calibration of ensemble forecasts.Mon. Wea. Rev., 137, 1717.

Kharin, V. V., and F. W. Zwiers (2003).Improved seasonal probability forecasts.J. Climate, 16, 1684.

Kharin, V.V., Boer, G. J., Merryfield, W. J., Scinocca, J. F. and Lee, W.-S. (2012).Statistical adjustment of decadal predictions in a changing climate.Geophys. Res. Lett.,39, L19705.

Vannitsem S. (2009).A unified linear Model Output Statistics scheme for both deterministic and ensemble forecasts,Quart. J. Roy. Meteorol. Soc.,135, 1801.

Vannitsem, S. (2011).Bias correction and post-processing under climate changeNonlin. Processes Geophys., 18, 911-924,

Van Schaeybroeck, B., S. Vannitsem (2011).Post-processing through linear regression.Nonlin. Processes Geophys., 18, 147.

Zunz, V., Goosse, H., and Massonnet, F. (2013).How does internal variability influence the ability of CMIP5 models to reproduce the recent trend in Southern Ocean seaice extent?The Cryosphere, 7, 451-468.


reliable probabilities through statistical post-processing ... · reliable probabilities through...

Documents