Assessing added value of high resolution forecasts

Emiel van der PlasMaurice Schmeits, Kees Kok

KNMI, The Netherlands

2/15

IntroductionQuestion: do high resolution (convection resolving) models perform better than the previous generation models?

T2M, wind, precipitation!

KNMI: Harmonie (2.5 km) > Hirlam(11, 22 km) ?Harmonie > ECMWF (deterministic run: T1279)?

Verification of high resolution NWP forecasts is challengingPrecipitation: highly localisedRadar/stationdata: double penalty!

If there is extra skill, how to demonstrate objectively?In this talk: Fuzzy methods and Model Output Statistics

3/15

Set-upHarmonie (‘ECJAN’):

2.5 km, 300x300 domain, AROME physics, 3DVARECMWF boundariesRun with 800x800 points, Hirlam boundaries: no sufficient archive available…

Hirlam (D11):22 km, 136 x 226, 3DVAR

ECMWF Operational (T1279)±16 km, global, 3DVAR

Radar: Dutch precipitation radar composite1 km

•Period: 1st February 2012 - 31st May 2012

All output resampled to Harmonie grid (nearest neighbour)

4/15

Example of Direct Model OutputE.g. frontal precipitation, 7th March 2012ECJAN, Hirlam, ECMWF, Radar

HarmonieECMWFD11

RADAR

Neo-classical verification: fuzzy methods• MET: suite of verification tools by NCAR (WRF)• Grid based scores: with respect to gridded radar observations

–Fractions Skill Score (Roberts, Lean 2008)–Hanssen-Kuiper discriminant, Gilbert Skill Score (ETS), …

• Object based scores (not in this paper)

5/15

FSS, 3x3, > 1mm/3h GSS, 25x25, > 2mm/3h

6/15

MOS: what is relevant in DMO?• How would a trained meteorologist look at direct model output?

?

7/15

Predictors• How would a trained meteorologist look at direct model output?

8/15

Predictors• How would a trained meteorologist look at direct model output?

9/15

Predictors• How would a trained meteorologist look at direct model output?

10/15

Model Output Statistics: predictive potentialConstruct a set of predictors (per model, station, starting and lead time):

For now: use precipitation onlyUse various ‘areas of influence’: 25,50,75,100 kmDMO, coverage, max(DMO) within area, distance to forecasted precipitation, … , threshold!

Apply (extended) logistic regression [Wilks 2009]Use threshold (sqrt(q)) as predictor:

complete distribution function (Wilks, 2009)Forward stepwise selection, backward deletion

using R: stepPLR (Mee Young Park and Trevor Hastie, 2008)

Verify probabilities based on coefficients of selected predictors in terms of reliability diagrams, Brier Skill Score

11/15

Results: example poor skill

Harmonie

ECMWF

D11

00UTC+003

12/15

Results: example good skill

Harmonie

ECMWF

D11

00UTC+012

Outlook• No conclusive results

• Grid-based, “fuzzy” methods suggest reasonable skill for high resolution NWP model (Harmonie)

• MOS: mixed bag Frontal systems (FMAM) well captured by hydrostatic models

• To do:Larger datasetTraining data, independent dataConvective season: more cases, higher thresholdsInclude Harmonie run on large domain…

13/15

Binary predictand yi (here: precip > q)

Probability: logistic:

Joint likelihood:

L2 penalisation (using R: stepPLR by Mee Young Park and Trevor Hastie, 2008):minimise

Use threshold (sqrt(q)) as predictor: complete distribution function (Wilks, 2009)

Few cases, many potential predictors: pool stations, max 5 terms

14/15

Extended Logistic Regression (ELR)

ECJAN

15/15

Period1st February 2012 -31st May 2012

The archive available for Harmonie was the limiting factorMostly frontal precipitation

ECMWFD11

RADAR

Period: base rate (HSS, HK, FBIAS)

HK

17/15

Verification: classical, Fraction Skill ScoreClassical or categorical verification, eg:

Hanssen-Kuiper discriminant, (aka True Skill Statistic, Peirce Skill Score)(a d – b c)/(a + c)(b + d)

Fraction Skill Score:(Roberts & Lean, 2008)

Straightforward interpretationbut: Double penalty

CTS Observed

yes no

Forecast yes| a b

no | c d

18/15

Verification: MODE (object based), waveletsMET provides access to MODE analysis:“Method for Object-based Diagnostic Evaluation”

Forecast, observation: convolution, thresholded, …

FC OBS

19/15

Verification: MODE (object based), waveletsMET provides access to MODE analysis:“Method for Object-based Diagnostic Evaluation”

… merged, matched and compared.

Center

of mass

Area,

Angle,

Convex hull,

…FC OBS