medium-range flood forecasting and warning

A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 2008

Medium-range flood forecasting and warning

X General COSMO meetingCracow, 15-19 September 2008

Basin-oriented verification of

COSMO-LEPS system

Andrea Montani

ARPA-SIM Hydrometeorological service of Emilia-Romagna, Italy

Thanks to H. Asensio, R. Buizza, F. Pappenberger, B. Ritter, J.W.

Schipper


Workpackage Medium-Range Plain Flood of PREVIEW project

Aims:1. Set-up and validation of the probabilistic medium-range flood

forecasting using the meteorological products for the Upper-Danube in the hydrological year 2002, which includes the large flood of August 2002; to achieve this,

• reruns of a number of state-of-the-art of atmospheric models were performed,

• “convenient” (basin-oriented) scoring techniques for probabilistic forecasts were developed.

2. In terms of operational applications, the goal is to demonstrate the usefulness of probabilistic medium-range flood forecasting as a sound basis for early warning and decision making.


The region of interest


Verification methodology

MAIN FEATURES:• Verification performed in terms of 24-hour

precipitation (from 6UTC to 6UTC);

• fcst ranges: 18-42h, 42-66h, 66-90h, 90-114h, …;

• observations: gridded observations (about 5 km of horizontal resolution) provided by JRC;

• verification domain: full upper-Danube river basin and 3 (out of 19) sub-basins;

• verification period: 20 July - 31 August 2002.


Sub-basin verificationID name of Basin area (km2) no. of gridded obs

1 Bratislava 131.978 5278

13 Wiblingen 2.247 80

16 Passau-Ingling 25.977 1045

20 Hofkirchen 47.534 1896

For each gridded observation, it was identified the sub-basin it belonged to.


Deterministic measures for verification

• Catchment-based measures• centre of gravity: it is a specific point at which the system's mass behaves as if it were concentrated. The centre of mass/gravity is a function only of the positions and masses of the particles of the system.

distmean

ccdistCoG obsf CoG: Centre of Gravity measure

cf: forecasted Centre of gravity dist: distance

cobs: observed Centre of Gravity distmean: mean distance of grid elements to catchment outlet

cf=

sNpo

ii

sNpo

iii

p

rp

int

1

int

1

ri are the geographical coordinates at grid point i

pi : forecast precipitation at grid point i

CoG (dimensionless) ranges from 0 to infinity … the lower the better (in practice the score is limited by the value 2, indicating a location error twice the mean catchment distance of the grid elements to the outlet).


Ensemble systems in PREVIEW

– – – – VarEPS by ECMWF (global, ensemble)– COSMO-LEPS by ARPA-SIM (limited-area, ensemble)


CentreofGravity distance Aim: quantify the "location error" of the precipitation forecast (by the ensemble mean); the CoG is scaled with the "mean distance" in a catchment a value of cog=0.3 for the

catchment "Passau-Ingling" (mean distance =113 km) indicates a location error of the forecasted centre of gravity of precipitation in a catchment to the observerd centre of gravity of precipitation of about 30 km.

with the scaling, we get a dimensionless number (…the lower, the better…) to compare the absolute location error with respect to the catchment size.

CoG The absolute figures are very

low (distances of the order of a few tens of km).

Little dependence of the distance from the forecast range.

Slightly higher distances for the Passau-Ingling basin (possibly related to observation problems in the Alpine region).

Best results for the smallest basin.

CoG

dis

tan

ce


Brier Score for VarEPS and COSMO-LEPS

BS

ECMWF : solid. COSMO-LEPS: dotted. Low values for both models:

GOOD! COSMO-LEPS performs better

over the smaller basins (not well resolved by VarEPS).

Similar performance of the two systems over the largest basin.

BS measures the mean squared difference between forecast and observation in probability space.

equivalent to MSE for deterministic forecast. BS between 0 and 1; the lower the better …. the largest (Bratislava) and the smallest (Wiblingen) basins are considered (tp > 80% of obs

distribution).


For the other results, come and see the poster!


Thank you !


The 32-day unified VarEPS at ECMWF

Unification of the 15d VarEPS (50+1 members) and the 32d monthly forecast (MOFC) systems into the unified 32d VarEPS:

+768

15d and 32d VAREPS

T0 +240 +360 +768

NEW SYSTEM (since 11/3/2008)

Twice-a-day (at 00 and 12 UTC):

• d 0-10: TL399L62 uncoupled

• d 10-15: TL255L62 coupled at 00

Once a week:


• d 10-32: TL255L62 coupled

old system

Twice-a-day (at 00 and 12 UTC):



Once a week:

– d 0-32: TL159L62 coupled

15d VAREPS

T0 +240 +360

T0

32d MOFC


Skill of ECMWF predictions for hydrological modelling

Nash_Sutcliffe efficiency coefficient: normally used to assess the predictive power of hydrological models.

… the higher, the better (ranges from – to 1)

Timing of forecasts is good, although underestimation occurs.

The ensemble spread brackets observations only partially.

A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 200817th of July 2007


7 Panel version


Atmospheric models in PREVIEW

– – – – – COSMO-LEPS by ARPA-SIM (limited-area, ensemble)


COSMO-LEPS (developed at ARPA-SIM)

MAIN FEATURES:• initial time: 12 UTC (i.e. once a day);• bc and ic: “selected” VarEPS members;• COSMO-LEPS configuration

– 10 members;– hor. res. = 10 km; 32 vertical levels;– forecast length: 132h;– archived variables: surf and plev up to +132h, every 3h;– output fields archived at ECMWF;

• rerun period: 20 July – 31 August 2002.

COSMO-LEPS: the Limited-area Ensemble Prediction System (LEPS), based on COSMO-model and implemented within COSMO (COnsortium for Small-scale MOdelling, including Germany, Greece, Italy, Poland, Romania, Switzerland).

integration domain


Brier Skill Score vs forecast range BSS is written as 1-BS/BSref. Sample climate is the reference system. Useful forecast systems

if BSS > 0. BS measures the mean squared difference between forecast and observation in probability

space. Equivalent to MSE for deterministic forecast.BSS For low thresholds, better

performance over the smallest basin.

For higher thresholds, more difficult to assess a clear trend (possible sampling problems over small basins).

Worse than climatology only for d+5 range (10 and 20 mm threshold).


Brier Skill Score vs increasing threshold BSS is plotted vs increasing threshold to assess the COSMO-LEPS skill as a function of rainfall

intensity.

BSS positive for all thresholds at both forecast ranges. At low thresholds, it is confirmed the higher skill of COSMO-LEPS over the smallest basin. Stable skill of the system for intermediate basins. For higher thresholds, more difficult to assess a clear trend (sampling problems over the small

basin).

BSS +90h

BSS +42h


Ranked Probability Skill Score A sort of BSS, but “cumulated” over all thresholds. Useful forecast system, if RPSS > 0.

RPSS RPSS always positive.

Better (worse) performance of the system over the smallest (largest) basin.

Almost identical scores using either nearest grid-point (NGP) or bilinear interpolation (BILIN).


ROC area vs increasing threshold ROC area is plotted vs increasing threshold to assess the dependence of COSMO-LEPS HIT/FAR

diagram on rainfall intensity.

ROC area always above 0.7 for all thresholds at both forecast ranges. At low thresholds, it is confirmed the higher skill of COSMO-LEPS over the smallest basin (the

same as BSS). Scores increase with thresholds for intermediate and large basins. For higher thresholds, more difficult to assess a clear trend (sampling problems over the small

basin).

ROC +42h

ROC +90h


ROC area vs forecast range Area under the curve in the HIT rate vs FAR diagram. Valuable forecast systems have ROC area values > 0.6.

ROC ROC area always above 0.6; at low thresholds, better

COSMO-LEPS performance over the smallest basin;

possible sampling problems at 20 mm thresholds.


Outliers How many times the analysis is outside the forecast interval spanned by the ensemble

members. … the lower the better …

OUTL The absolute figures are quite

large (about 20% at d+5 range). Lower outliers for the smallest

basin. Similar percentages for all basins

at the longest forecast range.


Main results and future plans• Output files are of COSMO-LEPS, for the period 20/7/ to 31/8/2002, are

archived on ECFS (retrieval script disseminated) and ready to be used for hydrological purposes; for any kind of help with grib files, just ask.

• Verification against gridded observations (about 5300 in the full upper-Danube basin) indicates better performance of the system over the smallest basin, especially for low thresholds.

• OLD RESULT (Offenbach, 25-26-/9/2007): Verification against SYNOP stations (about 100 in the full upper-Danube basin) indicates slightly better performance of the system over larger basins, although results are not statistically robust.

• Difficulty to understand and use the CoG measure for a probabilistic system.

• Finish the work on the verification report.

• Provide COSMO-LEPS fields for 2-month real-time testing (no “bureaucratic” problems envisaged).

• Real-time testing of services will be done using the “improved” COSMO–LEPS (16 members; 40 vertical levels; physics perturbations).


Main results

• During the hydrological year 2002, reruns of a number of NWP systems, both global-scale and limited-area, both deterministic and probabilistic.

• Development of new

– basin-oriented scores (e.g. centre of gravity);

– hydrological-oriented products (e.g. rivergrams);

• All systems seem to provide useful guidance for the possible occurrence of flood events also for forecast ranges up to 7 (5) days for global (limited-area) systems.

• Verification vs gridded observations (about 5300 in the full upper-Danube basin) indicates better performance of the high-res system over the smaller basin.

• The verification period is probably too short to draw general (and statistically solid) conclusions about the overall skill of the different forecast systems; the “Plain-flood campaign” clearly shows the potential of state-of-the-art NWP systems in the field of weather forecasting for river flooding.


Probabilistic measures for verification

N

kkk of

NBS

1

21

• Brier Score The BS is the mean-squared error of the probability forecasts:

where:

• the observation is either ok = 1 (the event occurs) or ok = 0 (the event does not occur);

• fk is the fraction of ensemble members which forecast a precipitation amount exceeding that threshold

• k denotes a numbering of the N forecast/event pairs.

BS ranges from 0 to 1, the perfect forecast having BS = 0.

BS is computed for a fixed precipitation threshold.

• ROC areaIt is the area under the Relative Operating Characteristics curve in the HIT rate vs FAR diagram

The integral under the curve is used to indicate the skill of the forecast.

ROC area ranges from 0 to 1 … the higher the better …

Useful forecast systems have ROC area values greater than 0.6


Summary of measures

• Deterministic measures:• Root Mean Squared Error • Mean Error• Probability of detection• Probability of false detection• True Skill Statistics• Centre of gravity• Coverage

• Probabilistic measures:• Brier score• ROC area


Performance measures

1) continuous measures (RMSE and MAE for the ensemble mean);

2) catchment-based measures (CentreofGravity for the ensemble

mean);

3) probabilistic measures (Brier Skill Score, ROC area, Percentage of

Outliers, …).Centre of Gravity: the centre of gravity of a system of raster cells is a specific point at

which, for many purposes, the system's mass behaves as if it were concentrated. The centre of mass is a function only of the positions and masses of the particles that comprise the system.

CoG: Centre of Gravity measurec: Centre of gravitydist: Distance f: forecastdistmean: mean distance of grid elements to catchment outletobs: observedp: precipitation

c= this score indicates a location error of the forecast, a perfect score having the

value of 0.

sNpo

ii

sNpo

iii

p

rp

int

1

int

1


Catchment Information SystemRiver-grams

Sylvie Lamy-Thepaut, Enrico Fucile


Main results

• During the hydrological year 2002 (1/10/2001 to 30/9/2002), reruns of a number of NWP systems, both global-scale and limited-area, both deterministic and probabilistic.

• Coordinated efforts to provide state-of-the-art weather forecasts over the Danube river basin.

• Development of basin-oriented scores

• Development of hydrological-oriented products (e.g. rivergrams)

• All systems seem to provide useful guidance for the occurrence of flood events.

increase in ROC area scores and reduction in outliers percentages; positive impact of increasing the population from 5 to 10 members (June

2004); some deficiency in the skill of the system can be identified after the system

upgrades occurred on February 2006 (from 10 to 16 members; from 32 to 40 model levels), BUT

scores are encouraging throughout DPHASE Operations Period.

Time-series verification scores indicate the following trends:


• “Close the gap” between hydrological and meteorological communities.

• Learn from MAP D-PHASE experience

• Exploit the wealth of information provided by probabilistic forecasts:

– assess performance over different domains (North and South of the Alps),

– study individual case studies,

– consider basin-by-basin performance.

• “Think about” increasing horizontal resolution to 7 km.

• Calibrate COSMO-LEPS fcsts using reforecasts (F. Fundel , Meteoswiss).

What is left?


Sub-basin verificationID name of Basin area (km2) no. of “stations” approx no of grid points

1 Bratislava 131.978 5278 1320

13 Wiblingen 2.247 80 20

16 Passau-Ingling 25.977 1045 260

20 Hofkirchen 47.534 1896 470

For each gridded observation, it was

identified the sub-basin it belonged to.


Ranked Probability Skill Score A sort of BSS, but “cumulated” over all thresholds. Useful forecast system, if RPSS > 0.

RPSS RPSS always positive.

Better (worse) performance of the system over the smallest (largest) basin.

Almost identical scores using either nearest grid-point (NGP) or bilinear interpolation (BILIN).


CentreOfGravity distance Scaled (that is, dimensionless) distance between predicted (by COSMO-LEPS ensemble

mean) and observed centre of gravity for each catchment. the "cog" is an attempt to quantify the "location error" of the precipitation forecast; we have

scaled the "cog" with the "mean distance" in a catchment - so a value of cog=0.3 for the catchment "Passau-Ingling" with a mean distance distmean=113 km indicates a location error of the forecasted centre of gravity of precipitation in a catchment to the observerd centre of gravity of precipitation of about 30 km. With the scaling we get a (dimensonless) number to compare the absolute location error with respect to the catchment size.

… the lower the better …CoG

The absolute figures are very low (distances of the order of a few tens of km).

Little dependence of the distance from the forecast range.

Slightly higher distances for the Passau-Ingling basin.

Best results for the Wiblingen basin (the smallest one)

Similar distances for the other basins.

CoG

dis

tan

ce


Catchment Information System River-grams

• Proto type, currently implemented operational.• Database of catchments will be extended to more

catchments in Europe and to include all major World catchments.

• Variables are currently static (always the same for all catchments), but will be ‘dynamic’ – reflecting the usage of catchments.


Outliers How many times the analysis is outside the forecast interval spanned by the ensemble

members. … the lower the better …

OUTL The absolute figures are quite

large (about 20% at d+5 range). Lower outliers for the smallest

basin. Similar percentages for all basins

at the longest forecast range.


Verification of GME forecasts against rain gauge data from high density observation network for the four sub-catchments of the Danube with different catchment sizes: • increasing centre of gravity score („location error“) with increasing forecast time;• a positive mean coverage error for this period.

Catchment-based verification of GME (2)

period 2002-01-01 – 2002-09-30


the verification period might be too short for scores to be statistically significant.

Verification of GME and COSMO-EU forecasts against rain gauge observations for the catchment Hofkirchen (upper Danube): • increasing centre of gravity score („location error“) with increasing forecast time • mostly a negative mean coverage error for this period

Catchment-based verification of DWD models (2)

period 2002-07-19 – 2002-08-20


COSMO-1km – cont. measures

A.Montani; Novelties in weather forecastingPREVIEW training workshop - Mosonmagyaróvár - 19-20 June 2008


Dim 2

Initial conditions Dim 1 Dim 2

Possible evolution scenarios

Dim 1 Initial conditions

ensemble size reduction

Cluster members chosen as representative members (RMs)

LAM integrations driven byRMs

LAM scenario

LAM scenario

LAM scenario

COSMO-LEPS methodologyCOSMO-LEPS methodology


COSMO-LEPS (developed at ARPA-SIM)

• What is it?It is a Limited-area Ensemble Prediction System (LEPS),

based on COSMO-model and implemented within COSMO (COnsortium for Small-scale MOdelling, which includes Germany, Greece, Italy, Poland, Romania, Switzerland).

• Why?It was developed to combine the advantages of global-

model ensembles with the high-resolution details gained by the LAMs, so as to identify the possible occurrence of severe and localised weather events (heavy rainfall, strong winds, temperature anomalies, snowfall, …)

generation of COSMO-LEPS to improve the Late-Short (48hr) to Early-Medium (132hr) range forecast of

severe weather events.


ROC area vs forecast range Area under the curve in the HIT rate vs FAR diagram. Valuable forecast systems have ROC area values > 0.6.

ROC ROC area always above 0.6. Similar results to those

obtained in terms of BSS: at low thresholds, better COSMO-LEPS performance over the smallest basin.

Possible sampling problems at 20 mm thresholds.


Probabilistic measures for verification (1)

The probabilistic measures we used, are the Brier Score and the ROC area. The thresholds for their computation are based on both absolute values (>1mm) and on the percentiles of the observed cumulative precipitation distribution (> 80% of observed).

N

kkk of

NBS

1

21

• Brier Score The BS is the mean-squared error of the probability forecasts; it averages the squared difference between pairs of forecast probabilities and the correspondent binary observations:

where:

•the observation is either ok = 1 (the event occurs) or ok = 0 (the event does not occur);

•k denotes a numbering of the N forecast/event pairs.

BS ranges from 0 to 1, the perfect forecast having BS = 0.

BS is computed for a fixed precipitation threshold and fk is the fraction of the ensemble members which

forecast a precipitation amount exceeding that threshold.


Probabilistic measures for verification (2)

oN

oN

ca

aHR kk

kk

kk

)1(

)1(

oN

oN

cb

bFAR kk

kk

kk

• ROC areaIt is the area under the Relative Operating Characteristics curve in the HIT rate vs FAR diagram

Hit rate (HR) and false alarm rate (FAR) are computed for each probability class k:

where:

• the verification sample is subdivided into subsamples of size Nk, according to the probability with which the event is forecasted,

• ok is the frequency with which the event is observed, being forecasted with a given probability and is the sample climatology.

The cumulative HRk are plotted against the corresponding cumulative FARk, generating the ROC curve.

The integral under the curve is used to indicate the skill of the forecast.

ROC area ranges from 0 to 1, the higher the better.

Useful forecast systems have ROC area values greater than 0.6


COSMO-1km – Skill Scores (IMK)

A.Montani; Novelties in weather forecastingPREVIEW training workshop - Mosonmagyaróvár - 19-20 June 2008

A.Montani; Basin-oriented verification of COSMO-LEPSX COSMO meeting – Cracow – 15-19 September 200817th of July 2007


Outline

• Introduction• Measure for model performance (precipitation

only)

• Atmospheric models re-run for PREVIEW:– GME by DWD (global, deterministic),– COSMO-EU by DWD (limited-area, deterministic),– VarEPS by ECMWF (global, ensemble),– COSMO-LEPS by ARPA-SIM (limited-area, ensemble),– COSMO-1km by IMK (limited-area, deterministic)

(tomorrow’s talk by J.W. Schipper)

• Application to the Danube sub-basins• Lesson learnt


Deterministic measures for verification (1)

• Continuous measuresThe Mean Error (ME, bias) of the precipitation forecast against the measurement (F: Forecast; O:

Observations; N: sample size):

ME ranges from –infinity to infinity; the closer to zero, the better …

N

iii OF

NME

1

)(1

• Categorical measuresConsider thresholds (e.g. tp >1mm and tp > 80% of observed mean).

Scores are generated with the help of a contingency table:

o b s e r v e d

o b s e r v a t i o n > t h r e s h o l d

o b s e r v a t i o n t h r e s h o l d

f o r e c a s t > t h r e s h o l d h i t s f a l s e a l a r m s

F

orec

ast

f o r e c a s t t h r e s h o l d m i s s e s c o r r e c t n e g a t i v e s

P r o b a b i l i t y o f d e t e c t i o n misseshits

hitsPOD

; P r o b a b i l i t y o f f a l s e d e t e c t i o n

alarmsfalsenegativescorrect

alarmsfalsePOFD

T r u e S k i l l S t a t i s t i c POFDPODTSS ( u s e f u l f o r e c a s t f o r 0 < T S S < 1 … t h e h i g h e r t h e b e t t e r … )


Atmospheric models in PREVIEW

– GME by DWD (global, deterministic)– COSMO-EU by DWD (limited-area, deterministic)– – –


GME: grid spacing: 40 km368642 * 40 grid elementstime step : 133 sec.forecasts up to 7 days

COSMO-EU: grid spacing: 7 km665*657 * 40 grid elementstime step : 40 sec.forecasts up to 78 hours

COSMO-DE: grid spacing: 2.8 km421*461 * 50 grid elementstime step: 25 secforecasts up to 21 hours

DWD operational NWP models


Example: DWD forecast of 24h accumulated precipitation with GME (global) and COSMO-EU (regional) for 2002-08-12 06 UTC to 2002-08-13 06 UTC, the forecast start time is 2002-08-11 12 UTC.

• forecasts with the global model GME for the

hydrological year 2002 (2001-10-01 2002-09-30).• forecasts with the high-resolution regional model

COSMO-EU for the PREVIEW special period

(2002-07-19 2002-08-20).

DWD meteorological forecasts (1)


24h catchment mean precipitation for the Hofkirchen catchment (upper Danube) forecasts with GME and COSMO-EU and corresponding adjusted radar data and high-density network rain gauge observations in the period 2002-08-01 to 2002-08-20.

There are differences between the two observing systems the overall evolution for the flooding event is well captured by both forecast models (COSMO-EU slightly better).

DWD meterorological forecasts (2)


GME forecasts vs rain-gauge data from high-density network for the four sub-catchments of the Danube with different catchment sizes: • positive bias for all four catchments in this period (the model “rains” too much);• positive TSS (true skill statistics), i.e. the forecasts are skilful.

Catchment-based verification of GMEperiod 2002-01-01 – 2002-09-30


the verification period might be too short for scores to be statistically significant.

GME and COSMO-EU forecasts vs rain-gauge observations for the catchment Hofkirchen:• negative bias for both models (the models “rain” too little); • GME slightly better for low thresholds• positive TSS (true skill statistics), i.e. the forecasts are skilful.

Catchment-based verification of DWD models

period 2002-07-19 – 2002-08-20


• “Close the gap” between hydrological and meteorological communities.

• Consider the outcome of MAP D-PHASE experimentation (model

performance may vary considerably from basin to basin).

• Develop and exploit “basin-oriented” calibration of NWP systems (being

implemented in these months for ECMWF varEPS and COSMO-LEPS).

• Exploit the wealth of information provided by probabilistic forecasts:

– get more and more acquainted with the concept of probability;

– consider the “gain” (€, $, Ft,…) presentation by D. Richardson.

What is left?

medium-range flood forecasting and warning

Documents