1

Probabilistic Forecast Verification

Allen BradleyIIHR Hydroscience &

Engineering The University of Iowa

RFC Verification Workshop16 August 2007Salt Lake City

2

Advanced Hydrologic Prediction Service

Ensemble streamflow forecasts

3

Advanced Hydrologic Prediction Service

Ensemble streamflow forecastsMultiple forecast locations

4

Advanced Hydrologic Prediction Service

Ensemble streamflow forecastsMultiple forecast locationsThroughout the United States

5

Forecast Location

Forecast Date

Forecast Variable

How good are the

ensemble forecasts

produced by AHPS?

AHPS Verification

6

Outline

Illustrate a consistent diagnostic framework for verification of AHPS ensemble forecastsDescribe a prototype interactive web-based system for implementing this verification framework within an RFCPresent a future vision for the role of verification archives in AHPS forecasting

6

7

Forecast Verification Framework

8

Perspective: Forecast Users

Evaluate the quality of forecasts at a specific location for a particular forecast variable and date

Examine one “element” in the data cube

8

9

Elemental Problem

Use a distributions-oriented approach (DO) to evaluate probability forecasts for “events” defined by a flow thresholdForecast quality attributes quantified over a range of flow thresholds

9

10

0

2000

4000

6000

8000

10000

Jun/14 Jul/7 Jul/31 Aug /23 Se p /15

Da

ily F

low

Vo

lum

es

(cfs

-da

ys)

D a te

D es M oines R iver

E nsem ble Stream flow P red ictions

Ensemble Forecast

11

Ensemble Forecast

Probability forecast of a discrete event

C onditional D istribu tion Forecast

1 0 4

1 0 5

1 0 6

1 0 7

.01 .1 1 5 10 2030 50 7 08 0 9 0 9 5 99 9 9 .9

Pe rc e nt

D es M oines R iver

Sea

son

al F

low

Vo

lum

e (

cfs-

da

ys)

yy

ff

12

Ensemble Forecast

Probability forecast of a discrete eventProbability forecasts of multicategory events

C onditional D istribu tion Forecast

1 0 4

1 0 5

1 0 6

1 0 7

.01 .1 1 5 10 2030 50 7 08 0 9 0 9 5 99 9 9 .9

Pe rc e nt

D es M oines R iver

Sea

son

al F

low

Vo

lum

e (

cfs-

da

ys)

WetWet

Near Near AvgAvgDryDry

fdry favg fwet

12

13

Ensemble Forecast

Generalize by defining event forecasts as a continuous function of threshold

C onditional D istribu tion Forecast

1 0 4

1 0 5

1 0 6

1 0 7

.01 .1 1 5 10 2030 50 7 08 0 9 0 9 5 99 9 9 .9

Pe rc e nt

D es M oines R iver

Sea

son

al F

low

Vo

lum

e (

cfs-

da

ys)

f0.50

yy0.500.50

yy0.750.75

f0.75

yy0.250.25

f0.25

Index function by the threshold’s climatological probability

13

14

Ensemble Forecast Verification

Forecast y<y p?

Date f x

1949/09 0.805 1

1950/09 0.952 1

1951/09 0.128 0: : :

1964/09 0.804 0

1965/09 0.732 01966/09 0.962 1

: : :1999/09 0.365 02000/09 0.130 1

Compute forecast-observations pairs for specific thresholds yp

Evaluate forecast quality for a range of thresholds yp

15

Des Moines River near Stratford

Standard Errors

Skill Skill depends on the thresholdUncertainty is greater for extremes

April 1st Forecasts

15

16

Distributions-Oriented Measures

Skill Score Decomposition:

(SS)Skill

22

22

x

xf

x

ffxfxMSESS

(RES)Resolution

(CB)Conditional

Bias

(UB)Unconditional

Bias

SlopeReliability

StandardizedMean Error

PotentialSkill

17

-0 . 6

-0 . 4

-0 . 2

0

0 . 2

0 . 4

0 . 6

0 . 8

1

0 0 . 2 0 . 4 0 . 6 0 . 8 1

N o n e x c e e d a n c e p r o b a b i li ty ( )p

D e s M o i n e s R i v e r a t J a c k s o n ( J C K M 5 )

S S

P S

S R E L

S M E

April 1st

Implications for Verification

IncreaseProbability

forecast skill

Eliminatewith bias-correction

Minimum 7-Day Flow

SSRESCBUB

17

18

Perspective: NWS RFC Forecaster

Assess the overall performance of the forecasting system Diagnose attributes limiting forecast skill (e.g., biases)

Examine “slices” and “blocks” of the data cube

18

19

Multidimensional Problem

The forecaster needs summary verification measures suitable for comparing forecasts at different locations and/or forecasts issued on different datesSummary measures describe attributes of the skill functions derived from the elemental verification problem

19

20

Summary Verification Measures

Ranked Probability Skill Score (RPSS):

MSEiMSEi

i SSpSSwRPSS )(

Weighted-average skill over probability thresholds

iii

iii pp

ppw

)1(

)1(

21

Summary Verification Measures

Skill RPSS shows average skillCenter of mass shows asymmetries in the skill function

RPSS

Center ofMass

21

22

Hypothetical Skill Functions

All skill functions have same average skill

22

23

Hypothetical Skill Functions

All skill functions have same average skillSecond central moment shows shape

23

24

Hypothetical Skill Functions

All skill functions have same average skillSecond central moment shows shape

24

25

NCRFC Forecasts

7-day minimum flow forecasts for mainstem locations for three rivers

MinnesotaRiver (MIN)

Des MoinesRiver (DES)

Rock River (RCK)

25

26

Forecast Skill Attributes

Forecasts made at the 1st of the month

26

27

Forecast Skill Attributes

Average skill is highest for DESThe skill function is peaked in the middle

27

28

Summary Measure Decomposition

Skill Score Decomposition:

Skill

)()()()( iiii pUBpCBpRESpSS

Resolution ConditionalBias

UnconditionalBias

UBCBRESSSRPSS

Weighted-average measures of resolution and biases

29

AHPS Minimum 7-Day Flow

A single MIN site has large biases for low flowsThe largest biases for other sites centered on higher flows

29

30

Forecast Bias Attributes

Unconditionalbias is dominateSimple bias-correction can significantly improve forecasts

Simple bias-correction

Post-hoccalibration

30

31

Verification Framework

Forecast quality for ensemble forecasts (e.g., skill) is a continuous function of the forecast outcome (or its climatological probability)Summary measures can be interpreted as measures of the “geometric shape” of the forecast quality functionThis interpretation provides a framework for concisely summarizing the attributes of ensemble forecasts

31

32

AHPSVerification

System

33

AHPS Verification System

Web-based tools for online

access, analysis, and comparison of retrospective

AHPS forecasts for River Forecast

Centers (RFCs)http://www.iihr.uiowa.edu/ahps_ver

34

Map-Based Navigation

2

1

3

35

Verification Data Archive

35

36

0

2000

4000

6000

8000

10000

Jun/14 Jul/7 Jul/31 Aug /23 Se p /15

Da

ily F

low

Vo

lum

es

(cfs

-da

ys)

D a te

D es M oines R iver

E nsem ble Stream flow P red ictions

Verification Data Archive

Retrospective forecasts for a 50-year period

36

37

Verification Data Archive

Retrospective forecasts for a 50-year periodProcessed ensemble forecasts & observations

C onditional D istribu tion Forecast

1 0 4

1 0 5

1 0 6

1 0 7

.01 .1 1 5 10 2030 50 7 08 0 9 0 9 5 99 9 9 .9

Pe rc e nt

D es M oines R iver

Sea

son

al F

low

Vo

lum

e (

cfs-

da

ys)

37

38

Verification Data Archive

Retrospective forecasts for a 50-year periodProcessed ensemble forecasts & observationsVerification results

38

39

Verification System ConceptsRetrospective ensemble traces available in their native format (*.VS files)Processed ensemble forecasts & observations for a suite of variables

Uses *.qme files from the calb systemForecast quality measures based on the ensemble forecasts

39

40

Disk Requirements

• 6 Forecast periods per month (72 per year)• All segments have 50 years observed record

1 600Segment Segments

Elements (MB) (GB)Ensemble Traces (*.VS) 96.8 58.1Ensemble forecasts/obs 91.3 54.8Verification measures 89.3 53.6

Total Disk Usage 277.4 166.5

41

Advantages

Interactive exploration of verification results

Provides a diagnostic “report card” for sites within an RFC

Instant access to forecasts and quality measures for verification sitesSeamless integration with other components of the NWSRFS system

41

42

A Vision for theFuture

43

Vision

Generation and archival of retrospective forecasts will be a routine component of forecasting systems

Verification methods can assess qualityVerification results would form the basis for accepting (or rejecting) proposed improvements to the forecasting systemArchival information will form the basis for generating improved forecast products

44

Product Generation with ArchiveRaw ESP forecastArchive verification indicates biases and skillOptimal merging and bias correctionEnsemble Forecast

VerificationArchive

OptimizedCS

45

Conclusions

46

ConclusionsA consistent verification framework provides both users and forecasters with the means evaluating forecast products (exploring the “data cube”)AHPS-VS integrates retrospective forecast generation and forecast verification within the operational setting of an RFCRetrospective forecast archives will become a routine component of a hydrologic forecasting system, enhancing forecast evaluation and product generation

47

48

Des Moines Forecast Skill

Skill is higher (lower) downstream (upstream)Skill decline from April to June

JCKM5

DESI4

STRI4

TotalBias