A Multimodel Streamflow A Multimodel Streamflow Forecasting System for the Forecasting System for the

Western U.S.Western U.S.Theodore J. Bohn, Andrew W. Wood, and Dennis P. Theodore J. Bohn, Andrew W. Wood, and Dennis P.

LettenmaierLettenmaierUniversity of Washington, U.S.A.University of Washington, U.S.A.EGU Conference, Spring 2006EGU Conference, Spring 2006

Session HS23/NP5.04Session HS23/NP5.04

OutlineOutline

BackgroundBackground– UW West-Wide Forecasting SystemUW West-Wide Forecasting System– Bayesian Model AveragingBayesian Model Averaging

Multi-model vs Individual ModelsMulti-model vs Individual Models– Deterministic Retrospective ForecastsDeterministic Retrospective Forecasts– ESP Retrospective ForecastsESP Retrospective Forecasts

BackgroundBackground

UW West-Wide Stream Flow Forecast system (Wood UW West-Wide Stream Flow Forecast system (Wood and Lettenmaier, in review; Wood et al, 2002)and Lettenmaier, in review; Wood et al, 2002)– Developed in partnership with USDA/NRCS NWCCDeveloped in partnership with USDA/NRCS NWCC– Long-lead-time (1-12 months) stream flow forecasting for Long-lead-time (1-12 months) stream flow forecasting for

western U.S.western U.S.– Main component: Variable Infiltration Capacity (VIC) large-scale Main component: Variable Infiltration Capacity (VIC) large-scale

hydrological modelhydrological model

Probabilistic forecastsProbabilistic forecasts– Uses forecasts from multiple climate models to take into account Uses forecasts from multiple climate models to take into account

climate uncertaintyclimate uncertainty– Does not yet take into account uncertainty in hydrologic model Does not yet take into account uncertainty in hydrologic model

physicsphysics

Forecast data flowForecast data flow

Forecast Productsstreamflow soil moisture

runoffsnowpack

derived productse.g., reservoir system

forecasts

model spin-up

forecast ensemble(s)

climate forecast

information

climatology ensemble

1-2 years back start of month 0 end of mon 6-12

NCDC met. station obs. up to 2-4

months from current

2000-3000 stations in

west

LDAS/other real-time met. forcings for remaining

spin-up~300-400

stations in west

data sources

obs snow state information

(eg, SNOTEL)

initi

al

cond

ition

s

BackgroundBackground

Immediate goal: improve forecast skill at Immediate goal: improve forecast skill at long lead times (1-12 months)long lead times (1-12 months)

Problems:Problems:– Uncertainty grows with lead timeUncertainty grows with lead time– Greater uncertainty when making forecasts Greater uncertainty when making forecasts

before the snow pack has accumulatedbefore the snow pack has accumulated– How much of this uncertainty is due to How much of this uncertainty is due to

hydrologic model physics?hydrologic model physics?

Relative important of initial Relative important of initial condition and climate forecast condition and climate forecast error in streamflow forecastserror in streamflow forecasts

Columbia R. Basin

Rio Grande R. Basin

RMSE (perfect IC, uncertain fcst)

RMSE (perfect fcst, uncertain IC)RE =

ICs more impt

fcst more impt

Expansion to multiple-model frameworkExpansion to multiple-model framework

It should be possible to balance effort given to It should be possible to balance effort given to climate vs IC part of forecastsclimate vs IC part of forecasts

Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep

N ensembles

climate ensembles

IC ensembles

streamflow volume forecast period

low

high

climate forecastsmore important

ICs moreimportant

How to quantify uncertainty and How to quantify uncertainty and reduce bias?reduce bias?

Multi-model ensembleMulti-model ensemble– Average the results of multiple models – Average the results of multiple models –

reduces biasreduces bias– Ensemble mean should be more stable than a Ensemble mean should be more stable than a

single modelsingle model– Combines the strengths of each model - Combines the strengths of each model -

generally as good as the best model at all generally as good as the best model at all times/locationstimes/locations

– Provides estimates of model uncertaintyProvides estimates of model uncertainty

ESP

ENSO/PDO

ENSO

CPC Official Outlooks

Coupled Forecast System

CAS

OCN

SMLR

CCA

CA

NSIPP/GMAO dynamical

model

VIC Hydrology Model

NOAA

NASA

UW

Seasonal Climate Forecast Data Sources

Expansion to multiple-model framework

Expansion to multiple-model framework

ESP

ENSO/PDO

ENSO

CPC Official Outlooks

Coupled Forecast System

CAS

OCN

SMLR

CCA

CA

NSIPP/GMAO dynamical

model

Model 2

NOAA

NASA

UW

Multiple Hydrologic Models

Model 1

Model 3

weightings calibrated via retrospective analysis

Averaging of ForecastsAveraging of Forecasts

Bayesian Model Averaging (BMA) (Raftery et al, 2005)Bayesian Model Averaging (BMA) (Raftery et al, 2005)Ensemble mean:Ensemble mean:

E(y|fE(y|f11,…f,…fKK) = ) = ΣΣwwkkffkk

where where y = observationy = observationffkk = forecast of k = forecast of kthth model modelwwkk = weight of k = weight of kthth model model = expected fraction of data points for which k= expected fraction of data points for which k thth model forecast is best of the model forecast is best of the

ensembleensemble

Ensemble variance, for forecast at time t:Ensemble variance, for forecast at time t:Var(yVar(ytt|f|f1t1t,…,f,…,fKtKt) = ) = ΣΣwwkk(f(fktkt - - ΣΣwwiiffitit))22 + + ΣΣwwkkσσkk

22

where where σσkk

22 = uncertainty of k = uncertainty of kthth model, conditional on k model, conditional on kthth model being the best model being the best = weighted mean square error (MSE), favoring data points for which k= weighted mean square error (MSE), favoring data points for which k thth

model forecast is best of the ensemblemodel forecast is best of the ensemble

Spread among models Spread among models

Spread due toSpread due tomodel uncertainty model uncertainty

Averaging of ForecastsAveraging of Forecasts

Model 1Model 1

Model 2Model 2

Model 3Model 3

σσ11

σσ22

σσ33

p(y|fp(y|f11))

ff11

ff22

ff33

p(y|fp(y|f22))

p(y|fp(y|f33))

++

++

==

ww11ff11

ww22ff22

ww33ff33

ΣΣwwkkffkk

p(y|fp(y|f11,…f,…f33))

MultimodelMultimodelAverageAverage

wwkk, , σσkk reflect uncertainty due to model physics reflect uncertainty due to model physics

Computing Model WeightsComputing Model Weights

Parameters wParameters wkk and and σσkk – wwkk and and σσk k depend on each otherdepend on each other– computed via iterative maximum likelihood methodcomputed via iterative maximum likelihood method– Currently: determined from model performance in a retrospective Currently: determined from model performance in a retrospective

deterministic simulationdeterministic simulation– Future: determine from performance of retrospective probabilistic Future: determine from performance of retrospective probabilistic

forecastsforecasts– The The σσk k help define a distribution about the multimodel averagehelp define a distribution about the multimodel average

Reflect model uncertainty Reflect model uncertainty

This method assumes normally-distributed dataThis method assumes normally-distributed data– Discharge tends to have positive skewDischarge tends to have positive skewTherefore:Therefore:– Generate monthly wGenerate monthly wkk and and σσkk from log-transformed discharge from log-transformed discharge– Form multimodel average from log-transformed forecastsForm multimodel average from log-transformed forecasts– Transform multimodel average (and distribution) back to flow domainTransform multimodel average (and distribution) back to flow domain

UW West-Wide Forecast EnsembleUW West-Wide Forecast Ensemble

Models:Models:VIC - Variable Infiltration Capacity (UW)VIC - Variable Infiltration Capacity (UW)SAC - Sacramento/SNOW17 model (National Weather Service)SAC - Sacramento/SNOW17 model (National Weather Service)NOAH – NCEP, OSU, Army, and NWS Hydrology LabNOAH – NCEP, OSU, Army, and NWS Hydrology Lab

ModelModel Energy BalanceEnergy Balance Snow BandsSnow BandsVICVIC YesYes YesYesSACSAC NoNo YesYesNOAHNOAH YesYes NoNo

SAC does not compute PET; it uses PET computed by NOAHSAC does not compute PET; it uses PET computed by NOAH

Data:Data:Calibration parameters from NLDAS 1/8 degree grid (Mitchell et al 2004) – Calibration parameters from NLDAS 1/8 degree grid (Mitchell et al 2004) – no further calibration performedno further calibration performedMeteorological Inputs: Maurer et al. (2002), 1949-1999Meteorological Inputs: Maurer et al. (2002), 1949-1999

Three Test BasinsThree Test Basins

Salmon R.Salmon R.(Above Snake R.)(Above Snake R.)Drainage area: 33600 kmDrainage area: 33600 km22

Colorado R.Colorado R.(Above Grand Junction)(Above Grand Junction)Drainage area: 19900 kmDrainage area: 19900 km22

Feather R.Feather R.(Above Oroville Res.)(Above Oroville Res.)Drainage area: 8600 kmDrainage area: 8600 km22

1.0

1.0

1.0

0.0

0.0

0.0

0.5

0.5

0.5

Salm.Salm.

Colo.Colo.

Feat.Feat.

Model WeightsModel Weights Monthly MeanMonthly MeanDischargeDischarge

Monthly RMSEMonthly RMSE

Deterministic Retrospective 1956-1995Deterministic Retrospective 1956-1995Training Period: Even YearsTraining Period: Even Years

1.0

1.0

1.0

0.0

0.0

0.0

0.5

0.5

0.5

Salm.Salm.

Colo.Colo.

Feat.Feat.

Model WeightsModel Weights Monthly MeanMonthly MeanDischargeDischarge

Monthly RMSEMonthly RMSE

Deterministic Retrospective 1956-1995Deterministic Retrospective 1956-1995Validation Period: Odd YearsValidation Period: Odd Years

Deterministic Retrospective ResultsDeterministic Retrospective Results

Individual ModelsIndividual ModelsVIC is best in generalVIC is best in general– Best at capturing autumn-winter base flow (all basins) Best at capturing autumn-winter base flow (all basins) →→ high weights high weights– Best estimate of snowmelt peak in Colorado basinBest estimate of snowmelt peak in Colorado basin– Generally Lowest RMSEGenerally Lowest RMSE

SAC is secondSAC is second– Low autumn/winter base flow Low autumn/winter base flow →→ low weights low weights– In Salmon basin, snowmelt peak flow is early but magnitude is close to observed In Salmon basin, snowmelt peak flow is early but magnitude is close to observed

in May in May →→ high weight high weight– Best estimate of snowmelt peak in Feather basin Best estimate of snowmelt peak in Feather basin →→ high weight high weight

NOAH is lastNOAH is last– No autumn/winter base flow No autumn/winter base flow →→ low weights low weights– In Salmon and Colorado basins, snowmelt peak flow is 1-2 months early and far In Salmon and Colorado basins, snowmelt peak flow is 1-2 months early and far

too small (high snow sublimation, lack of elevation bands) too small (high snow sublimation, lack of elevation bands) →→ low weights low weights– Competitive in Feather basin (snowmelt is less dominant here)Competitive in Feather basin (snowmelt is less dominant here)– Generally highest RMSE and lowest weightsGenerally highest RMSE and lowest weights

Deterministic Retrospective ResultsDeterministic Retrospective Results

Multimodel Ensemble PredictionMultimodel Ensemble PredictionIn general, ensemble bias and RMSE are at least as In general, ensemble bias and RMSE are at least as small as those of the best individual modelsmall as those of the best individual modelNotable exceptions: June in Salmon Basin, February in Notable exceptions: June in Salmon Basin, February in Feather BasinFeather Basin– SAC beats the ensemble RMSE in both the training and SAC beats the ensemble RMSE in both the training and

validation sets – how?validation sets – how?– Model weights reflect each model’s Model weights reflect each model’s bestbest performance performance– SAC consistently good hereSAC consistently good here– VIC not as consistent, but when it is good, it is very good VIC not as consistent, but when it is good, it is very good →→ gets gets

equal weight to SACequal weight to SAC– This warrants further investigationThis warrants further investigation

ESP ForecastsESP Forecasts

Extended Streamflow PredictionExtended Streamflow Prediction– Start with I.C. of forecast yearStart with I.C. of forecast year– Run model with ensemble of historical meteorological forcings Run model with ensemble of historical meteorological forcings

(climatology)(climatology)– The distribution of results indicates uncertainty due to climateThe distribution of results indicates uncertainty due to climate– (but implicitly contains uncertainty due to the model)(but implicitly contains uncertainty due to the model)

Retrospective Retrospective simulationsimulation

Save state Save state vector herevector here

Forecasts using climatology, Forecasts using climatology, starting from saved ICsstarting from saved ICs

ESP forecast distributionESP forecast distribution

ESP forecast typically ESP forecast typically includes median and includes median and quartile valuesquartile values

ESP Forecasts and MultimodelESP Forecasts and Multimodel

Forcing 1Forcing 1

Forcing 2Forcing 2

Model 1Model 1

Model 2Model 2

Model 3Model 3

Approach 1:Approach 1:•FIRST form multimodel average of all models for each forcingFIRST form multimodel average of all models for each forcing•THEN form ESP distribution of the multimodel averagesTHEN form ESP distribution of the multimodel averages•Use weights determined in the training periodUse weights determined in the training period

Forcing 2Forcing 2

Forcing 1Forcing 1

MultimodelMultimodel

ESP distribution of ESP distribution of multimodel distributions multimodel distributions (“grand distribution”)(“grand distribution”)

Add these distributionsAdd these distributions

One multimodel One multimodel distribution for distribution for each forcingeach forcing

ESP Forecasts and MultimodelESP Forecasts and MultimodelApproach 2:Approach 2:

•FIRST form the ESP distribution for each modelFIRST form the ESP distribution for each model•THEN form multimodel average of the ESP distributionsTHEN form multimodel average of the ESP distributions•Determine wDetermine wkk and and σσkk based on each model’s ESP distribution based on each model’s ESP distribution

Model 2Model 2

Model 3Model 3

Model 1Model 1 ESP 1ESP 1

ESP 2ESP 2

ESP 3ESP 3

σσ11

σσ22

σσ33

++ww11ESPESP11

++

ww33ESPESP33

ww22ESPESP22 == ESP distribution of ESP distribution of multimodel distributions multimodel distributions (“grand distribution”)(“grand distribution”)

ESP Forecasts and MultimodelESP Forecasts and Multimodel

Approach 2 incorporates model forecast Approach 2 incorporates model forecast performance into the computation of wperformance into the computation of wkk, , σσkk

– Should be more accurateShould be more accurate

Approach 1 is simplerApproach 1 is simpler

We will start with approach 1We will start with approach 1

Example ESP Forecast, 1966-1967

Oct Dec Feb Apr Jun Aug

Oct Dec Feb Apr Jun Aug

Oct Dec Feb Apr Jun Aug

ESP Distribution ofESP Distribution ofmultimodel averagesmultimodel averages

Distributions ofDistributions ofIndividual modelsIndividual models

S.S.

C.C.

F.F.

Spread of multimodelSpread of multimodelaverages is similar toaverages is similar toindividual model ESPindividual model ESPspreads - mainlyspreads - mainlyreflects uncertainty inreflects uncertainty inclimatological forcingsclimatological forcings

the average reflectsthe average reflectswhich model is morewhich model is morereliable but does notreliable but does notquantify modelquantify modeluncertaintyuncertainty

Now add multimodel “grand distribution”

Oct Dec Feb Apr Jun Aug

Oct Dec Feb Apr Jun Aug

Oct Dec Feb Apr Jun Aug

Multimodel “grand distribution”Multimodel “grand distribution”

S.S.

C.C.

F.F.

Grand distribution has largerGrand distribution has largerspread than distribution ofspread than distribution ofmultimodel averages, due tomultimodel averages, due toaddition of model uncertaintyaddition of model uncertainty

(Note: grand distribution error (Note: grand distribution error bars extend from 1%-ile to 99 %-bars extend from 1%-ile to 99 %-ile)ile)

ESP distribution ofESP distribution ofmultimodel averages multimodel averages

Oct Dec Feb Apr Jun Aug

Oct Dec Feb Apr Jun Aug

Oct Dec Feb Apr Jun Aug

Mean 25Mean 25thth – 75 – 75thth %-ile Range %-ile Range

Grand distribution has larger range Grand distribution has larger range between 25between 25thth-75-75thth %-ile range than that %-ile range than that of the distribution of multimodel means of the distribution of multimodel means alone.alone.

This difference reflects the contribution This difference reflects the contribution of model uncertainty.of model uncertainty.

Grand distribution’s 25-75 range is Grand distribution’s 25-75 range is larger than most individual models larger than most individual models during spring snowmelt peak (May-during spring snowmelt peak (May-June), reflecting range of model snow June), reflecting range of model snow formulations.formulations.

S.S.

C.C.

F.F.

Aggregate ESPs, Odd years 1956-1995Aggregate ESPs, Odd years 1956-1995

ConclusionsConclusions

Multimodel averaging canMultimodel averaging can– reduce the bias of a hydrological forecastreduce the bias of a hydrological forecast

but not always – depends on the weighting schemebut not always – depends on the weighting scheme

– help quantify model uncertainty and/or identify where help quantify model uncertainty and/or identify where model uncertainty is importantmodel uncertainty is important

model snow formulation in snowmelt-driven basinsmodel snow formulation in snowmelt-driven basins

Future work:Future work:– Weights based on forecast performanceWeights based on forecast performance– Multimodel’s influence on dependence of skill on Multimodel’s influence on dependence of skill on

forecast start dateforecast start date

ReferencesReferences

Wood, A.W., Maurer, E.P., Kumar, A. and D.P. Lettenmaier, 2002. Long Range Wood, A.W., Maurer, E.P., Kumar, A. and D.P. Lettenmaier, 2002. Long Range Experimental Hydrologic Forecasting for the Eastern U.S. Experimental Hydrologic Forecasting for the Eastern U.S. J. Geophysical J. Geophysical ResearchResearch, VOL. 107, NO. D20, October., VOL. 107, NO. D20, October.

Raftery, A.E., F. Balabdaoui, T. Gneiting, and M. Polakowski, 2005. Using Raftery, A.E., F. Balabdaoui, T. Gneiting, and M. Polakowski, 2005. Using Bayesian Model Averaging to Calibrate Forecast Ensembles. Bayesian Model Averaging to Calibrate Forecast Ensembles. Monthly Monthly Weather ReviewWeather Review, 133, 1155-1174. , 133, 1155-1174.

Model Averaging: Process Flow