analysis of blue mesa inflow forecast errors
DESCRIPTION
Analysis of Blue Mesa Inflow Forecast Errors. aka: “Wha’ happa’???”. Tom Pagano, [email protected] 503 414 3010. Performance of 2008 forecasts. http://www.wcc.nrcs.usda.gov/wsf/daily_forecasts.html. NRCS daily forecasts. Official coordinated forecasts. Observed. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/1.jpg)
Analysis of Blue Mesa Inflow Forecast ErrorsTom Pagano,
[email protected] 503 414 3010
aka: “Wha’ happa’???”
![Page 2: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/2.jpg)
Performance of 2008 forecasts
![Page 3: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/3.jpg)
NRCS daily forecasts
Official coordinated
forecasts
Daily forecast skill
Observed
http://www.wcc.nrcs.usda.gov/wsf/daily_forecasts.html
![Page 4: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/4.jpg)
2008
OfficialhistoricalpublishedoutlooksissuedApril 1
![Page 5: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/5.jpg)
Blue Mesa + Taylor Park Reservoir Storage
Kac
ft
FullAug 1
Jan 1
Forecasts and storage have come up short since 2000
![Page 6: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/6.jpg)
2008 obs
Historical NRCS “End of season”
daily forecast skill
This takes into account
all available dataincluding
spring/summer precipitation.
Even then, this yearstill falls out as a big anomaly
(major over-forecast)
![Page 7: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/7.jpg)
April-July 2008 Blue Mesa Inflow Forecast Comparison
Forecast issue date
official1971-00 norm
Apr
il-Ju
ly in
flow
kac
-ft
NRCS dailyforecastsbased on:
Precip/Snow
obs
![Page 8: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/8.jpg)
April-July 2008 Blue Mesa Inflow Forecast Comparison
Forecast issue date
official1971-00 norm
Apr
il-Ju
ly in
flow
kac
-ft
NRCS dailyforecastsbased on:Snow only
Precip/Snow
Precip onlyobs
![Page 9: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/9.jpg)
April-July 2008 Blue Mesa Inflow Forecast Comparison
Forecast issue date
official
NWS ESP
1971-00 norm
Apr
il-Ju
ly in
flow
kac
-ft
NRCS dailyforecastsbased on:Snow only
Precip/Snow
Precip onlyobs
![Page 10: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/10.jpg)
NRCS Daily Forecasts for Apr-Jul Blue Mesa Inflowusing snow information available by April 11
Forecast too high
Forecast too low
2008 snowpack composite well beyond
recent historical variability
Daily forecast input is a composite of 7 snotel sites:North lost trail, butte, park cone, brumley, independence pass, porphyry creek, slumgullion
![Page 11: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/11.jpg)
2008
Historical range1981-2007
![Page 12: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/12.jpg)
Park Cone Historical April 1 Snow DataS
no
w w
ater
eq
uiv
alen
t (i
n)
2008…2nd biggest
in last 42 years
6th biggestin last 72 years
![Page 13: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/13.jpg)
End of season wyprecipEnd of season snow
Apr 11 snow Apr 11 wyprecip
![Page 14: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/14.jpg)
Sources of predictability 1950-99 VIC model skill(University of Washington)Explained variance in predicting Apr-July runoff
Blue – SnowpackGreen – Soil MoistureRed – El Nino
Darker colors- more important
(courtesy of M Dettinger, Scripps)
What basin processes are important?How have they been behaving recently?
(using actual and model data)
How is runoff driven by:
snow and precipitationsoil moisture
evaporation/temperature
Note: Model is not necessarily reality
![Page 15: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/15.jpg)
snow and precipitation
![Page 16: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/16.jpg)
Snow and precipitation are easily the dominant factor
R2
(% var expl)
1.0 = perfect
better
worse
Blue Mesa April-July Inflow Forecast Skill 1979-2007
Model swe + precip versus Model runoff
Univ Washingtonmodel Snow + Precip
NRCS Daily Forecasts
Real world SNOTEL vs Real world runoff
Issue date of forecast
Snow/precipitation explain upwards of 90% of
the year to year variability in runoff
![Page 17: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/17.jpg)
Gunnison BasinApril-June Precipitation
1920-2008
Per
cent
of
1971
-200
0 av
erag
e
Data from WESTMAP: http://www.cefa.dri.edu/Westmap/
200872%
200230%
1995206%
2000-08 avg:75% norm
Precipitation after April 1 is important
![Page 18: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/18.jpg)
Spring precipitation, especially the sequencing with snowmelt is also important
Runoff
Snowmelt Rainfall
Rainfall mixed with snowmelt“normal”
April July
![Page 19: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/19.jpg)
Spring precipitation, especially the sequencing with snowmelt is also important
Runoff
Snowmelt Rainfall
Rainfall mixed with snowmelt“normal”
Rainfall boosting snowmeltLarger volumes
Snowmelt and rainfall separateNot enough “momentum” to produce big volumes
All these complex interactions are tough to “cartoonize”;Simulation models can handle this… but still it’s tough to predict beyond 1-2 weeks.
April July
![Page 20: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/20.jpg)
Spring precipitation, especially the sequencing with snowmelt is also important
April July
Runoff
Snowmelt Rainfall
Rainfall mixed with snowmelt“normal”
Rainfall boosting snowmeltLarger volumes
Snowmelt and rainfall separateNot enough “momentum” to produce big volumes
Even then, however,high heat and no rain
can lead to “pouring sunshine”
All these complex interactions are tough to “cartoonize”;Simulation models can handle this… but still it’s tough to predict beyond 1-2 weeks.
![Page 21: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/21.jpg)
2008
Runoff
Blu
e M
esa
nat
ura
l in
flo
wka
c-ft
/day
Total:Independ PassSnowmeltRainfall
SchofieldSnowmeltRainfall
(inches)
Many fits and starts to snowmelt…Almost no spring rainfall
![Page 22: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/22.jpg)
2008
Runoff
Blu
e M
esa
nat
ura
l in
flo
wka
c-ft
/day
Total:Independ PassSnowmeltRainfall
SchofieldSnowmeltRainfall
(inches)
1999
Good mix of rainand snowmelt
1993
![Page 23: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/23.jpg)
2008
Runoff
Blu
e M
esa
nat
ura
l in
flo
wka
c-ft
/day
Total:Independ PassSnowmeltRainfall
SchofieldSnowmeltRainfall
(inches)
1999
Snowpack poor, but “perfect storm”
for runoff efficiency
![Page 24: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/24.jpg)
Soil moisture/GroundwaterAre we still feeling the effects of 2002?
![Page 25: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/25.jpg)
Blue Mesa Basin Soil Moisture 2001-2008(According to the Univ Washington Model- top 2 layers)
![Page 26: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/26.jpg)
Blue Mesa Basin Soil Moisture 2001-2008(According to the Univ Washington Model- top 2 layers)
(According to Park Cone Snotel- ~0-30” depth)
Snotel does poorly in frozen soils, so that has been censored
Model resembles snotel, but also remember we’re comparing basin average with point measurement
![Page 27: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/27.jpg)
Blue Mesa Basin Soil Moisture 2001-2008(According to the Univ Washington Model- top 2 layers)
(According to Park Cone Snotel- ~0-30” depth)
Snotel does poorly in frozen soils, so that has been censored
Univ Washington Model “deep” soil moisture layer
![Page 28: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/28.jpg)
Colorado Active Well Level Network
Only one Colorado USGS groundwater station
in realtime (Pueblo)
Gunnison:Period of record 2003-2008 + 1996
Measured 1x/year
Crested Butte
8/1996
Taken inMid-may
![Page 29: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/29.jpg)
What influence humans?Does it matter?
Blue Mesa
For each site, all measurements Jan-Jun, Jul-Dec are averaged by year. Station half-year data then converted into standardized anomaly (o-avg(o))/std(o) vs
period of record for the half year. Multiple stations are then averaged.
![Page 30: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/30.jpg)
Univ Washington model Blue Mesa inflow basin total soil moisture (mm)
January 1 1920-2008
Start of
2008
![Page 31: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/31.jpg)
Evapotranspiration/sublimation
![Page 32: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/32.jpg)
Butte SNOTEL sublimation (as modeled by NOHRSC)
Wat
er y
ear
to d
ate
cum
ulat
ive
subl
imat
ion
(inch
es)
Oct-Jul SublYear Precip Prcp2004 19.3” 30%2003 19.6” 29%
2005 24.5” 15%2006 22.7” 16%2007 18.1” 18%2008 29.3” 10%
Long-term averageevaporation + transpiration + sublimation = 73% of annual precipitation
![Page 33: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/33.jpg)
Gunnison BasinMarch-May Average Temperature
1920-2008D
epar
ture
fro
m 1
971-
2000
Data from WESTMAP: http://www.cefa.dri.edu/Westmap/
Warm spring temps recently
exceptlastyear
![Page 34: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/34.jpg)
How good are forecasts in general?
“Perfect forecasts are all alike;
Every bad forecast is bad in its own way.”
![Page 35: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/35.jpg)
“Published Official”-Subjective (based on objective guidance)-Humans actively involved-Coordinated by NRCS+NWS
![Page 36: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/36.jpg)
“NRCS Daily”-Objective-Statistical model-based-Highly automated-Only uses SNOTEL snow+wytd precip
http://www.wcc.nrcs.usda.gov/wsf/daily_forecasts.html
![Page 37: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/37.jpg)
http://www.hydro.washington.edu/forecast/westwide/sflow/index.shtml
University of Washington-Objective-Simulation model-based-Highly automated-Research grade
![Page 38: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/38.jpg)
NWS ESP-Objective-Simulation model-based-Human controlled/vetted (in realtime)
![Page 39: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/39.jpg)
Sources
Natural flow: (1968-2008)http://www.usbr.gov/uc/crsp/GetSiteInfoBlue Mesa outflow – Taylor Park change in storage
Published official: (1971-2008)ftp://ftp.wcc.nrcs.usda.gov/data/water/forecast/With gaps filled in from other sources
NWS ESP: (1981-2002)http://www.nwrfc.noaa.gov/westernwater/database/index.php?id=BMDC2#forecasts
University of Washington: (1971-2008)http://www.hydro.washington.edu/forecast/hepex/esp_cmpr/NRCS Daily: (1979-2008)
http://www.wcc.nrcs.usda.gov/wsf/daily_forecasts.htmlAll reforecasts available internally, some available online
![Page 40: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/40.jpg)
Blue Mesa April-July Inflow Forecast Skill 1981-2002
RMSE as % 1971-2000
Normal
0 = perfect
worse
better Published officialNRCS Daily
Issue Month of Forecast(e.g. January 1)
RMSE = sqrt(avg(f-o)2)
Period common to all datasets
![Page 41: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/41.jpg)
Blue Mesa April-July Inflow Forecast Skill 1981-2002
RMSE as % 1971-2000
Normal
0 = perfect
worse
better Published officialNRCS Daily
NRCS Daily (residual)
Issue Month of Forecast(e.g. January 1)
RMSE = sqrt(avg(f-o)2)
![Page 42: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/42.jpg)
ModelForecast
Observed
Model Simulated
Forecasting runoff from a start date in June…
![Page 43: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/43.jpg)
ModelForecast
Observed
Model Simulated
Forecasting runoff from a start date in June…
Is your April-July “forecast” your 1. observed + future only forecast or is it2. simulated + future only forecast?
Forecasting “residuals” (1.) is more accurate but is an “open book” exam
![Page 44: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/44.jpg)
Blue Mesa April-July Inflow Forecast Skill 1981-2002
RMSE as % 1971-2000
Normal
0 = perfect
worse
better Published officialNRCS Daily
NRCS Daily (residual)
Issue Month of Forecast(e.g. January 1)
RMSE = sqrt(avg(f-o)2)
![Page 45: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/45.jpg)
Blue Mesa April-July Inflow Forecast Skill 1981-2002
RMSE as % 1971-2000
Normal
0 = perfect
worse
better Published officialNRCS Daily
NRCS Daily (residual)
University of Washington
Issue Month of Forecast(e.g. January 1)
RMSE = sqrt(avg(f-o)2)
![Page 46: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/46.jpg)
Blue Mesa April-July Inflow Forecast Skill 1981-2002
RMSE as % 1971-2000
Normal
0 = perfect
worse
better Published officialNRCS Daily
NRCS Daily (residual)
University of Washington
NWS ESP
Issue Month of Forecast(e.g. January 1)
RMSE = sqrt(avg(f-o)2)
![Page 47: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/47.jpg)
Blue Mesa April-July Inflow Forecast Skill 1981-2002
R2
(% var expl)
100 = perfect
better
worse
Published officialNRCS Daily
NRCS Daily (residual)
University of Washington
NWS ESP
Issue Month of Forecast(e.g. January 1)
What’s different: This does not
penalize for bias
![Page 48: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/48.jpg)
Blue Mesa April-July Inflow Forecast Bias 1981-2002
ForecastBias as %1971-2000
Normal
Forecasts too high
Forecaststoo low
Issue Month of Forecast(e.g. January 1)
Published officialNRCS DailyUniversity of Washington
NWS ESP
Bias = avg(f) – avg(o)
![Page 49: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/49.jpg)
Blue Mesa April-July Inflow Forecast Bias 2000-2007
ForecastBias as %1971-2000
Normal
Forecasts too high
Forecaststoo low
Issue Month of Forecast(e.g. January 1)
Published officialNRCS Daily
University of Washington
NWS ESP(not available)
Bias = avg(f) – avg(o)
Note:2000-2007 were dry years.
Comparatively, the 71-00 normal had a “bias” of +36% (!)
![Page 50: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/50.jpg)
Conclusions
From a snow perspective, 2008 was an epic bust.Water year precipitation matched runoff however.
Lack of spring precip and sequencing of snowmelt important
All existing models have comparable skill.30% normal error is typical Jan 1, 20-25% error on April 1
Snow, rainfall, evap, soil moisture important to runoff. Some easier to quantify than others.
Recommendations
Have local/anecdotal observationsfeed quantitative historical analyses
to build appropriate models.
Collect the right data to support creation of objective operational guidance, tempered by reason.
Recognize the unknowable versus the unknown.
Remember too that higher accuracy is not the only way to improve forecasts
![Page 51: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/51.jpg)
Snow sublimation (inches/day)
Snow sublimation (inches/day)
Daily average wind speed (mph)
Daily average relative humidity (%)
Butte SNOTEL Site
Data fromNOHRSC model
Nov 2002-Aug 2008www.nohrsc.nws.gov
Note that winds don’t vary from year to year in most hydrologic models
(e.g. NWS and UWashington)
![Page 52: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/52.jpg)
Blue Mesa Basin Evaporation Losses by Month(according to the Univ Washington Model)
Monthly average temperatures (deg F)
Eva
po-t
rans
pira
tion
(kaf
/mon
th) April-June:
temperatures and ETrelated
July-September: ET moisture driven
November-March: ET constant, small
jan feb
mar
apr
may
jun jul
aug
sep
oct
novdec
![Page 53: Analysis of Blue Mesa Inflow Forecast Errors](https://reader036.vdocuments.mx/reader036/viewer/2022062322/56814f51550346895dbcf536/html5/thumbnails/53.jpg)
Blue Mesa Basin Water Year Water Balance(according to the Univ Washington Model)
Kac
-ft/
year
Long-term average Evapotransporation (ET)/Precipitation = 73%
ET doesn’t vary much from year to year. Even then, 65% of its variability depends annual precipitation
(at least in the model’s reality…)
PrecipitationRunoff
Evapotranspiration