joshua m. boustead and daniel nietfeld noaa/nws wfo omaha/valley, ne ray wolf
DESCRIPTION
An Experiment to Evaluate the Use of Quantitative Precipitation Forecasts from Numerical Guidance by Operational Forecasters. Joshua M. Boustead and Daniel Nietfeld NOAA/NWS WFO Omaha/Valley, NE Ray Wolf NOAA/NWS WFO Davenport, IA. Presentation Overview. Study purpose and methodology - PowerPoint PPT PresentationTRANSCRIPT
An Experiment to Evaluate the Use of An Experiment to Evaluate the Use of Quantitative Precipitation Forecasts from Quantitative Precipitation Forecasts from
Numerical Guidance by Operational Numerical Guidance by Operational ForecastersForecasters
Joshua M. Boustead and Daniel NietfeldJoshua M. Boustead and Daniel Nietfeld
NOAA/NWS WFO Omaha/Valley, NENOAA/NWS WFO Omaha/Valley, NE
Ray Wolf Ray Wolf
NOAA/NWS WFO Davenport, IANOAA/NWS WFO Davenport, IA
Presentation Overview
• Study purpose and methodology
• Data results– Survey results– Snowfall forecast– Watch/warning statistics – Gridded forecast results
• Forecasting implications, conclusions, and future work
Study Motivation
• Strong interest in the role of the future forecaster– Can we still add value to the everyday
forecast?– How can we better concentrate on high-
impact weather?– How can we better utilize increasingly high-
tech tools into the everyday forecast?• How does this increasingly high-tech information
affect the forecaster?
ExampleNSSL 4km WRF 00Z 8/14/07
Results
Study Purpose
• To evaluate if and how operational forecasters are biased by numerically generated quantitative precipitation forecasts (QPF)
• Use these results to develop an updated methodology for operational forecasters on how to approach a daily forecast and utilize the latest technology, including high resolution model output
Study Methodology
• Utilizing the National Weather Service’s (NWS) Warning Event Simulator (WES) operational forecasters from two NWS offices made two forecasts for two winter weather case– The forecasters first completed the forecast,
including making a warning decision, without the use of model QPF
– The forecasters then went through the same case again with model QPF, again making a snowfall forecast as well as a warning decision
• Once each scenario was completed, the forecasters completed a survey about the specific case
Study Methodology
• Two winter weather cases were chosen from the Central and Northern Plains– December 7-8, 2005
from the Pleasant Hill, MO (EAX) forecast area
– February 28 – March 1, 2004 from the Bismarck, ND (BIS) forecast area
Survey ResultsDistribution of Forecaster Experience
0%
10%
20%
30%
40%
50%
60%
0 to 3 yrs 3 to 10 yrs 10 to 20 yrs 20 + yrs
Years of Operational Forecasting
Perc
en
t o
f F
ore
caste
rs
• Forecaster Demographics:– Forecasters were from the NWS offices in
Omaha/Valley, NE and Davenport, IA– Operational forecasters involved were of a high
experience level
Survey Results
• Forecaster confidence without using model QPF:– Majority of operational forecasters felt confident in making a
forecast without model QPF– Potentially due to the high experience level of the forecasters
Forecaster Confidence Level without Using QPF
0%
10%
20%
30%
40%
50%
60%
Very Confident Confident Neutral Unsure Very Unsure
Survey Results
• Forecaster confidence after seeing QPF:– Most forecasters indicated that seeing QPF either
increased their forecast confidence or it was unchanged
Change in Forecaster Confidence with QPF
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
Sig increased Increase Unchanged Decreased Sig decreased
Snowfall Forecast Results
• MAE was computed for each location and then averaged for before and after the use of QPF
• MAE decreased 0.5 inches for both the EAX and BIS case post QPF
Pre and Post QPF Mean Absolute Error for EAX and BIS
00.5
11.5
22.5
3
3.54
4.55
Pre and Post QPF EAX Pre and Post QPF BIS
Inch
es Pre-QPF
Post-QPF
Snowfall Forecast Results
• Majority of the forecasts were unchanged post QPF
• Majority of the forecasts that did change their forecast, increased accuracy
Distribution of Change in Forecast Accuracy with QPF among Forecasters for EAX
34%
29%
37% Improved Degraded
Unchanged
Distribution of Change in Forecast Accuracy with QPF among Forecasters for BIS
38%
26%
36% Improved Degraded
Unchanged
Warning Results
• The probability of detection (POD) and false alarm ratio (FAR) were computed by county for each of the forecast areas
• Forecasters showed improvement in both the POD and in FAR ratio once QPF was used
Warning Statistics
0%
10%
20%
30%
40%
50%
60%
70%
80%
Pre-QPF POD and FAR Post-QPF POD and FAR
POD
FAR
Warning Results
Distribution of Change in Probability of Detection Accuracy with QPF for BIS
30%
13%
57%
Improved Degraded
Unchanged
Distribution of Change in False Alarm Ratio Accuracy with QPF for BIS
29%
25%
46%
Improved Degraded
Unchanged
Distrubution of Change in Probability of Detection Accuracy with QPF for EAX
29%
21%
50%
Improved Degraded
Unchanged
Distrubution of Change in False Alarm Ratio Accuracy with QPF for EAX
33%
22%
45%
Improved Degraded
Unchanged
Gridded Forecast ResultsEAX Case
• EAX Pre and Post QPF MAE– Forecasters had
the most confidence in the northern CWA
– Much better agreement over the southern CWA post QPF
– Also a 2 to 3 inch decrease in MAE over the south
Gridded Forecast ResultsEAX Case
• EAX Pre and Post QPF Standard Deviation– Forecast
differences decreased over the north and south
– Slight increase in differences over the center
Gridded Model ForecastsEAX Case
• Greatest agreement of snow band across central CWA
• Viewing QPF increased the forecast confidence in the southern CWA
Actual SnowfallEAX Case
Gridded Forecast ResultsBismarck Case
• BIS Pre and Post QPF MAE– Good agreement
and low error over the northwest forecast area
– Mean errors of 5 to 6 inches over the southern and eastern forecast area
Gridded Forecast ResultsBismarck Case
• Pre and Post QPF Standard Deviation– Significant
increase in forecaster clustering across the central forecast area
– Greater than 4 inch differences continue over the southern forecast area
Gridded Model ForecastsBIS Case
• Models agree northwest CWA to get least QPF
• Larger uncertainty in the south
• Forecasters tended to pick a model– Led to continued
large MAE in the southern CWA
Actual SnowfallBIS Case
Discussion• Only a slight improvement in snowfall forecasts was
noted once forecasters viewed QPF– When snowfall forecasts were modified, a higher percentage
were improved than degraded• Model QPF seemed best utilized to resolve snow-no
snow areas– This led to improvements in both FAR and POD
• High MAE did not always mean high standard deviation, which can indicate a systematic forecasting error
• Doesn’t clearly answer the question does model QPF bias forecasters– Some evidence in the BIS case where model agreement was
poor• Possible forecast methodology
– Make entire forecast without QPF– Utilize QPF for placement for defining snow-no snow areas
Future Work
• Conduct the study using two warm season convective cases
• Investigate forecaster philosophy from the surveys where standard deviation is low and mean absolute error is higher
• Investigate what, if any, synoptic patterns increase forecaster uncertainty and MAE
• Continue to increase the number of forecasters in the study, and from different areas of the CONUS