Skill Variability Diagnosis for Discriminating Skill Variability Diagnosis for Discriminating Use of CPC Long-Lead Seasonal ForecastsUse of CPC Long-Lead Seasonal Forecasts

Bob Livezey and Marina TimofeyevaNOAA/NWS/OCWWS/Climate Services Division

Climate Prediction Applications Science WorkshopMarch 9-11, 2004Tallahassee, FL

OutlineOutline

• IntroductionIntroduction• CPC Skill Graphs and Some Other CPC Skill Graphs and Some Other

StratificationsStratifications• ResultsResults• Conclusions and LessonsConclusions and Lessons

IntroductionIntroduction

• Users should only care about the performance of forecasts that can potentially benefit their decision process

• Livezey (1990): There are non-random subsets of seasonal forecasts that were skillful enough to be useful

• After 1997-98 a common presumption was

that forecasts are generally skillful enough to be useful

Seasonal Temperature Forecast Skill Seasonal Temperature Forecast Skill 1960s to 80s1960s to 80s

All Seasons 8.3

Winter 12.6Spring 8.6Summer 9.3Fall 2.8

Introduction (Cont.)Introduction (Cont.)

• This talk will make the point :

– That was made by Livezey (1990)– That there are many non-random subsets of

forecasts that do not have useful skill – That it is critical for this information to be shared

with potential users– That skill analyses with different stratifications are

highly informative while CPC’s web displays are not

Displays and StratificationsDisplays and Stratifications

• CPC Seasonal Forecasts– For 3-equally probable temperature and precipitation classes at 102

Climate Divisions– Made every month from 1995 to present for 0.5-, 1.5-, …, 12.5 month

leads

• Skill Measure: Modified Heidke Skill Score of Categorized Forecasts

• Displays and Stratifications– CPC: Summed over all forecasts for each lead and displayed with

times series for this lead– Here:

• Summed over all forecasts for each lead and all leads displayed together• Stratified further by cold seasons (DJF to FMA) and warm seasons (MAM

to NDJ)• Stratified further by strong ENSO years vs. other years• Stratified by region

Displays and StratificationsDisplays and Stratifications

• CPC Seasonal Forecasts– For 3-equally probable temperature and precipitation classes at 102

Climate Divisions– Made every month from 1995 to present for 0.5-, 1.5-, …, 12.5 month

leads

• Skill Measure: Modified Heidke Skill Score of Categorized Forecasts

• Displays and Stratifications– CPC: Summed over all forecasts for each lead and displayed with

times series for this lead– Here:

• Summed over all forecasts for each lead and all leads displayed together• Stratified further by cold seasons (DJF to FMA) and warm seasons (MAM

to NDJ)• Stratified further by strong ENSO years vs. other years• Stratified by region

Stratification by Lead and Stratification by Lead and Seasons: TemperatureSeasons: Temperature

Heidke Skill Scores for All Years

-10

0

10

20

30

40

50

0.5 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5 10.5 11.5 12.5

Lead (month)

Hei

dke

SS

.

All Seasons Cold Seasons WarmSeasons

Further Stratification by Strong-Further Stratification by Strong-ENSO vs Other Years: Temp.ENSO vs Other Years: Temp.

Heidke Scores for Cold Seasons(DJF, JFM, FMA)

-10

0

10

20

30

40

50

0.5 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5 10.5 11.5 12.5

Lead (month)

Hei

dke

SS

.

All Years 1997-2000 Other Years

Further Stratification by Strong-Further Stratification by Strong-ENSO vs Other Years: Temp.ENSO vs Other Years: Temp.

Heidke Scores for Warm Seasons(MAM - NDJ)

-10

0

10

20

30

40

50

0.5 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5 10.5 11.5 12.5

Lead (month)

Hei

dke

SS

.

All Years 1997-2000 Other Years

Displays and StratificationsDisplays and Stratifications

• CPC Seasonal Forecasts– For 3-equally probable temperature and precipitation classes at 102

Climate Divisions– Made every month from 1995 to present for 0.5-, 1.5-, …, 12.5 month

leads

• Skill Measure: Modified Heidke Skill Score of Categorized Forecasts

• Displays and Stratifications– CPC: Summed over all forecasts for each lead and displayed with

times series for this lead– Here:

• Summed over all forecasts for each lead and all leads displayed together• Stratified further by cold seasons (DJF to FMA) and warm seasons (MAM

to NDJ)• Stratified further by strong ENSO years vs. other years• Stratified by region

Stratification by Lead and Stratification by Lead and Regions: Temp.Regions: Temp.

Heidke Skill Scores for All Seasons / All Years

0

5

10

15

20

0.5 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5 10.5 11.5 12.5

Lead (month)

Heid

ke S

S .

Western States Eastern States Allstates

ResultsResults

• Seasonal Temperature:– Useable skill confined to strong ENSO years and mainly

at short to medium leads– Otherwise skill is dominantly level with lead (derived

from biased climatologies, ie long-term trend)– Best forecasts are for strong-ENSO cold seasons at

very short leads– Worst forecasts are for cold seasons at longer leads for

strong ENSOs and at very-short leads for other years– Skill is substantially higher than average in the West,

and substantially lower in the East– Short-lead forecasts are better now than for the 1960s-

80s, ~13 vs ~8 overall, ~20 vs ~13 for the winter

Stratification by Lead and Seasons: Stratification by Lead and Seasons: PrecipitationPrecipitation

Heidke Skill Scores for All Years

-10

-5

0

5

10

0.5 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5 10.5 11.5 12.5

Lead (month)

Hei

dke

SS

.

All Seasons Cold Seasons WarmSeasons

Further Stratification by Strong-Further Stratification by Strong-ENSO vs Other Years: Precip.ENSO vs Other Years: Precip.

Heidke Scores for Cold Seasons(DJF, JFM, FMA)

-10

-5

0

5

10

15

20

0.5 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5 10.5 11.5 12.5

Lead (month)

Hei

dke

SS

.

All Years 1997-2000 Other Years

Further Stratification by Strong-Further Stratification by Strong-ENSO vs Other Years: Precip.ENSO vs Other Years: Precip.

Heidke Scores for Warm Seasons(MAM - NDJ)

-10

-5

0

5

10

15

20

0.5 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5 10.5 11.5 12.5

Lead (month)

Hei

dke

SS

All Years 1997-2000 Other Years

Stratification by Lead and Stratification by Lead and Regions: Precip.Regions: Precip.

Heidke Skill Scores for All Seasons / All Years

-2

-1

0

1

2

3

4

0.5 1.5 2.5 3.5 4.5 5.5 6.5 7.5 8.5 9.5 10.5 11.5 12.5

Lead (month)

Heid

ke S

S .

Northern States Southern States Allstates

ResultsResults

• Seasonal Precipitation:

– Barely useable skill entirely confined to strong ENSO years in short to medium leads

– Otherwise skill is either negative or statistically indistinguishable from zero

– Best forecasts are for strong-ENSO cold seasons at short leads

– Skill is a little higher than average in the South, and a little lower in the North

– Short-lead forecasts overall seem to be no better now than for the 1960s-80s

Conclusions and LessonsConclusions and Lessons

• There are non-random subsets of seasonal forecasts that are skillful enough to be useful

• There are many non-random subsets of forecasts that do not have useful skill (should we be issuing them?)

• It is critical for this information to be shared with potential users

• Skill analyses with different stratifications are highly informative while CPC’s web displays are not

• Skills mainly reflect ENSO and trend signals• Temperature forecasts are getting better