Can we distinguish wet years from dry years?

Simon Masonsimon@iri.columbia.edu

Seasonal Forecasting Using the Climate Predictability ToolBangkok, Thailand, 12 – 16 January 2015

2 Seasonal Forecasting Using the Climate Predictability Tool

The ROC

• The ROC answers the question: Can the forecasts distinguish an event from a non-event?

• Are we more confident it will be dry when it is dry compared to when it is not?– Do we forecast less rain when it is dry compared to when it

is not dry?– Do we issue a higher forecast probability for below-normal

when it is below-normal compared to when it is not?

3 Seasonal Forecasting Using the Climate Predictability Tool

Two-Alternative Forced Choice Test

In which of these two Januaries did El Niño occur (Niño3.4 index >27°C)?

What is the probability of getting the answer correct?

50% (assuming that you do not have inside information about ENSO).

Year 1965 1966

4 Seasonal Forecasting Using the Climate Predictability Tool

Two-Alternative Forced Choice Test

In which of these two Januaries did El Niño occur (Niño3.4 index >27°C)?

What is the probability of getting the answer correct?

That depends on whether we can believe the forecasts. Select the forecast with the higher probability.

Year Forecast 1965 11% 1966 89%

5 Seasonal Forecasting Using the Climate Predictability Tool

Year Forecast 1991 319 1992 313 1993 309 1994 322 1995 304 1996 336 1997 321 1998 302 1999 363 2000 374 2001 367 2002 315 2003 295 2004 316 2005 305 2006 362 2007 308 2008 403 2009 369 2010 296

Two-Alternative Forced Choice Test

• Retroactive forecasts of MAM rainfall for Thailand.

• How well do the forecasts distinguish “dry” years (driest 20%) from other years?

• Is the forecast for “dry” years less than for other years?

6 Seasonal Forecasting Using the Climate Predictability Tool

Year Forecast 2003 295 2010 296 1998 302 1995 304 2005 305 2007 308 1993 309 1992 313 2002 315 2004 316 1991 319 1997 321 1994 322 1996 336 2006 362 1999 363 2001 367 2009 369 2000 374 2008 403

Two-Alternative Forced Choice Test• It is easier to calculate by sorting the

forecasts so the driest forecast are at the top. We can then count how many of the non-dry years are lower in the table than the dry years.

• For 2010: 14 of the 15 non-dry years have lower probabilities.

• For 1998: 14 of 15• For 1995: 14 of 15• For 2005: 14 of 15• For 1992: 12 of 15• In total: 68 of 75 ≈ 91%.

7 Seasonal Forecasting Using the Climate Predictability Tool

Year Forecast 2003 295 2010 296 1998 302 1995 304 2005 305 2007 308 1993 309 1992 313 2002 315 2004 316 1991 319 1997 321 1994 322 1996 336 2006 362 1999 363 2001 367 2009 369 2000 374 2008 403

Two-Alternative Forced Choice Test• If the forecasts could perfectly

discriminate the dry years, the forecastss would be drier than for all the non-dry years, and the dry years would be listed at the top of the table.

• If the forecasts could not discriminate the dry years at all, they would be randomly distributed through the table, and there would be a 50% chance of the forecast being drier than on a non-dry year.

8 Seasonal Forecasting Using the Climate Predictability Tool

Year Forecast 1991 319 1992 313 1993 309 1994 322 1995 304 1996 336 1997 321 1998 302 1999 363 2000 374 2001 367 2002 315 2003 295 2004 316 2005 305 2006 362 2007 308 2008 403 2009 369 2010 296

ROC

• Retroactive forecasts of MAM rainfall for Thailand.

• Which year are you most confident is a dry year?

9 Seasonal Forecasting Using the Climate Predictability Tool

Year Forecast 2003 295 2010 296 1998 302 1995 304 2005 305 2007 308 1993 309 1992 313 2002 315 2004 316 1991 319 1997 321 1994 322 1996 336 2006 362 1999 363 2001 367 2009 369 2000 374 2008 403

ROC• The most sensible strategy would be to

list the years in order of increasing forecast rainfall.

• If the forecasts are good, the “dry” years should be at the top of the list.

10 Seasonal Forecasting Using the Climate Predictability Tool

Year Forecast 2003 295 2010 296 1998 302 1995 304 2005 305 2007 308 1993 309 1992 313 2002 315 2004 316 1991 319 1997 321 1994 322 1996 336 2006 362 1999 363 2001 367 2009 369 2000 374 2008 403

ROCFor the first guess:

Repeat for all forecasts.

hitnumber of Hit rate

number of events

s05

false alarmnumber of FAR

number of non-eventss

115

11 Seasonal Forecasting Using the Climate Predictability Tool

ROCYear Forecast Correct Incorrect 2003 295 0 of 5 1 of 15 2010 296 1 of 5 1 of 15 1998 302 2 of 5 1 of 15 1995 304 3 of 5 1 of 15 2005 305 4 of 5 1 of 15 2007 308 4 of 5 2 of 15 1993 309 4 of 5 3 of 15 1992 313 5 of 5 3 of 15 2002 315 5 of 5 4 of 15 2004 316 5 of 5 5 of 15 1991 319 5 of 5 6 of 15 1997 321 5 of 5 7 of 15 1994 322 5 of 5 8 of 15 1996 336 5 of 5 9 of 15 2006 362 5 of 5 10 of 15 1999 363 5 of 5 11 of 15 2001 367 5 of 5 12 of 15 2009 369 5 of 5 13 of 15 2000 374 5 of 5 14 of 15 2008 403 5 of 5 15 of 15

12 Seasonal Forecasting Using the Climate Predictability Tool

ROC

The area beneath the red curve, 0.91, gives us the probability that we will successfully discriminate a “dry” year from a non-dry year.

The area beneath the blue curve, 0.85, gives us the probability that we will successfully discriminate a “wet” year from a non-wet year.

13 Seasonal Forecasting Using the Climate Predictability Tool

ROC

The bottom left indicates whether the forecasts with strong indications of dry (or wet) are good.Can they indicate that an event will occur?

The top right indicates whether the forecasts with strong indications of not dry (or not wet) are good. Can the indicate that an event will not occur?

14 Seasonal Forecasting Using the Climate Predictability Tool

Relative Operating Characteristics

15 Seasonal Forecasting Using the Climate Predictability Tool

Exercises• Diagnose the quality of your forecast models by analysing

the ROC graphs.