continued development of tropical cyclone wind probability products john a. knaff – presenting...

Continued Development of Tropical Cyclone Wind Probability Products

John A. Knaff – PresentingCIRA/Colorado State University

and Mark DeMariaNOAA/NESDIS

Presentation at 60th Interdepartmental Hurricane Conference

Mobile, AL22 March 2006

60th IHC 22 March 2006 - Mobile, AL 2

Outline

• Training Activities

• Verification Activities

• Results/insights from examination of hurricane warning break points


Training Activities

• Mark DeMaria coordinated with Rick Knabb to provide feedback on a TPC/NHC training session.

• Several cases rerun for 2004 and 2005– For Pablo Santos, Miami WFO for an experimental

algorithm that uses the probabilities.– Web page with examples and a product description

http://rammb.cira.colostate.edu/projects/tc_wind_prob




Verification: Current Status

• Developed:– Input data handling (GRIB, ATCF….)– Statistical Methods

• Scalar measures of skill, accuracy, confidence• Conditional measures

– Methods to assess deterministic forecasts

• Remaining– Treatment of the OFCL forecast & wind radii through

5 days.– Integrating the pieces.– How to Interpret the statistics and optimize use.


Statistical methods: Probability Bias

• Mean Forecast Probabilities (Fi) minus the Mean Observed Frequencies (Ei) = 1 or 0

Determines if the probabilities over/under forecast the outcome.

N

kk

N

kk

E

FBias

1

1


Statistical Methods: Brier Score

• Mean of square of the Forecast Probabilities (Fi) minus the Observed Frequencies (Ei) = 1 or 0

Measures the Mean Square Errors (Accuracy) associated with a probabilistic forecast

n

kkk EF

NBS

1

21


Statistical Methods: Brier Skill Score

• A scalar skill score comparing a given Brier Score with the Brier Score of a reference forecasts (OFCL, CLIPER etc.).

Assess relative accuracy (skill) of a probabilistic forecast with respect to a reference forecast.

refBS

BSBSS 1


Statistical methods: Discrimination Distance

• Distance between the mean forecast probability (F) for all event (E) and all non-events (E’).

Measures the ability of a forecast scheme to discriminate events.

'EFEFd


Statistical Methods: Conditional Distributions

Ob

serv

ed f

req

uen

cy o

f ev

ents

Bins of Forecast Probabilities


Statistical Methods: Relative Operating Characteristics

Forecasts (contingent on the forecast probability)

Observation Warning (W) No Warning (W’) Total

Event (E) h m E

Nonevent (E’) f c E’

Total w w’ N

A series of 2x2 contingency tables, which are conditional on a range of forecast probabilities are constructed.

For instance a warning would be issued if the probability exceeded 1,2,3,4,…100 %

The results of the contingency tables can be quantified in terms of hit rate (hr) = h/(h+m) and false-alarm rate (far)=f/(f+c)

A plot of far vs. hr can be created and a skill score created from the area under the curve.


Statistical Methods: ROC diagram & Skill Score

12 ASSROC

,where A is the area under the curve

Mason and Graham (1999)

skill

No ski

ll

ROC Skill Score


Verification Procedure: Test Dataset

• Dataset– 5-day cumulative 64-kt

wind probabilities were generated for 342 coastal break points (195 official breakpoints + 147 additional points)

– These were analyzed when warnings were issued for the 14 storms to the right

– N=128250 points

Storm Name Year

Alex 2004

Charley 2004

Frances 2004

Gaston 2004

Ivan 2004

Jeanne 2004

Arlene 2005

Cindy 2005

Dennis 2005

Emily 2005

Katrina 2005

Ophelia 2005

Rita 2005

Wilma 2005


Verification Procedure: Summary Skill Measures

Bias = 0.893 (under forecasts)BS = 0.0248 BSOFCL= 0.0346BSzero = 0.0392

ROC

Relative Operating Characteristics5-d Cummulative Probabilities for Landfalling Atlantic TC 2004-2005

BSSOFCL = 28.30%

BSSzero = 36.75%


Verification Procedure: Conditional Distribution of Break Point Probabilities

Slight under forecast of probabilities for this dataset – due to Wilma

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

0 to

55

to 1

010

to 1

515

to 2

020

to 2

525

to 3

030

to 3

535

to 4

040

to 4

545

to 5

050

to 5

555

to 6

060

to 6

565

to 7

070

to 7

575

to 8

080

to 8

585

to 9

090

to 9

595

to 1

00Forecast Probability Bins

Ob

serv

ed F

req

uen

cy o

f E

ven

ts

Forecast Probabilities

Observed Frequencies

Linear (ObservedFrequencies)


Verification Procedure: Summary

• The probabilities are skillful– Brier Skill Score 28% more accurate than the

OFCL deterministic forecast• Note 50% of the OFCL forecasts verified

– ROC Skill Score 88%– The discrimination distance d=26% is large– Probabilities slightly under forecast and are

well calibrated for this limited dataset


Can the wind speed probabilities be used do decrease the area warned

or increase lead time?


Probability Model for NHC Hurricane Warnings

0

200

400

600

800

1000

1200

1400

1600

1/1

/1963

1/1

/1967

1/1

/1971

1/1

/1975

1/1

/1979

1/1

/1983

1/1

/1987

1/1

/1991

1/1

/1995

1/1

/1999

1/1

/2003

Date

Warn

ing

Len

gth

(n

mi)

0

12

24

36

48

60

72

84

1/1/

1963

1/1/

1966

1/1/

1969

1/1/

1972

1/1/

1975

1/1/

1978

1/1/

1981

1/1/

1984

1/1/

1987

1/1/

1990

1/1/

1993

1/1/

1996

1/1/

1999

1/1/

2002

1/1/

2005

Date

War

ning

Lea

d Ti

me

(hr)

NHC storm total hurricanewarning lengths 1963-2005

NHC storm average hurricanewarning lead times 1963-2005

Since 2000 warning areas have decreased and lead times increased.


0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

0

0 to

5

5 to

10

10 to

15

15 to

20

20 to

25

25 to

30

30 to

35

35 to

40

40 to

45

45 to

50

50 to

55

55 to

60

60 to

65

65 to

70

70 to

75

75 to

80

80 to

85

85 to

90

90 to

95

95 to

100

Bins (%)

Per

cen

t O

ccu

ren

ce

Avg. Probabilities at Warned Break Points (Warnings in Place)

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

0

0 t

o 5

5 t

o 1

0

10

to

15

15

to

20

20

to

25

25

to

30

30

to

35

35

to

40

40

to

45

45

to

50

50

to

55

55

to

60

60

to

65

65

to

70

70

to

75

75

to

80

80

to

85

85

to

90

90

to

95

95

to

10

0

Bins (%)

Per

cen

t O

ccu

ren

ce

Avg. Probabilities at Unwarned Break Points (Warnings in Place)

Warned Break Points(Warnings in Place)

Unwarned Break Points(Warnings in Place)

%2.29%,8.27 EFEF %4.3%,4.1 '' EFEF

%4.26d

Discrimination Distance

N=5025 N=123225


0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

0

0 t

o 5

5 t

o 1

0

10

to

15

15

to

20

20

to

25

25

to

30

30

to

35

35

to

40

40

to

45

45

to

50

50

to

55

55

to

60

60

to

65

65

to

70

70

to

75

75

to

80

80

to

85

85

to

90

90

to

95

95

to

10

0

Bins (%)

Per

cen

t O

ccu

ren

ce

Avg. Probabilities at Warned Break Points (Warnings in Place)

%2.29%,8.27 EFEF

?

Why are there so many low probabilities at the break points?


Distribution of Probabilities at the Ending Break Points

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

0

0 t

o 5

5 t

o 1

0

10

to

15

15

to

20

20

to

25

25

to

30

30

to

35

35

to

40

40

to

45

45

to

50

50

to

55

55

to

60

60

to

65

65

to

70

70

to

75

75

to

80

80

to

85

85

to

90

90

to

95

95

to

10

0

Bins (%)

Per

cen

t O

ccu

ren

ce

Probabilities at the ending break points

9.14

8.8

EF

EF

end

end

?

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

MX01

TX/MEX B

ORDER

BROWNSVIL

LE

PORT ISABEL

STATE PARK P

ADRE ISLA

NDTX01

PORT MANSFIE

LDTX02

TX03TX04

BAFFIN B

AYTX05

TX06

CORPUS CHRIS

TI

PORT ARANSAS

TX07

ROCKPORTTX08

TX09

PORT OCONNOR

TX10

PALACIO

S

MATAGORDA

TX11

SARGENTTX12

FREEPORT

SAN LUIS

PASS

TX13

GALVESTON

BOLIVAR P

ENINSULA

TX14

HIGH IS

LANDTX15

PORT ARTHUR

SABINE P

ASS

SABINE R

IVERLA

01

CAMERON

LAKE C

HARLESLA

02LA

03LA

04LA

05LA

06

INTRACOASTAL

CITY

VERMIL

ION B

AYLA

07LA

08

MORGAN C

ITYLA

19LA

09LA

10LA

11LA

12

GRAND ISLELA

20LA

13

MIS

SISSIP

PI RIV

ERLA

21LA

14LA

15LA

16LA

17

NEW O

RLEANS

LA22LA

18

PEARL RIV

ER

BAY SAIN

T LOUIS

GULFPORT

Break Points

Pro

bab

ility

/War

nin

gs

(1 o

r 0)

-43.5 hr Prob-43.5 hr Warn-31.5 hr Prob-31.5 hr Warn-19.5 hr Prob- 19.5 hr Warn-7.5 hr Prob -7.5 hr WarnVerification

Example: Hurricane Rita

Warnings are brought down too slowly in this case.

00UTC 24 Sep.

12 UTC 23 Sep. 00 UTC

23 Sep.

12 UTC 22 Sep

t=0 at landfall


Summary of Warning Break Points

• Probabilities are useful in the watch/warning process.– Objectively assign of the warnings at fixed

lead time?– Average at warnings = 28%– Average at end points of the Warnings = 9%

• It appears that warnings can be dropped sooner, thus decreasing the area warned area.


Future Plans

• Seasonal Verification Code

• See what we can learn from the verification.

• Report to this audience

Questions?

continued development of tropical cyclone wind probability products john a. knaff – presenting...

Documents