object-based spatial verification for multiple purposes

The Centre for Australian Weather and Climate ResearchA partnership between CSIRO and the Bureau of Meteorology

Object-based Spatial Verification for Multiple Purposes

Beth Ebert1, Lawrie Rikus1, Aurel Moise1, Jun Chen1,2, and Raghavendra Ashrit3

1 CAWCR, Melbourne, Australia2 University of Melbourne, Australia3 NCMRWF, India

www.cawcr.gov.au

Object-based spatial verification

2

FORECAST OBSERVATIONS

Verifying attributes of objects

3

Other examples

4

HIRLAM cloud

AVHRR satellite

Climate features (SPCZ)Jets in vertical plane

Convective initiation

Vertical cloud comparison

What does an object approach tell us?

• Errors in• Location

• Size

• Intensity

• Orientation

• Results can• Characterize errors for individual forecasts

• Show systematic errors

• Give hints as to source(s) of errors

• I will discuss CRA, MODE, "Blob"• Not SAL, Procrustes, Composite (Nachamkin), others

5

OBS

FCST

6

Contiguous Rain Area (CRA) verification

• Find Contiguous Rain Areas (CRA) in the fields to be verified– Choose threshold– Take union of forecast and

observations– Use minimum number of points

and/or total volume of parameter to filter out insignificant CRAs

Observed Forecast

• Define a rectangular search box around CRA to look for best match between forecast and observations

• Displacement determined by shifting forecast within the box until MSE is minimized or correlation coefficient is maximized

• Error decomposition MSEtotal = MSEdisplacement + MSEintensity + MSEpattern

Ebert & McBride, J. Hydrol., 2000

Heavy rain over India

Met Office global NWP model forecasts for monsoon rainfall, 2007-2012

7

Ashrit et al., WAF, in revision

Heavy rain over India

8

CRA threshold: 10 mm/d 20 mm/d 40 mm/d 10 mm/d 20 mm/d 40 mm/d

Errors in Day 1 rainfall forecasts

Heavy rain over India

9

Error decomposition (%) of Day 1 rainfall forecasts

Climate model evaluation

10

Delage and Moise, JGR, 2011 added a rotation component

Can global climate models reproduce features such as the South Pacific Convergence Zone?

Climate model evaluation

"Location error" = MSEdisplacement + MSErotation

"Shape error" = MSEvolume + MSEpattern

Applied to 26 CMIP3 models11

etc.

Climate model evaluation

Correcting the position of ENSO EOF1 strengthens model agreement on projected changes in spatial patterns of ENSO driven variability in temperature and precipitation

12 Power et al., Nature, 2013

13

Method for Object-based DiagnosticEvaluation (MODE) (Davis et al. MWR 2006)

Identification

Merging

Matching

Comparison

Measure attributes

Convolution – threshold process

Summarize

Fuzzy Logic Approach

Compare forecast and observed attributes

Merge single objects into clusters

Compute interest values*

Identify matched pairs

Accumulate and examine comparisons across

many cases

*interest value = weighted combination of attribute matching

CRA & MODE – what's the difference?

14

CRA MODE

Convolution filter N Y

Object definition Rain threshold Rain threshold

Object merging N Y

Matching criterionMSE or correlation coefficient

Total interest of weighted attributes

Location error X- and Y- error Centroid distance

Orientation error Y Y

Rain area YY, incl. intersection, union, symmetric area

Rain volume Y Y

Error decomposition Y N

Comparison for tropical cyclone rainfall

15

CRA MODE

Chen, Ebert, Brown (2014) – work in progress

Westerly jets

"Blob" defined by percentile of local maximum of zonal mean U in reanalysis Y-Z plane

16

5th percentile 10th percentile 15th percentile

Rikus, Clim. Dyn., submitted

Westerly jets

17

Westerly jets

18

Global reanalyses show consistent behaviour except 20CR.

Can be used to evaluate global climate models.

Future of object-based verification

• Routinely applied in operational verification suite• Other variables• Climate applications

19

Future of object-based verification

Ensemble prediction – match individual ensemble members

20

8 ensemble members

Johnson & Wang, MWR, 2012, 2013

Prob(object)=7/8

Brie

r sk

ill s

core

Ensemble calibration approaches

Future of object-based verification

Weather hazards

21

Tropical cyclone structure

Pollution cloud, heat anomaly

Blizzard extent and intensity

Flood inundation

Fire spread

WWRP High Impact Weather Project

Thank you

The Centre for Australian Weather and Climate ResearchA partnership between CSIRO and the Bureau of Meteorology

Thank youwww.cawcr.gov.au

Extra slides

23

25

Spatial Verification Intercomparison Project

• Phase 1 – understanding the methods

• Phase 2 – testing the methods

• "MesoVICT" – precipitation and rain in complex terrain

• Deterministic & ensemble forecasts

• Point and gridded observations including ensemble observations

• MAP D-PHASE / COPS dataset

Core

Determ. precip+ VERA anal

+ JDC obs

Tier 1

Det

erm

. win

d

+ V

ERA a

nal

+ JD

C o

bs

Ensemble precip

+ V

ERA anal

+ JD

C obs

Ensemble wind+ VERA anal

+ JDC obs

Tier 2a

Tier 2b

Deter

m. w

ind

+ V

ERA e

nsem

ble

+ JD

C obs

Determ. precip

+ VERA ensem

ble

+ JDC obs

Ensemble wind

+ VERA ensem

ble

+ JDC obs En

sem

ble

prec

ip

+ V

ERA e

nsem

ble

+ JD

C obs

Tier 3

Oth

er v

aria

ble

s ense

mble

+ V

ER

A e

nse

mble

+ JD

C o

bs

Sensi

tivit

y t

est

sto

meth

od p

ara

mete

rs

MODE – total interest

26

M

i jiji

M

i jijijij

wc

FwcI

1 ,,

1 ,,,

M = number of attributes

Fi,j = value of object match (0-1)

ci,j = confidence, how well a given attribute describes the forecast error

wi,j = weight given to an attribute

Attributes:•centroid distance separation

•minimum separation distance of object boundaries

•orientation angle difference

•area ratio

•intersection area

Tropical cyclone rainfall

27

CRA:•Displacement & rotation error•Correlation coefficient•Volume•Median, extreme rain•Rain area•Error decomposition

MODE:•Centroid distance & angle difference•Total interest •Volume•Median, extreme rain•Intersection / union / symmetric area