Verification in NCEP/HPC Using VSDB-fvs

Keith F. BrillNovember 2007

2

Overview

Introduce VSDB-fvs

VSDB component

fvs component

Examples of fvs output

3

VSDB-fvs Major ComponentsVSDB = Verification Statistics DataBase

fvs = forecast verification system

VSDB-fvs refers to a collection of software

1. Scripted programs to create VSDB records

Grid (GEMPAK) to grid comparison (“perfect obs” assumption)

Polygon (VG) to grid comparison

Grid to point (GEMPAK) comparison

Image overlay visual comparison as a by-product

2. fvs (command line or scripted)Define VSDB search conditions

Search VSDB for matching data

Compute and display (GEMPLT) performance measures

4

Brief History of VSDB-fvs

~1995: EMC begins to specify VSDB format 1997: I began coding fvs in the EMC 1998: I delivered first version of fvs 1999: I accepted a position in HPC 1999 – present: I continued the following...

Support fvs for EMC use Develop HPC oriented VSDB components Support fvs for HPC use Add new capabilities to fvs as time permits

5

VSDB Records Space delimited text, single-line records, multiple records per file

9 identifier fields “=” count_value data_value1 . . . data_valueN

Number of data values depends on statistic type in ID field 7

data_values (summary statistics) are sums scaled by count_value

these may be combined arbitrarily by fvs

count_value is number of ob-forecast pairs used for data Example record (split into two lines):

V01 WWD/46 24 2007021400 CSQ G212 PBS_WWD/4 SF SFC =

vrsn frcst_source fhr valid_date-time obs_src region statistic_type parm level

5533 0.20025 0.20965 0.16411 0.00795 0.00343 0.03072 0.09615

count <----------------------------------------- data --------------------------------------------------->

Defect: “Perfect obs” assumption.

6

Partial List of Statistic Types and Corresponding Data

L1L2 – means of forecasts, observations, their squares, their products, and absolute error.

FHO – fractions of events Forecast, Hit, and Observed for dichotomous forecasts

PHSE – east-west phase error: phase error (km), amplitude error, variances, analysis phase angle and amplitude, more variances

PBS – partitioned Brier Score: fractions forecast in probability categories followed by fractions both forecast and observed in categories

7

HPC Verification Regions

Defect: No stratification by elevation in West.

8

fvs – the tool for using VSDB data “fvs u” lets user set search conditions

specify statistic type

specify lists of identifier strings for data combination (forecast hours, time, areas, etc.)

“fvs s” executes the search through VSDB

applies event equalization consistency checking “fvs p”

computes user requested performance measures

displays graphics under user control

performs statistical significance test for paired data fvs can be run in “batch” mode to populate web pages

9

Examples of Graphs Produced Using VSDB-fvs

10

Example Visual Comparison For HPC Basic Wx

Forecasts of Broken Precipitation

Purple is QPE based

estimate

Broken Precip area

forecast

11

Computed from SSAL1L2 statistic type for HPC PMSL

12

Significance Test: Does HPC beat the 00Z ENS mean?STATISTICAL SIGNIFICANCE ANALYSIS

FOR PAIRED DATA FROM TWO TRACES

STAT=SSAL1L2 PARAM=SLP V_ANL=HPC/SFC V_RGN=MRDG VHHMM=1200

200707171200 TO 200710151200TRACE # 1 MODEL=MEDR/ FHOUR=120 SCALR_CORR

TRACE # 2 MODEL=ENSMN FHOUR=132 SCALR_CORR Null Hypothesis (trace1 = trace2): The mean of the differences of the paired values from the two traces is zero. Alternative Hypothesis (trace1 > trace2): The mean of the paired differences exceeds zero. Gaussian probability of wrongly rejecting null hypothesis: 0.000130 Test Level Decision (one-sided Gaussian) 0.100 REJECT null hypothesis 0.050 REJECT null hypothesis 0.010 REJECT null hypothesis 0.005 REJECT null hypothesis 0.001 REJECT null hypothesis ZRAW = 4.16671E+00 ZTST = 3.65271E+00 N = 90 Lag1 Corr = 0.130902

13

Significance Test: Does HPC beat the 12Z ENS mean?

STATISTICAL SIGNIFICANCE ANALYSISFOR PAIRED DATA FROM TWO TRACES

STAT=SSAL1L2 PARAM=SLP FHOUR=120 V_ANL=HPC/SFC V_RGN=MRDG VHHMM=1200

200707171200 TO 200710151200TRACE # 1 MODEL=MEDR/ SCALR_CORR

TRACE # 2 MODEL=ENSMN SCALR_CORR

Null Hypothesis (trace1 = trace2): The mean of the differences of the paired values from the two traces is zero. Alternative Hypothesis (trace1 > trace2): The mean of the paired differences exceeds zero. Gaussian probability of wrongly rejecting null hypothesis: 0.230557 Test Level Decision (one-sided Gaussian) 0.100 ACCEPT null hypothesis 0.050 ACCEPT null hypothesis 0.010 ACCEPT null hypothesis 0.005 ACCEPT null hypothesis 0.001 ACCEPT null hypothesis ZRAW = 7.43670E-01 ZTST = 7.37014E-01 N = 88 Lag1 Corr = 0.008990

14

COMPUTED from SSAL1L2 statistics for HPC PMSL

15

Computed from SSAL1L2 statistics for HPC PMSL

16

Computed from FHO statistics for HPC WWD

17

Computed from PHSE stats for HPC PMSL phase errors

18

Computed from PHSE stats for HPC PMSL phase errors

Brier Score

Decompositon

Attribute Diagram

Perfect

Reliability

Zero

Resolution

Histogram

shows forecast

use frequency

NO

SKILL

Brier Score

Brier Skill Score

NO SKILL SKILL

NO SKILL

SKILL

Sample

Climatological

Frequency

Generated for statistic type PBS_HHI – Heat IndexPartitioned Brier Score stats

20

SUMMARY VSDB-fvs is an end-to-end system for verifying

forecast objects against analyses or points VSDB is text data base of summary statistics VSDB data are combinable in a variety of ways fvs searches for VSDB data and combines it

under user control fvs computes performance metrics and

displays them in a variety of ways