climate service division / ocwws / nws l3mto qc and accuracy marina timofeyeva contributors: annette...

CLIMATE SERVICE DIVISION / OCWWS / NWSCLIMATE SERVICE DIVISION / OCWWS / NWS

L3MTO QC and AccuracyL3MTO QC and Accuracy

Marina Timofeyeva

Contributors: Annette Hollingshead, Dave Unger and Andrea Bair


CPC: CD forecast

RHQ:Collecting trouble ticketsCollecting trouble tickets

WFO:1.1. Ensure link to the productEnsure link to the product2.2. Secondary QC of the productSecondary QC of the product3.3. Local value addedLocal value added4.4. Customer services and outreachCustomer services and outreach

LFPS:1.Make forecast2.Primary QC3.3.Fix troublesFix troubles

WS:1. Post product on web

2.2. Fix TroublesFix Troubles

General QC FlowGeneral QC Flow

Archive


L3MTO Primary QCInterrogate CPC site and

check for new forecast data1. National 2. Alaska & Hawaii

Parse data: extract mean & standard deviation

Import data into Splus

Run Splus scripts

Export data

1. Parse Splus output 2. Create XML formatted files

Run QC scripts

Push data to web team

Pass

Troubleshoot and fix problems

Fail

Early am ET(~ 7am- Noon)

6-8 hours

45 minutes


Secondary QC (WFOs)Secondary QC (WFOs)

Erroneous Data Check

Approve data

Web & Functionality Error Check

Conspicuous Data Check

Send trouble ticket to WS

Pass

Pass

Pass

Fail

FailSend trouble ticket to Regional HQ

Start

Fail

Fail

Pass

Section A

Section B

Section CSend trouble ticket to

Regional HQ

Send trouble ticket to Regional HQ

Other Discrepancy:Detailed QC


WFO Technical Support Team

Who are they?Regional representatives available to help answer any L3MTO questions

What is their role?Answer customer and partner questions and inquiries that NWS personnel are unable to address (see Guideline doc 3.2.3)

Provide technical expertise & guidance:- interpreting L3MTO - comparing L3MTO to CPC 3-Month Temperature

Outlook

How to find your regional rep?Page 13 in the L3MTO Guidelines document


Web problem QC failureStation

metadata

L3MTO Frustration

Content and methodology

Contact Regional Technical

Support Team

WFO Quick Guide on Getting Help with L3MTO

Check CSSA, if error report to

Regional CSPM and RCPM

[email protected] Trouble Ticket

Procedure


WFOL3MTO Problem

Reporting Protocol ( )

Detailed QCSecondary QCWeb QCAdd

Stations Metadata Methodology

Web Team

Annette Hollingshead

1) Add Station (if not in CMS): report to CSPM;2) Metadata Errors: - If CSSA is up-to-date, then submit change to CSPM - If CSSA is NOT up-to-date submit change to CSPM and RCPM who will expedite CSSA change

TT collectors

Marina Timofeyeva

Issue Trouble Ticket

Report to CSPM / RCPMTechnical Support

Team

Technical Support Teams

Protocol in Guidelines document

Compare to CSSA Discussion

Inconsistencies, MethodologyTechnical Issues

Buffer issues


QC Tool: L3MTO Analysis Resource Center - ARC

http://ww2.wrh.noaa.gov/hsd/climate/L3MTO/ARC/index.phphttp://ww2.wrh.noaa.gov/hsd/climate/L3MTO/ARC/index.php

• Section A allows checking short leads L3MTO “sanity” with current climate trends• Section B allows checking L3MTO agreement with some of CPC tools• Section C allows checking L3MTO presentation and methodology correctness• Section D presents L3MTO long term performance evaluation



Season: MJJ

A CPC forecast of 49% (or greater) BELOW NORMAL probability is required to produce a significant* forecast for BELOW Normal category at the station. This automatically reduces the forecast for ABOVE Normal category to 28.3% or less.

A CPC forecast of 36% (or greater) ABOVE NORMAL probability is required to produce a significant* forecast for ABOVE Normal category at the station. This automatically reduces the forecast for BELOW Normal category to 28.3% or less.


L3MTO Verification

• L3MTO forecast archive

• Additional scores to coordinate with NWS verification practices

• Extended selection choices for years and leads

Univ. of AZ verification tool expanded to include:



ARC section D verification intended for WFO use, it shows results of 3 verification methods:

• Heidke Skill Score verifies 3 categorical forecast (pie chart format), indicates % improvement over climatology

• Reliability Diagram verifies long term performance of individual %-tile of POE

• Continuous Rank Probability Skill Score verifies accuracy, skill and resolution of POE, the most stringent score of the 3


QC Tool: L3MTO Analysis Resource Center - ARCDJF JJA

• Most stations have correct forecasts 30% (or more) of the time, and have little variability between the 13 leads

• For all years analyzed together, shorter leads in the winter seasons have higher scores than the same in the warm seasons

• Larger spread between stations’ performance in the warm seasons than cold



Bars show 95%Confidence Limits

0.0 0.2 0.4 0.6 0.8 1.0POE Percentile

0.0

0.2

0.4

0.6

0.8

1.0

Ob

serv

ed F

req

uen

cy

Reliability Diagram for San Jose for the Years 1994-2005 for the Season DJF

Perfect ReliabilityForecastClimatology

Bars show 95%Confidence Limits

0.0 0.2 0.4 0.6 0.8 1.0POE Percentile

0.0

0.2

0.4

0.6

0.8

1.0

Ob

serv

ed F

req

uen

cy

Reliability Diagram for San Jose for the Years 1994-2005 for the Season JJA

Perfect ReliabilityForecastClimatology

• For each station and forecast target season a reliability diagram was created to analyze each percentile’s performance.

• Confidence limits around the perfect forecast line, were drawn from a binomial distribution at 95% confidence.

• The forecast was compared with 1971-2000 climatology performance


Verification Summary

• Using multiple verification scores in climate forecast evaluation provides the most comprehensive assessment of long term forecast performance

• Verification analysis identified locations and 3-month periods with satisfactory forecast performance, and their spatial and temporal variability

• Overall, for the majority of stations and seasons, the L3MTO indicates higher skill than the 1971-2000 climatology


Verification Summary

• Although L3MTO in general proves to be more skillful than 1971-2000 climatology, it is still important to identify station/season combinations where L3MTO does not have long term skill. At such stations/seasons a Verification Filter will be applied.

• The verification filter is the test that compares the station Heidke Skill Score with cutoff value at 75% confidence level that depends on number of independent forecast made for the stations. If the HSS is greater than passing, then the forecast is reported, if less, then 1971-2000 climatology is reported.


Verification Filter

5. CPC produces CD forecasts monthly

6. CD outlook together with bi-variate statistics reported from the regression analysis, station forecast is produced

7. Verification filter is the test that compares the station Heidke Skill Score with cutoff value at 75% confidence level.

8. Data distribution and display are provided at the L3MTO web site

(6) Station Forecast Produced

(7) Verification Filter

(5) CPC Producesthe CD Forecast

Monthly Routine:

SufficientScore?

ProduceForecast Data

(8) Distribution and Display

Yes

ProvideClimatology Data

No


Additional Verification Information

Additional Verification InformationAdditional Verification Information

climate service division / ocwws / nws l3mto qc and accuracy marina timofeyeva contributors: annette...

Documents