datzilla usage ncdc perspective

http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov

Datzilla UsageDatzilla UsageNCDC PerspectiveNCDC Perspective

Karsten SheinKarsten Shein

NOAA National Climatic Data [email protected]

NWS Sub-regional data stewardship workshop – Fargo, ND 12-13 May 2007

22 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov

What is Datzilla?What is Datzilla?

Datzilla is a Web-based tool to report and track errors in NOAA-held data, metadata, or data delivery systems.

It is maintained by the Southern Regional Climate Center and NCDC is a participant to the extent of addressing data it holds and systems it maintains.


Errors vs. CorrectionsErrors vs. Corrections

• Datzilla intended to correct Datzilla intended to correct errorserrors• CorrectionsCorrections to observations are to observations are not errorsnot errors..• For changes to recent observations (For changes to recent observations (commoncommon))

– William Angel (COOP / ASOS)William Angel (COOP / ASOS)– Blake Lasher (LCD ASOS)Blake Lasher (LCD ASOS)– Send amended form to NCDC via normal channels.Send amended form to NCDC via normal channels.

• Changes to historical observations (Changes to historical observations (uncommonuncommon))– Amended formsAmended forms– Datzilla (limited to changing estimates of invalid obs)Datzilla (limited to changing estimates of invalid obs)– Must have a compelling reason to change a historical obs.Must have a compelling reason to change a historical obs.

• Corrections due to equipment problems may also require Corrections due to equipment problems may also require a metadata rendition.a metadata rendition.


When NCDC receives a Datzilla When NCDC receives a Datzilla ticket …ticket …

• Datzilla gatekeeperDatzilla gatekeeper

• Initial determinationsInitial determinations

• ReassignmentReassignment

• InvestigationInvestigation

• Course of actionCourse of action

• ResolutionResolution

• ClosureClosure Your friendly Datzilla Gatekeeper


Some possible reasons for a Some possible reasons for a Datzilla ticket to NCDCDatzilla ticket to NCDC

• Data issue (usually)Data issue (usually)– That TMAX of 74 should be a 47!That TMAX of 74 should be a 47!– Your #$&!@* QC clobbered my data!Your #$&!@* QC clobbered my data!

• System issueSystem issue– CDO inventory doesn’t match the data I got!CDO inventory doesn’t match the data I got!

• Metadata issueMetadata issue– This station’s COOP number is wrong!This station’s COOP number is wrong!


Good tickets and BadGood tickets and Bad

• Pre-report investigation (critical)Pre-report investigation (critical)– Is the error really an error?Is the error really an error?– Is it the right station?Is it the right station?– Is it reproducible (browsers, systems)Is it reproducible (browsers, systems)

• Do you have the right system?Do you have the right system?– ACIS, ThreadEx, NCDC data may be differentACIS, ThreadEx, NCDC data may be different

• Have you proposed a solution?Have you proposed a solution?


A good A good ticketticket

• Clear titleClear title• Good Good

description of description of problemproblem

• ID of stationID of station• Attached formAttached form• Proposed Proposed

solution.solution.


A bad ticketA bad ticket

• Vague titleVague title

• Vague description of problemVague description of problem

• No ID of specific station, element, dateNo ID of specific station, element, date

• No supporting materialNo supporting material

• No suggested fixNo suggested fix

• HunchesHunches

• Data “corrections”Data “corrections”


Investigating the errorInvestigating the error• xmACIS is a display tool for NOAA dataxmACIS is a display tool for NOAA data

– Interprets data flags from NCDC data setsInterprets data flags from NCDC data sets– Prioritizes simultaneous values from different data setsPrioritizes simultaneous values from different data sets– Displays “M” for invalid as well as missing dataDisplays “M” for invalid as well as missing data– May thread data from several stationsMay thread data from several stations

• Primary, Secondary, Tertiary forms:Primary, Secondary, Tertiary forms:– 1001, B-91, 10-A, …1001, B-91, 10-A, …– 1014, LCD, CD, CRB1014, LCD, CD, CRB– Media articles, eye-witness accounts, proxy dataMedia articles, eye-witness accounts, proxy data

• Paper, WSSRD, IPS …Paper, WSSRD, IPS …


My ticket’s been open forever!My ticket’s been open forever!

• Some are easy to fixSome are easy to fix– Small-batch keying errorsSmall-batch keying errors– A few missing valuesA few missing values

• Some are notSome are not– May be part of a larger issueMay be part of a larger issue– May exceed NCDC resourcesMay exceed NCDC resources– May not have a straightforward fixMay not have a straightforward fix– May involve parties/systems outside NCDCMay involve parties/systems outside NCDC


Your #$&!@* QC clobbered my Your #$&!@* QC clobbered my data!data!

• You are right to alert us.You are right to alert us.

• However, before you do …However, before you do …– Think about Think about whywhy our QC would flag that data. our QC would flag that data.– Most often, the problem is not our QCMost often, the problem is not our QC

• Observer date shiftingObserver date shifting

• Observing at time different than expected.Observing at time different than expected.

– Known climate anomalies?Known climate anomalies?– Unreported temporary anomalies?Unreported temporary anomalies?


Should that 850° F TMAX really be invalidated?

Fuel for the FireFuel for the Fire


One NOAAOne NOAA

• We’re all interested in having the best We’re all interested in having the best NOAA data possible.NOAA data possible.

• Better data means more accurate Better data means more accurate forecasts and research.forecasts and research.

• NCDC has limits on what it can do to NCDC has limits on what it can do to data.data.

• We will address and work to fix all We will address and work to fix all legitimate errors.legitimate errors.


Any questions?

Once again, your friendly (no, not that friendly) NCDC Datzilla Gatekeeper!

[email protected]

828.271.4223

NCDC Halloween, 2006

datzilla usage ncdc perspective

Documents

better data

different data setsdisplays

data delivery systems

best noaa data possible

noaa datainterprets

missing datamay thread

ips http

ncdc resourcesmay