datzilla

http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov

DatzillaDatzilla

NWS data stewardship workshop – Peachtree City, GA, 20-21 June 2007

Your friendly Datzilla Gatekeeper

A web-based error reporting and tracking tool

22 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov

What is Datzilla?What is Datzilla?

Datzilla is a Web-based tool to report and track errors in NOAA-held data, metadata, or data delivery systems.

It is maintained by the Southern Regional Climate Center and NCDC is a participant to the extent of addressing data it holds and systems it maintains.


Errors vs. CorrectionsErrors vs. Corrections• Datzilla intended to correct Datzilla intended to correct errorserrors

• CorrectionsCorrections to observations are to observations are not errorsnot errors..

• For changes to recent observations (For changes to recent observations (commoncommon))– William Angel (COOP / ASOS) – William Angel (COOP / ASOS) – [email protected]@noaa.gov– Blake Lasher (LCD ASOS) – Blake Lasher (LCD ASOS) – [email protected]@noaa.gov– Send amended forms to NCDC via normal channels.Send amended forms to NCDC via normal channels.

• Corrections to historical observations (Corrections to historical observations (uncommonuncommon))– Amended formsAmended forms– Datzilla (in some cases)Datzilla (in some cases)– Must have a compelling reason to change a historical obs.Must have a compelling reason to change a historical obs.

• Corrections due to equipment problems may also require Corrections due to equipment problems may also require a metadata rendition.a metadata rendition.


Some possible reasons for a Some possible reasons for a Datzilla ticket to NCDCDatzilla ticket to NCDC

• Data issue (usually)Data issue (usually)– That TMAX of 74 should be a 47!That TMAX of 74 should be a 47!– Your #$&!*@ QC clobbered my data!Your #$&!*@ QC clobbered my data!

• System issueSystem issue– CDO inventory doesn’t match the data I got!CDO inventory doesn’t match the data I got!

• Metadata issueMetadata issue– This station’s COOP number is wrong!This station’s COOP number is wrong!


Who should be submitting error Who should be submitting error reports to Datzilla?reports to Datzilla?

• NCDC personnel (yes)NCDC personnel (yes)• RCC climatologists (yes)RCC climatologists (yes)• State Climatologists (yes)State Climatologists (yes)• WFO DAPM/OPL (yes)WFO DAPM/OPL (yes)

• Non-NOAA researchers (no)Non-NOAA researchers (no)• COOP Observer (NO)COOP Observer (NO)

Outside parties should make reports to local WFO, SC or RCC.


Creating a DatzillaCreating a DatzillaError ReportError Report

Step 1: Log in and select “New” from Actions list.

If you do not have an account, you may request one here

http://datzilla.srcc.lsu.edu/datzilla

CLICK THERE


Good tickets and BadGood tickets and Bad

• Pre-report investigation (critical)Pre-report investigation (critical)– Is the error really an error?Is the error really an error?– Is it the right station?Is it the right station?– Is it reproducible (browsers, systems)?Is it reproducible (browsers, systems)?

• Do you have the right system?Do you have the right system?– ACIS, ThreadEx, NCDC data may be differentACIS, ThreadEx, NCDC data may be different

• Have you proposed a solution?Have you proposed a solution?– ““I think it’s too high.” is not a solution.I think it’s too high.” is not a solution.


A good A good ticketticket

• Clear titleClear title

• Good Good description of description of problemproblem

• ID of stationID of station

• Attached formAttached form

• Proposed Proposed solutionsolution


A bad ticketA bad ticket

• Vague titleVague title

• No supporting materialNo supporting material

• No suggested fixNo suggested fix

• HunchesHunches

• Data “corrections” Data “corrections”

• Vague description of problemVague description of problem

• No ID of specific station, element, dateNo ID of specific station, element, date

Please don’t drive us crazy

with wild goose

chases!


Investigating the errorInvestigating the error• xmACISxmACIS

– An RCC-maintained display tool for NOAA dataAn RCC-maintained display tool for NOAA data– Interprets data flags from NCDC data setsInterprets data flags from NCDC data sets– Includes recent data that may not be QC’dIncludes recent data that may not be QC’d– Prioritizes simultaneous values from different data setsPrioritizes simultaneous values from different data sets– Displays “M” for invalid as well as missing dataDisplays “M” for invalid as well as missing data– May thread data from several stationsMay thread data from several stations

• Primary, Secondary, Tertiary forms:Primary, Secondary, Tertiary forms:– 1001, B-91, 10-A, …1001, B-91, 10-A, …– 1014, LCD, CD, CRB1014, LCD, CD, CRB– Media articles, eye-witness Media articles, eye-witness

accounts, proxy dataaccounts, proxy data

• Paper, WSSRD, IPS …Paper, WSSRD, IPS …

Wheeling, WV flood of 1936


Who will investigate?Who will investigate?

• Karsten Shein (gatekeeper / historical)Karsten Shein (gatekeeper / historical)

• William Angel (operational data)William Angel (operational data)

• Henry Ray (metadata)Henry Ray (metadata)

• Bryant Korzeniewski (historical)Bryant Korzeniewski (historical)

• Mark Lackey/Deb Franklin (CDO)Mark Lackey/Deb Franklin (CDO)

• Keith Eggleston (xmACIS)Keith Eggleston (xmACIS)

• … … and others as neededand others as needed


My ticket’s been open forever!My ticket’s been open forever!• Some are easy to fixSome are easy to fix

– Small-batch keying errorsSmall-batch keying errors– A few missing valuesA few missing values

• Some are notSome are not– May be part of a larger issueMay be part of a larger issue– May exceed NCDC resourcesMay exceed NCDC resources– May not have a straightforward fixMay not have a straightforward fix– May involve parties/systems outside NCDCMay involve parties/systems outside NCDC

• Old pubs will not be reissuedOld pubs will not be reissued– But the data errors they may contain will be fixedBut the data errors they may contain will be fixed


Your #$&!*@ QC clobbered my Your #$&!*@ QC clobbered my data!data!

• You are right to alert us.You are right to alert us.

• However, before you do …However, before you do …– Think about Think about whywhy our QC would our QC would

flag that data.flag that data.– Most often, the problem is not our Most often, the problem is not our

QCQC• Observer date shiftingObserver date shifting

• Observing at time different than Observing at time different than expectedexpected

• Instrumentation problemsInstrumentation problems

– Known climate anomalies?Known climate anomalies?– Unreported temporary anomalies?Unreported temporary anomalies?


Any questions?

Once again, your friendly (no, not that friendly) NCDC Datzilla Gatekeeper!

Datzilla is maintained by:Kevin Robins (SRCC)[email protected]

Datzilla POC at NCDC:Karsten [email protected]

NCDC Halloween, 2006


Web Search, Store, Web Search, Store, Retrieve, DisplayRetrieve, Display

(WSSRD)(WSSRD)

NWS Data Stewardship Meeting – Peachtree City, GA, 20-21 June 2007


What is WSSRD?What is WSSRD?

• Electronic image archive (database)Electronic image archive (database)• Web-based interfaceWeb-based interface• http://noaa.imcwv.com/http://noaa.imcwv.com/• 50 + million images50 + million images• COOP network formsCOOP network forms

– Nearly all post-1984Nearly all post-1984– 14 of 50 states back to 189314 of 50 states back to 1893

• Access to:Access to:– U.S. Government and contractorsU.S. Government and contractors– Educational institutions doing env. ResearchEducational institutions doing env. Research– Researchers associated with NOAA projectsResearchers associated with NOAA projects


The WSSRD interfaceThe WSSRD interface

COOP Observation forms

CRBOld 1st order daily forms (SWO)CD and LCD

Some F-6 forms

Older 1st order monthly forms


WSSRD COOP SearchWSSRD COOP Search

Keep your searches

general, and use wildcards

(*) where possible

You’ll probably never use these fields

This is the (E)xpression builder to widen your search.

This is the (L)ist builder to widen your search.


WSSRD COOP Search ResultsWSSRD COOP Search Results

Note: A search by COOP number would not have returned this record!

Click here to view record


COOP form on WSSRDCOOP form on WSSRD


Alternatives and PlansAlternatives and Plans

• Most COOP forms available in NCDC’s IPSMost COOP forms available in NCDC’s IPS– Access via normal NCDC websiteAccess via normal NCDC website

• WSSRD in process of being replaced by WSSRD in process of being replaced by COTS software (in review)COTS software (in review)

• Transition process may take a while.Transition process may take a while.


Want An Account or More Info?Want An Account or More Info?

Please contact

Cindy Karl at NCDC

[email protected]

Or visit

http://www.ncdc.noaa.gov/oa/climate/cdmp/wssrd.html


WHEN MAKING A WSSRD ACCOUNT WHEN MAKING A WSSRD ACCOUNT REQUEST…REQUEST…

• Please Indicate via E-mail to Cindy Karl…Please Indicate via E-mail to Cindy Karl…– Your Job Title (e.g. Lead Meteorologist)Your Job Title (e.g. Lead Meteorologist)

– Your Organization (e.g. NWS-GSP)Your Organization (e.g. NWS-GSP)

– The Data That You’ll Need To Access (e.g. COOP)The Data That You’ll Need To Access (e.g. COOP)

• Will Accept Account Requests For…Will Accept Account Requests For…– Researchers and Those From Universities (*.edu Addresses)Researchers and Those From Universities (*.edu Addresses)

• Please Include Information on Your Research as it Relates To Your Please Include Information on Your Research as it Relates To Your Need for a WSSRD AccountNeed for a WSSRD Account

• Indicate the Time Frame Needed For Use of WSSRD (e.g. 90 days)Indicate the Time Frame Needed For Use of WSSRD (e.g. 90 days)

– Those Sponsored by NOAA (e.g. Contractors, Interns, etc.)Those Sponsored by NOAA (e.g. Contractors, Interns, etc.)• NOAA Sponsor Must Send E-mail Request on their BehalfNOAA Sponsor Must Send E-mail Request on their Behalf

datzilla

Documents

data delivery systems

different data setsdisplays

noaa datainterprets

missing datamay thread

datzilla ticket

webbased tool

metadata rendition

right station