Download - Datzilla
http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
DatzillaDatzilla
NWS data stewardship workshop – Peachtree City, GA, 20-21 June 2007
Your friendly Datzilla Gatekeeper
A web-based error reporting and tracking tool
22 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
What is Datzilla?What is Datzilla?
Datzilla is a Web-based tool to report and track errors in NOAA-held data, metadata, or data delivery systems.
It is maintained by the Southern Regional Climate Center and NCDC is a participant to the extent of addressing data it holds and systems it maintains.
33 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Errors vs. CorrectionsErrors vs. Corrections• Datzilla intended to correct Datzilla intended to correct errorserrors
• CorrectionsCorrections to observations are to observations are not errorsnot errors..
• For changes to recent observations (For changes to recent observations (commoncommon))– William Angel (COOP / ASOS) – William Angel (COOP / ASOS) – [email protected]@noaa.gov– Blake Lasher (LCD ASOS) – Blake Lasher (LCD ASOS) – [email protected]@noaa.gov– Send amended forms to NCDC via normal channels.Send amended forms to NCDC via normal channels.
• Corrections to historical observations (Corrections to historical observations (uncommonuncommon))– Amended formsAmended forms– Datzilla (in some cases)Datzilla (in some cases)– Must have a compelling reason to change a historical obs.Must have a compelling reason to change a historical obs.
• Corrections due to equipment problems may also require Corrections due to equipment problems may also require a metadata rendition.a metadata rendition.
44 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Some possible reasons for a Some possible reasons for a Datzilla ticket to NCDCDatzilla ticket to NCDC
• Data issue (usually)Data issue (usually)– That TMAX of 74 should be a 47!That TMAX of 74 should be a 47!– Your #$&!*@ QC clobbered my data!Your #$&!*@ QC clobbered my data!
• System issueSystem issue– CDO inventory doesn’t match the data I got!CDO inventory doesn’t match the data I got!
• Metadata issueMetadata issue– This station’s COOP number is wrong!This station’s COOP number is wrong!
55 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Who should be submitting error Who should be submitting error reports to Datzilla?reports to Datzilla?
• NCDC personnel (yes)NCDC personnel (yes)• RCC climatologists (yes)RCC climatologists (yes)• State Climatologists (yes)State Climatologists (yes)• WFO DAPM/OPL (yes)WFO DAPM/OPL (yes)
• Non-NOAA researchers (no)Non-NOAA researchers (no)• COOP Observer (NO)COOP Observer (NO)
Outside parties should make reports to local WFO, SC or RCC.
66 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Creating a DatzillaCreating a DatzillaError ReportError Report
Step 1: Log in and select “New” from Actions list.
If you do not have an account, you may request one here
http://datzilla.srcc.lsu.edu/datzilla
CLICK THERE
77 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Good tickets and BadGood tickets and Bad
• Pre-report investigation (critical)Pre-report investigation (critical)– Is the error really an error?Is the error really an error?– Is it the right station?Is it the right station?– Is it reproducible (browsers, systems)?Is it reproducible (browsers, systems)?
• Do you have the right system?Do you have the right system?– ACIS, ThreadEx, NCDC data may be differentACIS, ThreadEx, NCDC data may be different
• Have you proposed a solution?Have you proposed a solution?– ““I think it’s too high.” is not a solution.I think it’s too high.” is not a solution.
88 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
A good A good ticketticket
• Clear titleClear title
• Good Good description of description of problemproblem
• ID of stationID of station
• Attached formAttached form
• Proposed Proposed solutionsolution
99 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
A bad ticketA bad ticket
• Vague titleVague title
• No supporting materialNo supporting material
• No suggested fixNo suggested fix
• HunchesHunches
• Data “corrections” Data “corrections”
• Vague description of problemVague description of problem
• No ID of specific station, element, dateNo ID of specific station, element, date
Please don’t drive us crazy
with wild goose
chases!
1010 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Investigating the errorInvestigating the error• xmACISxmACIS
– An RCC-maintained display tool for NOAA dataAn RCC-maintained display tool for NOAA data– Interprets data flags from NCDC data setsInterprets data flags from NCDC data sets– Includes recent data that may not be QC’dIncludes recent data that may not be QC’d– Prioritizes simultaneous values from different data setsPrioritizes simultaneous values from different data sets– Displays “M” for invalid as well as missing dataDisplays “M” for invalid as well as missing data– May thread data from several stationsMay thread data from several stations
• Primary, Secondary, Tertiary forms:Primary, Secondary, Tertiary forms:– 1001, B-91, 10-A, …1001, B-91, 10-A, …– 1014, LCD, CD, CRB1014, LCD, CD, CRB– Media articles, eye-witness Media articles, eye-witness
accounts, proxy dataaccounts, proxy data
• Paper, WSSRD, IPS …Paper, WSSRD, IPS …
Wheeling, WV flood of 1936
1111 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Who will investigate?Who will investigate?
• Karsten Shein (gatekeeper / historical)Karsten Shein (gatekeeper / historical)
• William Angel (operational data)William Angel (operational data)
• Henry Ray (metadata)Henry Ray (metadata)
• Bryant Korzeniewski (historical)Bryant Korzeniewski (historical)
• Mark Lackey/Deb Franklin (CDO)Mark Lackey/Deb Franklin (CDO)
• Keith Eggleston (xmACIS)Keith Eggleston (xmACIS)
• … … and others as neededand others as needed
1212 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
My ticket’s been open forever!My ticket’s been open forever!• Some are easy to fixSome are easy to fix
– Small-batch keying errorsSmall-batch keying errors– A few missing valuesA few missing values
• Some are notSome are not– May be part of a larger issueMay be part of a larger issue– May exceed NCDC resourcesMay exceed NCDC resources– May not have a straightforward fixMay not have a straightforward fix– May involve parties/systems outside NCDCMay involve parties/systems outside NCDC
• Old pubs will not be reissuedOld pubs will not be reissued– But the data errors they may contain will be fixedBut the data errors they may contain will be fixed
1313 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Your #$&!*@ QC clobbered my Your #$&!*@ QC clobbered my data!data!
• You are right to alert us.You are right to alert us.
• However, before you do …However, before you do …– Think about Think about whywhy our QC would our QC would
flag that data.flag that data.– Most often, the problem is not our Most often, the problem is not our
QCQC• Observer date shiftingObserver date shifting
• Observing at time different than Observing at time different than expectedexpected
• Instrumentation problemsInstrumentation problems
– Known climate anomalies?Known climate anomalies?– Unreported temporary anomalies?Unreported temporary anomalies?
1414 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Any questions?
Once again, your friendly (no, not that friendly) NCDC Datzilla Gatekeeper!
Datzilla is maintained by:Kevin Robins (SRCC)[email protected]
Datzilla POC at NCDC:Karsten [email protected]
NCDC Halloween, 2006
1515 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Web Search, Store, Web Search, Store, Retrieve, DisplayRetrieve, Display
(WSSRD)(WSSRD)
NWS Data Stewardship Meeting – Peachtree City, GA, 20-21 June 2007
1616 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
What is WSSRD?What is WSSRD?
• Electronic image archive (database)Electronic image archive (database)• Web-based interfaceWeb-based interface• http://noaa.imcwv.com/http://noaa.imcwv.com/• 50 + million images50 + million images• COOP network formsCOOP network forms
– Nearly all post-1984Nearly all post-1984– 14 of 50 states back to 189314 of 50 states back to 1893
• Access to:Access to:– U.S. Government and contractorsU.S. Government and contractors– Educational institutions doing env. ResearchEducational institutions doing env. Research– Researchers associated with NOAA projectsResearchers associated with NOAA projects
1717 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
The WSSRD interfaceThe WSSRD interface
COOP Observation forms
CRBOld 1st order daily forms (SWO)CD and LCD
Some F-6 forms
Older 1st order monthly forms
1818 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
WSSRD COOP SearchWSSRD COOP Search
Keep your searches
general, and use wildcards
(*) where possible
You’ll probably never use these fields
This is the (E)xpression builder to widen your search.
This is the (L)ist builder to widen your search.
1919 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
WSSRD COOP Search ResultsWSSRD COOP Search Results
Note: A search by COOP number would not have returned this record!
Click here to view record
2020 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
COOP form on WSSRDCOOP form on WSSRD
2121 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Alternatives and PlansAlternatives and Plans
• Most COOP forms available in NCDC’s IPSMost COOP forms available in NCDC’s IPS– Access via normal NCDC websiteAccess via normal NCDC website
• WSSRD in process of being replaced by WSSRD in process of being replaced by COTS software (in review)COTS software (in review)
• Transition process may take a while.Transition process may take a while.
2222 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
Want An Account or More Info?Want An Account or More Info?
Please contact
Cindy Karl at NCDC
Or visit
http://www.ncdc.noaa.gov/oa/climate/cdmp/wssrd.html
2323 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov
WHEN MAKING A WSSRD ACCOUNT WHEN MAKING A WSSRD ACCOUNT REQUEST…REQUEST…
• Please Indicate via E-mail to Cindy Karl…Please Indicate via E-mail to Cindy Karl…– Your Job Title (e.g. Lead Meteorologist)Your Job Title (e.g. Lead Meteorologist)
– Your Organization (e.g. NWS-GSP)Your Organization (e.g. NWS-GSP)
– The Data That You’ll Need To Access (e.g. COOP)The Data That You’ll Need To Access (e.g. COOP)
• Will Accept Account Requests For…Will Accept Account Requests For…– Researchers and Those From Universities (*.edu Addresses)Researchers and Those From Universities (*.edu Addresses)
• Please Include Information on Your Research as it Relates To Your Please Include Information on Your Research as it Relates To Your Need for a WSSRD AccountNeed for a WSSRD Account
• Indicate the Time Frame Needed For Use of WSSRD (e.g. 90 days)Indicate the Time Frame Needed For Use of WSSRD (e.g. 90 days)
– Those Sponsored by NOAA (e.g. Contractors, Interns, etc.)Those Sponsored by NOAA (e.g. Contractors, Interns, etc.)• NOAA Sponsor Must Send E-mail Request on their BehalfNOAA Sponsor Must Send E-mail Request on their Behalf
2424 http://www.ncdc.noaa.govhttp://www.ncdc.noaa.gov