DATA MANAGEMENT UPDATE

Steve Williams, Linda Cully, and Scot Loehrer

NCAR/Earth Observing Laboratory (EOL)

NAME SWG (7.5) Meeting

28 October 2005

NAME Data Management Timeline

8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1

2003 2005

Finalize Operations Plan

Complete Operations Summary (WWW)

Operational Data Staged to CODIAC

On-line Catalog Operational (CODIAC)

Final Specs

from PIs

Field Ops and Data Collection QC’d data from PIs

Final Data Staged

Finalize Data Management Plan

OPS Planning Meeting

Planning Meeting

Preliminary data staged Integrated Data StagedDraft

DM Plan

2004

Data AnalysisWorkshop

http://www.joss.ucar.edu/name/dm

NAME Data Flow

OperationalData Ingest• GTS• Satellite• Radar• Model

SpecialResearchData Ingest• Aircraft• Ships• Profiler• ISS

NAME PIData andProducts

NAMEOnline Field Catalog

NAME DataArchive CenterUCAR (JOSS)

SMEX-04Data Center(NSIDC)

DistributedData Centers• CICESE• IMTA• NASA/DAACs• NASA/GSFC• NASA/JPL• NASA/LaRC• NOAA/ETL• NOAA/NCDC• NOAA/NODC• NOAA/NSSL• NSIDC• SMN• Univ. of AZ

DelayedOperational Data

“Final” PI Research Data

CatalogInformation

“Preliminary”Data Sets

“Prelim

inary”

Pro

du

cts

• • •

• • •

• • •

EOL Data Archives

metadataquality controlspecial processing

EOL Interactive Data System

EOL Interactive Data System

Data Sources

Research

Community

Archive andDistribution

NAME DATA BASE STATUS(as of 26 October 2005)

• 239 of 266 Data sets (90%) have been submitted or linked.

• 11 Data sets are in progress

• Field Catalog (w/reports and “browse” products) is available on-line

• Precipitation (hourly and daily) and Upper Air “Composites” are under preparation

• Data Access: http://www.joss.ucar.edu/ihop/archive/

http://www.joss.ucar.edu/name/namap2/

www.joss.ucar.edu/name/catalog/

.

NAME FIELD CATALOG SAMPLE PRODUCTS

NAME 2004 Field Catalog Statistics

• Reports/Summaries (Status, Mission, and Operations)

3,785 documents/imagery (0.27 GB)

• Research Platform Products (Aircraft, Surface, Upper Air, Ship)

17,606 Imagery files (1.25 GB)

• Operational Products (Satellite, Surface, Upper Air) 367,092 Imagery files (17.83 GB)

• Model Output Imagery (Analysis and Forecast Fields)

99,387 imagery files (6.76 GB)

• TOTALS: 487,870 Files (26.11 GB)

http://www.joss.ucar.edu/name/catalog/

Composite Data Sets at NCAR/EOL

A composite dataset is a collection (over some time period and region) of similar data (e.g. surface meteorological) from a variety of sources, put into a common format, and passed through a uniform quality control.

Why does NCAR/EOL develop composites?

- Provides data in a uniform format with QC.

- Allows determination of network/site problems.

- Useful for model applications.

- Prevents duplication of effort.

“Composite” Data Sets Being Developed by NCAR/EOL for

NAME

• Surface Meteorological – Datasets being converted

• Precipitation - Hourly, and Daily (June-Sept 2005)

• Soundings - Highest Res & 5 hPa (June – Sept 2005)

• Complete Station List for all Composites

NCAR/EOL Atmospheric Sounding Processing Procedures

Observing System Output e.g. NWS Micro-ART

Convert to JOSS Quality-control Calculate Derived

Format (ASCII) Parameters

Vertical Consistency Checks

QC Flags

Gross Limit Checks

Automated Checks QC Flags

e.g., u, v, dz/dt

Visual Examination

Error Files

High-resolution and/or Interpolated Vertical Composite e.g., 5 hPa

Inventory, Archive, and Develop Metadata

Data and Metadata Available Online

Fixed High Resolution (+)

Pibal (o)

R/V Altair (+)

R/V Ulloa (+)

R/V El Puma (+)

NAME TIER-1 EOL UPPER AIR “COMPOSITE” DATA

Fixed High Resolution (+)

Pibal (o)

R/V Altair (+)

R/V Ulloa (+)

R/V El Puma (+)

NAME TIER-2 EOL UPPER AIR “COMPOSITE” DATA

NAME TIER-3 EOL UPPER AIR “COMPOSITE” DATA

Fixed High Resolution (+)

Pibal (o)

R/V Altair (+)

R/V Ulloa (+)

R/V El Puma (+)

Belize - 140 soundings R/V Ulloa - 91 soundings R/V Altair - 154 soundings R/V El Puma - 32 tethersonde SMN - 2468 soundings ISS - 462 soundings Loreto GLASS - 176 soundingsNWS - 16079 soundings Phoenix - 360 soundings San Jose - 324 soundings Yuma - 938 soundings pibal - 3235 pibals

NAME ARCHIVED SOUNDINGS BY SOURCE

Total of 24,459 soundings, pibals, & tethersondes to be included in the Final “composite” data set for Tier 3*.

* Need NSSL Tezopaco and Pibal data to complete composite

Enhanced sites:Number Type Number Estimate of percent of sites of sondes number missing missing10 USA-NWS enhanced 1380 17 1.26 ISS, Loreto,2 ships 815 10 1.27 SMN enhanced 1057 30* 2.81 Yuma (4 sites comb.) 377 9 2.31 San Jose 173 12 6.925 subtotal enhanced 3802 78 2.1

Operational sites:33 USA-NWS operational 3030 41 1.38 SMN operational 402 57** 14.21 Belize 75 17 22.742 subtotal 3507 115 3.3

67 Total 7309 193 2.6* 14 missed due to unpaid electric bill at Empalme** 11 missed due to broken generator at Isla Socorro

CSU UPPER AIR COMPOSITE DATASET (V2.0)

(1 July - 15 August 2004)

Embedded Accumulative Precipitation

“Preliminary” Tier-1 Hourly Precipitation (872 stations)

ALERT (+ 405)

Army Range (o 40)

AZMET (o 27)

ISS (* 3)

MADIS (+ 84)

METAR (o 18)

NCDC Coop (* 58)

NCEP (* 59)

NERN/SMN/Sonora

(+ 119)

NMSU (+ 59)

“Preliminary” Tier-2 Hourly Precipitation (3913 stations)

ALERT (+ 414)

Army Range (o 40)

AZMET (o 28)

ISS (* 3)

MADIS (+ 891)

METAR (o 327)

NCDC Coop (* 736)

NCEP (* 1163)

NERN/SMN/Sonora

(+ 182)

NMSU (+ 129)

“Preliminary” Tier-3 Hourly Precipitation (10328 stations)

ALERT (+ 414)

Army Range (o 43)

AZMET (o 28)

ISS (* 3)

MADIS (+ 2855)

METAR (o 1089)

NCDC Coop (* 2328)

NCEP (* 3249)

NERN/SMN/Sonora

(+ 190)

NMSU (+ 129)

“Preliminary” Tier-1 Daily Precipitation (2050 stations)

GASIR (* 195)

Hourly Stations (+ 872)

NCDC Daily Coop (+ 208)

NCEP (* 557)

Simple Raingage (o 218)

“Preliminary” Tier-2 Daily Precipitation (11955 stations)

GASIR (* 1072)

Hourly Stations (+ 3913)

NCDC Daily Coop (+ 2544)

NCEP (* 4208)

Simple Raingage (o 218)

“Preliminary” Tier-3 Daily Precipitation (30232 stations)

GASIR (* 1106)

Hourly Stations (+ 10328)

NCDC Daily Coop (+ 7326)

NCEP (* 11254)

Simple Raingage (o 218)

DATA ISSUES FOR DISCUSSION

• Mexican Navy SEMAR Data approval letter received; Request to collaborate with data processing.

• SMEX-04 Archives are being merged Priorities?• Are there any datasets not included in NAME Master List?• Are there other “composites” that are needed?• Should U.S. and Mexican Streamflow data be “composited”?

Period of Record?• Need re-organization (and cleanup) of NAME web pages.• Need to establish “cutoff” date for datasets to be included in

the “composites”.• Others?