content, discovery, and accessibility enhancements to the ncar research data archive
DESCRIPTION
Content, Discovery, and Accessibility Enhancements to the NCAR Research Data Archive. Doug Schuster and Steve Worley NCAR. Topic Outline. Archive Content Data Discovery and Access. Archive Content. Common Data Management Tools Data acquisition Metadata extraction and mapping - PowerPoint PPT PresentationTRANSCRIPT
Content, Discovery, and Accessibility Enhancements to
the NCAR Research Data Archive
Doug Schusterand
Steve WorleyNCAR
Topic Outline
Archive ContentData Discovery and Access
Archive ContentCommon Data Management Tools
Data acquisition Metadata extraction and mapping Data archival
Data Management Tools• Old System: Specialized Software to manage each data
input.• Inefficient• Difficult to Scale
RDAMetadataDatabase
Unidata Server
University Server
NWPServer Online Disk
Tape Storage
GCMDMetadata
Server
RDA Data Server
Specialized Software
Package 2
Specialized Software
Package 3
Specialized Software
Package 1
• New System: Common RDA tools that homogenize data management.• Efficient• Scalable
RDA Data Management
Common Tool Set
Archive ContentPopular Datasets
In situ and satellite based observations Gridded analysis and forecast model data High resolution re-analysis
Popular RDA Datasets
Platform ObservationsDataset Title Coverage Update
FrequencyNCEP GDAS observations (PREPBUFR) Global 1999 - Present Daily
NCEP GDAS observations (BUFR) Global 1999 - Present Daily
NCEP GDAS Satellite Radiance Data (BUFR) Global 2004 - Present Daily
Unidata IDD GTS based observations (NetCDF) Global 2002 - Present Daily
NCEP operational observations (ON-29 Format) Global 1975 - 2007 Fixed
International Comprehensive Ocean-Atmosphere Data Set (ICOADS)
Global 1662 – Present
Monthly
1662 2010Global Platform Observations
Popular RDA Datasets
Analysis and Forecast Model DataDataset Title Coverage Update
FrequencyThorpex Interactive Grand Global Ensemble
(TIGGE) Global 2006 - Present Hourly
Unidata IDD (GFS 0.5deg, RUC 20km, NAM 12km) Global and Regional 2002 - Present Daily
NCEP ETA/NAM (40km) North America 1995 - Present Monthly
ECMWF Operational Deterministic (1.25 x 1.25 Deg) Global 1985 - Present Bi-Yearly
NCEP GDAS Final Analysis (1x1 Deg) Global 1999 - Present Daily
NCEP OI Global SST (1x1 Deg) Global 1981 - Present Weekly
NOAA OI Global SST (0.25 x 0.25 Deg) Global 1981 - Present Monthly
Hadley Centre Global Sea Ice and SST Global 1850 - Present Monthly
1850 2010Analysis and Forecast Model Data
Popular RDA Datasets
High Resolution Re-AnalysisDataset Title Coverage Update Frequency
ERA-15 (N80 Gaussian) Global 1979 - 1993 Static Set
ERA-40 (T159) Global 1957 - 2002 Static Set
ERA-Interim (N128 Gaussian) Global 1989 - Present Yearly
1890 2010High Resolution Re-Analysis
JRA-25 (1.125 Deg Gaussian) Global 1979 – Present Yearly
NCEP/DOE (T62) Global 1979 - Present Static Set
NCEP/NCAR (T62) Global 1948 - Present Quarterly
NARR (32 x 32 km) North America 1979 - Present Quarterly
CFSR Global 1979 - Present TBD
NOAA-CIRES 20th Century Global 1890 – 2008 Static Set
Data Discovery and AccessData Discovery
Popular dataset listGoogle style searchDrill down style searchEnhanced content metadata
Data DiscoveryLog inPopular Dataset ListGoogle Style
SearchDrill Down Style
SearchEnhanced Content
Metadata
Data Discovery and Access
Data Access MethodsUser created “dynamic” file list
and download optionsServer default “static” file list and
download optionsDelayed mode request and
download optionsSpecialized project portals
Data AccessInternet Download Options Matrix
Data Group TitleDiscover and Access Online Data Files for Immediate Download (40
TB)
Online Data Example 1: User Created “Dynamic” File List and
Access Options
Data AccessExample 1: User Created “Dynamic” File List
Select Desired Parameter(s)Select Type of
ProductSelect Vertical LevelEnter Desired Date RangeUpdate the List
Data AccessExample 1: User Created “Dynamic” File List
Select Files for downloadVolume SummaryDownload as
Single Tar FileDownload Full Files
with Wget ScriptDownload Select
Parameters with cURL Script
Data AccessExample 2: Server Default “Static” File List
Online Data Example 2: Server Default Static File ListClick on Magnifying Glass Icon
to Access File Level Content Metadata
Internet Download Options Matrix
Data AccessInternet Download Options Matrix
Discover and Request for Offline Data Files to be Staged Online for Delayed
Mode Download (500 TB)Example 3: Request Access to
Offline Data Files
Data Access
Example 3: Request Access to Offline Data FilesEmail Notification when
Delayed Mode Request is Ready for Download
Data AccessInternet Download Options Matrix
Email a Question Regarding Data Access
Special Project Portals• TIGGE Portal
– THORPEX Interactive Global Grand Ensemble Archive (TIGGE)
– Supports Weather research leading to an integrated Global Interactive Forecast System• Integrated across multiple international NWP
Centers • 10 NWP Centers contributing to TIGGE
• Future– Model Validation Portal (under design)– Upper Air Observations Portal (under
development)
Special Project PortalsTIGGE Portal
http://dss.ucar.edu