Ground Systems Data in the World Data Center Archives

Justin Mabie (NGDC) and Anatoly Soloviev (GCRAS)2 July 2011

1

Ground System Data Holdings

Solar

Indices Magnetometer2

Ionosonde

3

Old School Data in Climate Controlled Storage

Boxes and folders and files – oh my!

Some are neat, some are a mess, some are so old our computers can’t read them.

In the NGDC Warehouse:

4

To get space in the warehouse, we spend countless hours bringing the data together on a single medium and creating inventories and metadata records.

In the past, these data have been migrated to film or packed away in boxes. Modern digital data is packed away neatly in our ADIC tape library system, which is cheaper, less labor intensive, and does not require a large climate controlled warehouse.

Geomagnetic Data Ingest System

5

NGDC workflow for geomagnetic observatory data

Solar Data Ingest System

NGDC workflow for solar observations

ISO Metadata Workflow

Metadata is a real challenge. This workflow shows how NGDC ingests metadata, performs quality control, ports it to the ISO metadata standard, loads it into databases, and disseminates it to our customers.

NorthwesternResearchAssociates

USAF Weather AgencyResearch Lab

NWSSWPC

Australia Ionospheric Prediction Services

Univ. MA, LowellCenter for Atmos. Research

Space EnvironmentTechnologies

Users Include

Ionosonde – User ServicesReal Time Applications

9

NGDC is developing databases so our users can make use of our metadata in ways we can’t think of.

We traditionally focus on the needs of the environmental scientist – what about the instrument and hardware developers, the social scientist, the politicians?

For further information, contact:geomag.obs@noaa.gov

GRAILS/ORACLE Databases for Metadata

XML – Metadata Format

10

CDMP – Climate Data Modernization Project

CDMP Data Recovery for Magnetometer, Solar, Ionosonde, and Airborne Imagery

Example – Geomag:Title: CDMP - GeomagnetismVolume: Approximately 4TB per year of data Storage: ADIC tape libraryDissemination: Reduced resolution images will be put on FTP site; full resolution data available by request.

STATUS:All efforts De-Funded due to fiscal crisis. Procedure is in place, providing an opportunity for future funding.

Notes:Data volume is very large (many terabytes)

Full filenames are included to reduce NGDC overhead (lesson learned from earlier projects)

Holdings range in date from 1957 – 1983 for Geomag.

NGDC film magnetograms were on track for completion in 10 years. Solar data is near completion, and Ionosonde data is aprroximately 10% modernized.

11

Geomagnetic and Solar Indices Bulletins

Title: Geomagnetic Indices Bulletin

Description: Monthly publication containing Kp, Ap, Cp, An, As, Am.http://ngdc.noaa.gov/stp/geomag/indices.html

Solar Indices bulletin contains Sunspot Number, Sunspot Prediction and Solar Radio Flux data.

Media: online only NGDC Holdings: 1996 – Present

URL: ngdc.noaa.gov/stp/geomag/geoib.htmlftp.ngdc.noaa.gov/STP/SOLAR_DATA/Publications/

DATA SETS•Printed for research purposes

12

Geomag Observatory Metadata Form

Page 1/5Page 2/5

XML Page

Return

13

RSS feeds to notify users of product changes

14

What is RSS?

RSS is a way for a user to automatically retrieve information from selected websites and aggregate it.

The feeds (in XML format) are collected with an RSS reader. Readers for RSS include email applications such as Mozilla Thunderbird and fancier readers such as Google Reader.

Subscribers to RSS feeds maintain their privacy.

How will RSS feeds be used at NGDC?

Users will select RSS feeds for specific data types.

Subscribers will get occasional automatic alerts for:

format changes reprocessed data sets new data sets

Coming soon!

Thank You!

15