GSICS EP Meeting 2015 – GSICS Archiving Strategy 1

GSICS Archiving Strategy

Peter Miu / Boulder, USACMA, CNES, EUMETSAT, ISRO, IMD, JAXA, JMA, KMA, NASA, NIST, NOAA,

ROSHYDROMET, USGS, WMO

GSICS Data Preservation Strategy

GSICS EP Meeting 2015 – GSICS Archiving Strategy

Overview

Clarification: Archive vs Preservation

Identifying what GSICS Products to preserve.

Data Preservation Strategy.

Requirements and Summary.

2

GSICS EP Meeting 2015 – GSICS Archiving Strategy

Data Archiving vs Data Preservation

The term data archiving not only implies storage but additional services such as:

data discovery – services for users to detect the data (making the data explorable from multiple sources).

data identification – services for users to clearly understand what the data is (e.g. meta-data, landing pages, etc.).

data search – services for users to find the data. data migration – obsolescence mitigation services to ensure the components of the

archive maintain a reasonably level of integrity e.g. media change.

These services make data archiving expensive.

Data preservation is less of a formal endeavour to ensure that digital information of continuing value remains accessible and usable. It only imply reliable access to the data.

For GSICS products, a data preservation strategy shall apply.

3

GSICS EP Meeting 2015 – GSICS Archiving Strategy

Identifying what to Preserve

Do we need to preserve everything?

4

GSICS EP Meeting 2015 – GSICS Archiving Strategy

GEOLEOIR NRTC Products Analysis

EUMETSAT GEOLEOIR Near Real Time Correction (NRTC) Product Analysis:

One product has one correction applicable to support real time product generation.

Products have a shelf life after which time they are no longer useful.

For an ATBD, EUMETSAT can regenerate all NRTC products if needed.

Do we need to preserve NRTC GSICS products?

Recommendation: No.

5

GSICS EP Meeting 2015 – GSICS Archiving Strategy

GEOLEOIR RAC Products Analysis

EUMETSAT GEOLEOIR Re-Analysis Corrections (RAC) Product Analysis:

RAC is a combined product replaced daily with a new correction.

RAC products are expected to be available indefinitely i.e. no shelf life.

A RAC product will continue to grow until an ATBD update.

• A new MAJOR VERSION of the RAC combined product file is then created for the same time period as the version it replaces. The old version of the RAC product is moved but still accessible for reference.

All versions of a RAC product should be available for reference.

EUMETSAT can recreate any ATBD version of a RAC product if needed.

Do we need to preserve RAC GSIC products?

Recommendation: Yes.

6

GSICS EP Meeting 2015 – GSICS Archiving Strategy

Data Preservation Strategy (1)

For the latest RAC products’ version, product regeneration is done through:

By design as a RAC product is generated daily with a new correction which replaces the existing RAC file.

In addition to this, RAC products are expected to be replicated :

Across all collaboration servers. Through System administration of the servers where cluster technology

is used to ensure service always available and high availability disk arrays.

In addition to this, backups of disk are periodically taken to simplify data recovery.

Worst case scenario is the RAC product can be regenerated by the product generation software.

7

GSICS EP Meeting 2015 – GSICS Archiving Strategy

Data Preservation Strategy (2)

For the Old RAC products’ versions, product regeneration is done through the replication as described for the latest version:

Resulting from the mirroring across all collaboration servers. Through System administration of the servers where cluster

technology is used to ensure service always available and high availability disk arrays.

In addition to this, backups of disk are periodically taken to simplify data recovery.

Worst case scenario is the RAC product of any version is regenerated by the product generation software.

8

GSICS EP Meeting 2015 – GSICS Archiving Strategy

Requirements and Summary To implement GSICS Product Preservation, the following Data

Management Systems are required to be in place:

Mirroring on the collaboration servers for all GSICS products.

Collaboration Server Administrators need to ensure service and backup strategies are in place.

GSICS products generation software should be developed to be able to regenerate an version of a GSICS product.

The last point is a mandatory requirement in the development of the GSICS product generation software.

9

GSICS EP Meeting 2015 – GSICS Archiving Strategy 10

End of Presentation: Thank you for your attention