NPPNational Polar-orbiting Operational Environmental

Satellite System Preparatory Project

Interface Data Processing Segment (IDPS)

Joe Mulligan, IPO Ground Systems Division Deputy Chief

Marge Ripley, IPO IDPS IPT Lead

NPPNational Polar-orbiting Operational Environmental

Satellite System Preparatory Project

Interface Data Processing Segment (IDPS)

Joe Mulligan, IPO Ground Systems Division Deputy Chief

Marge Ripley, IPO IDPS IPT Lead

Timely, Accurate, Reliable Data from Sensors to Users

C3SDHN &

FEP(@ IDPS)

IDP@

Centrals

C3S

IDPS

Centrals SMD

Field Terminals (LRD, HRD)

FieldTerminalSoftware

CONUS Gateways

(4x)SMD

HRD

GroundReceptor

GroundStation

Mission Management Center

Stored Mission Data flow forCentrals and Science Data Users

C3SDHN &

FEP(@ IDPS)

IDP@

Centrals

IDP@

Centrals

C3S

IDPS

Science Users

SMD

Field Terminals (HRD)

FieldTerminalSoftware

FieldTerminalSoftware

HRD Data flow for Field Users

Field Users

CONUS Gateways

(2x)

Svalbard Ground

Mission Management Center

Deliver DataProducts

CLASS

Deliver DataProducts

Sensors Bus

Space Vehicle 1

Bus

NPP Spacecraft

Sensors

NPP Mission Data Flow

SDS

A/D Conversion

Packetization

Compression

Anc.

Sensor

Data

CCSDS (mux, code, frame)

Comm

Xmitter

SEN

SOR

S

OT

HE

R

SU

BS

YS

TE

MS

Cal.Source

\ Data Store

ENVIRONMENTALSOURCE

COMPONENTS

SPA

CE

SE

GM

EN

T

RDR

Production

EDR

Production

SDR

Production

EDR Level

SDR Level

RDR Level IDP

S

Comm

Receiver

Comm

Processing

CCSDS Level

C3S

NPP Data Product Definitions

NASA’s Science Data Segment will independently assess the quality of the science and environmental data records from the NPP mission for their

ability to support climate research.

All data products will be provided to NOAA’s Comprehensive Large-Array data Stewardship System (CLASS) for distribution to the broader scientific

user community.

The plan for the follow on mission is to add two more Centrals: FNMOC and NAVO.

NPP will provide data for processing at each of the two centrals: AFWA and NESDIS. Each Central will then disseminate the data to their user

communities.

The data products generated on IDPS are delivered to the Centrals, CLASS and SDS based on their data subscriptions.

NPOESS Preparatory Project (NPP) Science Data

Segment (SDS)

Air Force Weather Agency (AFWA)

NOAA Satellite and Information Service

(NESDIS)

Fleet Numerical Meteorology and

Oceanography Center (FNMOC)

Naval Oceanographic Office (NAVO)

Comprehensive Large Array-data Stewardship

System (CLASS)

Interfaces

Infrastructure

Ingest

Data Management

Data Delivery

Processing

Data QualityMonitor

DQM performs quality monitoring of the data products. It allows automated and ad-hoc processing in support of data quality notifications from the PRO subsystem. It also provides the interface and tools for the data quality engineers to analyze the data and generate reports.

This subsystem is implemented only at the NESDIS.

The Data Delivery Subsystem is the primary mechanism for data delivery. Users have the capability to submit data requests via a GUI or use of an API. HDF5 formatting, packaging and

aggregation of NPOESS Data products is performed.

The Data Management Subsystem manages the data within IDPS. Its primary function is sharing data between processes and the other subsystems. It provides on-line repository of

data, inventory and metadata.

The Processing (PRO) subsystem performs the data processing for product generation. PRO takes RDRs, ancillary (ANC) and auxiliary (AUX) data as inputs to generate Sensor Data

Records (SDRs), Temperature Data Records (TDRs), geolocation information, Intermediate Products (IPs), and Environmental Data Records (EDRs). PRO assesses the data quality of the SDR, TDR, and EDRs and sends real-time quality notifications to the Data Quality Monitoring

(DQM) subsystem to alert the operators of possible quality degradation.

The Ingest subsystem is the primary input path for data into IDPS. Stored Mission Data and Mission Support Data are ingested and used to create Raw Data Records (RDRs).

The Infrastructure subsystem manages IDPS by initiating tasks, monitoring processes, monitoring product generation, and maintaining log files of Status, Error, and Informational

messages. It also provides common utilities, tools and a scheduler. It interfaces with NPOESS Enterprise Management (EM) as well as providing the main GUI for IDP Operator management,

control and status.

The Interface Data Processing Segment (IDPS) consists of the data processing elements (DPEs) located at each of the four Centrals to generate and deliver NPOESS and NPP products and

associated metadata to the Centrals for additional application specific processing and dissemination. Each IDPS consists of six subsystems: Infrastructure, Ingest, Processing, Data

Management, Data Delivery and Data Quality Monitoring.

IDPS Functional Overview

•The current PRO design is based on an I-P-O model– Input (I) retrieves all needed input data –Processing (P) creates EDRs and IPs from the input data–Output (O) sends products and metadata to the DMS for

storage, formatting & delivery • The use of modular software isolates impacts of changes to the

basic architecture and provides for rapid, low impact recovery from hardware and software failures

• Additionally, granule based processing allows the generation of high quality products within latency timelines

• This construct allows the decoupling of algorithms from I/O

Input OutputProcessing

I-P-O Construct

NPP EDRs

NPP System LatencyNPP System Latency for 47th Orbit

4-copy Robustness Test executed 6/18 - 6/24/2009

0:00

10:00

20:00

30:00

40:00

50:00

60:00

70:00

80:00

90:00

100:00

110:00

120:00

130:00

140:00

150:00

Granule ID (increasing)

Sys

tem

Lat

ency

(m

m:s

s)

CrIMSS EDR

VIIRS EDR

ATMS SDR

CrIS SDR

OMPS SDR

VIIRS SDR

ATMS TDR

ATMS RDR

CrIS RDR

OMPS RDR

VIIRS RDR

Age @ Arrival

Max Allowable Latency for most EDRs

- NHF, Surface Albedo, & Sea Ice Characterization get 10 more minutes

- Final product for each chain shown for clarity - CERES is not delivered to the ACI.

IDPS is optimized to reach data latency requirements

Nominal Robustness TestLatency Orbit (Orbit 47)

0:00

10:00

20:00

30:00

40:00

50:00

60:00

70:00

80:00

90:00

100:00

110:00

120:00

CrIMSS-EDR

VIIRS-Aeros-EDR

VIIRS-AF-EDR

VIIRS-CBH-EDR

VIIRS-CCL-EDR

VIIRS-CEPS-EDR

VIIRS-COT-EDR

VIIRS-CTH-EDR

VIIRS-CTP-EDR

VIIRS-CTT-EDR

VIIRS-I1-IMG-EDR

VIIRS-I2-IMG-EDR

VIIRS-I3-IMG-EDR

VIIRS-I4-IMG-EDR

VIIRS-I5-IMG-EDR

VIIRS-IST-EDR

VIIRS-LST-EDR

VIIRS-M1ST-EDR

VIIRS-M2ND-EDR

VIIRS-M3RD-EDR

VIIRS-M4TH-EDR

VIIRS-M5TH-EDR

VIIRS-M6TH-EDR

VIIRS-NCC-EDR

VIIRS-NHF-EDR

VIIRS-OCC-EDR

VIIRS-SA-EDR

VIIRS-SCD-BINARY-SNOW-FRAC-EDR

VIIRS-SCD-BINARY-SNOW-MAP-EDR

VIIRS-SIC-EDR

VIIRS-SST-EDR

VIIRS-ST-EDR

VIIRS-SusMat-EDR

VIIRS-VI-EDR

CrIMSS-Latency-EDR

VIIRS-Latency-EDR

IDPS Data Latency

• Each Central will have an IDPS − High performance IBM computing hardware− POWER architecture

• Consists of two hardware suites− Operational string− Integration and Test string

IDPS IBM Architecture

•The HW is scalable and expandable− Can add new capabilities without impacting the existing system

• Combining software and hardware flexibility and robustness− Allows stringent performance requirements to be met

• Fault tolerant hardware and software − Ensures high quality data products− Available to users within operational latency timelines

IDPS Flexibility

Summary

• IDPS generates the data products for end users.•Central Users will get data directly from IDPS•Science Users will get data from CLASS•SDS will get data directly from IDPS

•25 EDRs will be produced by IDPS

•IDPS is meeting Latency requirements

• IDPS is installed at AFWA and NESDIS and is currently undergoing system level testing in preparation for launch.

Questions ?

Questions ?