1 microwave integrated retrieval system for npoess preparatory project: mirs npp/atms integration...

1

Microwave Integrated Retrieval Systemfor NPOESS Preparatory Project:

MiRS NPP/ATMS Integration into NDE

Test Readiness Review

January 19, 2011

Prepared By: Kevin Garrett1

Chris Grassotti1

Sid-Ahmed Boukabara2

Limin Zhao4 Flavio Iturbide-Sanchez1

Wanchun Chen3

Leslie Moy1

1 I.M. Systems Group2 NOAA/NESDIS/STAR

3 Dell4 NOAA/NESDIS/OSPO/SSD

2

Review Agenda

1. Introduction 1:30 – 1:40 K. Garrett

2. CDR Report 1:40 – 2:05 C. Grassotti

3. Software Architecture 2:05 – 2:35Context and System Layer C. Grassotti

4. Unit Test Readiness 2:35 – 3:15MiRS DAP Test Readiness K. Garrett QC DAP Test Readiness C. Grassotti

5. Risks/Actions 3:15 – 3:25 K. Garrett6. Summary and Conclusions 3:25 – 3:30 K. GarrettDiscussion 3:30 – 4:00 All

3

· INTRODUCTION· CDR Report· Software Architecture· Unit Test Readiness

» Test Readiness of Individual Units: MiRS

· Risks/Actions· Summary and Conclusions· Discussion

4

Section 1 – Introduction

Presented by

K. Garrett

5

Project Objectives

· Technical Objective» Adaptation of MiRS to NPP ATMS and integration within NDE and OSPO

· Science Objectives

» Improved temperature and moisture profile retrievals» The extension of the retrieved products to non-standard surfaces including

sea-ice and snow-covered land » The retrieval in all-weather conditions including cloudy and precipitating

conditions» An improved set of retrieved surface properties whose derivation is based on

the retrieved emissivities instead of directly from the brightness temperatures

6

TRR Objectives

· Objectives of the Test Readiness Review» Goal #1: Gather all MiRS stakeholders to review the

overall system integration of MiRS into the NPOESS Data Exploitation (NDE) environment

» Goal #2: Review of MiRS Software Architecture» Goal #3: Review of MiRS Software Unit Test Readiness» Goal #4: Review of CDR action items and actions taken» Goal #5: Identify new or outstanding risks w/mitigation

strategies

· Follow the STAR EPL Guidelines for TRR

7

MiRS Stakeholders

· Development Team» S.-A. Boukabara, K. Garrett, F. Iturbide-Sanchez, C. Grassotti, W. Chen, L. Moy

· OSPO Partners» L. Zhao, J. Zhao, T. Conrad

· NDE Partners» P. MacHarrie, L. Fenichel, D. Powell, J. Silva, G. Goodrum

· MiRS Oversight Board» F. Weng (chair), R. Ferraro (STAR), L. Zhao (OSPO), J. Silva (NDE), T. Schott (OSD)

· Oversight Panels» SPOP, PREPOP, ICAPOP, LSPOP

· MiRS Users» Dan Pisut (NOAA EVL), Tony Reale (STAR), Joe Turk (JPL), Ben Ruston (NRL), Sheldon

Kusselson (SAB), Stan Kidder (CIRA), Kevin Schrab and Andy Adman (NWS), Denise Hobson (AFWA), B. Yan and M. Kim (JCSDA), G. Huffman and W. McCarty (NASA/GSFC), J. Tesmar (FNMOC), P. Wang (Taiwan Weather Bureau), J. Janowiak (UMD), Paul Field (UKMET), K. Okamoto (JMA), M. V. Engel (IAO SB), B. Lambrigtsen (JPL), Peiming Dong, Qui Hong and Hu Yang (CMA), Universities (Graduate School of Chinese Academy of Sciences, Universia degli studi di Roma), Franklin Robertson and Clay Blankenship (NASA/MSFC), Tom Auligne (NCAR), D. Vila (CPTEC, Brazil), W. Han (Chinese Met. Admin.), D. Cimini (IMAA/CNR), M. Itkin (MPI-M, Hamburg), T. Greenwald (SSEC)

8

MiRS Timeline/History(1/3)

· Development Phase Begins (Sep. ’05-Sep. ’06)» Go-ahead with product development (Sep. ’05)» Preliminary Design Review (Oct. ’05)» Critical Design Review (Sep. ’06)

· Pre-Operational Phase Begins (Jul. ’07)» Operational/backup processing capabilities in place. » SPSRB approves product to go operational

· Operational Phase Begins (Aug. ’07)» Operational/backup processing capabilities reach ops status» Code transitions to operations (MiRS Release I)» SPSRB updates product metrics web pages» OSD updates Satellite Products database

· Transitioned to Operations (Aug’07 – Jun’10)» N18, N19, Metop-A, F16, F18» Also running in STAR: NPP/ATMS, AMSR-E, FY3» Recently processing TRMM/TMI, Simulated GPM/GMI

9

MiRS Timeline/History (2/3)

· NPP ATMS Preliminary Design Review (Sept ’09)» Running with proxy data in daily processing at STAR» Identified Risks/Actions

· NPP ATMS Critical Design Review (June ’10)» Provided technical data regarding overall system (operations, requirements,

ATB, system design, architecture, quality assurance)» Identified Risks/Actions

· QC DAP Delivered: Dec 2010» Testing in Progress

· Product Oversight Panel Meetings:» SPOP: S. Boukabara» PREPOP: F. Iturbide» ICAPOP: K. Garrett

• Requirements for MiRS NPP/ATMS and NDE integration outlined in the “Research to Operations Project Level 1 Requirements Document for Microwave Integrated Retrieval System for NPOESS Preparatory Project” (Updated April, 2010)

10

Jul’09 Jul’10Jan’10

Feb’09Integration into NDE

work begins

Mar’09Initial version

of MiRS Integrated

at NDE

May’09Official

version of MiRS

integrated

Jun’09Development

with proxy data begins

Sep’09PDR for NPP

ATMS

Oct’09Process multiple

sample data granules from

IDPS

Nov’09-Dec’10Enhanced MiRS

development (advanced footprint matching, proxy data SDRs, QC

DAP)

Oct’11NPP

scheduled for launch (was

Jan’11)

Oct’11-L+90Post launch activities

CalibrationValidation

FG/bias/preclassifierSRR (Jan ‘12)

etc

L+90Official DAP delivery to

NDE

Jan’11 Oct’11Jan’09

Aug’09Updated version of

MiRS implemented at NDE and STAR

(full ATMS functionality)

Jun’10CDR for NPP

ATMS

MiRS in NDE Timeline (3/3)

Jan’10QC for NPP

Review

Jan’11TRR for NPP

ATMS

Dec’10QC DAP Delivery

Mar ‘11CUTR for NPP

ATMS

11

· Introduction· CDR REPORT· Software Architecture· Unit Test Readiness



12

Section 2 – TRR Report

Presented by

C. Grassotti

• Review of CDR Action Items• TRR Entry Criteria• TRR Exit Criteria• CDR Summary

13

CDR Action Items

· Total of 7 Action Items

· For each AI, response was drafted (when available) describing the item, the action(s) (to be) taken, and status

· Response sent to AI author(s)

· At present:» 5 Items Closed» 2 Remain Open

14

CDR Action Items

Action Item

Description Author Lead Org Status

1 MiRS Operations Manual in OSPO A.K. Sharma STAR Closed

2Mitigation Plan for Missing/Bad Ancillary Data

A.K. Sharma NDE/STAR Closed

3QC DAP for Trend Analysis and QC Monitoring

A.K.SharmaSTAR Closed

4NDE/OSPO Define Product Tailoring Responsibilities

G. Goodrum NDE/OSPO Open

5 Obtain Risk Analysis Matrix J. Silva STAR Closed

6Data Submission Agreement for MiRS NPP/ATMS in CLASS

T. Schott OSPO Closed

7Feasibility of Implementing Tier-3 Processing in MiRS QC DAP

T. Schott, L. Zhao

OSPO Open

15

Action Item # 1: MiRS Operations Manual

Submitted by: A. K. SharmaDescription: Determine who will be responsible for writing the Operations Manual,

one of the required SPSRB documents. When will this document be delivered to OSPO?

Lead Organization: STARResponse:

(From STAR): Members of STAR, OSPO, NDE attended SPSRB Documentation Guidelines Mtg on 2 Nov 2010. Based on discussions the following guidance was obtained:

» Total of 4 Documents to be generated: Algorithm Theoretical Basis Doc (ATBD), System Maintenance Manual (SMM), Internal Users Manual (optional, depending on need), External Users Manual

» The Operations Manual is no longer a required document, as it provided redundant information already contained in the ESPC Operations Procedures documentation. However, information relevant to the operation of a specific science algorithm (installation, monitoring, etc.) will be contained in the SMM.

» SMM contents to be based on “Document Objects” (section reuse across some documents); SPSRB document guidelines outline which organizations (STAR/OSPO/NDE) are responsible for each document object. http://projects.osd.noaa.gov/spsrb/doc/System_Maintenance_Manual_Guidelines.doc

» STAR is lead organization for all docs up until delivery to operations, but responsibility for individual sections is shared between STAR/OSPO/NDE. After delivery, OSPO/NDE will manage docs

» For MiRS, MS Word may be used, following SPSRB version 2 guidelines. May be ported to Foswiki upon SPSRB approval of the software.

» Document delivery upon promotion of NPP to operational status

Status: CLOSED

http://projects.osd.noaa.gov/spsrb/doc/System_Maintenance_Manual_Guidelines.doc

16

Action Item # 2: Mitigation for missing or bad ancillary data

Submitted by: A. K. SharmaDescription: Explain the mitigation plan for missing ancillary data or bad ancillary

data (e.g. GFS required for the MIRS QC DAP) in NDE.Lead Organization: NDE/STAR Response: (From STAR): MiRS core retrieval algorithm does not rely on any

ancillary data, however, the QC DAP will require a GFS forecast file for the Tier-2 QC. If the latest GFS forecast file is unavailable, then the NDE system will provide the latest available GFS file. The QC DAP software will have the ability to extract forecast fields valid for the periods around the observation (initial, 6-hr,12-hr forecast etc.), up to a 24-hr forecast. If the GFS data is more than 24 hours old (highly unlikely), then the QC DAP should not be executed due to uncertainty in the forecast fields. If the GFS data are available but are bad for given collocated ATMS Fields-of-View (FOVs), the MiRS will set quality flags, and those FOVs will not be included when generating statistical metrics to monitor. If all GFS data are bad within a whole collocated NPP ATMS granule, processing will still commence, but QC metrics will be set to fill values (-999) signifying to the operator that the ancillary data, and not the retrieval, are bad.

Status: CLOSED

17

Action Item # 3: MiRS QC DAP for trend analysis and QC

monitoringSubmitted by: A. K. SharmaDescription: Explain how the MIRS QC DAP will be useful for trend analysis for

monitoring data quality.Lead Organization: STAR Response: The MiRS QC consists of several components. For trend analysis, the more useful metrics would be: QC

flag monitoring, which looks at rates of QC=0 and QC=2 occurrence, the channel NEDT monitoring, which looks at radiometric noise as a function of channel and time, as well as the radiometric monitoring, which looks at differences between observed and simulated brightness temperatures as a function of channel, scan angle, and time. Time series of these metrics are produced and updated each day within the STAR environment. In the STAR environment, geophysical performance monitoring (part of so-called Tier-3) is also being done in daily processing along with the radiometric QC. If it is deemed to be feasible by OSPO and NDE, and added as a requirement, then this level of QC monitoring, Tier-3, may be included with the QC DAP. This would include geophysical performance time series of temperature and water vapor (bias and standard deviation compared to NWP) at selected atmospheric layers. Slides # 75 and #81 of the MiRS NPP/ATMS CDR show time series examples of NEDT and radiometric bias monitoring, respectively. Many more examples may be found on the STAR MiRS website itself under "Products Monitoring: Data Quality", "Radiometric Performance: Calibration Bias Monitoring", "Sensors Quality: NEDT", and "Geophysical Performance: Performance Time Series". In the context of NDE, the time series for all QC monitoring will be constructed as granules are processed by MiRS and the QC DAP initiated, with the first point of the time series dependent on NDE data retention policies.

Status: CLOSED, Geophysical trend monitoring capability at NDE will be dependent on decisions by OSPO/NDE on whether to implement Tier-3 QC at NDE and add it as a requirement (see AI #7)

18

Action Item # 4: NDE and OSPO product tailoring

responsibilities

Submitted by: Geof Goodrum

Description: NDE and OSPO to coordinate and determine product tailoring responsibility. Meet with NDE and ESPC architects to clarify concepts and answer the following:» What is the ESPC architecture approach to sharing and blending products?» Use NDE distribution or the shared SAN to hand-off products to other OSPO

systems?» What is the schedule for ESPC “To Be” architecture?

Lead Organization: NDE/OSPO Response: (1) Limin, Geof and Joel met and discussed above questions. Data could be made

available to ESPC through NDE DDS, but eventually the data should be made available through the shared SAN in the ESPC “to be architecture”, which no final decision has been made yet. The decision is expected to come out in Dec, 2010 time frame, but no final word yet so far. (2) Limin presented a briefing on the detailed tailoring capability that is provided by ESPC for current POES/Metop/DMSP version of MIRS products, and discussed the cost and benefits based on different tailoring scenario that NDE would provide. NDE team will finalize their responsibility and present their final tailoring solution to STAR/OSPO. (3) As to the blended products that need the MIRS ATMS products, since it involves ingesting products from multi-satellites, NDE will feed the MIRS ATMS products to ESPC to allow the heritage application merge MIRS ATMS products with that from other satellites and continue to generate and distribute the blended products.

Status: OPEN

19

Action Item # 5: Risk Analysis and Probability Matrix

Summary

Submitted by: J. SilvaDescription: MIRS team to get the Risk Analysis Probability and Impact matrix

template from Jim Silva to summarize project risks.Lead Organization: STARResponse: Obtained template for Risk Analysis/Probability Matrix and updated to

reflect MiRS CDR Risk Analysis. This analysis has been added to the risks/actions section of the CDR slides.

Status: CLOSED

20

Action Item # 6: Data Submission Agreement for MiRS NPP/ATMS in CLASS

Submitted by: T. SchottDescription: OSPO to review current ESPC/CLASS Submission Agreement (SA)

with NDE to prepare for updated SA.Lead Organization: OSPO Response: Limin checked with Phil Jones at NCDC, and confirmed that a new

SA is needed for the NDE MIRS products. Options are: (1) the existing MIRS SA can be referenced and/or provided as attachment; (2) write a completely new SA, starting by deleting all legacy references in the existing MiRS DSA. Question as to whether NCDC requires restarting process at Step 1 with a “request for archive”. Given the situation that archive data will be produced by NDE for the first 18 months after launch, and also NDE has information about how and what the products will be made available to CLASS, the SA will need be coordinated between NDE and OSPO. OSPO can take a lead on this, but Limin needs to know whom she can work with in the NDE team on this matter. Action is recommended for closing, as current SA has been reviewed.

Status: CLOSED

21

Action Item # 7: Assess feasibility of implementing Tier-3 QC processing within

operations

Submitted by: T. Schott and L. ZhaoDescription: OSPO to draft RFA to assess feasibility of implementing Tier-3

(science quality) into operations if it was to be included in the MiRS QC DAP.Lead Organization: OSPO Response: OSPO needs detailed information from STAR, regarding effort, cost,

data resources, CPU and storage requirements, to determine if it is feasible to implement the Tier-3 QC monitoring into operation. STAR is available to discuss and provide additional information off-line on Tier-3 QC to OSPO.

Status: OPEN

22

CDR Action Items

Action Item


1 MiRS Operations Manual in OSPO A. Sharma STAR Closed


A.Sharma NDE/STAR Closed


A.SharmaSTAR Closed







T. Schott, L. Zhao

OSPO Open

23

MiRS for NPP/ATMS TRR Entry Criteria

· Entry # 1 – Review of CDR Report with Action Items, Responses, Status

· Entry # 2 – Review of the TRR for MiRS NPP/ATMS in NDE» Software Architecture» Unit Test Readiness» Risks/Actions

24

MiRS for NPP/ATMS TRR Exit Criteria

· Exit # 1 – Test Readiness Review Report» TRR Report will be compiled and delivered after TRR» TRR Report to contain:

– TRR Presentation– Actions– Comments

25

CDR Report Summary

· This CDR Report closes the CDR· Total of 7 Action Items:

» At present, 5 of 7 items are closed, 2 remain open· 2 TRR Entry Criteria have been established· 1 TRR Exit Criterion has been established

26

· Introduction· CDR Report· SOFTWARE ARCHITECTURE· Unit Test Readiness



27

Section 3 –Software Architecture

Presented by

C. Grassotti

28

Software Architecture: MiRS Background

· MiRS software already tested, implemented for OSPO (N18,N19, Metop, F16, F18)» High code maturity» A risk reduction for NPP/ATMS» Code unit testing was also done during MiRS development for POES and

DMSP before delivery to OSPO· Also running daily for AMSR-E and TRMM/TMI· Implemented for NPP/ATMS within STAR

» Runs daily using proxy data· Installed and tested using sample and proxy data at NDE

» TBD: integration with “live” data flow, daily processing, retention of multiple days/granules for QC testing.

· Modular design facilitates unit testing down to each major processing unit of the system-level data flow (flow diagram to follow)

· QC DAP components treated separately. MiRS retrieval codes independent from QC, but QC assumes that all MiRS products have been generated

29

Software Architecture:Standards Compliance (1/3)

• ISO Fortran 95 standards adherence is assessed using the Forcheck utility

• Manual Verification that SPSRB F95 coding standards are adhered to as well (standards and guidelines)» http://projects.osd.noaa.gov/spsrb/standards_software_coding.htm

• Standards upheld:» Use Implicit None in all modules and main programs.» Documented well enough. All modules/main programs/subroutines/functions have

a header:– Description of the code– Modules used– Subroutines or functions contained– Data types included– Intrinsic functions used– Argument description (input/output)– Code history

» In the code, major sections have their own comments to describe what the section of the code is doing.

» No hard coded numbers. All Constants are defined in module Const.

30


· Standards upheld (continued)» All variables are initialized. » Use free format syntax but the maximum length in each line is less

than 128 ( g95 will cut off any line longer than 128 characters).» No tab characters in all Fortran codes. » No GOTO statements in all codes. » All unused variables are removed.» Use private to ensure proper encapsulation.» No conditional statements using == or /= with floating points.» No numbered DO loops. Labeled DO loops are used.» All intrinsic functions are declared before using them.

31


· Use valgrind to help detect any memory leaks. Valgrind is a Linux program used to “detect many memory management and threading bugs, avoiding hours of frustrating bug-hunting, making your programs more stable”. Valgrind detailed information and usage:

http://www.valgrind.org/· Since last version, no memory leaks in MIRS (v7)

» All dynamically allocated memories are deallocated when they are not used anymore.

» No array index out of bounds. Array index out of bounds will usually cause “segmentation fault”.

· All code changes are managed by Subversion: a source version control system

· Full compliment of code review results related to coding standards to be provided in the Code Unit Test Review (March 2011)

32

Software Architecture:Resource Requirements

Minimum*# of CPUs

Minimum RAM

Min. Hard Disk Space

Platform Type

MIRS Requirements

1 1GB 3GB Any

Hardware Requirements

Operating System

C Compiler

F95 Compiler

Commercial Software

Freeware

MIRS Requirements

Linux orIBM AIX

xlC orxlc++ orgcc org++

ifort org95 orgfortran or Xlf95

IDL v6 and Up

BASH

Software RequirementsNot necessary for standard outputs

BUT required for QC DAP

*This is minimum required. Number of CPUs will affect the speed of execution. 8 CPUs are used in STAR to run three sensors daily.

33

Software Architecture:Overview

· The software architecture describes the structure of the system software elements and the external and internal data flows between software elements.

· The MiRS software architecture is described in the MiRS System Description Document created for POES and DMSP and delivered to OSPO. Updated documentation will be delivered by STAR as specified in the CDR Report.

· 3 Layers of design will be presented (STAR EPL Guidelines):» Context Layer - 0: External Interfaces

» System Layer - 1: Flow Between Units

» Unit Layer - 2: Flow Within Units

Section 3 (this section)

Section 4 (next section)

34

The Context-Layer:STAR EPL Guidelines

· The Context-Layer describes the flows between the system and its external interfaces

· An external input is defined as a data source needed by the system that is produced or made available by a process external to the system

· An external output is defined as a data sink that is produced by the system for an external user

· External interfaces must meet standard criteria to be included in the system architecture

MiRS Context Layer:External Interfaces

1

MiRS/QC

SAN

NDE Product Generation Manager

MiRS External Interfaces

Product Generation

Specifications

Working Directory

Systems Configurations

Forensics Repository

Input Files & PCF

InvocationProcess Req.

Rule SetsOutput Files

& PSF

Product Files

PSF (MiRS output)

Return Code

Working Directory Output

NDE Distribution ServerInput Files (HDF5, GRIB)

ESPCInput (HDF5) Files GFS (GRIB)

PCF (MiRS input)

IDPS

DAP Specifications

Data AreasConfigurations InfoMIRS SystemNDE Production Manager

NDE DHS Boundary

DDS ECMWF,GDAS (GRIB)

NWP not necessary for Core Products (QC only)

36

External Interfaces: Assumptions

· Tailoring and (re)formatting done by NDE and OSPO (Context Layer)

· Interactions with operational users and CLASS handled by NDE and OSPO (Context Layer)

· No incompatibility between NDE and OSPO ESPC environment» Single version of MiRS to maintain

37

· File format requirements for NPP ATMS» ATMS level 1b granules/geo formatted in HDF5» PCF ascii file generated by NDE DHS» MiRS product outputs formatted in netCDF4 (CF conventions)» MiRS output PSF ascii file listing output files» MiRS readers and encoders support these formats and have been

tested in NDE· Metadata:

» Current MiRS Collection Level metadata available in ISO 19115 at CLASS (for POES/Metop/DMSP)

» Expectation is for a similar Collection Level file for MIRS NPP ATMS products to be stored at CLASS

» Metadata requirements for MiRS NPP ATMS will be outlined by updated Submission Agreement in the future

» MiRS NPP ATMS Granule Level metadata to be contained inside the MiRS netCDF4 output header (following STAR metadata template)

MiRS External Interfaces

38

External Interfaces: Status

· All external interfaces have been reviewed and approved; no changes since CDR

· No open actions on external interfaces

39

The System-Layer: STAR EPL Guidelines

· The System-Layer data flow expands upon the Context-Layer data flow, showing the first layer of decomposition. » In addition to the System-Layer inputs and outputs, the

major processing units are shown along with their inputs and outputs.

» Each unit is designed as a stand-alone program for ease of testing and integration into a System-Layer scheduler.

40

The MiRS System Layer: Processing Units

· Each major step in the MiRS processing sequence is a stand-alone bash script and a corresponding Fortran 95 executable and namelist file and constitutes a Layer-2 Test Unit

Code Unit Purpose

r2r2tdr Convert raw data records to temperature data records (antenna temperatures); generate sensor NEDT files

tdr2sdr Convert temperature data record to sensor data record (TBs or radiances)

fm Footprint matching

chopp Chop fm files into sub-files

applyRegr (retrRegr) First guess generation using TB-based regression (applied on chopped fm files)

fmsdr2edr (1dvar) 1dvar: converts footprint matched SDRs to EDRs

mergeEdr Merge EDR files into 1 file

vipp Postprocessing converts EDRs to derived environmental parameters (DEPs)

convertMirs2nc Converts files from MiRS binary to netCDF4

41

The MiRS System-Layer:npp_scs.bash

· Some or all units in the system layer may be invoked by the top level driver script npp_scs.bash (in operations all 9 units run from same invocation of driver script).

· The system layer is where NDE will invoke the MiRS software units.· Each unit is a bash script-function that drives a low-level fortran processing program.· When the system’s input data is available (ATMS granule), the NDE PGM will run the

top level driver script.» Execute npp_scs.bash passing the “working directory” path as the argument» The NDE PGM will generate a Process Configuration File (PCF) which contains all input file locations and

parameters required for processing, and is read in by the driver script.

· All code units process sequentially, one ATMS granule at a time. · The NDE DHS must be able to run multiple instances of these units to process

concurrently available granules.· Each instance will produce a PSF file containing a list of output product files if they

were created successfully.· In STAR, the driver script is invoked through crontab and the PCF variables are

specific to the STAR environment.

42

MiRS System-Layer Process Flow: NDE Environment

d

Local Processing Directories(working directory)

NDESAN

(HDF5)

L1

B S

en

so

r D

ata

(H

DF

5)

rdr2tdr

TDRs

TDRs

SDRs

SDRsFMSDRs

FMSDRs

Chop FMSDRs

REGRESS Retr

Chopped FMSDRs

Chopped FMSDRsEDRs

EDRs + Ancillary`

DEPs

SND (netCDF4 EDR)

IMG (netCDF4 DEP)

npp_scs.bashPGM

PCF

Return value to PGM

Process Status File

Log FileLocal Processing Directories

tdr2sdr

fm

chopp

applyRegress

1dvar

mergeEdr

vipp

mirs2nc

Merged EDR

EDRs

EDRs + DEPs

L1B Sensor Data

43

System Layer Architecture: Status

· All System Layer components have been reviewed and approved; no changes since CDR

· No open actions on system layer components

44

· Introduction· CDR Report· Software Architecture· UNIT TEST READINESS



45

Section 4 –Unit Test Readiness

Presented by

K. Garrett - MiRS DAP C. Grassotti - QC DAP

46

The Unit-Layer: STAR EPL Guidelines

· The Unit-Layer data flow expands upon the System-Layer data flow, showing the second layer of decomposition. » This layer shows the detailed data flow into and out of

each software unit.» And, it shows data flow within the software units (if

applicable).

47

Test Readiness: Environment and

Configuration

· For all unit tests the testing environment will be both the STAR Linux servers and the NDE computational systems

· STAR Environment: Full end to end testing of MiRS DAP (RDRs to SND/IMG netCDF4) and QC DAP using proxy data for ~ 1 day of data and sample data using 1 granule

· NDE Environment:» MiRS DAP: Testing of ~ 1 day of proxy data independent of DHS (minimum);Testing of ~ 1

day of proxy data interfaced with DHS; 1 granule of sample data independent of DHS» QC DAP: Testing of ~ 1 day of proxy data independent of DHS

· For testing in STAR, configuration is the same as that used in the daily processing for all other sensors (e.g. f95, IDL, and bash scripts)

· For testing in NDE, configuration is identical, except for high level process management (interaction with DHS). Output files for all processing steps to be compared against benchmark data produced in STAR test.

48

Test Readiness: Test Data and Truth Data

· Test data will include:» Proxy data generated using MIT/LL software from Metop-A (1 day 2011-01-15)» Proxy data generated using GRAVITE (1 day 2011-01-15)» Sample data generated by IDPS (1 granule TBD)

· All input test data in HDF5 format following NPOESS Common Data Format Control Book (vol. III)

· Truth data: NWP analysis from GDAS as well as EDRs and DEPs obtained from the operational sensors valid at the same time and location (i.e. not all parameters available for validation in NWP data; e.g. sea ice)

49

All Units:Test Method, Sequence

· The test method is to run using ATMS proxy data in stand-alone mode (single execution of bash script and f95/IDL code) with all the required input/output files defined at the script level.

· For many steps the upstream input data files are simply the output generated by the previous required Layer 2 MiRS processing unit (e.g. fm→fmsdr2edr)

· Confirmation of successful test will be determined by verifying creation of intermediate and output data (comparison with benchmark files), using log files, and/or diagnostic graphical tools (IDL) available in the STAR and/or NDE environment (maps, error plots, etc.)

50

Unit Tests

· The following will be presented for each of the 9 units:» Unit Description : Purpose and Function» Process Flow» Test Items» Input/Output Data» Test Risks

51

Unit rdr2tdr: Purpose and Function

· Convert raw sensor data records to temperature data records (antenna temperatures)

· Reformatted into MiRS internal format· Computes sensor radiometric noise values (used to update the

instrument noise matrix used in the 1dvar minimization)· Input: Level 1b in HDF5 format produced by external process at NDE

(current testing with proxy data generator uses HDF5). For NPP ATMS, the Level 1b data may be actual TDRs or SDRs, rather than RDRs.

· Output: TDR and NEDT files in internal format produced by rdr2tdr· It reads in a namelist which specifies operating parameters (passed

from the PCF by npp_scs.bash)

52

Unit rdr2tdr: Process Flow

d

Local Processing Directories

rdr2tdr


Input Files Specified in f95 Namelist

NDE SAN

Input: Level 1b Sensor Data (HDF5)

Output: • TDR Files• NEDT Files (MiRS Internal)

rdr2tdr unit script

Process Control FileExecution from PGM

Log File

Return value to PGM if failure

npp_scs.bash

53

Unit rdr2tdr Test Items

Software Unit Purpose File Input File Output

npp_scs.bash This is a driver script that: (1) Reads in the PCF, parses its contents, and passes necessary information to rdr2tdr.bash. (2) Calls rdr2tdr script(3) Writes out log files, the PSF file, and returns error status to the NDE DHS.

PCF PSFDriver script log file

rdr2tdr.bash Performs local file management , staging, resources file generation (e.g.f95 namelist) and error handling forrdr2tdr.

Script log fileExecutable program log fileExecutable resource input file (F95 namelist)

rdr2tdr.f90 For a single ATMS granule, it must read both the ATMS HDF5 RDR*and Geo files.

* In the case of NPP ATMS, rdr2tdr code can read in either ATMS TDRs or SDRs and convert to MiRS TDR format.

1 ATMS TDR or SDR HDF5 file1 ATMS Geo HDF5 fileNamelist file

1 MiRS TDR_* file1 MiRS NEDT_* file

54

Unit rdr2tdr –Input/Output Test Data Table

Test Data Item Type Filename* Description

1 Input npp_atms_rdr2tdr_2011-01-15.in F95 Namelist file

4 Raw

Sensor data

SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5 NPP HDF5 granule

5 Geo File GATMO_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5 NPP HDF5 Geolocation

2 Output TDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5

MiRS internal format file containing output TDR

data

3 Output npp_atms_nedt_2011_01-15_befFm.dat NEDT file

*All sensor data filenames used in the following tables pertain to proxy data, which mimics IDPS sample data naming conventions.

55

Unit rdr2tdr –Test Risks

· Test Risk #1: Simulated data and data formats may not fully represent the actual data (identified in CDR)

· Mitigation: Use of GRAVITE data sets from IPO or latest sample data from IDPS

56

Unit tdr2sdr: Purpose and Function

· Apply antenna pattern correction (if selected) to convert tdr files to sensor data record (TBs or radiances)

· In operations, no correction applied and the step is simply a reformatting to MiRS internal format (differences accounted for in radiometric bias corrections)

· Input: TDR files in internal format produced by rdr2tdr· Output: SDR files in internal MiRS format· It reads in a namelist which specifies operating

parameters (passed from the PCF by npp_scs.bash)

57

Unit tdr2sdr: Process Flow

d


tdr2sdr


Input: TDR Files(MiRS Internal)

Output: SDR Files(MiRS Internal)

tdr2sdr unit script


Log File

Return value to PGM


Process Status File

npp_scs.bash

58

Unit tdr2sdr Test Items


npp_scs.bash This is a driver script that: (1) Reads in the PCF, parses its contents, and passes necessary information to tdr2sdr.bash. (2) Calls tdr2sdr script(3) Writes out log files, the PSF file, and returns error status to the NDE DHS.


tdr2sdr.bash Performs local file management , staging, resources file generation (e.g.f95 namelist) and error handling fortdr2sdr.


tdr2sdr For a single ATMS granule, it must read the ATMS TDR file, apply attenna pattern correction (optional) and write out to SDR file (MiRS internal format).

1 ATMS TDR_* fileNamelist file

1 MiRS SDR_* file

59

Unit tdr2sdr –Input/Output Test Data Table

Test Data Item Type Filename Description

1 Input npp_atms_tdr2sdr_2011-01-15.in F95 Namelist File

2 Input TDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5

MiRS internal format file to containing TDR data

3 Output SDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5

MiRS internal format file containing SDR data

60

Unit tdr2sdr –Test Risks

· Test Risk #1: Proxy data reliability (identified in CDR)

61

Unit fm: Purpose and Function

· Footprint matching to ensure all channels view the same location on the Earth

· Three options:» Use fm code (to be available from IPO) to tailor resolution to available CPU

resources and requirements (Best option)» Use resampled data from IDPS (step would then be a simple copy)» MiRS heritage code: FOV averaging

– High resolution: 96 FOVs per scanline. Each channel measurement retains its value for each FOV

– Low resolution: 3x3 average for each FOV at each channel (3 FOVs x 3 scanlines), 32 FOVs per scanline

· Input: SDR files produced by tdr2sdr· Output: Footprint-matched SDRs (FMSDRs) in the MiRS internal RAD

file format and NEDT file· It reads in a namelist which specifies operating parameters (passed


62

Unit fm: Process Flow

d


fm


Input: SDR File(s)(MiRS Internal)

Output: FMSDR File (MiRS Internal)

fm unit script


Log File

Return value to PGM


Process Status File

npp_scs.bash

63

Unit fm Test Items


npp_scs.bash This is a driver script that: (1) Reads in the PCF, parses its contents, and passes necessary information to fm.bash. (2) Calls fm script(3) Writes out log files, the PSF file, and returns error status to the NDE DHS.


fm.bash Performs local file management , staging, resources file generation (e.g.f95 namelist) and error handling forfm.


fm For a single ATMS granule, it must read in all necessary files, perform footprint matching or averaging and write out to FMSDR file (MiRS internal format).

1 ATMS SDR_* file1 NEDT file (befFm)Namelist file

1 MiRS FMSDR_* file

64

Unit fm –Input/Output Test Data Table


1 Input npp_atms_fm_2011-01-15.in F95 Namelist file

2 Input SDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5

MiRS internal format file to containing SDR data

3 Output npp_atms_nedt_2011_01_15_aftFm.dat NEDT File

4 Output FMSDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5

MiRS internal format file containing FMSDR data

65

Unit fm –Test Risks

· Test Risk #1: Proxy data reliability (identified in CDR)· Test Risk #2: Optimal footprint matching codes not

available from IPO for testing within MiRS framework.

66

Unit chopp: Purpose and Function

· The chopp step allows for the simultaneous retrieval of multiple sub-orbits or sub-granules derived from the input file

· Chopping divides the orbit or granule FMSDR files into X number of files, where X is defined in the PCF

· Input: FMSDR files produced by fm step· Output: Chopped FMSDRs· It reads in a namelist which specifies operating parameters (passed

from the PCF by npp_scs.bash)· Note: This step is optional, and may be configured to optimize the

efficiency of the processing environment and take advantage of unused resources.

67

Unit chopp: Process Flow

d


chopp


Input: FMSDR File(s)(MiRS Internal)

Output: Chopped FMSDR Files (MiRS Internal)

chopp unit script


Log File

Return value to PGM


Process Status File

npp_scs.bash

68

Unit chopp Test Items


npp_scs.bash This is a driver script that: (1) Reads in the PCF, parses its contents, and passes necessary information to chopp.bash. (2) Calls chopp script(3) Writes out log files, the PSF file, and returns error status to the NDE DHS.


chopp.bash Performs local file management , staging, resources file generation (e.g.f95

namelist) and error handling for chopp.


chopp For a single ATMS granule, it must read in all necessary files in order to chop the FMSDR into sub-granules (MiRS internal format).

1 ATMS FMSDR_* fileNamelist file

X MiRS CHP_FMSDR_*_00X file

69

Unit chopp –Input/Output Test Data Table


1 Input npp_atms_chopp_2011-01-15.in F95 Namelist file

2 Input FMSDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5

MiRS internal format file to containing SDR data

3 OutputCHP_FMSDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.

h5_00X

MiRS internal format files containing FMSDR data

70

Unit chopp –Test Risks

· None Identified

71

Unit applyRegress: Purpose and Function

· First guess generation· Needed to start 1dvar minimization (iterative)· Estimate of the geophysical state vector (T,q, Tskin, Emis, etc.)

needed at each fmsdr location· applyRegress: Regression (using real observed TBs based on

off-line training)· Input: FMSDR (or chopped FMSDR) files, tuning files, covariance

matrix files, bias correction files, modeling error file, topography file· Output file format: Same as EDR Scene files (internal MiRS)· It reads in a namelist which specifies operating parameters (passed


72

Unit applyRegress: Process Flow

d


applyRegress


Input: • FMSDR File• Regression Coefficients (6

params * 4 sfc types)• Topography• Atm and Sfc bg cov• Tuning file• Rad bias file(MiRS Internal)

Output: REGR File (MiRS Internal EDR)

applyRegress unit script


Log File

Return value to PGM


Process Status File

npp_scs.bash

73

Unit applyRegress Test Items


npp_scs.bash This is a driver script that: (1) Reads in the PCF, parses its contents, and passes necessary information to applyRegress.bash. (2) Calls applyRegress script(3) Writes out log files, the PSF file, and returns error status to the NDE DHS.


applyRegress.bash Performs local file management , staging, resources file generation

(e.g.f95 namelist) and error handling forapplyRegress.


applyRegress For a single ATMS granule, it must read in all necessary files, perform the regression using FMSDR s(TBs) and write out to REGRESS file (MiRS EDR internal format).

1 ATMS FMSDR_*fileor X CHP_FMSDR_*00XCoefficient filesNamelist file

1 MiRS internal format REGRESS_*file or X REGRESS_*_00X files

74

Unit applyRegress –Input/Output Test Data Table


1 Input npp_atms_ApplyRegress_2011-01-15.in F95 Namelist file

2 Input Oc_regressCoeffs_npp_temp.dat* Coefficient Files

3 Input Topography.bin Topography file

4 Input TunParams_npp_atms.in, TunParams_npp_atms_2.inTuning Parameters

files

5 InputCovBkgMatrxTotAtm_all.dat,

CovBkgMatrxTotSfc_all_npp_atms.datCovariance Matrix files

w/background

6 InputbiasCorrec_npp_atms.dat, ModelErrFile_npp_atms.dat

Bias correction and modeling error

* Coefficient files for Ocean, Sea-ice, Land, and Snow surfaces, and for Temperature, Water Vapor,Skin Temperature, Emissivity, and Cloud Liquid Water

75

Unit applyRegress –Input/Output Test Data Table


7 Input

FMSDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5

OrCHP_FMSDR_SATMS_npp_d20110115_t0000311_e0000506_b000

15_c20110116000000020002_proxy_pop.h5_00X

MiRS internal format file(s) containing

FMSDR data

8 Output

REGR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5

OrCHP_REGR_SATMS_npp_d20110115_t0000311_e0000506_b00015

_c20110116000000020002_proxy_pop.h5_00X

MiRS internal format file(s) containing EDR

data

76

Unit applyRegress –Test Risks

· Test Risk #1: Proxy data reliability (identified in CDR)

77

Unit fmsdr2edr: Purpose and Function

· 1dvar: converts footprint matched SDRs to EDRs (MiRS core products)· All elements of geophysical state vector retrieved simultaneously: T(p),

q(p), Tskin, Emis, CW(p), RW(p), IW(p)· Variational solution optimal assuming error characteristics of geophysical

background, forward model (CRTM), sensor NEDT, and Jacobians (derivatives) obtained from forward model

· Tuning file can adjust characteristics of retrieval (nEOFs, channels used, etc.)

· QC flags assigned to each EDR location· Input: FMSDRs (MiRS RAD file format) produced by fm code (or

chopped FMSDRs), tuning files, NEDT file, first-guess, covariance matrix files, CRTM files, bias correction files, modeling error file, topography file

· Output: EDR Scene files (internal MiRS)· It reads in a namelist which specifies operating parameters (passed from

the PCF by npp_scs.bash)

78

Unit fmsdr2edr: Process Flow

d


1dvar


fmsdr2edr unit script


Log File

Return value to PGM NDE SAN

Output:• EDR File (Internal format)

Input :• Tuning Files (2)• CRTM Coef Files (3)• Rad Bias• Sfc Topography• BG Mean, Covariance,

Eigenvectors (2)• Fwd Model Error• NEDT• FMSDR(MiRS Internal)

Static

Dynamic

(NetCDF4)


Process Status File

npp_scs.bash

convertMirs2nc

79

Unit fmsdr2edr Test Items


npp_scs.bash This is a driver script that: (1) Reads in the PCF, parses its contents, and passes necessary information to fmsdr2edr.bash. (2) Calls fmsdr2edr script(3) Writes out log files, the PSF file, and returns error status to the NDE DHS.


fmsdr2edr.bash Performs local file management , staging, resources file generation (e.g.f95

namelist) and error handling for 1dvar.


1dvar For a single ATMS granule, it must read in all necessary files (dynamic and static), perform the 1dvar retrieval of core products and write out to EDR file (MiRS internal format).

1 ATMS FMSDR_* file(or X chopped FMSDRs)Namelist fileNEDT file (aftFm)Tuning FilesCovariance MatricesCRTM filesTopography fileBias Corr./Modeling ErrorFirst-guess

1 MiRS EDR_* file or X chopped EDR_*_00X

80

Unit fmsdr2edr –Input/Output Test Data Table

(1/2)

Test Data Item

Type Filename Description

1 Input npp_atms_CntrlConfig_1dvar_2011-01-15.in F95 Namelist file

2 InputbiasCorrec_npp_atms.dat, ModelErrFile_npp_atms.dat

Bias correction and modeling error

3 Inputnpp_atms_SpcCoeff.dat, npp_atms_TauCoeff.dat,

mw_cloud_opt.datCRTM files

4 Input npp_atms_nedt_2011_01_15_aftFm.dat NEDT File

5 Input topography.bin Topography file

6 Input TunParams_npp_atms.in, TunParams_npp_atms_2.in Tuning Parameters files

7 Input

REGR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5.LR.ORB

OrCHP_

REGR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5.LR.00X

First-guess file(s)

81

Unit fmsdr2edr –Input/Output Test Data Table

(2/2)

Test Data Item


8 InputCovBkgMatrxTotAtm_all.dat,

CovBkgMatrxTotSfc_all_npp_atms.datCovariance Matrix

files

9 Input

FMSDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5

OrCHP_FMSDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c2

0110116000000020002_proxy_pop.h5_00X

MiRS internal format file(s) containing

FMSDR data

10 Output

EDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5.LR.ORB

OrCHP_EDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c201

10116000000020002_proxy_pop.h5.LR.00X

MiRS internal format file(s) containing EDR

data

82

Unit fmsdr2edr –Test Risks

· Test Risk #1: Proxy data reliability (identified in CDR)· Test Risk #2: Radiometric bias generation using proxy data

will require regeneration when sufficient real data become available (identified in CDR)

83

Unit mergeEDR: Purpose and Function

· The mergeEDR step combines all sub-orbit or sub-granule retrieval (EDR) files back into 1 file, if chopping was applied.

· X files are merged together, where X is defined in the PCF· Input: EDR files produced by fmsdr2edr step· Output: 1 Merged EDR file· It reads in a namelist which specifies operating parameters (passed

from the PCF by npp_scs.bash)· Note: This step is optional, and may be configured to optimize the

efficiency of the processing environment and take advantage of unused resources. Must be enabled if the chopping step is applied.

84

Unit mergeEDR: Process Flow

d


mergeEDR


Input: EDR Files(MiRS Internal)

Output: Merged EDR File (MiRS Internal)

mergeEDR unit script


Log File

Return value to PGM


Process Status File

npp_scs.bash

85

Unit mergeEDR Test Items


npp_scs.bash This is a driver script that: (1) Reads in the PCF, parses its contents, and passes necessary information to mergeEDR.bash. (2) Calls mergeEDR script(3) Writes out log files, the PSF file, and returns error status to the NDE DHS.


mergeEDR.bash Performs local file management , staging, resources file generation (e.g.f95

namelist) and error handling for merge.


mergeEDR For a single ATMS granule, it must read in all chopped EDR files in order to merge the EDRs into 1 EDR file (MiRS internal format).

X ATMS EDR_*_00X filesNamelist file

1 MiRS EDR_* file

86

Unit mergeEDR –Input/Output Test Data Table


1 Input npp_atms_chopp_2011-01-15.in F95 Namelist file

2 Input EDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5_00X

MiRS internal format files containing EDR data

3 Output EDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5

MiRS internal format file containing EDR data

87

Unit mergeEDR –Test Risks

· None Identified

88

Unit vipp: Purpose and Function

· Post-processing converts EDRs to derived environmental parameters (DEPs)· Two types of processing:

» 1. Vertical Integration: – q(p) → TPW, CL(p) → CLW, RW(p) → RWP, IW(p) → IWP

» 2. Post-processing: – Rain Rate: Regression RR=f(CLW,RWP,IWP)– Sea Ice Concentration (FY,MY): Look-up in emissivity catalogs– Snow Water and Grain Size: Minimized cost function (emissivity LUT

based on CRTM physical snow emissivity model, uncertainties in SW and GS).

· Input: EDR Scene files produced by fmsdr2edr or mergeEDR (internal MiRS)· Output: DEP (Derived product) file (internal MiRS)· It reads a PCF, writes a PSF, and returns error handling information to the calling

NDE Data Handling System (via npp_scs.bash).

89

Unit vipp: Process Flow

d


vipp


vipp unit script


Log File

Return value to PGM

Output Files:• DEP File (Internal format)

Input Files:• Sea Ice Emis Catalog• Snow Emis Catalog

• EDR File(s)(MiRS Internal)

Static

Dynamic

(NetCDF4)


Process Status File

npp_scs.bash

NDE SAN

convertMirs2nc

90

Unit vipp Test Items


npp_scs.bash This is a driver script that: (1) Reads in the PCF, parses its contents, and passes necessary information to vipp.bash. (2) Calls vipp script(3) Writes out log files, the PSF file, and returns error status to the NDE DHS.


vipp.bash Performs local file management , staging, resources file generation (e.g.f95 namelist) and error handling forvipp.


vipp For a single ATMS granule, it must read in all necessary files (dynamic and static), perform the perform the vertical integration and postprocessing of core products and write out to DEP file (MiRS internal format and netCDF4).

1 ATMS EDR_* file 1 MiRS DEP_* file

91

Unit vipp –Input/Output Test Data Table

Test Data Item


1 Input npp_atms_CntrlConfig_1dvar_2011-01-15.in F95 Namelist file

2 Input EDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5.LR.ORB

MiRS internal format file containing EDR

data

3 InputSeaIceEmissCatalog_npp_atms.dat, SnowEmissCatalog_npp_atms.dat

Snow and sea-ice Emissivity Lookup

Tables

4 Output DEP_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5.LR.ORB

MiRS internal format file containing DEP

data

92

Unit vipp –Test Risks

· Test Risk #1: Proxy data reliability (identified in CDR)· Test Risk #2: Catalog files based on proxy data will require

real data for regeneration (identified in CDR)

93

Unit convertMirs2nc: Purpose and Function

· Converts EDR and DEP files from MiRS internal binary to netCDF4 format

· Input: EDR from fmsdr2edr output· Input: DEP from vipp output· Output: SND and IMG product files in netCDF4 format

94

Unit convertMirs2nc: Process Flow


mirs2nc


Input: EDR or DEP File(s)(MiRS Internal)

Output: EDR (SND) or DEP (IMG) Files (netCDF4)

convertMirs2nc unit script


Log File

Return value to PGM


Process Status File

npp_scs.bash

NDE SAN

IMG

/SN

D.n

c

95

Unit convertMirs2nc –Input/Output Test Data Table

Test Data Item


1 Input EDR_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5.LR.ORB

MiRS internal format file containing EDR

data

2 Input DEP_SATMS_npp_d20110115_t0000311_e0000506_b00015_c20110116000000020002_proxy_pop.h5.LR.ORB

MiRS internal format file containing DEP

data

3 Output MIRS-SND_v7_NPP_s201101150311_e201101150506_c201101160215.ncSND products in

netCDF4 file

4 Output MIRS-IMG_v7_NPP_s201101150311_e201101150506_c201101160215.ncIMG products in

netCDF4 file

96

Unit convertMirs2nc –Test Risks

· None Identified

97

MiRS Design Assumptions Summary

· The MiRS design assumes the NDE PGM will run each instance of a MiRS unit in a single working directory that is unique to that run.

· The PCF will be made available to each unit driver script locally in the working directory at run time.

· The MiRS processing units (rdr2tdr, tdr2sdr, fm, chopp, applyRegress, fmsdr2edr, mergeEDR, vipp, convertMirs2nc) are designed to process the data for a single ATMS granule set. » The NDE PGM will therefore be able to run multiple instances of the same

MiRS bash script to process several granule sets at a time.

· The NDE DHS will perform all file management for MiRS outside of the working directory where MiRS runs.

· NDE will perform all tailoring and/or reformatting

98

MiRS DAP Software Architecture Summary

· All MiRS products will be generated by 9 software units» rdr2tdr» tdr2sdr» fm» chopp» applyRegress» fmsdr2edr» mergeEdr» vipp» convertMirs2nc

· These units and the layers of data flow will be documented in the MiRS Documentation (System Maintenance Manual, External Users Manual)

· These units will be:» Delivered to NDE in the form of a DAP» Integrated into the NDE DHS by the NDE team» NDE will run MiRS in production mode at NSOF

99

MiRS QC Overview

· Overall Objective: Comprehensive real-time monitoring at various stages of MIRS data processing; selective flagging and notification of anomalies via e-mail

· Run daily in STAR environment; Applied to N-18, N-19, Metop-A, NPP/ATMS, DMSP · Tier-1 (“Autonomous”) QC: Easily implemented in operations

» NEDT monitoring (time series, e-mail alerts) → QC DAP

» QC flag monitoring (time series, e-mail alerts) → QC DAP

» Convergence monitoring (1dVAR) (time series, e-mail alerts) → QC DAP· Tier-2 (“Intermediate”) QC:

» Radiometric performance (time series, e-mail alerts for dynamic biases) → QC DAP· Tier-3 (“Science”) QC: Appropriate for R&D environment

» Geophysical performance (comparisons with GDAS, ECMWF, GFS)» Comparisons w/ RAOBS, field experiments, other sensors/algorithms

· All monitoring results displayed on STAR MIRS website: time series, maps, statistics (+ other assessment plots)

· Off-line tools to assess system performance sensitivity to channel selection, EOFs, noise levels, etc.

Not part of QC DAP

100

MiRS QC: Implementation Status at

NDE

· MiRS QC DAP Prepared and Delivered (Dec 2010) to NDE» Contains all codes and files necessary to run Tier-1 and 2 QC» Requires f95 compiler and IDL v6 license» Currently testing

· What are the NDE requirements for product registration? QC processing assumes that:» Core and derived products have been generated (MiRS binary format

retained)» NWP (GFS) fields are available (for Tier-2 radiometric QC only)» If time series desired, then QC,NEDT,Radiometric bias files must be

retained

101

QC Layer-2 Units: Summary

· Each step in the MiRS Tier-1 and Tier-2 QC processing sequence is a stand-alone bash script and one or more corresponding Fortran 95 executables, or IDL procedures (v6) driven by namelist files

Code Unit Purpose Tier Comment

qcNedt Parses NEDT files (generated by rdr2tdr) and generates an e-mail if thresholds exceeded

1 Autonomous

qcRetrieval Parses QC files (generated by fmsdr2edr) and generates an e-mail if thresholds of QC or conv rate exceeded

1 Autonomous

prepNwp Convert NWP files from GRIB to binary 2 NWP required

nwp Interpolates NWP fields to FMSDR location and time 2 NWP required

fwd Runs forward model on NWP output (generated by nwp) to simulate TBs at each FMSDR location

2 NWP required

biasGen Compares FWD to FMSDR TBs and generates bias files 2 NWP required

qcRadiometricBias Parses bias files (generated by biasGen) and generates an e-mail if thresholds exceeded

2 NWP required

biasMonitor Produces time series of radiometric bias from bias files (generated by biasGen)

2 NWP+product retention

dataQualityMonitor Produces time series of nedt, qc, convergence from NEDT and QC files (generated by rdr2tdr and fmsdr2edr)

1 Product retention required

102

MiRS Tier-1 and 2 QC System-Layer Process Flow: NDE

Environment

dLocal Processing Directories

qcNedt (T1)

Unless specified, all formats are MiRS internal

qcRetrieval (T1)


DDS

NEDT (rdr2tdr)

QC (fmsdr2edr)

NB: Time series require archiving biasCorr, NEDT and QC files

npp_qc.bash


E-Mail if threshold exceeded

E-Mail if thresholds exceeded

T1

T2

fwd (T2)

biasGen (T2)

nwp (T2)

qcRadiometricBias (T2)

FMSDR`(fm)

NWP

biasMonitor (T2)

dataQualityMonitor (T1)

GFS (deGRIBed)

NWP

FWD

FMSDR (fmsdr2edr)+FWD

biasCorr

biasCorr

biasCorrTime series radiometric bias (*.png)

NEDT (rdr2tdr)+QC (fmsdr2edr)Time series qc, conv, nedt (*.png)

E-Mail if threshold exceeded

prepNwp (T2)GFS (GRIB)

GFS (deGRIBed)

103

Unit qcNedt: Process Flow

d


qcNedt.pro

Input Files Specified in IDL Namelist

Input: NEDT File(MiRS Internal)

qcNedt unit script


Log File

Return value to PGM

Yes


Process Status File

npp_qc.bash

E-Mail alert !Threshold exceeded?

104

Unit prepNWP: Process Flow

d


prepNWP

Namelist

Input: • 1 NWP file (GFS) Grib format

Output: 2 NWP Files (MiRS Internal Binary)Valid at 2 consecutive forecast times (0, 6, 12, 18 UTC)

prepNWP unit script


Log File

Return value to PGM


Process Status File

npp_qc.bash

105

Unit qcRetrieval: Process Flow

d


qcRetrieval.pro


Input: QC File(MiRS Internal)

qcRetrieval unit script


Log File

Return value to PGM

Yes


Process Status File

npp_qc.bash

Thresholds exceeded?

E-Mail alert !

106

Unit qcRadiometricBias: Process Flow

d


checkBias.pro


Input: biasCorr File(MiRS Internal)

qcRadiometricBias unit script


Log File

Return value to PGM

Yes


Process Status File

npp_qc.bash

E-Mail alert !Thresholds exceeded?

107

Unit nwp: Process Flow

d


colocNWPwRad


Input: • FMSDR file• 2 NWP files (4 valid times: 0, 6,12,18 UTC)

Output: NWP File (MiRS Internal EDR)

nwp unit script


Log File

Return value to PGM


Process Status File

npp_qc.bash

108

Unit fwd: Process Flow

d


fwd


fwd unit script


Log File

Return value to PGM

Output:• RAD File (Internal)

Input :• Tuning Files (2)• Instr Config• CRTM Coef Files (3)• NEDT• NWP(MiRS Internal)

Static

Dynamic


Process Status File

npp_qc.bash

109

Unit biasGen: Process Flow

d


Calib_generic_rad.pro


biasGen unit script


Log File

Return value to PGM

Output:• biasCorr File (Internal)

Input :• NWP File• FMSDR• FWD(MiRS Internal)

Dynamic


Process Status File

npp_qc.bash

110

Unit dataQualityMonitor: Process Flow


MonitorQC_Mirs.pro

Inputs Specified in IDL Namelists

dataQualityMonitor unit script


Log File

Return value to PGM

Outputs:• Time Series Files

(2) (png)

Input :• Dir name containing QC

files• List of NEDT files

MonitorNEDT.pro


Process Status File

npp_qc.bash

111

Unit biasMonitor: Process Flow


biasMonitor.pro

Inputs Specified in IDL Namelists

biasMonitor unit script


Log File

Return value to PGM

Outputs:• Time Series File (png)

Input :• Dir name containing

biasCorr filesLocal Processing Directories

Process Status File

npp_qc.bash

112

MiRS QC Test Readiness: Summary/Risks

· MiRS QC DAP Prepared and Delivered (Dec 2010) to NDE» Contains all codes and files necessary to run Tier-1 and 2 QC» Requires f95 compiler and IDL v6 license» Currently testing

· Risks: Availability of GFS for Tier-2 testing

113

· Introduction· CDR Report· Software Architecture· Unit Test Readiness

» Test Readiness of Individual Units: MiRS/NDE

· RISKS/ACTIONS· Summary and Conclusions· Discussion

114

Section 5 –Risks and Actions

Presented by

K. Garrett

115

Open CDR Risks and Actions

· Risk #1: Unclear if resources available to run MiRS in high-resolution mode» Risk Mitigation: Already incorporated heritage (MSPPS) algorithms into MiRS

and can be optionally turned on/off to produce high resolution rain rate within MiRS on heritage sensors (at OSPO); Workload scenarios at NDE can determine resource adequacy for MiRS NPP/ATMS.

» Status: Open

· Risk #2: NPP launch date may be postponed beyond Oct 2011; Budget planning assumes minimal schedule slippage.» Risk Mitigation: Project lead to monitor schedule vis-à-vis MiRS project plan for

FY10 and FY11 and, if needed, request additional funding in consultation with MiRS Oversight Board via Annual Review for Satellite Product Development

» Status: Open

116


· Risk #3: Footprint Matching (FM) codes not integrated» Risk Mitigation: (1) Use resampled data from IDPS; (2) Expected that MiRS will

process highest resolution data available and can do simple FOV averaging.» Status: Open

· Risk #4: Proxy data fidelity; Preclassifier and post-processing algorithms partially dependent on calibration with proxy (or real) data.» Risk Mitigation: MiRS performing well with current nominal values and

parameters. Calibration and tuning to be performed immediately post-launch.» Status: Open

117


· Risk #5: SciTech Renewal» Risk Mitigation: MiRS IPT expected to remain in place during any transition.

Project will have advance notice and can reallocate resources as necessary to maintain continuity.

» Status: Open

118

MiRS NPP/ATMS Risk Summary(at CDR)

Impact *

5 1

4

3 2

2

1 3,4,5

1 2 3 4 5

Likelihood

Risk No.

(Rank)

Risk Category Risk Risk

LevelRisk Index

1External Risks

If NPP launch is delayed further funds may not be available for post-launch cal/val efforts.

Moderate 15.00

2Project Management Risks

Running MiRS in high-resolution mode requires additional computing resources which may slow data processing time. Low 9.00


If footprint matching codes not available or SDRs not footprint-matched MiRS processing of non-FM SDRs would be sub-optimal. Very Low 3.00


If real NPP/ATMS data has significantly different characteristics than proxy data, additional efforts will be needed post-launch to update MiRS processing algorithms. Very Low 3.00

5External Risks

If SciTech renewal is not awarded to incumbent there may be an impact on MiRS IPT continuity.

Very Low 3.00

Risk Level Likelihood

1 Very Low P < 10%

2 Low 10% ≤ P < 30%

3 Moderate 30% ≤ P < 70%

4 High 70% ≤ P < 90%

5 Very High P ≥ 90%

Impact Level* Schedule Impact (months)

Cost Impact($ x 10K)

1 Very Low < 1 ~ 0

2 Low 1 - 3 1 - 5

3 Moderate 3 - 6 5 - 10

4 High 6 - 12 10 – 50

5 Very High ≥ 12 > 50

*Risk Impact may be on schedule, cost, and/or science product quality and availability. Approximate impact levels for schedule/cost only are in table below. Not all risks will have schedule/cost impacts.

119

CDR Action Items

Action Item


1 MiRS Operations Manual in OSPO A. Sharma STAR Closed


A.Sharma NDE/STAR Closed


A.SharmaSTAR Closed







T. Schott, L. Zhao

OSPO Open

120

· Risk 1: Simulated data and format may not represent actual NPP/ATMS data (also a CDR risk)

· Risk Assessment (probability): Moderate· Impact: Low· Mitigation: Testing will utilize both STAR generated

proxy data and GRAVITE generated proxy data (official) to ensure consistency. IDPS generated sample data will also be used in testing (current build, and Platinum72 when available), as those should be the true data formats (CDFCB).

· Status: Open

TRR Risk

121

· Risk 2: Coefficient files require real data for training and algorithm tuning (bias, covariances, first-guess, catalogs) (also a CDR risk)

· Risk Assessment (probability): Low· Impact: Low· Mitigation: Post-launch efforts will focus on cal/val;

software and correction coefficients based on proxy data already in place, which is adequate for testing purposes.

· Status: Open

TRR Risk

122

· Risk 3: Available resources in STAR and NDE environments are not consistent (IDL versions, compiler versions, etc.)

· Risk Assessment (probability): Medium· Impact: Low· Mitigation: Identify and address issues during the

code unit testing. Efforts have already been taken to ensure MiRS software is compatible with a variety of operating environments (Linux, AIX, multiple compilers).

· Status: Open

TRR Risk

123

· Risk 4: QC DAP lacks ability to monitor radiometric bias for upper-atmospheric ATMS temperature sounding channels using GFS at input. GFS model profiles only extend to 10 mb, below the sensitivity of ATMS channels 14 and 15 (57 GHz).

· Risk Assessment (probability): High· Impact: Low· Mitigation: Broader instrument issues will be captured

through NEDT, QC, or radiometric monitoring of other ATMS channels. STAR will also perform radiometric monitoring using other data sources which extend to the upper atmosphere where ATMS channels 14 and 15 are sensitive (e.g. ECMWF analysis).

· Status: Open

TRR Risk

124

Review Items Summary

· 5 Risk Items from the CDR were identified, 5 of which are still open

· 7 Action Items from the CDR were identified, 5 were closed, 2 remain open

· 4 TRR Risk Items have been identified , 4 remain open; NB: 2 are already contained in CDR risks

125



· Risks/Actions· SUMMARY AND CONCLUSIONS· Discussion

126

Section 6Summary and

Conclusions

Presented by

K. Garrett

127

Summary and Conclusions

· Following have been reviewed:» CDR Report» Software Architecture » Unit Test Plans» Risks and Actions

128

Next Steps

· Prepare TRR Report (including action items)· Reviews:

» Code Unit Test Review in late March, 2011 [9.1.4 in the Level 1 Reqs Doc]

» System Readiness Review in January, 2012 [9.1.5]

· Address action items identified in TRR

129



· Risks/Actions· Summary and Conclusions· DISCUSSION

130

Open Discussion

· The review is now open for discussion

131

Back-up Slides

132

MiRS Official Products

Observational Parameter Imagery Product Sounding Product PackageAtmospheric Temperature profile xAtmospheric Water Vapor profile xTotal Precipitable Water XLand Surface Temperature XSurface Emissivity Spectrum xSea-ice Concentration xSnow Cover Extent xSnow-Water Equivalent xIntegrated Cloud Liquid Water xIntegrated Ice Water Path xIntegrated Rain Water Path xRainfall Rate x

133

Operational Products

Product Proc Format User Coverage Output TypeIMG/SND NDE netCDF4 CLASS/

OSPOGranule Direct

RR/TPW/TB NDE netCDF4 OSPO Orbital Tailored

RR/TPW NDE netCDF4 OSPO AOI* Tailored

TPW/RR/TB Composite

OSPO McIDAS SAB Orbital Tailored

TPW/RR Composite

OSPO AWIPS NWS AOI* Tailored

Blended TPW/RR

OSPO AWIPS/McIDAS

NWS Global/Orbital Tailored

IMG/SND ? HDF-EOS CLASS Granule/Orbital Tailored

*AOIs : HI, AK, Puerto Rico, Super National, Tropical

NDE or OSPO

134

MiRS: Anticipated ATMS Hi-Res Processing

Platform/Sensor Machine Single CPU (core)

Number of CPUs (cores)

Time: 1 orbit(multi CPU)

Number of Profs:1 orbit

Time: 1 granule(single CPU)

Time: 1 granule (multi CPU)

N18/AMSU-MHS orbit272L 2.67 GHz 24 11.0 min 232470 N/A N/A

NPP/ATMS orbit272L 2.67 GHz 24 10.4 min* 224640* 1.28 min* 3.23 sec*

*Based on: • 1 granule = 1152 profiles• 1 orbit = 195 granules• MiRS Outputs: 22 MB• 0.0028 sec/profile

Benchmarks based on current processing capacity

Note: MiRS can distribute processing over multiple CPUs on same node (chopping or not)

Actual Estimated

135

MiRS: Current and Planned Processing Environment in

STAR

Machine Single CPU (core)

Number of CPUs (cores)

Memory (GB)

Disk Storage (GB)

Jobs Running

orbit227L 3.16 GHz 8 16 800 • Daily Global Low-Res: N18,N19,Metop,F16,F18,NPP (30 min/satellite)• Daily Regional High-Res*: N18,Metop,F16,AMSRE

orbit232L 2.83 GHz 4 8 1200 • Development

orbit227L 3.16 GHz 8 16 800 • Daily Global Low-Res: N18,N19,Metop• Daily Regional High-Res: AMSRE

orbit232L 2.83 GHz 4 8 1200 • Development

orbit273L 2.8 GHz 16 32 3000 • Daily Global Low-Res: F16,F18,NPP

orbit272L 2.67 GHz 24 128 4500 • Daily Global High-Res: N18,N19,Metop,F16,F18,NPP

orbit267L 3.46 GHz 16 40 3000 • GPM, development

3 additional servers purchased to enhance MiRS processing capacity within STAR

Current Planned

* Current high-res 15 km daily processing for 4 ROIs/Satellites» N18: Gulf of Mexico» MetopA: U.S. West Coast» F16: China Southeast Sea» AMSR-E: CONUS

1 microwave integrated retrieval system for npoess preparatory project: mirs npp/atms integration...

Documents

garrett slide

nde test readiness review

test readiness review

mirs dap test readinessk

mirs nppatms integration

mirs risksactions summary

review agenda

trr slide