Ocean Color Data Processing System:Status

2 November 2006

NASA’s Goal

To make available the highest quality ocean color (and sst) data to the broadest user community in the most timely and efficient manner possible.

•

• Expertise:

Requirements for Success

• Expertise: internal

- highly integrated project structure with all elements co-located - continuous

communication.

Requirements for Success

• Expertise: internal and external

- highly integrated project structure with all elements co-located - continuous

communication.

- Strong links with mission-specific expertise (MCST) and research community (algorithms, validation data, new products)

Requirements for Success

• Expertise• Infrastructure

- flexible data processing system that constantly upgrades procedures, technologies and equipment

Requirements for Success

• Expertise• Infrastructure

- flexible data processing system that constantly upgrades procedures, technologies and equipment

SCIENCE drives the system rather than the SYSTEM driving the science

Requirements for Success

Processing Cluster34 processing nodes 1.5 TB

9 storage nodes 17.1TB

Ingest Servers2 SeaSpace groundstations 78 GB9 storage nodes (1xx,2xx) 16.6 TB

Distribution Servers (ftp)6 storage nodes (2xx) 25.5 TB

1 server 250GB Distribution Servers (web)Production: large server 3 TB 3 processing nodes (2xx, 3xx)

46 storage nodes (2xx, 25x) 303 TBDevelopment: 2 superstore nodes 10 TB

Testing Cluster8 test nodes (1xx) 480GB

7 processing nodes (2xx) 365GB2 storage nodes 2.4 TB

Network Support Systems

Database ServerProduction: 1 large server 876 GB

Development: 1 superstore node 5.8TB

Backup Servers1 large server 876 GB+2 TB tape

13 storage nodes 15.6 TB

Extreme NetworksBlack Diamond 6816

Gigabit Ethernet switch280 ports

User Desktops

Cal/Val & QC Systems

Mission Operations Systems

Ocean Color Processing System Overview

Development Servers1 processing node

2 storage nodes 2.4 TB

Ingest queuetable

L0Ingest

process

L1A-L1B(MOD_PR02)

L1B-L2(MSl12)

MODISA L2table

Archive - DistribServer

L3BIN(l2bin)

MODISA L3-bintable

Archive - DistribServer

MODISA L3-maptable

Archive - DistribServer

Operational MODIS-Aqua Data Flow (NRTPE)Terra and Aqua

Software process

Hardware system

Database table

Ancillary input data

SensorCAL

Sensorattribs

Atmcorr

BrowserCGI /httpd

Ocean ColorWeb Server

UserCommunity

MySQLDB

Ozone

L3MAP(smigen)

MET

MET, Ozone, and OISST data

are dynamically selected for each L1A granule

Product meta data are

populated from production DB

October 30, 2006

OISST

Ocean ColorIngest

MODISA L0table

Archive - DistribServer

ATT/EPHIngest

process MODISA attephtable

Archive - DistribServer

L0-L1A(MOD_PR01)

Geo-Location(MOD_PR03)

MODISA L1table

Archive - DistribServer

Full-resolution day- and nighttime via SEN (~60 GB per

day/mission)

NOAA RealtimeSystem

NASA EDOSSystem

MissingL0

process MODAPS L0Archive

MODAPS L0Archive

Granules not received from

NOAA realtime flow

• Expertise• Infrastructure• Data

- Most efficient and ultimately most cost effective when source data is available online for all needs including processing, reprocessing, evaluation testing, distribution.

Requirements for Success

CZCS: Volume = 582.72 GB1978-1986 Files = 15499OCTS: Volume = 137.62 GB 1996-1997 Files = 7641SeaWiFS GAC: Volume = 935.48 GB 1997-present Files = 51077SeaWiFS MLAC: Volume = 6.70 TB Files =

39257SeaWiFS LAC: Volume = 358.90 GB

Files = 42231

Current Source Data Holdings

MODIS/AQUA Level 1A Subset 7/02 - now

Volume = 19 TB Files = 449823MODIS/AQUA Level-0 (approx 20TB / year)

Volume = 58 TB Files = 332765

MODIS/TERRA Level 1A Subset 1/05 - now Volume = 8 TB Files = 183623

MODIS/ TERRA Level-0 - since August 2006 Volume = 5 TB Files = 27374

Current Source Data Holdings

• Expertise• Infrastructure• Data• Communication

Requirements for Success

OceanColor Web

oceancolor.gsfc.nasa.go

v

Consolidated data access, information,

services andcommunityfeedback

• Expertise• Infrastructure• Data• Communication• Intuitive and efficient data

distribution

Requirements for Success

• Expertise: internal and external• Infrastructure• Data• Communication• Intuitive and Efficient data

distribution • Community access to data

processing/analysis tools

Requirements for Success

1- Extensive user support withover 500 sites, active online Forums. > 3000 posts

2- SeaDAS ported to the Macintosh OS X includingnew Intel architecture. Alsoruns on Linux, Sun, SGI

3- Redesigned GUI and website are now much more user-friendly

4- New simple online installer and four SeaDAS ftp mirrors (Australia, Brazil, Japan, UK)

5- SeaDAS-lite option for display/analysis only (~50MB download)

6- Modis Direct BroadcastAnd High Resolution processing module.

7- User training workshops