ooi ci overview and status

OOI CI R2 Life Cycle Objectives Review

Aug 30 - Sep 1 2011

Ocean Observatories Initiative

1

OOI CI Overview and Status

Matthew Arrott

DMAC-STWashington DC, Jan 18-19, 2012


Aug 30 - Sep 1 20112

Agenda

• Review of the OOI CI Construction Objectives

• Introduction of Release 2 Functional Objectives

• Computational Infrastructure Objectives

• External Observatory Integration

• What will OOI-CI mean to me


Aug 30 - Sep 1 20113

OOI CI Construction Objectives


Aug 30 - Sep 1 20114

OOI Project Scope

Building/deploying science infrastructure for understanding a changing water planet

• Multi Dimensional Engineering Concern• Social• Organizational• Functional• Procedural• Technical

Cyberinfrastructure fulfills the integrative mechanism across these dimensions


Aug 30 - Sep 1 20115

The Hubble Telescope for the Oceans


Aug 30 - Sep 1 20116

The Hubble Telescope for the Oceans

OOI CI Release 2 KickoffMay 2, 20117

Core Science & Education Capabilities

• Interactive Ocean Observing (R3 - R4)

• Interactive Ocean Modeling & Data Assimilation (R3 - R4)

•Discipline-Driven Semantic Organization of Data (R3 - R4)

•Automated Data Product Generation (R1 - R3)

• Interactive Instrument Network (R1 - R3)

• Integrated Observatory Management (R1 - R2)

•User-Driven Integration of Resource (R1 - R3)

ObservatoryActivity Model

OOI CI R2 Life Cycle Objectives ReviewAug 30 - Sep 1 2011 8

OOI Integrated Observatory Product•The fully operational research observatory will meet the following •Goals:

• Continuous observations at time scales of seconds to decades• Spatial measurements from millimeters to kilometers• Sustained operation during storms and other severe conditions• Real-time or near-real-time data as appropriate• Two-way transmission of data and remote instrument control• Power delivery to sensors between the sea surface and the seafloor• Standard plug-n-play sensor interface protocol• Autonomous underwater vehicle dock for data download/battery recharge• Access to deployment and maintenance vehicles that satisfy the needs of

specific observatories• Facilities for instrument maintenance and calibration• A management system that makes data publicly available• An effective education and outreach program


Aug 30 - Sep 1 20119

Extending the Traditional Data Access Model

10

Multi-Scale Research Specific Ocean Laboratories

OOI Review Year 2May 16-20, 201111

Release Schedule

Inception Phase

Elaboration Phase

Construction Phase

Transition Phase


Release Schedule by Subsystem

Inception Phase

Elaboration Phase

Construction Phase

Transition Phase


CI Components Developed for R1

• Python Capability Container

• Java Capability Container Access Library

• Web UI Platform

• Exchange Messaging System

• Distributed State Infrastructure

• Resource Registry Framework

• Science Data Persistence and Transport Format

• Data Publish-Subscribe Framework

• Event Notification Framework

• Instrument Agent Framework

• Elastic Processing Unit

• Virtualized Cloud Management Tools


Capabilities of the Managed Instrument Network

•Operate Marine Observatories

•Operate Platforms and Instruments

•Manage Instrument Lifecycle

•Test and Troubleshoot Instruments

•Acquire Data and Generate Data Products

•Search Data

•Visualize Data

•Manage the Integrated Observatory Network


R1 Integrated Observatory Network Web UI

https://confluence.oceanobservatories.org/display/CIPUB/OOI+Review+2011+Demos




Aug 30 - Sep 1 201116

Release 2 Functional Objectives


Aug 30 - Sep 1 201117

Observatory Operations & Administration


• CI Release-1: Data Distribution Network

• Provide framework for discovery and interaction based on

data-intensive users’ needs

• CI Release-2: Managed Instrument Network

• Provide support for operations and maintenance of marine

networks based on RSN and CGSN needs

• CI Release-3: On Demand Measurement Processing

• Provide dynamic analysis and visualization tools based on

science communities’ user needs

• CI Release-4: Interactive Ocean Observatory

• Provide mission control and collaboration support based on

science communities’ needs

User Experience Incremental Development Plan

Scientific CollaborationScientific Collaboration

General Science General Science UsersUsers

ScienceScience ExpertsExperts

Resource Resource ProvidersProviders


UXUX

R2 Product Description: List of Use CasesUC.R2.01 Acquire External Data SourceUC.R2.02 Derive Data ProductUC.R2.03 Produce Real-Time QC DataUC.R2.04 Browse to Get Data ProductUC.R2.05 Register and Connect InstrumentUC.R2.06 Command InstrumentsUC.R2.07 Direct Instrument Access IIUC.R2.08 Manage Instrument LifecycleUC.R2.09 Activate Instrument DriverUC.R2.10 Manage Marine PlatformUC.R2.11 Operate Marine ObservatoryUC.R2.12 Deploy Agents On Remote PlatformUC.R2.13 Acquire Data From InstrumentUC.R2.14 Monitor an InstrumentUC.R2.15 Qualify Instrument InterfaceUC.R2.16 Install Instrument Automatically

UC.R2.17 Define Visualization MethodUC.R2.18 Visualize Data ProductUC.R2.19 Produce Matlab Visualization

UC.R2.20 Annotate ResourcesUC.R2.21 Transform Data in WorkflowUC.R2.22 Version ResourceUC.R2.23 Ingest Dataset SupplementUC.R2.24 Search for ResourceUC.R2.25 Advanced Resource SearchUC.R2.26 Navigate Resources and MetadataUC.R2.27 Manage Replicated ArchiveUC.R2.28 Manage Resource MetadataUC.R2.29 Integrate External Data Source

UC.R2.30 Define InteractionUC.R2.31 Define New ServiceUC.R2.32 Conduct NegotiationUC.R2.33 Enroll in an OrgUC.R2.34 Share an Org ResourceUC.R2.35 Share Affiliated Orgs' ResourcesUC.R2.36 Create an OrgUC.R2.37 Control Service InteractionsUC.R2.38 Define Resource Life CycleUC.R2.39 Manage ION UsersUC.R2.40 Monitor ION ResourcesUC.R2.41 Recover Failed ProcessUC.R2.42 Define Resource PolicyUC.R2.43 Operate Message Brokers

UC.R2.44 Put Services Anywhere EasilyUC.R2.45 Replicate Activated ServiceUC.R2.46 Operate Integrated SystemUC.R2.47 Deploy Versioned User ProcessUC.R2.48 Schedule User-Defined ProcessUC.R2.49 Deploy Distributed ProcessesUC.R2.50 Define Scaling PolicyUC.R2.51 Define Execution EngineUC.R2.52 Manage ION Processes

UC.R2.53 View Modeler-Submitted ProductsUC.R2.54 Access NEPTUNE CA Data

UC.R2.55 Manage Help TicketUC.R2.56 Monitor ION NetworkUC.R2.57 Configure Start Page

UC.R2.58 Display Arbitrary ResourceUC.R2.59 Generate New Screen

DMDM

S&AS&A

A&SA&S

COICOI

CEICEI

EOIEOIOpsOps


Aug 30 - Sep 1 201120

Managed Instrument Network


Sensor Set #1 Instrument AgentsClass # Family Instrument Class Long Name

ADCPS 16 Water Velocity Velocity_profile_600m

CTDGV 24 Conductivity, Temperature, Depth CTD_glider

CTDMO 135 Conductivity, Temperature, Depth CTD_mooring

CTDPF 30 Conductivity, Temperature, Depth CTD_profiler

DOSTA 61 Dissolved Oxygen oxygen_dissolved_stable

FDCHP 7 Direct Covariance Flux flux_direct_cov_HP

FLORD 35 Fluorometer Fluorometer_two_wavelength

METBK 18 Meteorology Meteorology_bulk

NUTNR 22 Nitrate nutrient_Nitrate

OPTAA 35 Optical Properties attenuation_absorption_optical

PCO2A 10 Partial Pressure CO2 pCO2_air-sea

PCO2W 25 Partial Pressure CO2 pCO2_water

PHSEN 33 Partial Pressure CO2 pH_stable

PREST 3 Seafloor Pressure pressure_SF_tidal

SPKIR 22 Optical Properties spectral_irradiance

VEL3D 32 Water Velocity Velocity_point_3D_turb

VELPT 21 Water Velocity Velocity_point

WAVSS 10 Surface Wave Spectra wave_spectra_surface

ZPLSG 4 Zooplankton Sonar plankton_ZP_sonar_global



BOTPT 3 Seafloor Pressure pressure_bottom_tilt

CAMDS 12 Cameras x camera_digital_still_strobe

CAMHD 1 Cameras x camera_digital_video_HD

HYDLF 5 Hydrophone Hydrophone_LF_passive

MASSP 2 Mass Spectrometer mass_spectrometer

OBSBB 3 Ocean Bottom Seismometer Seismometer_BB_triaxial_accel

OBSBK 2 Ocean Bottom Seismometer Seismometer_BB_triaxial_keck

OBSSP 8 Ocean Bottom Seismometer Seismometer_shortperiod

PPSDN 1 Seafloor Fluid and DNA DNA_particulate

RASFL 1 Seafloor Fluid and DNA watersample_chem_trace_H2S_pH

THSPH 1 Seafloor Properties Temp_H2_H2S_pH

TMPSF 1 Seafloor Properties Temperature_seafloor

TRHPH 1 Seafloor Properties Temp_resist



ADCPA 15 Water Velocity velocity_profile_mobile_asset

ADCPT 17 Water Velocity Velocity_profile_300m

CTDAV 3 Conductivity, Temperature, Depth CTD_AUV

CTDBP 18 Conductivity, Temperature, Depth CTD_bottom_pumped

DOFST 22 Dissolved Oxygen oxygen_dissolved_fastresp

FDCLP 3 Direct Covariance Flux flux_direct_cov_LP

FLORT 42 Fluorometer Fluorometer_three_wavelength

HPIES 2 Inverted Echo Sounder IES_pressure_velocity

HYDBB 6 Hydrophone Hydrophone_BB_passive

NUTR4 3 Nutrients nutrient_four_channel

OSMOI 2 Seafloor Properties watersample_chem_trace

OTISF 1 Seafloor Properties flow_benthic

PARAD 30 Optical Properties PAR

PRESF 9 Seafloor Pressure pressure_SF

VADCP 2 Water Velocity Velocity_profile_50m_turb

ZPLSC 8 Zooplankton Sonar plankton_ZP_sonar_coastal


CI Architecture for Instrument Integration

25

Data Processing and Product Generation


COL

OOI's Sensor Life Cycle

OOI Final OOI Final System System DesignDesign

Core Core Sensor Sensor

ListListCore Core

Sensor Sensor SpecsSpecs

Sensor WorkingGroup

Core Core Sensor Sensor RFPsRFPs

COL

CGSNRSN

Common Common Sensor Sensor

SelectionSelectionss

COL

CGSNRSN

CI

Procure Procure Common Common SensorsSensors

CGSNRSN

Develop Develop QC/Xform QC/Xform AlgorithmsAlgorithms

CGSNRSN

Develop Develop Sensor Sensor DriversDriversCI

Integrate Integrate w/ Sensor w/ Sensor

AgentAgent

Test Test w/Instrumw/Instrument Test ent Test

KitKit

CI CI Integration Integration

TestTest

System System Integration Integration

TestTestCICICICI

Encode Encode AlgorithmsAlgorithms

CI

Deploy Deploy at Seaat Sea

RSN

CGSN


Instrument Development Kit

• Scheduled for Release-2

• Includes:

• Logical Test Facility

Workbench (dry testing)

• Marine Specific System

Test Facility (wet testing)

• Configurations for RSN

and CGSN observatories

• Access and Management

Portals for Interactive

access


Aug 30 - Sep 1 201128

Computational Infrastructure

Objectives


National Network Deployment (Year 3)


CyberPoPs and Network Infrastructure

• Acquisition Point CyberPoPs:• Portland, OR• Woods Hole, MA

• Distribution Point CyberPoPs:• Seattle, WA• Chicago, IL• San Diego, CA• McLean, VA (optional)

• Engineering Center:• San Diego, CA

• Network Infrastructure• Dedicated 10GE loop San

Diego, Portland, Seattle,

Chicago; branch to McLean

(optional)• Dedicated 1GE connection to

Woods Hole


San Diego Engineering Center

• Purchased and deployed

San Diego CyberPoP and

network equipment (see

rack drawing)

• Located in UCSD’s Atkinson

Hall server room 1101

(secure and protected)

• In use for

• Release-1 production and

QA

• Management tools

(Confluence, Jira, etc)


San Diego CyberPoP and Network Equipment


Acquisition Point CyberPoP Deployments (Y3)

•Sites•Portland•Woods Hole

•Function•Data acquisition from

marine observatories•Real-time data processing

and OOI data product

generation•Data preservation•High availability


CyberPoP Physical Layout (Year 3 Deployments)

•Sites•Seattle•Chicago

•Function•Content distribution (web

servers, data servers,

messaging)• Interconnects with major

national and international

network providers (layer 2

peering)•Links to compute clouds


Aug 30 - Sep 1 201135

Network Peering IntegrationChicagoSeattle


Aug 30 - Sep 1 201136

Multi-Tier Messaging Federation

37

Multi-Site Service Network Deployment


Aug 30 - Sep 1 201138

External Observatory Integration


Aug 30 - Sep 1 201139

IOOS Integration Model


External Observatory Integration Progress

• Identification of datasets and data sources needed to support early

adopter group (ESPRESSO/Rutgers, HiOOS/UHawaii) numerical model

workflows

•Classification of datasets by type and representation

•Development of Dataset Agents for automatic data and metadata

ingestion

•Translation of data/metadata into the Common Science Data Format

•Streaming of data packets and metadata update notifications

• Integration of real-time data sources in early-adopter on-site numerical

workflow generation processes, using the Integrated Observatory

Network Release-1


External Observatory Integration

• Exemplar: MARACOOS (Rutgers University)

• Experimentally substitute part of the scientists’ data assimilation workflow preceding numerical

model execution


External Observatory Integration: Master Dataset List

• Datasets

• NAVY : NRLSSC (DAP) : 1

• NOAA : NDBC SOS (HTML) : 34

• NOAA : NDBC SDF (DAP) : 1

• NOAA : PFEG (DAP) : 1

• Rutgers : tashtego (DAP) : 2

• UCSD : HFRNET (DAP) : 1

• UH : SOEST (DAP) : 1

• USGS : WaterService (HTML) : 26


Aug 30 - Sep 1 201143

“What will OOI-CI mean to me”


Regional Data Provider & Manager

• Effective mechanism to reliably at scale distribute your data

• to your user community and data management/preservation

facility

• Independently publish and receive data

• in a wide variety of syntactic formats and community vocabularies

• Publish and receive as near-real-time data streams and as historical

data sets

• Effective mechanism to reliably at scale aggregate and disseminate

data from your publishers and to your end user communities


Participating Agency

•Participate as providers, managers and consumers

•Anticipated some requirements for publishing on the OOI as

a separate Agency will require coordination with NSF

•Requirements are being addressed as a part of the IOOS

OOI Integration effort


How does one “join” – approval / certification?

•Open data policy network for consumers of data

•Anticipate publishers required to register and certified

• Data policies and rights management being established

•Open data policies and content limitations to science and

education data content

• Publishing requires acceptance of some form of open use policy

• Consuming requires some form of “as is”; “at your on risk”; “with

attribution of source if republished” policies


Relationship with IOOS, NEPTUNE and others observatories

•External Observatory Integration subsystem within the OOI

• Specifically focused on providing bi-directional integration

interfaces to IOOS, NEPTUNE-Canada and WMO (aka GTS)

• Integration continues on into the Operations and Maintenance

phase of the OOI as a user/observatory “on boarding” capability

•Provide the training and support to communities that wish to

integrate with OOI


OOI define standards for them to “plug in”?, e.g SensorML

• OOI provides a wide variety of syntactic formats and community

vocabularies by which to interface with the OOI

• OOI and the user community will be able to extend these interfaces

• Capability facilitated by architectural choice of using controlled set

of canonical data models into and out of which all data are

transformed

• New transformations are added, “plug ins”, to the network for a

new format and/or vocabulary to one of the controlled data models


Will a region be able to host an instance of the OOI “stack”?

• The OOI “Integrated Observatory Network” is a federation architecture

• Comprising participants from multiple domains of authority operating from their local

domain of authority

• The OOI “stack” exists at a couple of levels:

• It is a messaging protocol

• It is a component software implementation that implements the messaging protocol

and process management containers (think modern web server) that publish and

consume services

• “Capability Containers”, can be and are written in multiple languages

• achieve their interoperability through use of the common OOI messaging protocol

• OOI message protocol and any of the OOI supplied Capability Containers will be

provided under one of the standard open source licenses


Aug 30 - Sep 1 201150

Thanks!

ooi ci overview and status

Documents

globalooi review year

orgdisplaycipubooi review

instrument agentsclass

ci components

cgsn needsci release

instrumentsacquire data

collaboration support

maintenance of marine