mbari’s ssds data management for ocean observatories brian schlining ブライアン...
TRANSCRIPT
![Page 1: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/1.jpg)
MBARI’s SSDS
Data Management for Ocean Observatories
Brian Schlining ブライアン シュリニング
![Page 2: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/2.jpg)
Overview
• About MBARI
• Overview of MOOS
• Functional Requirements of SSDS
• Metadata use by SSDS
• Dataflow through SSDS
![Page 3: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/3.jpg)
Monterey Bay Aquarium Research Institute
Established in 1987
![Page 4: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/4.jpg)
Santa Cruz
Monterey
Monterey Canyon
MBARI
![Page 5: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/5.jpg)
MOOS History
Monterey Ocean Observing System• 1989: Began mooring-based observations.
• 1999: Adopted the term “MOOS” (Monterey Ocean Observing System)
• 1999: Began development of an integrated ocean observatory.
![Page 6: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/6.jpg)
MOOS – The Present Concept
Benthic Node
Mooring
Autonomous Underwater Vehicle (AUV)
MBARI
![Page 7: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/7.jpg)
MOOS – The Present Concept
Benthic Node
Mooring
AUV
Mooring
‘Plug and Work’ Instruments
![Page 8: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/8.jpg)
Future of Ocean Observing
Variety of platforms
![Page 9: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/9.jpg)
Data Management
Challenges:• Large number of data sources
• Large variety of data sources
• Dynamic systemData sources may appear and disappear
• No standard data formatData can be instrument ‘native’
![Page 10: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/10.jpg)
SSDS
Shore-side Data System• Data management system for MOOS
![Page 11: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/11.jpg)
SSDS – Functional Requirements
1. Capture and store data from MOOS data sources Includes files and ‘streams’
2. Capture information (i.e. metadata) about the stored data Location, instrument, platform, data format, etc.
3. Capture and store data products Derived products, quality controlled data, plots, etc.
![Page 12: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/12.jpg)
SSDS – Functional Requirements
4. Provide access to the original ‘raw’ data
5. Convert data to common formats for user application tools (Excel , Matlab, ARCView, …)
6. Present simple plots of any well-described data.
7. Provide access to data through an application program interface (API) and a web interface.
![Page 13: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/13.jpg)
Metadata – What we have learned
• Metadata must accompany the data– A system’s power relies on good knowledge of its data– Without metadata, the data quickly becomes unusable
• Metadata must accompany the instrument– Every connector between the two increases error rates– Once data and metadata detached, reattaching is painful
• Metadata must be flexible and structured– Flexible: you’ll need to define new kinds of data sources– Structured: consistency => automation => value
![Page 14: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/14.jpg)
SSDS – Metadata (Object View)
![Page 15: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/15.jpg)
‘Deployment’ information.
SSDS tracks:• Where the data was collected.• When it was collected.• What other data was used.• Relation to other deployments
SSDS – Metadata (Object View)
![Page 16: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/16.jpg)
The data source.
SSDS tracks:• Software or hardware source• Unique identifier• Manufacturer information• References to documentation
SSDS – Metadata (Object View)
![Page 17: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/17.jpg)
References to the data.
SSDS tracks:• The data storage location.• The deployment that produced this data.
SSDS – Metadata (Object View)
![Page 18: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/18.jpg)
Format and contents of a DataContainer.
SSDS tracks:• The contents of a data-set.• The data format (To allow parsing by software).
SSDS – Metadata (Object View)
![Page 19: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/19.jpg)
Why not?• Multiple overlapping standards. Which to
choose?– None have achieved broad community support– Some are not yet complete (Marine XML)– Excessive documentation requirements (FGDC)
• May use with SSDS later (Dublin Core)
SSDS does not currently use metadata standards such as ESML, ESRI’s Marine Data Model, FGDC, Dublin Core or Marine XML.
SSDS – Metadata Standards
![Page 20: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/20.jpg)
Mooring Portal SSDS
Example SIAM to SSDS Data Flow
![Page 21: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/21.jpg)
Mooring Portal SSDSDevice
Example SIAM to SSDS Data Flow
A device is connected to a platform, such as a Mooring.
![Page 22: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/22.jpg)
Mooring Portal SSDSDevice
<?xml version="1.0" encoding="UTF-8"?><Metadata> <Deployment name="2003.30.01" role="platform“> <Device id="9873"/> <Deployment role="instrument" > <Device id="101"/> <output> <DataStream> <RecordDescription bufferStyle="binary" bufferLengthType="fixed" endian="little"> <RecordVariable name="time" columnIndex="1" format="double"
longName="Time(GMT)" units="milliseconds since Jan 01, 1970"/> <RecordVariable name="temperature" columnIndex="3" format="double"
longName="Seawater temperature" units="degrees C"/> </RecordDescription> </DataStream> </output> </Deployment> </Deployment></Metadata>
Example SIAM to SSDS Data Flow
The mooring retrieves the metadata from the device.
![Page 23: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/23.jpg)
Portal SSDSDevice
Metadata Packet
<?xml …> <Metadata> … </Metadata>
Mooring
Example SIAM to SSDS Data Flow
The metadata is packaged and sent to a portal on shore before any data is sent to shore.
![Page 24: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/24.jpg)
Portal SSDSDevice Mooring
Example SIAM to SSDS Data Flow
Metadata Packet
<?xml …> <Metadata> … </Metadata>
The portal forwards the metadata to SSDS.
![Page 25: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/25.jpg)
PortalDevice Mooring
DB
SSDS
<?xml …> <Metadata> … </Metadata>
Example SIAM to SSDS Data Flow
SSDS stores the metadata in a database.
This allows applications to query for and use data.
![Page 26: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/26.jpg)
PortalDevice Mooring
DB
SSDS
Example SIAM to SSDS Data Flow
![Page 27: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/27.jpg)
PortalDevice Mooring
DB
SSDS
34,56.234,0.0023,...
Example SIAM to SSDS Data Flow
The device produces a data record.
![Page 28: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/28.jpg)
PortalDevice Mooring
DB
SSDS
Data Packet
34,56.234,0.0023,...
Example SIAM to SSDS Data Flow
The data is packaged and sent to SSDS.
![Page 29: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/29.jpg)
PortalDevice Mooring
DB
SSDS
VersionID,DeviceID,MetadataID,RecordType,PlatformID,SystemTime,SequenceNumber,DataBuffer(34,56.234,0.0023,…)
Serialized
Example SIAM to SSDS Data Flow
SSDS uses information in the packet to sort and store the data in a ‘raw’ format.
![Page 30: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/30.jpg)
PortalDevice Mooring
DB
SSDS
netCDF
Example SIAM to SSDS Data Flow
Serialized
VersionID,DeviceID,MetadataID,RecordType,PlatformID,SystemTime,SequenceNumber,DataBuffer(34,56.234,0.0023,…)
The ‘raw’ data is parsed and stored as netCDF for easier access.
![Page 31: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/31.jpg)
MBARI Software
SSDS
Software applications allow users to discover and obtain data in formats useful to the typical MBARI user. (netCDF, text, etc.)
PortalDevice Mooring
DB
netCDF
Example SIAM to SSDS Data Flow
Serialized
netcdf parosci {dimensions: time = UNLIMITED ; // (17761 currently)variables: double time(time) ; time:long_name = "Time (GMT)" ; time:units = "seconds since 1970-01-01 00:00:00" ; double depth(time) ; depth:long_name = "depth" ; depth:units = "UNKNOWN" ;
// global attributes: :title = "AUV data" ; :created = "2003-06-12T23:34:58Z" ; :history0 = ": Deployment information for parosci.log" ; :deploymentName = "2003.099.10" ; :instrumentId = "3699" ;}
![Page 32: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/32.jpg)
PortalDevice Mooring
DB
SSDS
netCDF
Example SIAM to SSDS Data Flow
Serialized
MBARI Software
Software applications also provide simple visual representations of data
![Page 33: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/33.jpg)
PortalDevice Mooring
DB
SSDS
netCDF
Example SIAM to SSDS Data Flow
Serialized
Web Pages
MBARI Software
Provide internet access
![Page 34: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/34.jpg)
PortalDevice Mooring
DB
SSDS
netCDF
Existing netCDF Software
Example SIAM to SSDS Data Flow
Serialized
MBARI Software
Web Pages
Save development time by using existing software applications
![Page 35: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/35.jpg)
Requirements Development
2002 2003 2004
MTM
AUV CTD
MTM II
Relational DB Schema
Infrastructure & Capabilities
Access & VisualizationExtreme Week
Tools Training
MTM Deployed
AUV Science Ops Start
SSDS Schedule
MetadataMetadata
MOOS ‘Lite’
![Page 36: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/36.jpg)
Development Team
• John Graybeal
• Kevin Gomes
• Michael McCann
• Brian Schlining ( 武頼庵 洲美守 )
• Rich Schramm
• Daniel Wilkin
![Page 37: MBARI’s SSDS Data Management for Ocean Observatories Brian Schlining ブライアン シュリニング](https://reader033.vdocuments.mx/reader033/viewer/2022061602/56649e4d5503460f94b43945/html5/thumbnails/37.jpg)
• John Ryan• Francisco Chavez• Ken Johnson• Drew Gashler• Mark Chaffee• Tom O’Reilly• Mark Chaffee
Science and Technical Advisors
Funding provided by the David and Lucile Packard Foundation