NEESGrid Data and MetaDataTechnology

Kincho Law, Jun Peng, Jim Eng, Terry Weymouth, Paul Hubbard, Charles

Severance

Goals Data is online and persistent Data and Metadata are supported together Data migrates transparently including security, and metadata Data is completely secure with access controls but security does not

get in the way Data provenance - how was it gathered, how has it been manipulated? Data in support of research publication Support for repeatable experiments Data oriented research computation support Support for workflow

The Slide

gas

mile

gallon

car

model

ratio

mileage

fluid

volume

denominator

numerator

measurementdistance

rate

travel

consumption

fuel

unit

efficiency

vehicle

estimate

Metadata

Data

Data Viewers

Data

Map

pers

Data Ingestors

There is a layer is where we develop tools which take advantage and begin to depend on of the “meaning” of the data – where we begin to depend on the meaning of a second.

Where we make a viewer capable of viewing a certain type of object.

This is where we build things which make use of knowledge.

This layer will never be complete but it is a large focus of the coming months.

Concept

s

Search

09/2003

NEESgrid Data – Core Elements Local Repository Central Repository JAVA APIs – Run locally on the same system as a

repository or over OGSA Web Services– NEES File Management Services– NEES Meta Data Services

Data Viewers– Streaming (numeric, X/Y graph)– Stored (X/Y graph, 2-D structure, video)

NEESdata

NEESpop

LocalRepository

Core Elements

AP

I

CentralRepository

DataTeamlets

Data Acquisition

Workstation

AP

I

DataTeamlets

AP

I

Data/MDIngestTools

Data tools

Data viewers

Grid and Web Services

A Simple Experimental Scenario

DAQ System

Glue

Test S

pe

cimen

Lab

view

Developer System

Researcher System

Simulation System

Code

Simulation System

Code

A Simulation ScenarioDeveloper System

Simulation System

Code

NCSA NEESPop (1.1)

Colorado

NEESPop (1.1)

Incoming FTP

NEESMost (Win XP)

UIUC/Newmark

NEESPop (2.0)

LabView DAQ

MatLab Host And Real-Time Target Control

SystemSim Controller

CO

LabView DAQ

NSDS

UIUC

Test Specimen

Matlab Computational

Model

Shore-Western

Test Specimen

Incoming FTP

NSDSRepository

CO

NCSA

UIUC

Meta

Series of files

Complete file (aggregated)

NTCP

Site / LocationComputer Process

NCSA

Ingest

Ingest

NTCP

Matlab

NTCP

Ftp

NTCP

Wires

NFMS/NMDS

NSDS

File I/O

Plug In

Ingest

UIUC

MOST Scanario

Overview of tools and technologies Model development in RDF Project Browser Repository Browser Electronic Notebook Data Mappings Data Viewer Data Turbine Data As Video Still Cameras

11/2003

RDF Integration

Some of the data and meta data task force members are using Protégé-2000 to develop their models and expressing them in RDF.

RDF and NEESML are very similar but not identical so it may be challenging to ingest any arbitrary RDF

We expect that we will be able to map a subset of RDF to NEESML for ingestion or adapt an RDF parser (Jena or Raptor) to ingest that subset directly into the repository

11/2003

Models + Data Model

Repo

Models

Configure

Data

Load

Con

figur

e

RDF/OWL

RDF<owl:ObjectProperty rdf:ID="hasPublications"> <rdfs:domain> <owl:Class> <owl:unionOf rdf:parseType="Collection"> <owl:Class rdf:about="#Project"/> <owl:Class rdf:about="#Task"/> </owl:unionOf> </owl:Class> </rdfs:domain> <rdfs:range rdf:resource="#Publications"/> </owl:ObjectProperty>

Protégé - 2K

Models + Data Model

Repo

Models

Configure

Data

Load

Con

figur

e

RDF/OWL

RDF

Repository Browser

Electronic Notebook

Collaborative effort with the DOE SciDAC– Electronic notebook - metadata entry– Data mapping– Data provenance– Data display– Slide data/metadata jakarta.apache.org/slide/

Ultimate integration will be via JSR-170 www.scidac.org/SAM/ collaboratory.emsl.pnl.gov/docs/collab/sam/samtechoverview.html

DOE ELN / Example

DOE ELN / NEESgrid Integration (to date)

SAM /Slide

Repository

ChefGrid

Security

NEESgridRepository

MyProxy

Technology Celebration

DOE ELN / NEESgrid Integration (ultimate)

ChefGrid

Security

NEESgridRepository

MyProxy

Mappings and the Data Viewer NSDS (ISO 8601 Time channel) Column data with time recorded as a column Column – generate time Column – generate time – trigger filter

Channel units: g,g,in,kipTime ATL1 ATT12002-11-13T15:48:55.26499 -0.006409 0.004272 2002-11-13T15:48:55.36499 -0.005798 -0.003662

100.0000.435 0.161 -1.016 -0.981 0.430 0.161 -1.016 -0.9770.435 0.161 -1.016 -0.977

public class NEESDataMap{ public static boolean repoMap(File mainFile,

File mappingFile, String mapping) {

// Code here }}

Data Turbine

Commercial, free data streaming toolkit

Data Turbine (cont)

Existing data viewers will be adapted to access and display data from data turbine

Data acquisition software will be adapted to place information in Data Turbine Channels

Metadata elements will be developed to represent data turbine live, stored, and derived channels

New efforts (video as data) will be developed from the ground up using Data Turbine

outlet.creare.com/rbnb/

11/2003

Video as Data

Follow on to initial demonstration at ORST Experiment based development: Minnesota Design phase complete Joint effort, NEESGrid SI, ORST, Minnesota,

UC Davis, Texas, Buffalo, and others

DT Main System

CTLNTCPPlugin

Data Turbine - Today

NTCPControl

DAQData

CaptureDT Client

BT848 rbnbjcap

DT Client

NEESNSDSDriver

ControlPlugin

AXIS / DTGateway

Axis

DT Main System

Data Turbine

NTCPControl

ControlPlugin

Thumbs

Control

Make Smoothie

Technology Celebration

DT Main System

PTZ/USB

StillCapture

DT Client

DT Capturing

CameraControl

DAQData

CaptureDT Client

AudioAudio

EncoderDT Client

BT848 rbnbjcap

DT Client

Still Capture - Minnesota / Paul HubbardVideo capture - From CreareAudio capture - From Creare (TBD)Data Capture - From sites (upwards compatible)NEES NSDS Driver - Paul HubbardCamera Control Plugin - Mich / Minn

NEESNSDSDriver

Each still capture produces two channels - Small 1-5fps stream + large single images when picture is taken

ControlPlugin

DT Main System

User Views / Still Camera

Quicktime Storage System

QuicktimeSlicingTool

Still Image / Camera Control

~

< >^

^

< >

ControlPlugin

Data Viewer

ThumbnailProcess

Thumbnail - uBuffalo / UmichiganThumbnail viewer - Creare / MichCamera Control Applet - Minn / MichQuicktime Slicing tool - Mich (low)Stored Data Viewer - MichJPEG Viewer - CreareQuickime Viewer - Apple

Thumbnail + Audio + Data

< > +

MinnesotaMock - up

If you area developer and interested in following / helping / participating in this activity, join the mailing list

neesgrid-dv@neesgrid.org

Tool List - To Do

Next release of repository Integrate ELN into repository DAQ Control Panel in CHEF

– Set/Retrieve Metadata– Start / Stop– Ingest data from staging space

Data Turbine Control Panel in CHEF– Start / Stop / Configure Sources

• Video | Audio | Data | Thumbnail– Control permanent storage of video

To Do (cont)

NTCP Debugging and Monitoring in CHEF– Needed Data Turbine

Data as Video Client Tools in CHEF– New Monitor Tool– Still Image– Camera Control

Data Turbine Audio Capture We may need to support XML Schema QuickTime Capabilities

– Archive, retrieve, slice, dice, convert, present– Probably will not be completed as part of SI effort

Data Model Work

Data Curation Summit– Understand issues form go-forward plan– Meeting 3/18/2004

Data and Metadata Task Force– Finish the tsunami and centrifuge models

DSAC Committee– Meeting 3/19/2004

Summary

In September 2003, we met and “re-visioned” data

A bunch of requirements gathering and development has been done– The “high risk” elements are working now

There is more to do - We will run out of time– Evolutionary development approach - there will

always be usable working code - we will stop when we run out of time

The people…

Gokhan Peckan - Data Models Kincho Law - Data Models / Software design Jun Peng - Data Models / Software design Jim Eng - Parse / Ingest / RDF / Project

Browser Jim Myers - Electronic Notebook Terry Weymouth - Data Turbine Paul Hubbard - DAQ and NTCP Joe Futrelle - Data / Metadata Repository