http://forge.gridforum.org/projects/dfdl-wg/ dfdl wg session 1 summary of status wg co-chairs: mike...

http://forge.gridforum.org/projects/dfdl-wg/

DFDL WG Session 1Summary of StatusWG Co-Chairs:

Mike Beckerle, Ascential Software Alan Chappell, PNNL Martin Westhead

(Two note takers please?)

Tuesday, 2004-09-21 17h30+02:00 Brussels (BE.CEST) in Room D005.

11h30 New York, (US.EDT, UTC-4) 08h30 San Francisco (US.PDT, UTC-7)


Abstract

Progress and Status Review - This session will review the progress since GGF11 which is substantial, and also provide enough context for those interested in getting involved who haven't been following along.


ADF BoF

Sep 22 11h00-12h30ACCESS TO DATA IN FILES-WG CHARTER DISCUSSION BOF (DATA) Calendar: QB (200p)Details:Title: ADF - Access to Data in FilesSession Leaders: Neil Chue Hong, Brian M CollinsAgenda:- Summary of General Focus, Milestones and Objectives- Discussion of Relationships specifically with DFDL and DAIS as well as other GGF Groups- Charter Update


DFDL WG Session 2

Room D008

Thursday, 2004-09-23 11h00+02:00 Brussels (BE.CEST)

05h00 New York (US.EDT, UTC-4)

02h00 San Francisco (US.PDT, UTC-7)


Since GGF11 (June 2004)

Latest working examplesdocument/jdm20040720.zip

DFDL Represenation Properties: Basic Set document/ggf-dfdl-rep-properties-

proposal-basic-001.doc


DFDL-WG Session 1 Summary of Status

Agenda Review of purposes/goals

XML / XSD impact: Data ModelExamples w/latest updates

• as evolved now: different from primer document

General discussion (40 mins) Overview of other sessions (5 mins)


Data Interchange Formats

Prescriptive: Put your data in this format! XML – textual Binary – ASN.1, XDR, NetCDF, HDF, EBML,…

Descriptive: What format is your data in? Commercial products ASN1 Encoding Control Notation

• ITU-T X.692

DFDL


Why Descriptive?

Allows us to achieve two goals simultaneously:

1. Interoperability Modern and Legacy data formats

2. Performance! Density

• Fewest bytes to represent data without resorting to compression

Optimized I/O• Seekable random access• Memory mapped, aligned

• Without sacrificing general access


Why the GGF for DFDL?

Grids are about big-data and big-computation problemsSimplistic solutions like “use XML”

won’t cut it! Grids are about universal data

interchange


General Features

Basic ASCII/Binary Read capabilities Inclusion of static info, e.g. “units” Validated input (from XML Schema) Defaulted input for missing values Reference – use of a previously read value in subsequent

expressions Choice – use of a previously read value to select among format

variations Multi-layer – description of an intermediate representation not

exposed in the final result Multiple input streams Basic Math – in DFDL expressions Basic Math – to generate new representations/values “Push-back” capability New type/transform specification


Desires

Support very efficient parsers Hints about lengths so data can be accessed

randomly Avoid requiring data copies But allow caching (e.g. based on hints) Allow ‘black-box’ transforms (e.g. unzip)

Support round-tripping Link transforms and inverses Infer inverses for lossless transforms

Keep simple cases simple Default transforms and parameters


Related Standards Efforts

Prescriptive systems:W3C binary XML

(http://www.w3.org/XML/Binary/)• Formed, but discussion group has no

items.• CORRECTION - there is activity now.

Descriptive systems:ASN1 Encoding Control Notation

• ITU-T X.692


XML Synergy

Use XSD to describe the logical data Use annotations within the XSD to

describe the representation of it.


XSD Types

Elements A.k.a. fields

Sequence groups, All groups All = unordered group

Choice A.k.a. union, redefine,

Vectors Use element with minOccurs, maxOccurs.

Nillability A.k.a. Nullable values


XML/XSD – basic types

anySimpleType

string QName NOTATION float double decimal boolean base64Binary hexBinary anyURI

normalizedString

token

language Name NMTOKEN

NMTOKENSNCName

ID IDREF ENTITY

IDREFS ENTITIES

integer

long nonPositiveInteger nonNegativeInteger

negativeInteger positiveInteger unsignedLong

unsignedInt

unsignedShort

unsignedByte

int

short

byte

date time dateTime gYear gYearMonth gMonth gMonthDay gDay duration


Example 1: XML

<w>5</w><x>7839372</x><y>8.6E-200</y><z>-7.1E8</z>


Example 1: XSD

<xs:sequence><xs:element name="w" type=“int"/><xs:element name="x" type=“int"/><xs:element name="y" type=“double"/><xs:element name="z" type=“float"/>

</xs:sequence>


Example 1 DFDL - binary

0000 0005 0077 9e8c

169a 54dd 0a1b 4a3f

ce29 46f6


Example 1 DFDL - binary

<xs:complexType name="example1"> <xs:annotation> <xs:appinfo> <repType>binary</repType> <byteOrder>bigEndian</byteOrder> </xs:appinfo> </xs:annotation> <xs:sequence> <xs:element name="w" type="int"/> <xs:element name="x" type="int "/> <xs:element name="y" type="double "/> <xs:element name="z" type="float "/> </xs:sequence> </xs:complexType>


Example 1 DFDL - textual

“5, 7839372, 8.6E-200, -7.1E8”


Example 1 DFDL - textual

<xs:complexType name="example1"> <xs:annotation> <xs:appinfo> <repType>text</repType> <charset>UTF-8</charset> <decimalSeparator>.</decimalSeparator> <separator>,</separator>

</xs:appinfo></xs:annotation><xs:sequence>

<xs:element name="w" type="int"/><xs:element name="x" type="int"/><xs:element name="y" type="double"/><xs:element name="z" type="float"/>

</xs:sequence></xs:complexType>


More Examples

Currently working on a set of examples which illustrate each of the DFDL constructs


Examples (so far)

BasicMathInExpressions.xsd BasicMathInRepresentations.xsd Choice.xsd DefaultValuePropagation.xsd DefaultedInput.xsd IncludeTransform.xsd IncludeType.xsd & NewDFDLType.xsd MultiLayer.xsd MultiStreams.xsd NewDFDLTransform.xsd NewExternalDFDLTransform.xsd NewMixedDFDLTransform.xsd Reference.xsd StaticInformation.xsd ValidatedInput.xsd


Current Status

Fairly clear idea of required features and data types

Large library of simple, real world, and pathological examples

Converging on an annotated XML Schema syntax for DFDL


Status

We think we have:A good handle on requirements/use casesCommon idea of functionality/ overall

designA not quite complete syntax based on

XML SchemaA set of “unit-test” examples to help in

finalizing the syntax


Opportunities

Good time to help with syntax Once that exists:

Standardized language bindings/APIsDFDL implementationsDFDL-based ADF Data Virtualization

ServicesExtensions for specific domains/uses…


Next Steps

Session 2: Charter-revisions discussion Review of specific examples IBM Websphere Business Integrator

DFDL presentation


END

http://forge.gridforum.org/projects/dfdl-wg/ dfdl wg session 1 summary of status wg co-chairs: mike...

Documents