extensible and dynamic topic types for dds

Extensible and DynamicTopic Types for DDSMARS – Jacksonville, FL – March 2010

Presenter: Rick Warren, RTI

Submitting POCs: Angelo Corsaro, PrismTech: [email protected] Gerardo Pardo, RTI: [email protected] Robert Poth, PrismTech: [email protected] Rick Warren, RTI: [email protected]

Supporting POCs: Virginie Watine, Thales: [email protected]

document number: 2010-03-02

mailto:[email protected]







Copyright © 2010 RTI - All rights Reserved 2

Structure of this Presentation

Overview of specification and process

Specification contents Very detailed discussion last time This presentation focuses on what’s changed

AB feedback

DDS Types: Assessment and Challenges

Good

Unmatched type safety => fewer defects

Unique type awareness => faster integration

Unique type awareness => higher performance

Bad

No standard API to define / access types at runtime

Hard to change types without rebuilding, redeploying

Hard to upgrade systems one subsystem at a time


Vendor-Specific Solutions

OMG-Standard Solutions


Where We Are

RFP issued: Jun. 2008

Initial submission: Nov 2008

Revised submission extended: Feb. 2009

Revised submission extended: Aug. 2009

Revised submission extended: Nov. 2009

Revised submission deadline: Feb. 2010 Submitters back on the same page Responded to comments from 2 (now 3) AB reviewers Discussed with Task Force in Long Beach meeting


Design Principles

Don’t break anybody. Many people depend on DDS

Don’t modify existing APIs Don’t modify existing compliance points

Many people depend on RTPS/DDSI 2 Don’t modify existing protocols or formats Don’t modify existing compliance points

Keep it fast. New capabilities must permit efficient implementations. Don’t use them? Don’t pay for them.

Keep it orthogonal. Type compatibility doesn’t depend on: Types’ definition language(s) Data representation(s) Programming language(s) QoS configuration


Overview

Type System

Type Representation

Language Binding

Data Representation


Overview: Example

TypeRepresentation

LanguageBinding

DataRepresentation

IDL:Foo.idl

struct Foo { string name; long ssn;};

IDL to Language Mapping:Foo.hFoo.cFooTypeSupport.c

struct Foo { char *name; int ssn;};

Foo f = {"hello", 2};

IDL to CDR:

0000000668656C6C6F00000000000002


Conformance Changes

Feedback on previous submission: conformance criteria, relationships to other specs unclear

New submission more precise, better organized: Conformance (part II, section 3):

2 conformance levels correspond to DDS, RTPS Programming language conformance:

Type system, type representations, language bindings, DDS Minimum Profile

Network interoperability conformance:Type system, data representations, RTPS

Relations to other specs (part III, section 8) Extensions to DDS:

“Programming language conformance” sections constitute new optional conformance level (“profile”)

No changes to existing DDS profiles

Extensions to IDL: Normative to impl.’s of this spec only;

no change to CORBA conformance


Type System

Extensible, Compatible 6.5.1: (a) Type System as UML meta-model.

(b) Type substitutability rules. 6.5.2: Type extensibility, mutability rules

6.5.3: Type compatibility in presence of change 6.5.17: Allow type to evolve without breaking interoperability

6.5.5: Support keys. How does extensibility apply?

More Expressive 6.5.4: Map data type 6.5.6, 6.5.19: Sparse data: object contains only subset of fields defined

by its type


Type System Overview

class Type System

NamedElement

Type

+ nested: boolean {readOnly}

PrimitiveType

«enumeration»TypeKind

CollectionNamespace

Aggregation

Namespace

Enumeration

NamedElementNamespace

Module

Namespace

BitSetAlias

ConstructedType

«enumeration»ExtensibilityKind

FINAL_EXTENSIBILITY {readOnly} EXTENSIBLE_EXTENSIBILITY {readOnly} MUTABLE_EXTENSIBILITY {readOnly}

+/container

0..*{frozen}

+/containedType

*{addOnly}

+extensibility_kind

1{frozen}

+element_type

1{frozen}

+kind

1{frozen}

+base_type

1


Type System Overview

Entirely familiar from IDL: Primitives Strings, narrow and wide Arrays and sequences Aliases (typedefs) Unions Modules

As in IDL, but extended: Structures—including single inheritance

(for substitutability: 6.5.1) Enumerations—specify bit width and constant values

New relative to IDL: Maps—like std::map or java.util.Map;

required by RFP (6.5.4), fundamental OO collection Annotations—for extensibility (6.5.2, 6.5.3)

module Foo { struct Bar { long baz; }; typedef Bar Quux;}


Type System Changes

Clearer, more-flexible type extensibility kinds1. Previous kind (Boolean): “extensible”=true:

Add/remove/reorder type members Renamed to “mutable”

2. Previous kind (Boolean): “extensible”=false: Add to end, but don’t add elsewhere, remove, or reorder Renamed to “extensible” Now allow removing from end as well as adding to end

3. New kind “final”: Don’t allow any modifications

Explicit annotations identify all levels in IDL Un-annotated types presumed “extensible” (2) Can be overridden by IDL compiler option

Several concrete examples added


Type Extensibility Example

@Extensibility(

EXTENSIBLE)

struct Coord2D {

long x;

long y;

};

@Extensibility(

EXTENSIBLE)

struct Coord3D {

long x;

long y;

long z;

};Assignable


Type Extensibility Example

@Extensibility(

FINAL)

struct Coord2D {

long x;

long y;

};

@Extensibility(

FINAL)

struct Coord3D {

long x;

long y;

long z;

};Not Assignable


Type Representations

IDL XML, XSD TypeObject

Static Definition 6.5.11: Programming-language-

independent type serialization format(s) 6.5.9: Represent types in IDL and XML 6.5.7: Support IDL subset used by DDS 1.2

Dynamic Definition 6.5.12: Type discovery


IDL Type Representation Changes

Added optional pragmas to delimit IDL segments using new extended syntax#pragma dds_xtopics begin [<version_number>]

// IDL definitions

#pragma dds_xtopics end Document conformance requirements for human readers Help non-conforming compilers recognize problems

sooner, give more meaningful errors No particular errors or warnings mandated



Use struct syntax for inheritance, not valuetypestruct MyStruct : MyBaseStruct { … }; Type system doesn’t include all valuetype features, so

syntax was confusing Unclear whether valuetype could extend struct



Cleaner user-defined annotation syntax Previously used methods, like Java annotations:

@Annotation local interface MyAnnotation {

long my_annotation_member();

}; Now uses attributes:

@Annotation local interface MyAnnotation {

attribute long my_annotation_member;

}; Still uses local interface (alternative: struct)

Emphasizes that no runtime state exists Allows future support for multiple inheritance, if desired


XML Type Representations (No Change)

XML Type Representation Terse XML syntax designed ground-up for this Type

System Easy to read, plays nicely with auto-completing editors

XSD Type Representation Based on existing IDL-to-WSDL mapping specification Extended for new Type System concepts from this spec


TypeObject Changes

Type Representation suitable for DDS discovery

Renamed from “TypeCode” Name similarity to CORBA TypeCode deemed A Bad

Thing: this spec doesn’t depend on that one .Net also has “TypeCode,” something else entirely…

Improved string sharing to make representations much more compact


Data Representations

Extended CDR XML


CDR Data Representation (No Change)

Requirements: Continue supporting existing (compact) CDR used by user-

defined types (6.5.16) Continue supporting existing (extensible, parameterized) CDR

used by discovery types (6.5.16) Provide extensibility to user-defined types (6.5.2, 6.5.3, 6.5.17)—

all Data Representations should have equivalent expressiveness

Solution: Make existing extensible, parameterized CDR available to all

Same rules for every type, built-in or user-defined 100% compatible with existing types and serialization

Parameterized CDR: structs & unions marked “mutable”

Each member == parameter Member ID == parameter ID

Compact CDR: everything else


XML Data Representation (No Change)

Optional Requirement (6.6.1):Dynamic data access based on XML or JSON

Given: XSD Type Representation

Data object == XML document

Validated by XSD Representation of its type

SAX, DOM-based access follow naturally

New RTPS encapsulation ID lets you use XML on the network

struct OriginalLandData { long x; long y;};

<OriginalLandData> <x>5</x> <y>42</y>;</OriginalLandData>

Language Bindings

Plain Language Binding Dynamic Language Binding



Plain Language Binding (No Change)

Defined by existing IDL-to-language mappings:1. Start with IDL Representation of a type

2. Apply mapping to desired programming language

Mappings expanded to cover new concepts e.g. maps, annotations, optional members For C, C++, Java as required by RFP (6.5.8)

// IDLstructOrigLandData { long x; long y;};

// Javapublic class OrigLandData { public int x; public int y;}


Dynamic Language Binding (No Change)

Classes DynamicTypeFactory: Creates new types (6.5.20) DynamicType: Introspect type definitions (6.5.13)

Informed by CORBA TypeCode API

DynamicData: Introspect objects (6.5.18) Informed by CORBA DynAny, JMS MapMessage,

TIBCO Rendezvous self-describing messages

DynamicTypeSupport: Registers types with DDS (6.5.20) DynamicDataWriter: Writes objects of any (single) type,

each represented by a DynamicData object (6.5.21) DynamicDataReader: Reads objects of any (single) type,

each represented by a DynamicData object (6.5.21)


Use by DDS

Data Representation Compatibility Type Compatibility Built-in Types


Data Representation Compatibility

Reader and writer must agree on which Data Representation to use

6.5.14: Encapsulation specification, negotiation 6.5.15: Is encapsulation request-offer?

6.6.2: Optional: Custom encapsulation API

Largely unchanged from previous submission

Solution: DataRepresentationQosPolicy List of representation IDs (standard = XCDR, XML) Applies to both readers and writers Included in corresponding built-in topic data types Request-Offer semantics

Writer offers single representation it will use Reader requests a list of representations Compatible iff writer’s representation is in reader’s list

Can describe proposed representations, future additions, and vendor extensions


Type Compatibility

If I write type “Foo” and you read type “Bar”, can we communicate?

New submission clearer, more flexiblebased on feedback on previous submission

3 parts:1. Physical: type assignability

2. Intentional: type signature

3. Enforcement: QoS policy governs how above are applied (NEW in this submission)


Type Compatibility Example: What I Want

struct Coord2D

{

long x;

long y;

};

struct Coord3D :

Coord2D

{

long z;

};

Read Write

Write Read


1. Type Assignability

Structural compatibility:Can sample serialized according to one type be deserialized according to another? Reminder: Rules designed to require no writer-specific

runtime checks at deserialization time

Intrinsic property of types’ definitions Governed in part by extensibility kind No action required by application programmer


2. Type Signature

Do I intend the types to be compatible?

Lists of equivalent types, base types provided at registration time

Clarification to previous submission: both reader and writer declarations taken into account

New since previous submission: lists can contain patterns, not just literal names


3. Type Consistency Enforcement Policy

QoS policy on Topic, DataReader, DataWritergoverns how (1) and (2) take effect RxO: reader and writer must have the same setting

EXACT_TYPE: Types must also be equal(as if extensibility were “final”)

Registered names must be equal Like DDS of today but with extra safety Assumed during endpoint matching if remote policy

is unspecified

EXACT_NAME: Assignability assumed without inspection Registered names must be equal This is what DDS guarantees today.

DECLARED: Types must be assignable Type signatures must be compatible Default for conformant implementations

ASSIGNABLE: Types must be assignable Type names, signatures ignored: talk to any

compatible type


Type Compatibility Example: How to Get It

1. Is Coord2D assignable from Coord3D? Yes.

2. Type Signature:

Coord3DTypeSupport.register_type( my_participant, "Coord3D : Coord2D")

3. Type Consistency Enforcement: DECLARED(default)

struct Coord2D

{

long x;

long y;

};

struct Coord3D :

Coord2D

{

long z;

};


Built-in Types (No Change)

Goal: Improve out-of-box experience for new users, those with simple needs

Background: Usability gap vs. competition Other middlewares allow sending textual or binary data

without explicit type definition, registration

Response: Provide simple types built in Pre-defined, pre-registered String: single string payload Bytes: octet sequence payload KeyedString: string payload + string key KeyedBytes: octet sequence payload + string key


Summary


AB Feedback

Incorporates feedback from 3 AB reviewers:

Define IDL pragma to delimit extensions: Done

Clarify relationships to other specs: Done

Provide more detail about modules and namespaces: Done

Make annotation type definition syntax more natural: Done

Standardize VerbatimText behavior for C, C++, Java languages: Done

Assorted typos, wording changes, organizational suggestions: Done


Summary

The proposal:

Satisfies all mandatory and optional requirements andsupports identified real-world use cases

Without sacrificing backwards compatibility Without sacrificing performance

Is easily extensible w/ new type, data representations to handle additional use cases

Only small changes since previous submission

Incorporates all AB feedback

We will be asking for a vote to recommend adoption.

Q & A


extensible and dynamic topic types for dds

Technology