niem and xml for architects and developers
DESCRIPTION
Maplesoft event in OttawaTRANSCRIPT
<Insert Picture Here>
Delivering NIEM-aligned data services
David Webber, Information Architect, Oracle Public Sector
NIEM and XML
for Architects
and Developers
The following is intended to outline Oracle general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.
The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
Disclaimer Notice
Slide 2
© 2012/13 Oracle Corporation – Restricted Distribution 3
Agenda
•Background - What are the business drivers and needs?•NIEM today – Technical Introduction•The Delivery Life Cycle – Lessons Learned•Software Delivery - Anatomy of a NIEM “IEPD”•Tooling to the Rescue!•Key Pillar of Dictionaries•Summary
BACKGROUNDBusiness Drivers and Needs
© 2012/13 Oracle Corporation – Restricted Distribution 5
XML is everywhere…
•PMIX – Prescription Drug information sharing•Elections Results Reporting•Pharmacy and Vaccines locating•COBie Buildings Information sharing•PESC – Student Post-Secondary Education•City 311 Services•Vehicle spare parts inventory•Public Safety information sharing•Healthcare services delivery coordination•eDevice control templates
© 2012/13 Oracle Corporation – Restricted Distribution 6
Public Sector Challenges
• Cost of building information exchanges
– Reduction of development sunk costs through minimizing coding tasks
– Allowing rapid prototyping and proof of concepts (agile development)
– Flexibility - dynamic adaptable templates instead of rigid fixed code
– Reuse – templates for common systems / solutions easily shared and adapted
• Government transparency and public information sharing
– Structured content from data stores – e.g. election results reporting, monthly cost
reports, scheduled road maintenance; obvious metadata and semantics
– Simple search requests on data – e.g. part numbers, codes, licenses lookup
– Not intended for unstructured and textual content document delivery
• Practical real world data sharing anywhere
– Direct simple tools with short learning curve and plug and play deployment
© 2012/13 Oracle Corporation – Restricted Distribution 7
Technology Capability Requirements
• Delivering on the Open Data Digital Strategy Vision for government
• Open Standards and Open Source based
• Plug and play with code-free templates and rapid development
• Supports NIEM and Open XML exchanges
• Leverages deployed technology today
• Workflow integration
• Secure messaging delivery and partner management
• Delivered across-platform and device
• End-to-end security & governance
NIEM TODAYTechnical Introduction
The NIEM FrameworkNIEM connects communities of people who share a common need to exchange information in order to advance their missions, and provides a foundation for seamless information exchange between federal, state, local, and tribal agencies. Much more than a data model, NIEM offers an active user community as well as a technical and support framework.
Support FrameworkTechnical FrameworkCommunity
Formal Governance Processes
Online Repositories
Mission-Oriented Domains
Self-Managing Domain Stewards
Data Model
XML Design Rules
Development Methodology
Predefined Deliverables (IEPD)
Tools for Development and Discovery
Established Training Program
Implementation Support
Help Desk & Knowledge Center
The NIEM Data Model
NIEM’s data model is a set of common, controlled, and approved XML data structures and definitions vetted through the Federal, State, Local, Tribal and Private Sectors.
Data elements are organized into core and domain-specific components
Core components are used by multiple
domains and can be described by structure,
semantics, and definition universally
Domain-specific components are
continually updated by subject matter experts that are actual NIEM
participants and industry experts for
their particular domain
NIEM Naming and Design Rules (NDR) specify how each of
these components are defined and utilized
NIEM’s Domains View
NEWEST DOMAINS
NIEM Government Resource
Management Domain
NIEM Health Domain
NIEM Human Services Domain
DOMAINS IN PLANNING
NIEM Agriculture
NIEM Education
NIEM Transportation
11
© 2012/13 Oracle Corporation – Restricted Distribution 12
Available XML Dictionaries
•NIEM 2.1 dictionaries– CBRN dictionary– Emergency dictionary– Family dictionary– Immigration dictionary– Infrastructure dictionary– Intelligence dictionary– Justice dictionary– Maritime dictionary– Screening dictionary– Trade dictionary– Immigration blueprint– NIEM core dictionary
Available from download site
direct link:http://sourceforge.net/projects/camprocessor/files
XMLXML XMLXML XMLXML XMLXML XMLXML XMLXML
+ includes spreadsheets and sample models
Note: Those marked in bold are model style dictionaries with recursive components.
© 2012/13 Oracle Corporation – Restricted Distribution 13
NIEM uses W3C Schema Collection
XSD Collection
*Code list xsd
Namespace* OptionalXML Schema
xsd subsetxsd
subset *Constraint xsd
*Constraint xsd
*Extension xsd
*Extension xsd
*Restriction xsd
*Restriction xsd
Industry Domain
schemas
Code list xsd
LEGEND:
Exchange xsdExchange xsd
NIEM Exchange Schema
© 2012/13 Oracle Corporation – Restricted Distribution 14
NIEM Past and Present Challenges
•Vertical domain vocabulary for DHS and DOJ
•One dozen domain dictionaries
•Technology limited - built using XSD schema
•Core components highly contextual to DHS/DOJ
•Surprisingly – no actual logical models of information!
•Gap with semantic technologies integration
•Limited data content rules and code lists
•Mapping automation support missing
•Multi-year development life cycles
© 2012/13 Oracle Corporation – Restricted Distribution 15
NIEM is still improving…
•Collections of complex XML Schema•Verbose components•Embedded context in names•Currently facing significant scaling challenges
– Inconsistencies; too much manual management; slow lifecycles
•Dictionary technology incubating•Enhanced code lists mechanism incubating•UML profile is evolving initiative with OMG
– Attempt to marry modelling techniques and XSD Schema syntax
•Policy and Security mechanisms incubating•Need to embrace enabling tools for rapid expansion of
domains and community collaboration
DELIVERY LIFECYCLE
How to build NIEM solutions
Lessons learned
Foundation Pillars
© 2012/13 Oracle Corporation – Restricted Distribution 17
The 8 “D”s and NIEM
•Design•Develop•Deploy•Document•Dictionaries•Discovery•Differentiate•Diagnose
Repeatable, Reusable Process(Exchange Specification Lifecycle)
NIEM IEPD Process
*IEPD - Information Exchange Package Documentation
©2012 Oracle Corporation 18
NIEM IEPD Reports
DictionariesDiscovery
NIEM IEPD / Exchange Delivery Lifecycle
Diagnose
Differentiate
Design
Develop
Validated
Templates /
Schema
XSD Schema
XML Samples
XMI / UML Models
Production
Results
Requirements
Updates
Documentation
Drag and Drop Visual
Designer
DocumentDeploy
Exchange
Templates 1
2
3
4
5
Required IEPD artifacts
©2012 Oracle Corporation 19
Integrating into your existing processes and control• Critical to manage and steer use of NIEM
• Ensure business outcomes match requirements• Lessons learned with NIEM
• The earlier in the software development process that NIEM is introduced the better the overall outcomes
• Each step has deliverables – business plan, delivery architecture, data model, alignment reports, schema, test results that can be monitored in your SDLC*
• Monitoring products reinforces reuse as best practice• Tools essential in review processes
• Collaboration tools can coordinate artifacts across projects and participants
• Test bed tools and open APIs reduce integration alignment costs *SDLC – Software Development Life Cycle
SOFTWARE DELIVERY
Anatomy of a NIEM “IEPD”
It is not just all about the XSD schema
©2012 Oracle Corporation 21
IEPD Components & Requirements
<Exchange_Schema/>
<Extension_Schema/>
<Subset_Schema/>
IEPD IEM
IEPD MPD
NIEM Core Schema(s)
Domain Schema(s)
Main Document
Catalog
Change Log
Sample XML Instance
In order to be NIEM-conformant, the IEPD must adhere to:1. NIEM Conformance Document
2. NIEM Naming and Design Rules (NDR) v1.3
3. NIEM Model Package Description (MPD) Specification v1.0
Formal NIEM IEPD Packaging Details
IEPD Artefact Description
Exchange Files
Subset Schema (SSGT) Subset of the full NIEM schema—a compressed directory of schemas to distinguish from other schema sets.
Wantlist User requirements -distinguishes user-required data components from components they depend on for conformance. Generated by and up-loaded to the Schema Subset Generation Tool (SSGT). This is an open specification and the SSGT is not required to create a wantlist, though it is easier.
Exchange Schema Base document schema that defines the XML root element and is generally named after the IEPD itself. Also known as the document schema, reference schema, or root schema.
Constraint Schema Constraints for separate constraint validation path—a compressed directory of schemas to distinguish from other schema sets.
Extension Schema Specification for extended components—separate local name-spaces of components not contained in NIEM.
Sample XML Instance Example instance(s) – may reference optional stylesheet.
Stylesheet (generic) Example stylesheet for display of instance(s).
Documentation
Master Documentation The Master Document is the main document for which all of the context and details around the exchange are explained. This document includes, the overview, as well as details surrounding the exchange, business drivers and requirements
Exchange model XMI Exchange model in standard open format (xmi, vsd, zargo) and standard open graphic (jpg, pdf, etc.) preferably a Unified Modeling Language (UML) model.
Business Rules Business rules in one of the following formats: (1) plain or structured English, (2) written into master documentation, (3) Schematron or other formal business rule language, or (4) generated by a development tool.
Mapping to NIEM Mapping of domain components, tagged with constraints (i.e., cardinality, etc.) to NIEM components ; preferably the Component Mapping Tool (CMT).
Extended Components Components created because they were not in NIEM—may be part of mapping spreadsheet and include structure and definitions of new components. Prefer CMT.
Change log Record of cumulative changes from previous IEPD versions. The initial IEPD simple records its creation date.
Catalog
Catalog XML file A machine-readable list of artifacts in the IEPD in an open, portable and browser-displayable format (simple manual edit).
Metadata XML file All metadata of owner and domain to be registered with the IEPD (simple manual edit).
Generated using CAM
Never Used
Word Document
Manual Entry
©2012 Oracle Corporation 23
XML Technology ComponentsNIEM: need to manage complete aspects of exchange delivery, not just the structure and content data model
©2012 Oracle Corporation 24
Swim Lane View
UML Technologies XML TechnologiesW3C Schema
This is what NIEM exchanges are delivered with on-the-wire!
UMLXMI
XSD Namespace
EnumerationsXSLT
XQueryXPath
XML
Patterns
Associations
CardinalityStereotypes
InheritanceFacets
Elements
Attributes
ContentComplex Types
Audience:
Data Modellers / Designers
Audience:
Software Developers and Middleware Engineers
Audience:
General content delivery and applications; content engineers; rules representation systems; multimedia delivery; geospatial systems
Objects
Web
HTML
JavaScript
JSON
REST
SQLSOAP
Audience:
Web application developers
Open-XDX
24
TOOLING TO THE RESCUEAccelerating and Improving the NIEM process
©2012 Oracle Corporation 26
IEPD Delivery Tasks / Tools Matrix
CAM CAMVOpen-XDX
(OPX)Exchange Schema Development X
Canonical Model Development / Generation X
Exchange Schema Generation X
NIEM Compliance Checking / Reporting X
WSDL / JAXB Binding Generation X
Operation Message Test Case Generation X
Production Services Message Generation X
SQL Schema Mapping X X
NIEM IEPD Artifact Generation X
- Cross-reference spreadsheet X
- UML model X
- Business Rules Report X
- XSD schema package X
Conformance Test Suite X
Run time XML instance validation X
In order to be NIEM-conformant, the IEPD must adhere to:
1. NIEM Conformance Document
2. NIEM Naming and Design Rules (NDR) v1.3
3. NIEM Model Package Description (MPD) Specification v1.0
© 2012/13 Oracle Corporation – Restricted Distribution 27
CAM Toolkit for NIEM
•Concepts behind the Content Assembly Mechanism
approach
•Understanding the steps involved in top down /
bottom up exchange design, testing and delivery
•Applicability to NIEM and IEPD / MPD delivery
* NIEM – National Information Exchange Model
* IEPD – Information Exchange Package Documentation
*MPD – Model Package Description
© 2012/13 Oracle Corporation – Restricted Distribution 28
CAM Templates – WYSIWYG Exchange
Structure Rules
Documentation
CAM Template
http://www.oasis-open.org/committees/cam
© 2012/13 Oracle Corporation – Restricted Distribution 29
INGESTINGEST
CAM and W3C Schema Wizards
EXPORTEXPORT
XSLT tools
process
entire XSD
collection
XSLT tools
process
entire XSD
collectionXSD Collection
Structure
Rules
Documentation*Code list xsd
Namespace* OptionalXML Schema
xsd subsetxsd
subset*Constraint
xsd*Constraint
xsd
*Extension xsd
*Extension xsd
*Restriction xsd
*Restriction xsd
Industry Domain
schemas
Code list xsd
LEGEND:
Exchange xsdExchange xsd
Exchange Schema
CAM Template
htt
p:/
/ww
w.c
amed
ito
r.o
rgCAM Editor – Main Screens
© 2012/13 Oracle Corporation – Restricted Distribution 31
CAM Toolkit Features
• Dictionary “Drag and Drop” of components to build exchanges
• Visual editor and structure editing
• Cross-platform deploy - Windows, Ubuntu/LINUX, Mac; 32/64bit
• Mind map models – integration with Freemind tool
• UML/XMI export – import to UML tools / ArgoUML
• Code list table lookups – rule based and selectable
• This allows people to fully integrate their exchange development from dictionaries through to working deployed exchange schema.
31
© 2012/13 Oracle Corporation – Restricted Distribution 32
XML Exchange Package Artifacts
•Template Evaluation Report
– Naming and Design Rules assessment
– Interoperability factors review
– Spell checking
•Comparison to Dictionary
– Reuse report, cross-reference spreadsheet, subset “wantlist”
•Generate Schema Bundle with layout styles (e.g. NIEM)
•Realistic test samples generation
•Testing with runtime validation engine (CAMV)
•Business information exchange documentation
•Models – XMI and mind maps
© 2012/13 Oracle Corporation – Restricted Distribution 33
CAM Template Evaluation Report
•The purpose of the CAM template evaluation report is
to provide a suite of checks against a template that
are difficult, tedious or otherwise to determine by
manual inspection of a template structure and its
associated rules and annotation details
•The report covers the following areas:– alerts users to potential issues with their template and
provides suggestions for improving the functional details
– addresses logical and conceptual issues that simple syntax
checking alone does not flag
– can be used as part of a formal review process to ensure a
baseline for quality control
© 2012/13 Oracle Corporation – Restricted Distribution 34
CAM Toolkit Capabilities
© 2012/13 Oracle Corporation – Restricted Distribution 35
Online Tutorials – www.niemtrainingvideos.org
KEY PILLAR OF DICTIONARIESIntroduction and Concepts
© 2012/13 Oracle Corporation – Restricted Distribution 37
The Semantic Framework and Vision
XML/NIEM Components in
Neutral Dictionary
Representation
UML Technologies
XML Technologies
W3C Schema Tools
Semantic Representations
Rules Technologies
HTML 5 rendering tools
Security solutions
Middleware solutions
© 2012/13 Oracle Corporation – Restricted Distribution 38
Dictionaries: Key Pillar of Information Sharing
Canonical XML Components DictionaryCanonical XML Components Dictionary11
Component Associations and CouplingsComponent Associations and Couplings22
Exchange Templates and RulesExchange Templates and Rules33
W3C Schema and Model RepresentationsW3C Schema and Model Representations44
Delivery Control, Messaging, SecurityDelivery Control, Messaging, Security55
Presentation
Collaboration S
ervices
Artefact relationships
Implementation Artifacts and ExamplesImplementation Artifacts and Examples66
Navigation and Query
Tools
Domains Reuse Library
Components
Automatic Rendering Tools
Deployment Environments and Middleware
XML
User Inserts
Relationship Lookups
TemplateCatalog
Testing Workbench, Rules Engine, Data Samples, Integration ETL
SchemaXML
Models Components
Canonical Dictionary Collections
©2012 Oracle Corporation 39
Dictionaries Semantics Approach• Abstract dictionary representations with strong
semantics, rules and code lists support
• Schema, models, documentation are generated from
dictionary definitions automatically
• Contextual mechanisms allow linear number of true
core components – not exponential growth
• Easier for practitioners to leverage with more
predictable and repeatable results
• Enable domain use horizontally across government
• ETL mapping automation support
• Continuous collaboration environment
©2012 Oracle Corporation 40
Developing domain dictionaries (EIEM)
• Allows domains to manage their components libraries• Provides consistency for project development teams
• Sets of NIEM consistent XML exchange components• Aligned to enterprise data stores• Optimized for reuse and interoperability
• Save time and effort across the enterprise• Perennial question for developers – when should I use NIEM
components, and when our own local ones?• Are there components already available for that purpose?
• Provide formal mechanisms and procedures to share components and collaborate across SDLC process
• Provide external parties consistent data views
*EIEM – Enterprise Information Exchange Model
©2012 Oracle Corporation 41
EIEM/BIEC
• Business Information Exchange Components (BIEC)• NIEM-conforming XML schema data
component definition that meets a particular recurring business requirement for an enterprise
• Enterprise Information Exchange Model (EIEM)• NIEM-conforming schemas that
define data components to be reused in IEPDs developed by an enterprise; collection of enterprise BIECs organized into a subset and one or more extension schemas
© 2012/13 Oracle Corporation – Restricted Distribution 42
Managed Dictionaries – Authoritative Sources
Slide 42
Centralized
Distributedschema
schema
schema
SQL
SFIS Values Library Service, NIEM, Metadata Registry
AL
LO
WE
D V
AL
UE
Sschema
Managed Dictionaries
Publish
SUMMARYNext Steps
© 2012/13 Oracle Corporation – Restricted Distribution 44
Why CAM and NIEM?
“Can you give us a brief overview of how you expect IT professionals and line of business managers to use CAM*?”
– Van Ristau, CTO DLT Inc.
*CAM editor toolkit – http://www.cameditor.org and OASIS Content Assembly Mechanism standard – http://docs.oasis-open.org/cam
© 2012/13 Oracle Corporation – Restricted Distribution 45
Response
•Line of business managers can use CAM to drive better
alignment to NIEM and faster more consistent and
predictable delivery of NIEM information exchanges
•CAM provides tools that allow business analysts to work
collaboratively with software developers to build exchanges
•Teams use CAM to perform the 8 “D’s” tasks when
implementing NIEM exchanges
•CAM automates the delivery of key XML artifacts for NIEM
exchange services
•XML validation can be deployed using the CAMV engine
•Open-XDX provides rapid flexible open data web services
© 2012/13 Oracle Corporation – Restricted Distribution 46
Public Sector Open Data Services DeliveryOpen Platform Solutions
• Visual Editor
• SQL Browser
• Dictionaries
• Drag and Drop
• Query/Response
• Generate Data
• Validate Data
• Send Data
• JAX-WS
• soapUI
• CAM Editor
• Content Models
• OER Integration
• Web Forms UI
Open Data Test Bed Services Dictionaries
DB
Web
Open XDX
CAMV
Open-XDX / CAM Editor
Dictionary
Security
SQL
XML
XML
SQL
Repository
• CAM Editor
• Templates
• Policy Rules
• XACML
Templates
XACML
© 2012/13 Oracle Corporation – Restricted Distribution 47
Technology Components and Features
•Providing open data sharing– Open-XDX component accelerates deployment of interfaces
•Aligning meaning and definitions– Collaboration using XML component dictionaries and NIEM
•Formal information verification and content rules– CAMV rules engine provides runtime handling– Industry domain test bed facilities– Rules documentation and sharing
•Business process integration– Oracle BPM and Forms combined with CAMV and Open-XDX
• In progress – Security profiles and policy templates– Upcoming release of CAM Editor and XACML integration
© 2012/13 Oracle Corporation – Restricted Distribution 48
CAMV interactive
testing
Cloud Services Deployment
Open-XDX web
services
XML Technologies
W3C Schema Tools
Semantic Representations
Rules Technologies
Security solutions
Middleware solutions
CAMV test bed for
industries
Registry for Dictionaries
Security Profile
Templates
Web site and collaboration tools
Policy Rules
CAMeditor.ORG Project StatisticsSNAPSHOT OF
PROJECT ACTIVITIES
135,000+ CAMeditor.org page
visits to site
165+ countries have downloaded tools; 30% of visitors are
from U.S.;800+ downloads
weekly
2000+ student views of online video training
resources
8 languages now available
www.cameditor.orgwww.niemtrainingvideos.org
©2012 Oracle Corporation 50