Chuck MosherDirector, Federal Architecture &

Solutions

Abstraction As an Organizing Principle for Agile Enterprise Architectures

3rd SOA for E-Government ConferenceMay 2, 2007

2

Agility is Key

• Agility is the right precept

• Change is a constant– Technologies– Requirements– Standards– Models– Processes– ???

3

The Business Inflexibility Trap

• Inflexibility is the Mother of All Business Problems

• If you’re flexible enough, you can solve all the other problems

Ron Schmelzer, ZapThink

4

Multiple Dimensions to Address

• Governance– Leadership and oversight

• Policy– Rules for sharing, regulations, authority

• Technology– Systems, protocols, standards, security

• Culture– Potentially the most challenging issue

• Economics– Incenting the correct behavior

5

Focus of this talk

• Governance– Leadership and oversight

• Policy– Rules for sharing, regulations, authority

• Technology– Systems, protocols, standards, security

• Culture– Potentially the most challenging issue

• Economics– Incenting the correct behavior

6

Progressive Evolution of Solutions

• IT was supposed to help! And it has:– Assembly Language– Programming Languages, Procedural– Relational Databases– Programming Languages, OO– Messaging Frameworks– CORBA, IDL, RMI, UML– Java, J2EE, .NET, Application Servers– HTTP, XML, SOA, ESB, BPEL

Magic Bullets, All!

7

Magic Bullets, Current Crop

• XML

• Service Oriented Architecture (SOA)

• Enterprise Service Bus (ESB)

• Standardized Data Interchange Models

The Evolution of Computer Science has been as a result of more and more sophisticated thinking about appropriate levels of abstraction in how to solve problems in information management, and the technologies that result.

8

XML

• XML provides a transport-independent abstraction to interchange data between applications and across the enterprise

• Self-describing data• Human and machine-readable• Easily extendable• No more proprietary data formats

• It does not, however, address the difficult problem of accessing highly distributed data assets and resolving the many disparate structural and semantic differences between them

9

SOA

• SOA provides an effective abstraction to encapsulate business functionality & provide interoperability across the enterprise

• Decouples data from applications & user views• Standardized interfaces improves info sharing• Leverage existing investments

• It does not, however, address the difficult problem of accessing highly distributed data assets and resolving the many disparate structural and semantic differences between them

10

Enterprise Service Buses

• ESBs define an abstraction for machine and OS-independent messaging

• Pub/Sub, Point-to-Point, Content-based routing• Highly fault/failure-tolerant, assured delivery• Extensive logging/auditing capability• Complex event processing

• They do not, however, address the difficult problem of accessing highly distributed data assets and resolving the many disparate structural and semantic differences between them

11

Standardized Data Models

• Enterprise-wide or COI-specific data abstraction definitions

• Simplifies information sharing between communities

• Many examples: GJXDM, NIEM, C2IEDM, JC3IEDM

• Concept of COI very powerful

• They do not, however, address the difficult problem of accessing highly distributed data assets and resolving the many disparate structural and semantic differences between them

12

Information Sharing – THE Problem

• “It’s all about the data” – Hon. John Grimes, ASD(NII)/CIO

• “It’s all about the data” – Mr. Dave Wennergren, Deputy CIO, DoD

• “The biggest challenge is the data” – BGN Gary Salisbury, US European Command (ret)

• “It’s not the chain of command, it’s the chain of information” – Gen Cartwright, US STRATCOM

• The central concern of NCES• Thesis: If we can solve the data problem with

agility, adaptability, and cost-effectively, we have solved the most difficult (technical) problem

13

We Need One More Abstraction!

• A virtual data repository layer focused on turning the data we have into the data we want

• Data as an abstract service– Accessible in multiple formats (XML, relational)

• An abstraction that provides access to diverse distributed resource stores– Don’t move the data, virtualize it

• Middleware handles federation, structural, and semantic reconciliation to a reusable data abstraction

• Create agility precisely where it is needed, at the data layer

14

Information Virtualization

Information Resources

Information Consumers

Abstract Data TierInformation Virtualization Layer

15

Information Virtualization Layers

Unified Semantic Layer

Information Virtualization

Data Federation Layer

Data Access/Connectivity Layer

Enterprise Data Sources

Unification of different system conceptsSingle-query access to heterogeneous systemsUniform, standardized access to any system

16

Data Service Layer in SOAClient Process & Applications

Data Sources

Data Services Layer

Message Services (ESB)

Business Services

Business Process Services

App App App App App App

Data Service Data Service Data Service Data Service Data Service

17

Designing data services

Creating a Virtual Data Layer

xml

databases

warehouses

spreadsheets

services

<sale/> <value/></ sale >

geo-spatial

rich media

…Enterprise Enterprise Information Information

Sources (EIS)Sources (EIS)

Information Information ConsumersConsumers

Reusable,Reusable,Integrated Data Integrated Data

ObjectsObjects

ExposedExposedDataData

ServicesServices

<WSDL><WSDL>(contract)

<WSDL><WSDL>(contract)

<WSDL><WSDL>(contract)

Custom Apps

Web Services,Business Processes

Packaged Apps

Reporting, Analytics

EAI, Data warehouses

OD

BC

JDB

CS

OA

P

Intelligence

CRM, Logistics

18

Multiple layers of abstraction

19

Not Just a Notional Abstraction

• Multiple vendors provide a data management layer that efficiently and reliably enables federated access to a wide range of heterogeneous data sources

• Sophisticated security controls• Standards-based• Model-driven development

– Model, not code, your way to agility– Rich metadata management

• Data abstraction layer can concretize enterprise or COI-specific data models

20

MetaMatrix Integration Server

Information Consumers

Web Svc XML RDBMSPackaged Connectors

Siebel,SAP

OracleApps

CICSVSAM

MetaMatrix Catalog

MetaMatrix Designer- Design and deploy data services

MetaMatrix Product Architecture

JMSODBC JDBC SOAP

QueryProcessor

ProcessorProcessorOptimizerOptimizer

Integration ServerVirtualDataBases

VDBVDBVDBVDB

IntegratedSecurity

UsersUsers

RolesRoles

EntitlementsEntitlements

AccessModels

ViewsViews XMLDocsXMLDocs

<a>

</a>

<b>

</b>…

ServicesServices

in outproc

MetaMatrix Connector Framework

MetaMatrixServer

21

T

Authoritative Sources:• Mapped to logical

Multiple Internal/External Information Sources

Application views of information:

• Relational, XML

T T

XML Document<a>

</a>

<b>

</b>…

T

TT

ODBC/JDBC JDBC SOAP

WebServices

WebServices

Search Applications

Search Applications

BusinessIntelligence

Applications

BusinessIntelligence

Applications

Logical Data Model:• Agency or COI-specific• Rationalize, harmonize,

mediate

C2, Logistics, Intelligence, …

Data Abstractions and Data Models

bldg_id SITENUM Facility_ID

Location_ID

bldg_type Depot_Number

Location_Type

Semantic Data Services!

22

23

24

25

Metadata-aware Applications

Metadata management & federation

XML Models

RDBMS

MetaMatrix Repository

Metadata Sources

• Establish data dictionary• Define integration paths• Documentation• Version Control

Impact Analysis

Virtual Metadata

Metadata Search& Reporting

ModelingTools

MetadataRepositories

MetaMatrixServer

MetaMatrix Designer

26

Impact Analysis

27

Search Engine Index / Metadata Catalog

Master Data Person / Facility / Vehicle

MetaMatrix Enterprise

Stage SOAApp’s

Federal Agencies

Data Access Services• SQL, Web Service/XML• Staged Data (optional)

OntologyMgmt /Reasoning

Enterprise Service Bus

MetaMatrix Dimension

Local Commands

State/Local• Security/Authentication• Operations Management • Error / Exception Management

• Orchestration• Encryption• High Availability

MediationXSLT, Multi-source

Information Exchange Topology

28

US Defense Logistics Solution

• Global Combat Support System (GCSS)– Mission: supply the war-fighter with access to

accurate and timely logistics information

• Focused Logistics– Fusion of information technologies to enable forces of

the future to be more mobile and versatile – Provides the joint war fighter with a single capability to

manage and monitor units, personnel, and equipment– Deployed at 23 sites around the world– Networked environment allows DoD users to access

shared data & applications, regardless of location

29

DISA GCSS-J IDE = Abstract Data Layer

30

CFDB

CSDS DMDC

GSORTSIDE/AVNGA

FLIS

CSDS_PL

CSDS_VBL

Facilities_VMLMaterial_VML

Facilities_VQLMaterial_VQL

GDSS

Plans_VQL

Pri

vate

Dat

a an

d M

etad

ata

Pu

bli

c D

ata

Virtual Mid Layer (VML)

Virtual Query Layer (VQL)

(Exposed Views)

JOPESClassic

JOPES4.0

Virtual Base Layer

(VBL)

Physical Layer

(PL)GTN

GCSS Modeling Approach

31

MetaBase

• Data dictionary• Integration paths• Portal metadata

Intelligence Agency: Information Portal

Data Host(s)

Managed DataManaged

DataManaged Data

MissionDataMission

DataMissionData

MetaMatrixQuery Engine

Information IntegrationHost(s)

Application ServerHost(s)

Query Service(Business Layer)

Presentation Layer

Query Page

Control

ResultsPage

Control

FormPage

Control

Browser

Browser

Client(s)

T

Bottom-up(harmonization)

<X>

</X>

<X>

</X>

<X><X>

Top-down (mapping)

Org, Person, Image,

Location

Portal:• Entities• Components• RelationshipsMetadata:• Name• InformationContext• Usage• Description• Display Name• Default Value• Attribute Units• Presentation Type• Sort Order• Visible

ExternalFeeds

…

32

DCGS-A Services Network

En

terp

rise S

erv

ice B

usMetadata

DiscoveryService

SIGINTGateway

Service

MetadataPublishing

Service

MetadataCatalog

AlertSubscription

Service

EventAssessment

Service

WeatherEffect

Service

IMETS(IWEDA)

E-SpaceServices

WeatherEffects

EWData

AlertCriteria

Alerts /EventsMetadataMetadata

Searches

InfrastructureInfrastructure ServicesServices

ISR DataListenerService

Async“Callbacks”

Filters

WorkflowEngine

ServiceManagement

HUMINT Data

Service(s)

HDWS (CHAMS)

Map /Coverage

Google EarthRich Client

Handheld

NCES Service

Discovery

TransformationEngine

BC GatewayService

ForceTracking

MIP Blue Force Tracking

Intelligence Architecture: MetaMatrix & ESB

Google EarthRich Client

33

Long Term Software Cost Comparison

Competitive (it’s the business model!)

MediumHigh

ZeroMediumHigh

0++

++++

Competitive (it’s the business model!)

MediumHigh

ZeroMediumHigh

0++

++++

SI’s COTS Fully Supported Open Source

Code Quality

Profit Margin/Cost

Support

“Proprietary-ness”

34

Summary

• Leverage the power of abstraction to simplify, clarify, and realize your EA vision– Abstraction is how you design for change– Be very aware of the level you are working at

• Leverage the increasing power that COTS and open source tools can provide

• The rubber meets the road at the data layer– That you have, not what you want– Abstract it, don’t move it!

• Model, not code your way to agility! -RS

• Metadata management is a must-have– For repositories too

35

Unacceptable Risk???

36

Not if you know the lay of the land