1

Enabling Data as a Service with

JBoss Data Services

Prajod Vettiyattil

Twitter: @prajods

Gnanaguru Sattanathan

Twitter:@gnanagurus

Website: bushorn.com

2

What this session is about

The why and what of data services How data services work Use cases JBoss Data Services Platform

3

Why

4

Proliferation of data

SQL File Mainfra

me NoSQL Email

Content

Manage

ment

System

Custom

er portal Employee

portal ERP CRM Accounting Billing

ERP

Vendor

Management Partner

Management Sales Finance Marketing

Data Consumers

Data Sources and Data Managers 5

Proliferation: so what ?

6

• Multiplicity of connections – High development cost – Huge operational overhead – Difficult and risky to change Data Sources/Managers

• Dispersed data connectors • Data duplication

– Too much ETL – Lines of Business copies data

• Duplicated data aggregation • Impossible to create “Single source of truth” • Data ownership issues • No comprehensive view

– No data movement dashboards – Location of data and its status

What

7

Data Services and DSP The basic view

• DSP = Data Services Platform

• Abstracts the data

managers/sources

Data Managers

Data Consumers

Data Services Platform Data

Service 1

Data

Service 2

Data

Service 3

Data

Service 4

C1 C2 C3 C4 C5 C6

D1 D2 D3 D4 D5 D6

8

• Presents the data as a service to the

consumer

• ETL++

Dashboard in a DSP

Data Dashboard

Data

Connections

Data

movement

status

Errors

Error

Corrections Failures Alerts

9

How it works

10

Features of a DSP

• Enables architecture principles – Separation of concerns – Protected variations

• Data adapters • Data mapping tools and standards • Data caching

– Local and distributed

• Service search and reuse • Data security and data usage audit • Data access control • Central channel for all data requirements • Data dashboard • Configurable performance and reliability

11

Use cases

12

Auto manufacturing supply chain:

Requirements

• Vehicle ownership experience • Business Process Automation • Disparate data sources • Multiple data feeds

– Parts catalog – Prices

• Dealer updates – Parts consumed – Parts replaced – Part failure statistics

• Customer feedback – Post purchase – Breakdown support – Service Quality Dashboards

• Integration solutions based on batch transfers – Unreliable – Not traceable

13

Auto manufacturing supply chain:

Layer Diagram

14

Business Processes

Data Services Platform

Customer feedback

Customer Master

Parts Catalog

Dealer feeds

Dealer Info

Business Activity Monitoring Customer Experience Dashboards

Breakdown reports

Parts supplier

feeds

Enterprise Service Bus

Enterprise Data Access Layer: Requirements

15

• Golden copy / System of Record / Single source of truth

• Shared services team for Enterprise Data Management

• Data usage audit • Data access control • Reduce request load on Data Management team • Reduce data maintenance costs

Enterprise Data Access Layer:

Layer Diagram

16

Enterprise Data Consumers

Data Services Platform

Mainframe

Data Services

Customer Master

Partner Data

Content Management

System

Data Access Control

Data base drivers

Auditing Data Aggregation

Partner Info

Employee Info

Virtual DB Metadata

Reporting risk for securities:

Requirements

• Internal and external reporting – Risk and margin

• Centralized risk capture and management • Calculate risk from different customer activities • Report consolidated data to comply with regulation

– Dodd Frank – Sarbanes Oxley Act (SOX)

• Dashboards for higher management

17

Reporting risk for securities:

Architecture without DSP

18

COTS Trading Systems

Enterprise Middleware Systems (MQ, ESB, FTP, File shares)

Risk Management

Settlement Ref Data

Mgmt Custom built

Apps

Customer facing Apps

Partner Apps Government

Systems

Price feeds

Accounting

Trade feeds

Reporting Applications

Payment Systems

Margin Mgmt Trade

Matching

Order Mgmt Execution

Mgmt Liquidity

Mgmt Position Mgmt

Order Book

Clearing Ref Data

Feeds

Reporting risk for securities:

Patterns in this requirement

• Regulatory requirement for transparency – Cannot be met by opaque internal systems

• Data Sources – Large number of them – Internal and external

• Reports are read heavy • No real time data requirements

– once a quarter or once a year

• No excuses for incorrect data in reports • Non-discretionary spending

19

JBoss Data Services

Platform

20

Architecture

21

• The EDS platform – v5 Runs on SOA-P

• Teiid • ModeShape

Data Adapters

Data interfaces

(JCR API, Web service, JDBC, ODBC, OData,..)

Data virtualization Metadata

repository

SAP Sybase Flat file SalesFo

rce

Oracle XML

Data Sources

Data consumers

(Custom Applications, COTS products, Business Processes, Business Services )

Cassan

dra

Mongo

DB

Data Services Platform

SOA Platform

• Parts of the architecture – Data interfaces – Data adapters – Data virtualization – Metadata repository

Oracle

DB IBM

DB2

MS

SQL

Server

MySQL PostgreS

QL Sybase

Greenpl

um Teradata Netezza Ingres Mondria

n MetaMa

trix

LDAP Salesfor

ce

Delimite

d file XML

file

Web

services Apache

Hive

MS

Excel MS

Access

JBoss

Messagi

ng

JBoss

HornetQ

TIBCO IBM

MQ

22

Data sources

Data Mapping

• Teiid Designer – Map actual data tables using transforms to virtual

tables – MDD; use Data Models, not SQL – Semantic mapping – Virtual procedures

• A set of SQL statements, similar to DB stored procedures

23

Data Standards

• JCR – Java Content Repository(JSR-283)

• OData – Open Data Protocol

• JDBC • ODBC • Others

– S-RAMP – An SOA repository spec, OASIS

– Web Services – REST – JMS

24

Access control and Audit

• Teiid

– passwords

– MembeshipDomains for authentication

– Data roles

• Fine grained access and visibility control of tables

– CRUD level permissions for VDB

– LDAP integration

• ModeShape

– LoginContext

– AuthenticationProvider

– Role to Action mapping

25

Teiid and ModeShape Data type Teiid ModeShape

Approach Relational Hierarchical

Metadata repository Not suitable Yes

Content repository Not suitable Yes

ACID transactions Yes Yes

SQL queries Yes Yes(JCR-SQL)

Flat file data source Yes Not suitable

Relational DB data source Yes Not suitable

Schema Fixed Optional

NoSQL data sources Not suitable Yes

Stores data No Yes

26

Summary

• Data Services

– Why

– What

– How

• Use cases

– Auto Manufacturer

– Enterprise Data Access Layer

– Regulatory Reporting

• JBoss DSP

– Data virtualization

– Teiid

– ModeShape 27

Questions

28

Our Open Source Middleware Group on LinkedIn

http://tinyurl.com/be6e93q

Prajod Vettiyattil

Twitter: @prajods

Gnanaguru Sattanathan

Twitter: @gnanagurus

Website: bushorn.com