red hat jboss data services platform: overview & roadmap · data virtualization – overview...

Red Hat JBoss Data Services Platform:

Overview & Roadmap

Ken Johnson Director, Product Management Red HatJune 12, 2013

Agenda

Data Virtualization

● What it is

● How can you benefit

JBoss Data Services Product Overview

Demo

Roadmap/Future Direction

Wrap-up, Q&A

Data Virtualization: Overview

Data Virtualization – Overview

Data Virtualization Process● Accessing data from disparate systems (databases,

files, services, applications, etc.)● Integrating and transforming the data, creating

canonical abstraction● Exposing information

● as a single-source● through a variety of data access methods to support different

tools and applications● access is in “real-time” from original sources

● Hide transformation and connectivity details from information consumers

Data Virtualization – Simplistic View

Data Virtualization

Data Virtualization – What it is Not

Time consuming – difficult/costly

No re-use of data logic

Any changes break the application

Hard CodedReplication/ETL

Data not fresh

Costly – additional licenses

More copies of data = more silos

Data Virtualization – Benefits

Why Data Virtualization?● Quickly derive value from your data assets

● Present it to your users in the form they want through the tools they use.

● Build a flexible, agile information infrastructure that lets you quickly respond to change

● Complements existing technologies and approaches (SOA, ETL, EAI)

● Gain control of your information sprawl

Data Virtualization

Data Virtualization – Common Use Cases

Architectural/Technical● Data Layer/Data Hub

● SOA/Data Services

● Virtual data mart/Data warehouse extension

● MDM augmentation

● Cloud data integration

Business Driven● Customer service, Single view

● Compliance, governance

● Analytics, Business Intelligence

● Mergers and Acquisitions

● Mobile enablement

● Operational Dashboards

Red Hat JBoss Data Services

Red Hat JBoss Data Services Platform

Usage pattern: Connect and Compose

● Connect to sources, integrate, federate, transform data

● Create foundational data views

xml

databases

warehouses

spreadsheets

services

<sale/> <value/></ sale >

files

applications

…

Existing sources and silos of data

Integrated set of standard data views

Finance, HR

Inventory, Logistics

Usage pattern: Compose and Consume

● Provide consistent information in the form required by different information-consuming BI/analysis applications, processes, services.

● Ensure complete and consistent information through all delivery modes/formats.

Forms: Relational Tables/Views Star schema Procedures Schema-compliant XML

Access Modes: JDBC, ODBC SOAP, REST Services POJO XML over HTTP

<WSDL><WSDL>(contract)



Custom Apps

Business Process

Packaged Apps

Reports, Dashboards

Analytics

O/R M

app

ing

JD

BC

/OD

BC

SO

AP

/RE

ST

Finance, HR


Red Hat JBoss Data Services Platform

• Turn the data you have into the information you want.

• Standards-based read/write access to heterogeneous data stores in real time.

• Speeds application development by simplifying access to distributed data.

• Transforms data structure and semantics through data virtualization.

• Consolidates data into a “single view” without the need for more data.

• Centralized access control, auditing through robust security infrastructure.

• Creates services that provision data to business process in your SOA.

• Enterprise-proven – flexible, scalable, high-performance.

JBoss Enterprise Data Services

Data Service Data ServiceData View

SQL WebServices

System view and work flow

Tooling VDB Engine Server

System view and work flow - Tooling


Users create data models based on metadata:

●Imported from data sources●Supplied via DDL●Provided by Engine●Specified by user

Models are packaged in a Virtual Database (VDB)

System view and work flow - VDB


Virtual Databases (VDBs) are deployment archives similar to .WAR.

VDBs contain●Source metadata and models●View metadata and models●System metadata●Connection information, which is bound to sources at deployment time

VDBs are deployed to the query engine (Teiid).



Teiid Query Engine is core data virtualization functionality: Federating relational query engine. Rule and cost based optimizer, advanced query planner, caching, hint processing.

Query Engine hosts VDBs, binds to data sources, performs query execution and results processing.



The server runtime environment is JBoss EAP. The Teiid Query engine is hosted in EAP and uses key container-provided services:● Transaction manager● JAAS security framework● Container managed data sources● EAP management infrastructure● EAP deployment

The Server: exposes views/services to consumers and managed connections and connection pools for data sources.

Caching/Materialization for Performance

● Multiple levels of caching to meet performance requirements and manage load on source systems.

● Fully automatic to fully user controlled.

Repository/Governance

S-RAMP: SOA Repository Artifact Model and Protocol

● OASIS specification that defines: ● a common data model for repositories, as well as ● an interaction protocol to facilitate the use of common

tooling and sharing data.● S-RAMP repository capabilities:

● Store and retrieve content and metadata● Classification of artifacts (e.g. XSD, WSDL, VDB, ...)● ATOM API

Repository/Governance

Core Models storing standard metadata and the file content itself

Derived Models are read only and data is derived from the content

XPath2 based QL

Content extractors/derivers

Clients interact via:

● ATOM/REST

● Java via S-RAMP client

● S-RAMP browser (web-based)

● Maven

● VDB (query/read)

ATOM BINDING (REST)

Derived Models

(Read Only)

Core Model Documents

JCR Storage (Modeshape + Infinispan)

Demo Scenario

Text with no bullets

● Bullets layer one● Bullets layer two

● Bullets layer three

Product Roadmap and

Future Direction

Data Services Roadmap: JBoss DS Platform v5 -> JBoss DS Platform v6

Current: DS-P v5 Next: DS-P v6

● Simplify packaging: bundle with EAP rather than SOA● EAP 5 base --> EAP 6 base● Honor customer investment, provide SOA-P to existing users● Timing of EDS-P v6:

● Alpha: July 2013● Beta: Aug 2013● GA: Q4 2013

JBoss DS Platform v5

BPEL Rules

EAP v5

JBoss ESB

Dev Studio

+SOA

+Datatools

JON+

EDSPlugins

+SOA

Plugins

Managed

UDDI

jBPM3

Dev Studio

+DataTools

JBoss DS Platform v6

JDBC/ODBC

EAP v6

JON+

EDSPlugins

Managed

ReposData Virt/Teiid

ReposData Virt/Teiid

JDBC/ODBC

Select Upcoming New Features

● DDL Based View Definitions

● VDB import/reuse

● VDB deployment updates

● Native Queries

● OData Support

● Source temp tables

● Source security domains

You can access these features in Teiid community releases

● Procedure Exception Handling

● NoSQL sources

● OBJECTTABLE

● JSON Production

● Column masking

● Teiid Designer 8.x

● Dashboard Builder

DDL-based Dynamic View Creation

● Unique among data virtualization technologies/vendors

● Programmatic configuration

● In Teiid 7.7/EDS 5.x, DDL could only specify integration between multiple physical sources.

● In Teiid 8/EDS 6, DDL can be used to specify view definitions.

DDL based Dynamic View Creation

<?xml version="1.0" encoding="UTF8" standalone="yes"?><vdb name="twitter" version="1"> <model name="twitter"> <source name="twitter" translatorname="ws" connectionjndiname="java:/twitterDS"/> </model>

<model name="twitterview" type="VIRTUAL"> <metadata type="DDL"><![CDATA[ CREATE VIRTUAL PROCEDURE getTweets(query varchar) RETURNS (created_on varchar(25), from_user varchar(25), source varchar(25), text varchar(140)) AS select tweet.* from

(call twitter.invokeHTTP(action => 'GET', endpoint =>querystring('',query as "q"))) w,

XMLTABLE('results' passing JSONTOXML('myxml', w.result) columns created_on string PATH 'created_at', from_user string PATH 'from_user', source string PATH 'source', text string PATH 'text') tweet;

CREATE VIEW Tweet AS select * FROM twitterview.getTweets; ]]> </metadata> </model></vdb>

DDL Support

● Creation of Views

● Creation of Virtual Procedures

● Creation of Functions

● Like SQL/MED, but more feature rich

VDB Import/Reuse Feature

VDB 1

VDB2

RDBMS

Flat FileFlat FileFlat File

ClientApplication

Services

VDB1

VDB2

combined

VDB Import/Reuse

VDB 1

VDB2

RDBMS

Flat FileFlat FileFlat File

ClientApplication

Services

combined

VDB Imports/Reuse

● Enables and promotes enterprise wide canonical data model

● Hides the complexity from application developers

● Improves performance

● Treats it as single source

● Materialized/cached data from individual VDBs is shared with others in combined VDB

Native Query Support

Allow pre-defined or “canned” queries to be passed to sources directly. Take advantage of source-specific behavior.

Two Forms

● Source Table metadata

● Native() procedure

Native Query Support

Source Table Metadata● Specified as metadata property on source table:

● nativequery="select c from g"

● Produces source query with inline view:

● "SELECT c FROM (select c from g) as x"

Native() Procedure● Allow ad-hoc native SQL to be passed to source. Metadata structure

defined at execution.

● SELECT x.* FROM (call pm1.native('select * from g1')) w,

ARRAYTABLE(w.tuple COLUMNS "e1" integer , "e2" string) AS x

OData Support

OData (OASIS Open Data Protocol)https://www.oasisopen.org/committees/tc_home.php?wg_abbrev=odata

Objective: OASIS OData TC works to simplify the querying and sharing of data across disparate applications and multiple stakeholders for re-use in the enterprise, Cloud, and mobile devices. A REST-based protocol, OData builds on HTTP, AtomPub, and JSON using URIs to address and access data feed resources. It enables information to be accessed from a variety of sources including (but not limited to) relational databases, file systems, content management systems, and traditional Web sites.

Data Services v6 supports Odata in two ways:

● Connect to and access Odata sources

● Act as an Odata server to client applications

Dashboard Builder

New component/application from the Polymita acquisition

● Generalized beyond BPM and BAM use cases

● Planned for inclusion in Data Services v6

● Provides the ability to create dashboards for visualizing data

Dashboard Builder

Product

Interoperability

Data Services and BRMS:Reasoning: with Business Rules

● Capture and codify key decision-making logic in business rules

● Facilitate collaboration between business analysts and IT

● Process information actively

Rule sets possibilities: Pricing Fraud detection Regulatory compliance Productivity/Efficiency Control systems Product configuration...

Insurance Rules:

Age

Sex

Health

Occupation

Lifestyle

= $ Price

Data Services and BRMS:Reasoning: with Business Rules

● Expose virtual data views through JDBC and Hibernate.

● Rule engine accesses Hibernate session to retrieve records and turn them into facts over which the engine can reason.

Hib

ernate

JDB

C

Finance, HR


Facts

Results

Data Services and JBoss Data Grid:Distributed cache as a data source

● Treat JBoss Data Grid as a data source

● Combine that with other data using standard product principles

databases

warehouses

services

applications

Existing sources and silos of data

Integrated set of standard data views

Inventory, Customers

JBoss Data Grid

Wrap-up

Red Hat JBoss Data Services – Business Value

✔Increased ROA

✔Greater agility, faster time to solution

✔Improved organizational performance

✔Better control of information

Improved utilization of data assetsDerive more value from existing investments

Complements existing systems

Better/faster than hand codingFaster, less costly than batch data movement

Data virtualization provides loose coupling

Right data at the right time to the right peopleDecision support, BI with a complete view of

information

Powerful security, Auditing, Data FirewallAvoid data silo proliferation

Central data access and policy, Compliance

Red Hat JBoss Middleware

MIDDLEWARE

Foundation

Dev

elop

men

t Too

ls

Accelerate

Man

agem

ent T

ools

Data Virtualization

Application Integration

Integrate

Business Process Management

User Interaction

Automate

JBoss EAP JBoss Web Server JBoss Data Grid

JBoss Data Services

JBoss Fuse JBoss A-MQ Jboss SOA

JBoss BRMS

JBoss Portal JBossDeveloper Studio

JBossOperations Network

Related Sessions

Wednesday

10:40 - Mainframe & Database Legacy Modernization

3:40 - Red Hat Storage & Big Data

4:50 - Get Mobile-ready Fast with Red Hat JBoss Data Services Platform

4:50 - Town Hall Discussion: Big Data & Traditional Databases

Thursday

10:40 - When Big Data Goes Beyond Hadoop

3:40 - Hadoop on Red Hat Storage Server: A Reference Architecture

Friday

9:45 - Interoperability Results from SAP & Red Hat Collaboration

9:45 - Design Business Intelligence Appliances Using Red Hat Technologies