jboss enterprise data services (data virtualization)

1. 1 JBoss Enterprise Data Services Peter Larsen JBoss Solutions Architect, Red Hat Inc. [email protected]

2. 2 Agenda Motivation for Data Services EDS in the community / History Positioning Data Services in an Enterprise Architecture Use Case Domains / Customer Examples Technical Architecture Demo

3. 3 The Business Inflexibility Trap Inflexibility: The essential Business Problem With agility all problems are solvable With enough eyes, all bugs are shallow

4. 4 Agility is Key Because change is the only constant Technologies Requirements Regulations Standards Models Processes ??? What is the single most important problem preventing Agility?

5. 5 Problem: Data Challenges Challenges Different physical structure Different terminology and meaning Different interfaces May need to federate/integrate May be locked in to database Must ensure performance Maintain/Improve security Tremendous value in existing information assets, but... Time consuming and costly to implement new applications that leverage this information Data Warehouse Packaged Applications Operational Data Stores Data Gap

6. 6 Problem: Data Challenges Alternatives Time consuming difficult/costly No re-use of data logic Any changes break the application Data Gap Data Gap Hard Code Replicate/Data Mart Data not fresh Costly additional licenses More copies of data = more silos Governance/security

7. 7 Solution: JBoss Enterprise Data Services JBoss Enterprise Data Services Data Service Data ServiceData Service SQL Web Services Access to multiple data stores in real time Standards-based read/write access Speeds application development Transform data structure and semantics Consolidates data into a single view Centralized access control Enterprise-proven flexible, scalable, high-performance Turns the Data You Have Into the Information You Need

8. 8 Data Services Platform Where it fits JBoss Enterprise Data Services Platform Other Vendor Portal / ESB/SOA Platforms Data Service Data Service Data Service Data Service Data Service

9. 9 Integration Technologies ETL SOA EDS Data ProcessIntegration Style DataIntegrationTimeliness Real Time Batch

10. 11 Data Services Platform Common Use Cases Service Oriented Architecture Federate/transform data efficiently use by higher-level services Insulate business processes from data access details Business Intelligence, Operational Analytics, Reporting Consolidated financial reports/dashboards/KPIs Virtual data marts Information Consolidation, Reference Data Management Single/360 view of Customer Single/360 view of Supplier Single/360 view of Employee Regulatory Compliance Provide common security, central access and auditing of data VISA PCI, Sarbanes Oxley, Basel II, HIPAA

11. 12 JDBC/ODBC Query Engine Data Virtualization, Federation JBoss Enterprise Data Services JBoss ModeShape Repository Services JBoss jBPM JBoss Rules JBoss Enterprise SOA Platform JBossESB JBoss Enterprise Application Platform Red Hat Enterprise Linux Windows, UNIX, other Linux Turns the data you have into the information you want Augments and extends SOA Platform to address data access, integration and abstraction. SOA Patterns, best practices Reporting/Analytics enablement Information Consolidation, Data Mgmnt Data Governance, Compliance Real-time read/write access to heterogeneous data stores Speeds application development by simplifying access to distributed data Centralized access control, auditing JBoss Enterprise Data Services

12. 13

13. 14 Data Services Platform Architecture

14. 15 Query Performance & Optimization Minimal overhead for simpler requests Control enforce mandatory criteria with certain requests enforce time and size limitations on requests Rule-based optimization use criteria to avoid unnecessary fields and records removal of unnecessary joins across data sources merge all transformation logic for a single source Cost-based optimization join algorithms (nested loop, merge, dependent, hash) cost profile of each data source Data caching and staging (materialized views) Manage dataflow buffer management

15. 16 Designer Tooling Virtual Models Physical Models representing actual data sources Shows structural transformations Defines transformations with Selects Joins Criteria Functions Unions User Defined

16. 18 Semantic Mediation/Integration T Authoritative Sources: Mapped to logical view Multiple Internal/External Information Sources Application views of information: Relational, XML, Java T T XML Document T T T Web Services Web Services Workflow/ ESB Workflow/ ESB Business Applications Business Applications Claims, Billing, Policies, bldg_id SITENUM Facility_ID Location_ID bldg_type Depot_Number Location_Type Semantic Data Services Data Dictionary: Based on logical data model or XML schema Support for multiple COIs Support for multiple versions

17. 19 Data Services and the ESB User-facing Logic (Service Consumers) Business Logic Data Logic Process and Other Integration Logic Rich or Thin Desktop Process, Integraion Services Business Services ESB Direct ODBC, JDBC ODBC, JDBC WSDL, SOAP, MOM, other WSDL, SOAP, MOM, other Process Orchestration Services Data Services

18. 20 JBoss Enterprise SOA Platform Enables Business Process Automation by integrating and orchestrating application components and services running on JBoss Enterprise Middleware and/or any other standards-based AS Single distribution that integrates JBoss ESB, jBPM, JBoss Rules, Enterprise, Application Platform Enables multiple integration styles: SOA integration, EAI, EDA, process and business rules technologies to automate business processes to improve business productivity Certified Platform for Service Integration and Orchestration Simple, Flexible, and Scalable Light footprint, simple installation JBoss ON platform management and services monitoring Scalable clustering to support high transaction volumes A flexible, standards-based platform to integrate applications, SOA services, business events and automate business processes. Red Hat Enterprise Linux Windows, UNIX, other Linux Workflow Rules JBoss Enterprise SOA Platform JBossESB Transformation, Routing, Registry JBoss Enterprise Application Platform Container services, Hibernate, Web Services stack, Seam, Clustering, Cache, Messaging, Transactions

19. 22 Enterprise Data Services 5.2 EDS 5.2 Released December 2011 Tighter data services/ESB tooling integration Performance tweaks LOB handling Cost-based optimizer enhancements Programmatic view creation Repository enhancements Versioning support More artifact types Cloud-based data sources Fixes, minor enhancements, additional platform certs

20. 23 Business Value of Enterprise Data Services Greater agility, faster time to solution Increased ROA Improved organizational performance Better control of information Improved utilization of data assets Derive more value from existing investments Complements existing systems Jumpstart Your SOA Initiatives! Better/faster than hand coding Faster, less costly than data replication Data virtualization provides loose coupling The right data at the right time to the right people Decision support, BI with a complete view of information across the enterprise Powerful security, Auditing, Data Firewall Avoid data silo proliferation Central data access and policy, Compliance

21. 24 A Comprehensive Middleware Portfolio JBoss Enterprise Data Services Platform JBoss Enterprise SOA Platform JBoss Enterprise Application Platform JBoss Enterprise Web Platform JBoss Enterprise Web Server Red Hat Enterprise Messaging JBoss Enterprise Portal Platform JBoss Enterprise Business Rules Management System JBossDeveloper Studio Seam Hibernate WebFramework Kit JBoss Operations Network RedHatServices Cloud Implementation Cloud GovernanceCloud Strategy & Selection VMWare Microsoft Hyper-V Red Hat Enterprise Virtualization PrivatePublic Amazon EC2 Other RHEL,Unix,Windows

22. 25 Where did Teiid come from? Project lineage is from MetaMatrix starting in ~1999. Teiid - http://www.jboss.org/teiid Teiid Designer - http://www.jboss.org/teiiddesigner DNA - http://www.jboss.org/dna/ MetaMatrix was the leader in Enterprise Information Integration (EII) hence Teiid. Red Hat acquired MetaMatrix in 2007. Last major MetaMatrix product release, 5.5.4 11/09

23. 26 Project Status (March 2011) Open source 2/2009 heavily refactored from 5.5 line 7.0 Initial release 6/2010 7.1 Teiid / Teiid Designer release 8/2010 Basis for EDS 5.1 release with hundreds of issues resolved and targeted enhancements 7.4 Coming Soon! More source integration (MDX via XMLA, Ingres), expanded function support, etc. should be picked up along the work in 7.1-3 by the next service pack release.

24. 27 Community Version Community web site: www.jboss.org/teiid Teiid sub-projects: Teiid Runtime, Teiid Designer Teiid 7 is built for AS7

25. 28 JBoss Enterprise SOA Platform (SOA-P) and Enterprise Data Services (EDS) Roadmap Q4 08 Q1 09 Q2 09 Q3 09 2.8 SOA-P 5.0 Q4 09 3.0 Q1 10 Q4'09 Q1'10 Q2'10 Q3'10 Q4'10 Q1'11 Q2'11 Enterprise Q3'11 Q4'11 Q1'12 Platform Release March 2010 Calendar Quarters Platform Release January 2011 (target) SOA-P 5.1 Platform Release January 2011 (target) SOA-P/EDS 5.2 EDS 5.1 Platform Releases Mid-Q2 2011 (target) MetaMatrix 5.5.4 Platform Release November 2009

26. 29 Generating New Value with JBoss Enterprise SOA Platform and Enterprise Data Services

27. 30 The context Organizations Significant assets already deployed or otherwise in use Applications, databases, services, spreadsheets, file extracts, manual processes, tribal knowledge Not realizing full benefit Mandate Remove business impediments, improve status quo Control/reduce costs Derive greater value from the assets you already have

28. 31 Common Challenges Data Decision making Inflexible systems Manual processes

29. 32 Common Challenges Data Data sprawl Tied up in silos Not reconciled/integrated Not easily usable Decision making Inflexible systems Manual processes

30. 33 Common Challenges Data Decision making Insufficiently informed Missing key information Stale or out-of-context information Inflexible systems Manual processes

31. 34 Common Challenges Data Decision making Inflexible systems Logic hard-coded into applications Redundant logic, not standardized or shared Changes require development cycle, resources, time Unable to react quickly to business, market, IT changes Manual processes

32. 35 Common Challenges Data Decision making Inflexible systems Manual processes Business processes are manual Data entry, swivel-chair integration Overly dependent on individuals Inconsistent, prone to error, difficult to govern

33. 36 Common Challenges Data Decision making Inflexible systems Manual processes But... These data, systems, applications, decision-making processes, business processes and logic are your current assets waiting to be improved and put to better, more effective use. How?

34. 37 Solution Patterns 1. Pattern: Data Foundation 2. Pattern: Information Delivery 3. Pattern: Externalize Knowledge 4. Pattern: Automate Decision Making 5. Pattern: Codify Business Processes

35. 38 Solution Patterns: Data Foundation Liberate, integrate, mediate, transform data Tap silos, gain control over data sprawl Create foundation data layer through data virtualization xml databases warehouses spreadsheets services sale > files applications ExistingExisting sources andsources and silos of datasilos of data Integrated setIntegrated set of canonicalof canonical data objectsdata objects CRM, Employee SupplyChain, Logistics

36. 39 Solution Patterns: Information Delivery Provide consistent information in the form required by different information consuming applications, processes, services. Ensure complete information through all delivery modes/formats. Forms: Relational Tables/Views Star schema Procedures Schema-compliant XML Access Modes: JDBC, ODBC SOAP Web Services POJO XML over HTTP, JMS (contract) (contract) (contract) Custom Apps Business Processes Packaged Apps Reports, Dashboards Data warehouses O/RMappingJDBC/OSOAP/JMS CRM, Employee SupplyChain, Logistics

37. 40 Solution Patterns: Externalize Knowledge Externalize key business logic from application code Isolate and standardize rules that govern business decisions and operations Enable business analysts and development to collaborate in defining functional behavior Rule sets possibilities: Pricing Fraud detection Regulatory compliance Productivity/Efficiency Control systems Product configuration ... Insurance Rules: Age Sex Health Occupation = $ Price

38. 41 Solution Patterns: Automate Decision Making Move beyond reports to active analysis and decision making Extend rule sets to analyze information provided through earlier patterns. Process information on scheduled basis or dynamically as data is flowing through applications and on the bus. Raise alerts, initiate corrective actions, seize opportunities sale >

39. 42 Solution Patterns: Codify Business Processes Codify the processes actually followed by your organization Create standardized, reusable workflows/orchestrations Eliminate unnecessary manual steps, keep human tasks only where appropriate. Identify common business patterns both standard normal processes and exception remediation processes Extend automated decision making with business processes and vice versa

40. 43 Solution Patterns 1. Pattern: Data Foundation 2. Pattern: Information Delivery 3. Pattern: Externalize Knowledge 4. Pattern: Automate Decision Making 5. Pattern: Codify Business Processes

41. 44 How technologies map to patterns JDBC/ODBC Data Virtualization Data Access, Federation JBoss Enterprise Data Services Metadata Repository Repository Services Workflow Rules JBossESB Transformation, Routing, Registry JBoss Enterprise Application Platform Container services, Hibernate, Web Services stack, Seam, Clustering, Cache, Messaging, Transactions Red Hat Enterprise Linux Windows, UNIX, other Linux JBoss Enterprise SOA Platform 1. Data Foundation 2. Information Delivery 3. Externalize Knowledge 4. Automate Decision Making 5. Codify Business Processes

46. Architecture

47. 50 Architecture Socket transport and query engine have separate work queues and thread pools Deep integration with JBoss AS MC, Profile Service, JCA, JTA, Web Services (consume and produce), JAAS, standard logging

48. 51 Teiid Connector Architecture Teiid splits connectivity concerns into: Data Sources standard JCA based pooled resources configured on the server Translators a Teiid specific CCI (common client interface) that accesses a particular Data Source and is configured as part of the VDB Extended metadata from the translator directs the optimizer source query formation. In addition to out of the box offerings, our JDBC translator is easily extended. Can be thought of as a JDBC/ODBC toolkit since the end result is consumable through JDBC/ODBC

49. 52 Teiid Clustering Clustering is enabled in the SOA production/all profile Teiid does not require clustering, but will use it when available Clients will re-authenticate as needed in load-balancing/fail- over scenarios The default strategy for determining cluster members is by just using the URL. Deployments and jar updates need to happen on all nodes. Farming should help with this. The result set cache and internal materialized views can be replicated.

50. 53 Other Extension Points Logging (Log4j), specific contexts for audit and commands Configurable security domains for admin/query access Can utilize any container supported LoginModule User defined functions both source specific and for source/runtime execution via a Java method Groovy scripting through AdminShell Client discovery of Teiid instances Customizable WARs generated for Web Service access

51. 54 Questions?Questions?

52. 55 Use Cases

53. 56 Credit Suisse: Derivatives Trading Dashboard Challenge Monitor derivatives security trades to prevent rogue trades and financial loss Trading data spread across many databases/systems Solution Consolidate all trading data into single view Real-time access Transformation of data differences Business Benefit Prevent financial loss, lower risk Saved time and cost to develop Easier to manage data changes Data Services Platform Dashboard Data Sources Data Service One of many projects part of data layerOne of many projects part of data layerOne of many projects part of data layerOne of many projects part of data layer

54. 57 Smith Barney: Unified Customer View Challenge Branch Managers account notes in two very different applications (centralized DB2 on mainframe, and distributed SQL Server (600 servers) Cannot access extended account information from other offices. Cannot manage customer only individual accounts. Two years behind schedule in making all notes avail in one application Solution Enable CRM application to easily find customer information across all databases Real-time access Business Benefit Better management of customer, improved customer service Data Services Platform Brokerage CRM Application 600 MS SQL DBs - geographically distributed Data Service Single view of customer key component in new data architectureSingle view of customer key component in new data architectureSingle view of customer key component in new data architectureSingle view of customer key component in new data architecture

55. 58 Large Bank: Data Security/Governance Challenge VISA PCI mandates protection of card holder info Difficult to maintain common security policy across multiple data stores Solution Create data firewall across many data sources Federate rather than replicate Common access policy and common data definitions across sources Audit trail Business Benefit Single, central set of data security policies Prove to auditors and regulators that data protection requirements are being met. Data Services Platform WebFocus Portal Data Sources Data Service Data Firewall to protect and govern use of dataData Firewall to protect and govern use of dataData Firewall to protect and govern use of dataData Firewall to protect and govern use of data

56. 59 DISA GCSS-J: Unified Logistics Portal Challenge Combatant commanders need timely logistics Data spread across many databases/systems; each system owned & managed by different agency Solution Provide a single capability to monitor and manage personnel, equipment, and supplies across all databases Real-time access Networked environment allows DoD users to access shared data & applications regardless of location Business Benefit Single portal for integrated logistics Isolation/abstraction from silos Easier to manage units, personnel, equipment Data Services Platform Multiple Logistics Tracking Applications Data Sources Data Service Consolidated Logistics Information Deployed in TheaterConsolidated Logistics Information Deployed in TheaterConsolidated Logistics Information Deployed in TheaterConsolidated Logistics Information Deployed in Theater

57. 60 CFDB CSDS DMDC GSORTS IDE/AV NGA FLIS CSDS_PL CSDS_VBL Facilities_VML Material_VML Facilities_VQL Material_VQL GDSS Plans_VQL PrivateDataandMetadataPublicData Virtual Mid Layer (VML) Virtual Query Layer (VQL) (Exposed Views) JOPES Classic JOPES 4.0 Virtual Base Layer (VBL) Physical Layer (PL) GTN Structured Data Services Plans_VML

58. 61

59. 62 Global Insurer: SOA Data Services Layer Challenge Deploying SOA reference architecture Want common data model across sources Don't want tightly bound data sources Need to change sources without breaking applications Solution Data is accessed via data services DSP provides federation and consistent logical data model Data model exposed through Web Services and SQL Business Benefit All applications get the same data through use of common model Easier to consume data with new applications Easier to change/add data sources to architecture Data Services Platform Applications Data Sources Data Service Service-enabled, consistent data model for SOAService-enabled, consistent data model for SOAService-enabled, consistent data model for SOAService-enabled, consistent data model for SOA Data Service Common Data Model SOA Platform

60. 63 DISA ADNET: Anti-Drug Network Challenge Counter-narcotics and counter-narcoterrorism Statutory detection and monitoring Data is heterogenous & on multiple systems Solution MetaMatrix provides an abstracted view across multiple State/Local Law enforcement agencies. The virtual Database enables BI tools to get a complete picture of a "person of interest" from any history, warrants, jail, crimes, vehicles, etc... Also, MM is used as a federated search layer looking for possible persons of interest given general details (cars, addresses, license, aliases, etc...) Benefit Enable ADNET to deliver on its mission Data Services Platform BI tools, Portal, Federated Search Data Service Disparate, heterogenous State/Local databases

61. 64 HQ/Langley: MDM and SOA Enablement Challenge Need to find Person of Interest among disparate systems Adherence/mapping to common schema Data integration for SOA enablement Solution Created abstracted view of a Enterprise Schema that is focused on Master Data Entities (Domains) like Person, Organizations, etc. Provide data services layer of the SOA stack, feeding the ESB ESB facilitates sync/async capabilities and provides integrated enterprise data efficiently and rapidly to multiple consumers Benefit Simplified data access and decoupled services and apps from the underlying complex data infrastructure Single view of data enables migration of external sources into the Enterprise repository seamlessly and without application impact Data Services Platform Portal, ESB, Federated Search Data Service Disparate, heterogenous data sources with varying schemas/representations

62. 65 Intelligence Agency: Signal Analysis Portal Challenge Intelligence analysts have to navigate multiple systems to try to assess SIGINT Underlying data consists of both managed data assets and live feeds Security is mission-critical Solution Provide a single capability to monitor and analyze signal intelligence data across databases Metadata repository allows for metadata-aware/driven application Business Benefit Single portal for analysis end of swivel-chair integration Federated data also put on DCGS ESB Data Services Platform Metadata-driven Analyst Portal Data Service Consolidated SIGINT Data - DeployedConsolidated SIGINT Data - DeployedConsolidated SIGINT Data - DeployedConsolidated SIGINT Data - Deployed Over a dozen unique geospatial DBs mix of live & managed data

63. 66 Intel Architecture Data Services and the ESB Metadata Discovery Service SIGINT Gateway Service Metadata Publishing Service Metadata Catalog Alert Subscription Service Event Assessment Service Weather Effect Service IMETS (IWEDA) E-Space Services Weather Effects EW Data Alert Criteria Alerts / Events Metadata Metadata Searches InfrastructureInfrastructure ServicesServices ISR Data Listener Service Async Callbacks Filters Workflow Engine Service Management HUMINT Data Service(s) HDWS (CHAMS) Map / Coverage Google Earth Rich Client Handheld NCES Service Discovery Transformation Engine BC Gateway Service Force Tracking MIP Blue Force Tracking Google Earth Rich Client DCGS-A Services Network EnterpriseServiceBus

64. 67 Backup Slides

65. 68 Caching Overview

66. 69 ResultSet Caching Caching of user query results. Scoping of results is automatically determined to be either VDB (replicated) or session level. Configurable number of cache entries and time to live. Caching of XML document model results. Administrative clearing.

67. 70 CodeTable Caching Short cut to creating an internal materialized view table via the lookup function Way to get a value out of a table when a key value is provided. Example: Lookup(ISOCountryCodes,CountryName, CountryCode,US) Limitations (why use Materialized Views): No option to use the lookup function and not perform caching. No mechanism is provided to refresh code tables.

68. 71 Materialized Views Transformations are pre-computed and stored just like a regular table When queries are issued against the views, the cached results are used Improve Performance/Cost of accessing all the underlying data sources and re-computing the view transforms each time a query is executed Supports no cache queries(Fresh Data full or partial) SELECT * from vg1, vg2, vg3 WHERE ... OPTION NOCACHE Internal materialization creates Teiid temporary tables to hold the materialized table

69. 72 When to Use Materialized Views? Underlying data does not change rapidly It is acceptable to retrieve data that is "stale" within some period of time Access staged data rather than placing additional query load on operational sources.

70. 73 Cache Hints How They Are Used Indicate that a user query is eligible for result set caching Set the result set query cache entry memory preference or time to live Set the materialized view memory preference, time to live, or updatability /*+cache[([pref_mem][ttl:n][updatable])]*/ pref_mem - if present indicates that the cached results should prefer to remain in memory ttl - if present n indicates the time to live value in milliseconds updatable - if present indicates that the cached results can be updated

71. 74 Why JBoss Enterprise Data Services Platform Data Virtualization Technology Real-time integration of diverse data, federation Break down existing data silos, avoid creating new ones Decouple applications from data stores through data services Maintain control and security of information Value Maximize ROA - Return on Assets - get the most out of your existing information and data stores. Faster route to deployment, rapid prototyping, little/no coding Savings in long-tail maintenance costs Leverage skills/knowledge widely available in the industry (SQL/Eclipse) Open source community Available through JBoss subscription, includes JBoss SOA Platform

72. 75 Why Data Services Platform cont'd Flexibility Support for standards like JDBC, ODBC, SOAP make it easy to integrate with existing COTS applications and IT infrastructures. Numerous extension points available to meet varying customer needs: Connector API Custom User-defined Functions, language extensions Administrative API Maturity Based on MetaMatrix technology acquired by Red Hat in 2009. Industry leader in the space Technology under development for over 11 years. Many iterations, improvements, refinements Deployed in demanding production environments

73. 76 Repository: Metadata and more ModeShape Data Service metadata Rules repository SOA repository Includes: JCR Engine RESTful service WebDAV service JDBC driver Eclipse plug-in JBoss AS/EAP kit Sequencers JON plugin DB or file system storage

74. 77 Customer-Related Data Single customer view requires unified view of Customers and Accounts And Services and Transactions and Reference Data Data Volume Low High Frequency of Change, Complexity Static Dynamic Customer Master Data Customer Organization Customer Demographics Claims Transactions Call Center Transactions Benchmark Data Product/ Service Catalog Market Prices Transaction History Customer Accounts Customer Documents Text, Image Customer Contacts Customer Data Services Reference Data 360 view of the customer relationship

75. 78 Map Data Sources to XML and Deploy Model XML Docs, Schemas Build XML Doc. models from XML Schemas Map XML Doc. models to other data models Enable data access via XML Designer Tooling for XML-centric Data Services