think analytics with your ims applications & data.ppt analytics with your ims... · think...

© 2015 IBM Corporation

Think Analytics with your IMS Applications & Data

Hélène Lyon

Distinguished Engineer & CTO, z Analytics & IMS for Europe

IBM Systems, Software Sales, Europe *


Agenda

� Introduction

� A New Need: Analytics in Business Applications

� Focus On Analytics & Decision Management

� Focus on IDAA with your IMS Data

� Focus on Hadoop & Spark

2


Analytics have become Business Critical

These applications may support a large concurrent user population with a high volume of requests

Failure of these applications for any length of time can result in lost business, customer turnover, reputational risk, etc.

Today, analytics are integrated with transaction systems running on the mainframe and are critical to the business

These applications need to deliver insight in real-time or near real-time and integrate with business processes

Prevent Fraud Reduce Customer Churn

Business Critical Analytic applications require superior qualities of service, including a high degree of reliability, continuous availability, scalability,

security and low data latency

Cross-sell/up-sell to customers Operational Reporting

3


More users across the organization depend upon business critical analytics

Bu

sin

es

s C

riti

ca

lA

na

lyti

cs

Tra

dit

ion

al

Bu

sin

es

s A

na

lyti

cs

Customer Service & Support(e.g., call centers, sales personnel)

C-level Mgt.

Company Management

Customers

(e.g., external, web)

User Community

Analysts

(e.g., marketing, research)

CriticalSimpleVery Large

Analytic requirements have expanded

Many

Numberof Users

TransactionVolume

TransactionType

Qualities of Service

LessImportantFew Small Complex

4


… but IT remains aligned to the old way of doing business analytics.

� Some reluctances from the past–Core business is primary, analytics is secondary!

• On core business side: High volume transactions and batch processing running

concurrently, shared everything DB

• On analytics side: Low volume complex queries and batch reporting, shared nothing

DB

–Cost of running analytics on z … without looking at all hidden costs concerning data movement – latency, data governance, IT complexity

–Impact on operational performance & security

� Key drivers to change IT perception–Recognized business value of advanced analytics, including in-

transaction analytics and “big data” analytics–Awareness of z position as primary Systems of Records–Technology availability to build a fully-integrated, end-to-end system

that executes “intelligent” business processes

5


Traditional z Systems Customer Analytics Landscape

� Today challenges– Complexity

• both in systems management and in applications

– Difficulties in supporting real time analytics

– Inability to match ever more demanding SLA requirements

– Data protection & governance issues– High total cost of ownership

� Historical reasons– Different data access patterns

• impact on performance

– EDW as the data integration hub– Different life-cycle characteristics– Different Service Level Agreements

(SLA)• Lack of broadly available workload

management capabilities• Choice of lower cost-of-acquisition

offerings

zData

zAppsz/OS based Transaction &

Batch Workloads

“Transaction & Apps” Data

Core Business Applications and Data

ODS

EDW &Data Mart

ETL or ELTExternal Data & Master Data

Existing BA Infrastructure

ETL or ELT

“IT” Data IT Data

6


z Systems Analytics Areas – Evolution & Extension

zData

zAppsz/OS based Transaction &

Batch Workloads

“Transaction & Apps” Data

Core Business Applications and Data

New Business Critical AnalyticsAdd, Extend, Modernize Analytics Services

New ODS

EDW &Data Mart

ETL or ELTExternal Data & Master Data

Optimized Analytics

Infrastructure with z in mind

Analytics in Business Applicationsor

In-Transaction Real Time Analyticsor

Embedded analytics

Data Serving analytics including “Big Data”

for both structured and unstructured data

Analytics on “IT” Data

ETL or ELT

“IT” Data

Unstructured data

Enhanced data

Analytics Platform

ODS, DW, DM

Data lakes

…

IT Data

A

A

A

A

A Analytics Components

7


� Analytics in Business Applications– Just ReThink business applications with Hybrid Transaction & Analytical influence

– Bring analytics in z/OS based applications and don’t limit your thinking about what are

analytics! You can really do things you couldn’t do before!

– Discuss about zApps improvement even if analytics is today on distributed.

� Analytics on “IT” Data– Improve IT Operation & Run with IT Analytics – Predict / Search / Optimize

� Analytics Data Serving for both structured and unstructured data– Bring analytics to the data – Don’t extend what has been done in the past.

– Make zData easily available for reporting tools, distributed applications or ERPs

– Don’t miss the Spark / Hadoop trend – Make it just relevant with zData without moving

data off z

– Provide a way to improve data governance & data security

z Systems Analytics Solutions Areas

8


IMS Data supports a range of analytics solutions today

Use Case Solution

IBM DB2 Query Management Facility• Real-time BI and dash

boarding of IMS Data

• Analyze IMS data with

unstructured data

InfoSphere Big

Insights

• Deep and fast analysis

of IMS data

• Visualize all data assets

including IMS data

Watson Explorer

72%say greatest value will come

from analyzing transactional data

IBM DB2 Analytics

Acceleraor

9


… and we continue to integrate analytics with core IMS transactions

� Right-time insight at point of impact

� Increased business agility

55%of enterprise applications need

mainframe to complete transactions

Use Case Solution

• In-transaction analytics

Operational decision

management with

externalized business

rules

IBM Operational

Decision Management

on z/OS

• Execution of predictive

analytics in real time

IBM DB2 Analytics

Acceleraor

• Imbedded queries on

historical data

SPSS Real-Time

scoring & Zementis

solution

10

• Callout to non z/OS

based analytic serviceSpark Services

Watson Services, …


Agenda

� Introduction





11


Focus on “Analytics in Business Applications” What is “real-time analytics?”

�This is the time during which a transactional

event is still occurring

–Someone is shopping at a store

–Someone is on the phone with a customer

service representative

–An electronic payment is being processed

–…

�By acting on threats or opportunities as they

arise…

–Revenues can be increased (up-sell, cross-

sell)

–Customer churn can be reduced

It’s about leveraging the power of analytics “in the transactional moment” to achieve a more favorable outcome for a transactional event, while the event is in progress

The person visiting a

store buys more than he

or she otherwise would

have

What would have been an

over-payment is stopped

before it gets out the door

The person on the phone,

who was about to cancel

a service, instead re-ups

A commission of fraud is

stopped before it is

effected

12


Simple Performant AdaptiveIterative

Focus on “Analytics in Business Applications”Technology Pillars - A 100% z/OS based solution for Real-Time Analytics

BusinessRules

� Policy

� Regulation

� Best Practices

� Know-how

Business Critical Queries

Tra

nsa

ctio

n &

B

atc

h W

ork

loa

d

� API Programmable Orchestration to coordinate all activity in the same unit of work and ensure best performance

� IBM DB2 Analytics Accelerator augment analytics capabilities on historical data.

� Predictive modeling, business rules and orchestration together enable the most effective decisions

1

2

3

Add-on

Integration with other Service Provider, BPM

or ESBs

Orc

he

str

ati

on

w

ith

Sim

ple

A

PI

� Risk� Clustering� Segmentation� Propensity

Predictive Analytics

Model Execution

Predictive Analytics

Model Creation

13


Focus on “Analytics in Business Applications”A key point about the real-time analytics technology pillars

�Three technology pillars enable real-time analytics in a z/OS runtime environment:1. Business Rules with Operational Decision Management – ODM on z/OS

2. Predictive analytics with SPSS Real-Time scoring adaptor or with Zementis solution

3. Business critical queries with IBM DB2 Analytics Accelerator for z/OS based data as

well as distributed data when needed by zApps

–Optionally: integration with Hadoop or Spark solutions, CPLEX mathematical algorithm for

Optimization and Watson-based cognitive services

�Each of the three pillars just mentioned delivers significant value on its own.

�Combined, they can work together to–Uncover patterns of events and behaviors

–Use those patterns to develop models to predict future occurrences

–Use those models to identify threats and opportunities as they arise

–Respond immediately to identified threats and opportunities in an autonomic fashion

An organization can go right to “ultimate” real-time analytics capability, or take a phased approach – and realize benefits at each intermediate stage (and the order of enabler implementation is up to the client)

14


Implementation - Today Architecture for z/AppsA simple View of a Unit of Work

� An input layer processes the data coming from a client.

– At the end of this process we have an input area filled with EBCDIC characters.

� An orchestration layer analyses the input area and implements a sequence of calls to business logic.

– Usually subroutines ☺

� The business logic layers implement “services”, directly calling resource managers with simple API

– Access and update data managed by DB2, VSAM, or IMS DB

– Eventually send a message to MQ

� An output layer builds the data to be sent back to the client and finishes the unit of work.

– All updates are committed using the two phase commit protocol.

� Business Logic

� Data Layer

z/OS Resource Manager

CICS or IMS transaction

GettingRequest

Sending Answer

B

z/OS Apps & Data

Orchestration

Business Logic n°1

Business Logic n°p

D

D

Business Logic n°2 D

…

Mainframe = Backend

B

D

15


Implementation - Adding Analytics Services in z/AppsPositioning z as Service Requester

� Co-Location for optimum performance & scalability

� Real-Time Analytics Simple API – SQL API for IDAA

– SQL API for SPSS RTS

– ODM API for ODM

� Other Integration Options– Java Integration for

• Zementis score execution• Spark supported API

– Web service calls also available• WOLA API to call local web services• Many integration options for

CICS or IMS Apps

– Using an external Integration Layer• IBM Integration Bus

z/OS Resource Manager

CICS or IMS transactionGettingRequest

Sending Answer

B

z/OS Apps & Data – The Benefit of Co-Location

InitialOrchestration

Business Logic n°1

Business Logic n°p

D

D

DB2IDAA

RT

A

Orc

hestr

ati

on Historical

Queries

Mainframe = Frontend

Score

Rules

16


Agenda

� Introduction





17


Operational & Analytical Decision Management

Operational Decision Management Analytical Decision Management

Business Processes, Applications & Solutions

DecisionServices

Business

Rules

Internal & External Data

� Policy� Regulation� Best Practices� Know-how

Scenario Analysis& Simulation

Learn from the factsBuild automatically a predictive

model by self learning from data

Learn from the expertsAuthor a rule-based model capturing expert knowledge

Business rules and predictive scoring together enable most effective decisions

Externalized decisions are easy to change

Centralized decisions enable reuse and consistency..

Predictive

Analytics

� Risk� Clustering� Segmentation� Propensity

18


Express the decision logic with business rules

Business language

1 rule for the business = 1 IBM ODM artifact

Transparency, visibility

ifthe product type is car insurance and the client has a car insurancethen

do not recommend the product;

Callable operational decision services

Retention next

best action

service

Shared and platform-agnostic services

Stateless: The calling application passes the context

Synchronous

Externalization, consistency and traceability

Controlled access

Decoupled from the application logic

Externalization and agility

Externalize & centralize the business logic Rule Repository

Bring the IT and the lines of business together Rule Designer Decision Center

One single view on rules

Test, simulation, versioning

Collaboration and governance

WebSite

CallCenter

The 4 IBM ODM Business Rules Essentials

19


Business Problems & Benefits of ODM

Challenges for most z Systems clients

1. Consolidation, isolation, extension of COBOL & PL/I application portfolios

2. Ability to react to increasing pace, variety and volume of change requests

3. Sharing business rules across platforms & channels

4. Ensuring seamless business experience in migration/ application evolution

Benefits of the ODM Approach

� Cost savings

– Shorter change cycle, without increased business risk

– Rule engine processing is zIIP eligible

� Improved agility

– Improved Time to Market

– Manage business decisions in natural language

– Decouple development and business decision change

lifecycles

� Single version of the Truth

– Consolidated and shared expression of business policy

– Maintainable with a Center of Competency model

� Incremental Adoption

– Deploy decision methodology one decision at a time

– Focus on decisions that need to change often & quickly

– Expand adoption of “market validated” decisions

20


IBM Operational Decision Manager

Rule DesignerEvent Designer

Rule Solutions for Office

Decision Center Versioned Assets

Rule Execution Event Execution Decision Monitoring Connectors

Manage

Decision ServerConsole

Design Monitor

Decision Server

Deploy Measure

VisibilityCollaborationGovernance

Define Update

Web Services – API - GUI

DevicesEnterpriseApplication

POS BPM CRM

Social

Event Widgets Space Business Console

Enterprise Console

Access and ControlDecision Artifacts

21


Decision Management on z/OS Comprehensive Flexibility System z

z/OSDistributed or z

System

COBOL & PL/I

Applications

Dep

loy

zRES

Workstation

Rule Designer+ COBOL & PL/I Management

Decision

Service

Bu

IMS

COBOL & PL/I

Applications

Decision

Service

Business Rules

zRES

Decision

Service

Decision

Service

Business Rules

RES on WAS for z/OS

Decision

Service

z/OS Batch

COBOL & PL/I

Applications

Decision Center+ COBOL & PL/I Management

Architect,

Application

Developer

Business Analyst,

Business Manager

Decision Center Repository

CICS

Decision

Service

Business Rules

22


Simplified Integration with zRES API

� Connect to Execution Region– call ‘HBRCONN’ using HBRA-CONN-AREA

� Populate Header with parameter data

� Connect to Execution Server– call ‘HBRRULE’ using HBRA-CONN-AREA– IF HBRA-CONN-COMPLETION-CODE =

HBR-CC-OK THEN

. . .

� Disconnect from Execution Region– call ‘HBRDISC’ using HBRA-CONN-AREA

01 HBRA-CONN-AREA.

10 HBRA-CONN-EYE PIC X(4) VALUE 'HBRC'.

10 HBRA-CONN-LENTH PIC S9(8) COMP.

10 HBRA-CONN-VERSION PIC S9(8) COMP VALUE +2.

10 HBRA-CONN-RETURN-CODES.

15 HBRA-CONN-COMPLETION-CODE PIC S9(8) COMP.

15 HBRA-CONN-REASON-CODE PIC S9(8) COMP.

10 HBRA-CONN-FLAGS PIC S9(8) COMP VALUE +1.

10 HBRA-CONN-INSTANCE PIC X(24).

10 HBRA-CONN-RULE-COUNT PIC S9(8) COMP.

10 HBRA-CONN-RULE-MAJOR-VERSION PIC S9(8) COMP.

10 HBRA-CONN-RULE-MINOR-VERSION PIC S9(8) COMP.

10 HBRA-CONN-RULEAPP-NAME PIC X(256).

10 HBRA-RESPONSE-AREA.

15 HBRA-RESPONSE-MESSAGE PIC X(512).

10 HBRA-RA-PARMETERS.

15 HBRA-RA-PARMS OCCURS 32.

20 HBRA-RA-PARAMETER-NAME PIC X(48).

20 HBRA-RA-DATA-ADDRESS USAGE POINTER.

20 HBRA-RA-DATA-LENGTH PIC 9(8) BINARY.

10 HBRA-RESERVED.

15 HBRA-RESERVED02 PIC X(12).





23


Analytical Decision Management – Predictive ScoringA key-element of the real-time decisioning strategy

� Like the real world, predictive models are not binary– Understanding how closely a pattern of behavior matches a

known pattern of bad (or good) behavior can help uncover

crimes or non-obvious opportunities

� Predictive models detect patterns– Deviation from expected behavior can isolate bad (or good)

behavior, trigger additional actions or new targeted marketing and up-sell / cross-sell offers

� Predictive models can have many variations– Can be built to assess only specific transactions or more

generically for all transactions

– Multiple layers of models can be invoked for increasing

sophistication of analysis, triage leading to further inspection

of contributing factors and weights

24


Predictive Scoring - A 2 steps approach

AnalysesSegments

Profiles

Scoring models

...

Scoring CustomerService

Center

Data:Demographics

Account activity

Transactions

Channel usage

Service queries

Renewals

…

Identify predictive models/patterns found in

historical data

Use those predictive models

with variables to score

transactions & identify the best possible future

outcomes

Practical scoring approaches�Off-line: Batch Scoring

�On-line: External scoring function�On-line: Within a transaction, in-DB, real

time

Step 1 – Build the predictive model Step 2 – Execute the predictive model

Model Creation

enhanced with

IDAA V5SPSS Model

Model Execution

enhanced with

SPSS RTS

25


Real time scoring - What has changed?

CICS or IMStransaction

Select customer scoring data from database using customer id

Customer scoring data

Get score viaWeb Service using customer scoring data

Scoring Model

Customer score

CICS or IMStransaction

End of transaction

Option 1 : Real time analytics process with external scoring function


Get score via database using customer id

Customer score


End of transaction

Option 2 : Real time analytics process with in-database local scoring

Assumption : customer data needed to obtain scoring from model are located in operational database. If historical data are needed, process will vary.

Start of transaction Start of transaction

Scoring Engine

Scoring Engine

26


Real time scoring of the transactional data in DB2 for z/OSSPSS & DB2 for z/OS

� IBM SPSS Real-Time Scoring Adapter for DB2 on z/OS

– Enables customers to score predictive

models built by IBM SPSS Modeler

directly within a specific online

transaction processing transaction that

is running with DB2 for z/OS.

� Business Value– Delivers better, more profitable

decisions, using the latest data, at the

point of customer impact

– Enables more informed customer

interaction • Improves customer service • Increases revenue per customer ratio• Heightens customer retention

– Improves fraud identification and

prevention• Reduces risk and exposure

DB2 for z/OS

Application w/latest data

Real-Time Score/Decision Out

ETLData In

R-T, min, hr, wk, mth

Reduced Networking

End to end solution

Support for both in-transaction and in-database scoring on the same platform

Consolidated Resources

DB2 for z/OS Data

Historical Store

Copy

SPSS Modeler

For Linux on System z

Scoring Algorithm

Business System / OLTP

Scoring Engine

27


Core software for Real-Time Decision solutions on z Systems

� Predictive Analytics – Product: IBM SPSS Modeler with

Scoring Adapter for z or Zementissolution

– Delivers better, more profitable decisions, at the point of customer impact

– Improves accuracy by scoring directly within the transactional application against the latest committed data

– Delivers the performance needed to meet operations SLAs

– Avoid data governance and security issues, save network bandwidth, data copying latency, disk storage

– Same high qualities of service as operational systems

– Easier to incorporate scoring into applications

� Business Rules– Product: IBM Operational Decision

Manager for z/OS– Automate and manage frequently

occurring, repeatable business decisions

– Codifies business policies, practices and regulations

– Enables changes to be easily made by business people

– Automates decision making with the fidelity of an expert

– Centralized, externalized decisions enable consistency and reuse

– Manage business decisions in a natural language

– Decouple development and decision change lifecycle

28


More Information

� Paper from Decision Management Solutions – James Taylor– Transforming Operational Systems: From Transactional Systems to Decision

Management Systems

� Videos on the value of real-time scoring for in-transaction analytics– https://www.youtube.com/watch?v=_vII97Ylq0Y

– https://www.youtube.com/watch?v=CWQNJ2ystic&feature=youtu.be

� Link to Zementis partnership & solutions– http://zementis.com/products/uppi/z4ibmzsystems_goz4z/

– http://www.prweb.com/releases/2015/07/prweb12867889.htm#!

– http://enterprisesystemsmedia.com/article/zementis-announces-predictive-analytics-

integrated-with-ibm-z-systems#sr=g&m=o&cp=or&ct=-

tmc&st=%28opu%20qspwjefe%29&ts=1442215130

29


First National Bank of South Africa

� Business challenges – Increased business pressure due to international

expansion, regulation and consumer demand– Risk scoring rules replicated in each instance of engine with

poor reusability and increased testing needs– Inflexibility of the old rules engine to accommodate changes

� Solution– Shift from Business Rules to Decision Management– Start small with a single LOB – Personal Banking, grow

later based on success

� Benefits–Agility in order to rapidly cope with new regulations and

business conditions–Seamless integration into FNB Mainframe IMS Cobol

environment–Address change and complexity by revamping smoothly a

strategic system–Design of rules in a componentized format which forces

reusability.

Gaining business agility to better and faster adapt to changes with simplified and standardized decision management, shared between all Enterprise IT, including existing mainframe environment.

Solution components:� IBM IBM Operational Decision Management

� On z/OS

� On distributed

“ ODM provides us with a platform to deliver business rules through an agile framework making design decisions a lot simpler.” — Jay Prag - CIO Channel Integration | Core Technology Solutions | FNB Premium

30


European Insurance Company

� Business Challenges– Rethink the underwriting process to become more flexible

and to respond faster to market demand with new products and services.

– Adapt quickly tariffs on competition behavior to reduce severe losses in the corresponding segment.

– Reduce IT cost for tariff rule maintenance

� Solution– Implement a Decision Management System – Easy integration with z/OS based applications– Rules sharing between all Enterprise

� Business Benefits– Increase flexibility in pricing & underwriting rules

management to allow quick changes in response to evolving market conditions

• From 5 months to minutes

– Reduce Time-To-Market for new products– Integrate the end-client together with the broker and back-

office as part of the Digital Strategy– Reinforce cross & up-selling capabilities as a key element to

increase average premium– Providing all users with a coherent user applicative

experience

Business Benefits

Project Approach

1. Designed a decision management solution using ODM on IBM PureApplication Service on SoftLayer

2. Enabled that solution for ODM zRES called from IMS applications.

3. Sharing rules between mainframe and the cloud.

The project started in June and the first Eligibility Rules application went in production in October.

31


Agenda

� Introduction





Hybrid Analytics Solution using IBM DB2 Analytics Accelerator for z/OS: http://www.redbooks.ibm.com/redbooks/pdfs/sg248151.pdf

32


Hybrid Transaction & Analytical Data Processing

The hybrid computing platform on z Systems

Supports transaction processing

and analytics workloads concurrently, efficiently and

cost-effectively

Delivers industry leading performance for mixed workloads

The unique heterogeneous scale-out platform in the industry

Superior availability, reliability and security

TransactionProcessing

AnalyticsWorkload

33


Deep DB2 integration within z Systems & z/OS

Applications

Application Interfaces(standard SQL dialects)

DBA Tools, z/OS Console . . .

Operational Interfaces(e.g. DB2 Commands)

DataManager

BufferManager

IRLMIBM DB2 Analytics

Accelerator

. . .Log

Manager

DB2 for z/OS

Superior availability,

reliability, security

Workloadmanagement

z/OS onz Systems

Superior

performance

on analytic

queries

PureData for

Analytics

34


SPEED• Dramatically improve query response – up to

2000X faster – to support time-sensitive decisions• Right-time. Low latency. Trusted. Accurate.

SIMPLICITY• Simplify infrastructure, reduce ETL and data movement

off-platform • Non-disruptive installation

SAVINGS• Minimize data proliferation• Lower the cost of storing and managing historical data• Free up compute resources

SECURITY• Safeguard valuable data under the control and security

of DB2 for z/OS • Protected. Secured. Governed.

A workload optimized, appliance add-on to DB2 for z/OS that enables the integration of analytic insights into operational processes to drive business critical analytics & exceptional business value.

IBM DB2 Analytics Accelerator for z/OS

35


IBM DB2 Analytics Accelerator Query Execution Process Flow

DB2 for z/OS

Optimizer

IDA

A D

RD

A R

equesto

r

DB2 Analytics Accelerator

Application

Application

Interface

Queries executed with DB2 Analytics AcceleratorQueries executed without DB2 Analytics Accelerator

Query execution run-time for

queries that cannot be or should

not be off-loaded to IDAA

SPU

CPU FPGA

Memory

SPU

CPU FPGA

Memory

SPU

CPU FPGA

Memory

SPU

CPU FPGA

Memory

SM

P H

ost

36


DB2 Analytics Accelerator Loader for z/OS

� An IBM Branded product– PID: 5639-OLA, S&S:5639-OLB

� A Utility that supports loading “Any Data” into DB2 for z/OS front-end and the DB2 Analytics Accelerator, or only into the IDAA

� Allows loading of data from– DB2 image copy files

• Also: DB2 image copy + log files (i.e., “play an image copy forward”)

– Data from other sources • IMS, VSAM• DB2 for LUW, Oracle, SQL Server• And more – if you can get the data into a sequential file, the Loader can get it into the Accelerator

� Example: IMS Database• Initial Implementation: Routing IMS Queries thru DB2 only with data movement optimization

IMS

Catalog

IDAA

IMS

Databases

Define IMS tables & load data into IDAA

DB2

Catalog

DB2

OptimizerQueries

Define IMSTable Metadata

• Complete table refresh

• No Trickle Feed

37


DB2 Analytics Accelerator Loader for z/OS - Intended direction

DB2 for z/OS

DB2 LogsDB2 Image

CopiesDB2 Load

FilesVSAM

Other non-

relationalIMS DB

Accelerator Only TablesAccelerator Only Tables

IBM DB2 Analytics Accelerator

AcceleratorTables

AcceleratorTables

DB2 for z/OS

DB2 Base Tables

IBM DB2 Analytics Accelerator Loader for z/OS intended directionIBM DB2 Analytics Accelerator Loader for z/OS intended direction

DRDA and Federated

• Load DB2 data and Non-DB2 data directly to IDAA• Easy consolidation of enterprise data on secure z

platform• Exploit Accelerator to join DB2 non-DB2 data

• New analytic workload on DB2 for z/OS• Trending, Fraud detection, Capacity Planning, etc.• Analytics of SMF data, DB2/IMS performance data,

etc.

Use Cases

38


More Use Cases than Ever

Hardware

evolution

More query

acceleration

Capabilities for

more use cases

Improved

transparency and

management

Examples

• Faster hardware

• Self-encrypting disks

• Static SQL support

• Closing gaps in

unsupported SQL

• Accelerator-only

tables

• Improved data

maintenance

• Call home

• Management of

failures

39


DB2 Analytics Accelerator – Usage scenariosHow organizations leverage the Accelerator today

Accelerate

existing

workload

Reduce

IT sprawl

Derive new

business insightInclude external

& historical data

Reduce IT sprawl for analytics

If the data is offloaded to a distributed data warehouse or data mart

• Simplify and consolidate

complex infrastructures, low latency, reliability, security and TCO

Rapid acceleration of Business Critical Queries

If the data is analyzed on the mainframe

• Performance improvements

and cost reduction while retaining z Systems security and reliability

Derive business insight from z/OS transaction systems

If the data is not being analyzed yet

• One integrated, hybrid platform,

optimized to run mixed workload

• Simplicity and time to value

Improve access to historical data and lower storage costs

If the analysis is based on a lot of historical data

• Performance improvements

and cost reduction

40


DB2 Analytics Accelerator Version 5.1Adding new dimensions in functionality to expand use cases

Rapid acceleration of Business Critical Queries

Adding application support fortemporary objects (QMF, Multi-step Reporting, IBM Campaign, etc.)

Individual ad-hoc analysis that provides a Data Scientist Work Area

• Insight into now to maximize

business opportunities in today’s

dynamic environment

Accelerate

existing

workload

Reduce

IT sprawl

Derive new

business insightInclude external

& historical data

Reduce IT sprawl for analytics

• Business agility through

simplified architecture

Improve access to historical data and lower storage costs

• Simplified access to information

– when you need it

Integrate more data sources

for analytics, using DB2 Analytics Accelerator Loader for z/OS to

assimilate with IMS data or data from other sources

In-database transformation to

support Data Stage Balanced Optimization and the

consolidation of ETL/ELT processing in DB2 for

z/OS

Derive business insight from z/OS transaction systems

In-database analytics to accelerate predictive analytics applications; SPSS/INZA data mining and in-database modeling can be processed within the Accelerator

• Real-time, actionable business processes

• Environment to efficiently, continuously test and improve analytic results to drive better customer understanding

Deliver Right-

Time Analytics to drive better business outcomes

41


� IBM DB2 Analytics Accelerator – Primary Product Page

– Prerequisites and Maintenance

– Guides and manuals

– Knowledge Center

� Customer Testimonials– https://engage.vevent.com/index.jsp?eid=556&seid=68284&code=brand

� Redbooks and Redpapers– Reliability and Performance with IBM DB2 Analytics Accelerator Version 4.1

– Optimizing DB2 Queries with IBM DB2 Analytics Accelerator for z/OS

– Hybrid Analytics Solutions using IBM DB2 Analytics Accelerator for z/OS V3.1

– IBM DB2 Analytics Accelerator: High Availability and Disaster Recovery

– SAP Integration with IBM DB2 Analytics Accelerator for z/OS

� All TechDocs available at the following link.

Sources of Information

42


IBM DB2 Analytics Accelerator Strategy

• Complement DB2's industry leading transactional processing capabilities

• Provide specialized access path for data intensive queries

• Enable real and near-real time analytics processing

• Execute transparency to the applications

• Operate as an integral part of DB2 and z Systems

• Reusing industry leading PDA's query and analytics capabilities and take advantage of future enhancements

• Extend query acceleration to new, innovative usage cases, such as:

– in-database transformations

– advanced analytical capabilities

– multi-temperature and storage saving solutions

Enable DB2 transition into a truly universal DBMS that provides best

characteristics for both OLTP and analytical workloads.

Ultimately allow consolidation and unification of transactional and analytical data stores

DB2 for z/OS

In-databaseTransformation

QueryAccelerator

StorageSaver

OLTP

AdvancedAnalytics

43


Agenda

� Introduction





44


Big Data, Hadoop & Spark History

Developed in 2009 at UC Berkeley AMPLab, open sourced in 2010, Spark has since become one of

the largest OSS communities in big data, with over 200 contributors in 50+ organizations

“Organizations that are looking at big data challenges – including collection, ETL, storage, exploration and

analytics – should consider Spark for its in-memory performance and the breadth of its model.

It supports advanced analytics solutions on Hadoop clusters, including the iterative model required for

machine learning and graph analysis.”

Gartner, Advanced Analytics and Data Science (2014)

45


Hadoop Introduction

� Open source software framework from the Apache Software Foundation that supports data-intensive highly parallel applications

– High throughput, batch processing

� Designed to run on large clusters of commodity hardware

– Lots of cores – inexpensive cores working all the time

• Processors fail – that’s ok – just replace them

– Lots of redundant disks – really inexpensive disks

• Disks crash – that’s ok – just replace them

– But nothing in Hadoop requires commodity cores and disks!

� Two main components– Hadoop Distributed File System (HDFS)

• Self-healing, high-bandwidth clustered storage

– MapReduce engine• A simple, powerful framework for parallel

computation

46


Spark processes and analyzes data from ANY data source

� Apache Spark is an open source, in-memory processing engine designed for Big Data.

– in-memory processing capability, – interfacing with multiple data sources, – ability to be written with multiple

programming languages.

� It was designed with 3 key tenants in mind– Fast, simple, able to run in many

environments.

� Apache Spark is NOT– A data store

• Spark attaches to other data stores but does not provide its own

– Only for Hadoop• Spark can work with Hadoop (especially

HDFS), but Spark is a separate, standalone system

– Only for machine learning• Spark includes machine learning and does

it very well, but it can handle much broader tasks equally well

– A replacement for Streams• Spark Streaming is micro-batching, not true

streaming, and cannot handle the real-time complex event processing that true streams do

Hadoop Database MainframeData-

warehouse

Business Applications and Business Intelligence

Source : TypeSafe , Apache Spark Survey 2015, Databricks - How Companies are! Using Spark

47


Spark Advantages over MapReduce

� Spark is complementary to Hadoop, but much faster with in-memory performance

� MapReduce limitation– Suitable only for batch processing jobs, implementing interactive jobs and models

becomes challenging.

– Iterative jobs and SQL queries involves a lot of disk I/O for each repetition!

� Spark Processing– Innovative execution engine that support cyclic data flow and in-memory computing.

– Based on a functional programming model similar to MapReduce with the ability to load

data into memory which makes it suitable for interactive analysis, SQL queries and

complex jobs.

Query 1

Query 2

Query 3

Query 4

Result 1

Result 2

Result 3

Result 4Distribute memory

One-time processing

…

48


z Systems & Apache Spark: Strategic Direction

Spark node

Spark node

Spark node

Linux on z

Systems

Leverage LoZ

virtualization benefits

Power

Spark node

Spark node

Spark node

Spark node

x86

Leverage call center,

external, social,

sentiment data …

Unified Analytics PlatformFlexibility & Agility with multi-language support

Efficient Structure – 100x vs. in-memory map reduce

Rich set of built-in functions with consistent APIs: Spark SQL, Spark Streaming, GraphX, …

DB2 VSAM

CICS

WAS

Spark node

z/OS

Spark node

Spark node

Leverage z/OS data and transactions

IBM DB2 Analytics Accelerator

IMS

Fast, expressive, cluster computing system leveraging in-memory framework for analytics

49


Hadoop & z Systems Integration – 2 Use cases

1 - Mainframe clients want to incorporate sensitive mainframe data

into exploratory analytic models

What has been holding them back?

z/OS

DB2 VSAM

QSAM IMS

SMF RMF

Logs

2 - Mainframe clients want to incorporate into zApps analytics based on non-z data like

social media, machine generated data, e-mail

What has been holding them back?

There is risk associated with having copies of sensitive data existing outside the mainframe

Performance & Integration are key inhibitors for real-time analytics.

50


Use Case 1: Challenges

� Address governance, security, and other operational practices

� Leverage Big Data without losing control of data

Challenge How to address?

Clients are worried about data

governance as the data moves off of

z. Data is considered secure as long

as it is on z. How do you secure

sensitive data once it has left z?

z needs to be in "control" of the data.

How can existing security policies be

applied?

The ingestion of data from z into the

Hadoop environment is turning into a

bottleneck

Need high speed / optimized

connectors between traditional z/OS

LPARs and a z-controlled-Hadoop

environment

zData

51


Use Case 1: Populating System z Hadoop clustersIBM & Veristorm partnership

� A secure pipe for data– Data never leaves the box– RACF integration – no need for separate or

special credentials– Data streamed over secure channel using

hardware crypto, SSL

� Easy to use ingestion engine– Native data collectors accessed via graphical

interface– Light-weight; no programming required– Wide variety of data sources supported,

including JDBC for non-mainframe data sources

– Automatic code page conversions– COBOL copybook Parsing and presentation,

Metadata translation– Automated job scheduling

� Fast and low resource utilization– HiperSockets and 10 Gbps internal transfer– Streaming technology does not load z/OS

engines or require DASD for staging

IBM InfoSphere System z

Connector for Hadoop

IBM and Veristorm are collaborating on tighter product integrations for System z

customers

52


Use Case 2: ““““Augmented Analysis””””

� Very large amounts of non-relational data originate outside z Systems– e.g. e-mails sent by customers, tweets, posts to company Facebook page

� Analyze sentiments and identify customers who are dissatisfied with company– Words ‘cancel’, ‘terminate’, ‘switch’ or synonyms thereof– Names of competitors

� Gather names and e-mail addresses of customers at risk

� Join these results with operational data– Alert agents of at-risk customers– Agents work with customer and offer a promotion to stave off defection

All of the

operational

applications &

data are here

There is also

potentially relevant

data here

zData

zApps

53


Business View - Claiming disability allowance

Hadoop or agency

Data from Social Media

sites analyzed with

Text analytics

“Unable to work”

Work Status

-

z Systems

“Dude – awesome

vacation”

Facebook Post

Core Business

Processes

DB2 for z/OSIMS DB

Deterrent for fraudsters -Cost Savings for the business

Make payment or investigate

Gateway to

Hadoop

54


Business View - Claiming disability allowanceStep by Step

� The claimant submit a transactional request to claim for disability.

� Later a batch application identifies all entitled parties for a disability payment.– This complex query could benefit of the DB2 Analytics Accelerator.

� We add in this batch a “fraud detection” step as a request for information on claimants from an Hadoop cluster or an amalgamator agency.

– A z/OS based Apps kicks off a Hadoop job by using Co:z product from Doventail for example.

– Same zApps can soon use Spark services to interrogate any source of data.

� Information on claimants is gathered from social media and other sources – and the results is sent to the requester.

– Any Hadoop Solution ingests data that usually is not ingested by established structured data analysis systems like DB2 for z/OS, .e.g. email from all clients sent to an insurance company, facebook, …

– Analytics can search for family of words, can identify customer sentiment from key words in emails or based on comments in social media.

� Data is sent back to z/OS in a predefined format and can be “joined” to the claimants main record.

– Then it can decide what to do as Next Best Action.

55


Agenda

� Introduction





Are you ready to modernize your IMS IT Environment to answer Business needs?

56


z Systems: vision, strategy and technology to fuse transactions and analytics to support enterprise decisions

Best of class integrated data life cycle management for: Generating customer

insights, fighting fraud and financial crimes, …

58

think analytics with your ims applications & data.ppt analytics with your ims... · think...

Documents