1

Drive Smarter Decisions with Big Data

Using Complex Event Processing

Eric Roch ▪ July 16, 2013

Our Speaker

Eric Roch

• Principal for Perficient’s Connected Solutions Practice –

SOA – Cloud - Mobile

• 25+ years of experience in Information Technology

• Previous roles include: executive level management,

technical architect, and software development in top tier

technology organizations including TIBCO Software and

Deloitte Consulting

• Strategic planning and commercialization of

methodologies and software

• Technical architecture for multi-platform application and

systems integration at organizations

• Guest speaker and author

2

3

Perficient is a leading information technology consulting firm serving clients

throughout North America.

We help clients implement business-driven technology solutions that integrate

business processes, improve worker productivity, increase customer loyalty and

create a more agile enterprise to better respond to new business opportunities.

About Perficient

4

• Founded in 1997

• Public, NASDAQ: PRFT

• 2012 revenue $327 million

• Major market locations throughout North America

• Atlanta, Boston, Charlotte, Chicago, Cincinnati, Cleveland, Columbus, Dallas, Denver,

Detroit, Fairfax, Houston, Indianapolis, Los Angeles, Minneapolis, New Orleans, New York

City, Northern California, Philadelphia, Southern California, St. Louis, Toronto and

Washington, D.C.

• Global delivery centers in China, Europe and India

• ~2,000 colleagues

• Dedicated solution practices

• ~85% repeat business rate

• Alliance partnerships with major technology vendors

• Multiple vendor/industry technology and growth awards

Perficient Profile

5

Business Solutions

• Business Intelligence

• Business Process Management

• Customer Experience and CRM

• Enterprise Performance Management

• Enterprise Resource Planning

• Experience Design (XD)

• Management Consulting

Technology Solutions

• Business Integration/SOA

• Cloud Services

• Commerce

• Content Management

• Custom Application Development

• Education

• Information Management

• Mobile Platforms

• Platform Integration

• Portal & Social

Our Solutions Expertise

Agenda

• Big Data Trends and Categories

• Analysis of large amounts of complex, unstructured and semi-

structured data

• Harnessing the power big data, social/mobile data stores and BI

projects for real-time decision-making

• Predictive Analytics and Event Processing for decision management

6

Evolution of Big-Data

• Mainframe

• Client-Server

• Web

• Mobile

• Cloud

• Social

• Internet of Things

7

Source: Go-Globe.com

State of Technology Adoption

8

Categories of Big-Data

9

Source: splunk

10

Characteristics of Big Data

• Data in motion

analyzes data

before storage

• Data at rest

analytics are

based on a

historic snapshot Source: IBM

Big Data Technologies

• MapReduce frameworks implements pattern recognition though classification algorithms – what happened?

• Data Visualization presents information views graphically and/or statistically – what happened and what might happen?

• Predictive Analytics uses mathematical pattern recognition in historical data – what’s going to happen?

• Complex Event Processing uses pattern recognition on event streams and can apply rules to predict logical events – what is going to happen and what do we do about it?

11

Source: TIBCO Spotfire

Log Analysis vs. Business Analytics

• Ingest – Versus ETL

• Big Data – Bidirectional integration with Hadoop

• Query language – MapReduce function on unstructured data

• Drill anywhere – Investigate on all the data versus a predefined schema or cube

• Information discovery – Discover relationships based on patterns in the data

• Ad-hoc versus dimensional – Log analysis is not based a predefined structure based a point-in-time set of requirements

12

Source: splunk Implementation

Predictive Analysis

• Predict the future state of

variables associated with

business goals

• Describe human

detectable patterns

• Data mining techniques • Rule Discovery – describe

• Pattern Discovery – describe

• Clustering – describe

• Classification – predict

• Regression – predict

• Deviation – predict

13

Source: InformationBuilders

Event-driven Architecture

• Event-driven architecture

(EDA) is a software

architecture pattern promoting

the production, detection,

consumption of, and reaction

to events

• Complex event processing

(CEP) consists in processing

many events happening

across all the layers of an

organization, identifying the

most meaningful events within

the event cloud, analyzing their

impact, and taking subsequent

action in real time.

14

14

A Holist View of Decision Optimization

15

Source: James Taylor

http://www.decisionmanagementsolutions.com

Barriers to Big Data Analytics

• Information throughout the

enterprise

• Silos of data

• Decentralized control

• No one single solution

• No cohesive strategy

• Legacy systems difficult to

make part of the strategy

16

SOA and Integration

17

HT

TP

HT

TP

/S

SO

AP

/HT

TP

SO

AP

/JM

S

FT

P

SM

TP

EM

S/J

MS

ED

I

Enterprise Service Bus (ESB)

Credit

Check Place

Order

Check

Quantity Issue

Invoice

Alert

Large

Order

Notify

Customer

Process

Order

Check

Customer

Account

• Connect

• Transport

• Route

Services Backbone Enterprise Service Bus

(ESB)

• Mediate

• Event notification

• Exception Handling

Abstract the data format and the behavior of legacy systems to publish events

The SOA Information Gap

″SOA by itself does nothing to address the question

of how data should be managed within this

architecture. ... data remains fragmented despite

the best efforts to rationalize it. This issue is

motivating the creation of a new class of

middleware that Forrester calls the information

fabric.”

The Forrester Report Information Fabric:

Enterprise Data Virtualization

18

″ You will waste your investment in SOA unless you have enterprise

information that SOA can exploit.”

Gartner

Data Virtualization Layer

19

Data Warehouse

Packaged Application

Legacy Application

• Master Data Management and Data Virtualization • Data federation for consistent packaging of data • Leverages understanding of metadata relationships • Applies consistent rules to data • Centralized control and maintenance • Flexibility to change information sources and formatsar

Create Quote

Process Flow

Trigger

Create Estimate

Process Flow

Trigger

Information as a Service (Shared Metadata)

Business Process Management and

Workflow

• The term Business Process Management refers to activities performed by businesses to optimize and adapt their processes.

• Although it can be said that organizations have always been using BPM, a new impetus based on the advent of software tools which allow for

• Direct execution of the business processes without a costly and time intensive development of the required software.

• In addition, these tools can also monitor the execution of the business processes, providing managers of an organization with the means to analyze their performance and make changes to the original processes in real-time

• BPM has a tight link to componentized and service oriented IT architecture

20

BPM and Services

21

Service

X

Service

U

Service

Y

Service

Z

Human Task

A

Human Task

D

Human Task

F

Human Task

B

Human Task

C

Workflow

Invoke

Invoke Invoke Invoke

• Workflows implement business processes

• Workflow engine navigates the network of activities

• Typically invoking automatic (service choreography) or manual activities

• Mostly visual programming/modeling

Process Orchestration Layer - BPMS

• Designer and repository

• Execution engine

• Database – case state

• Database – case history

• Case history reporting – KPIs,

task timings, timings by role

• Starting a new case is

resource intensive

22

State Management

Design Repository

Process History

Execution Engine

BRMS Architecture

• Manages the lifecycle of the

rules

• Author rules

• Execute stateless rules

• Statistical reports about rule

execution

• Rule execution is embedded in

business applications – e.g. a

decision service

23

Source: IBM

Using BRMS in BPMS

• Lifecycle of rules are

external to the BPMS

• Business processes “call”

rules e.g. via services

• Rules make a stateless

decision

• Rules have to have a

driving workflow or

application

24

Rule Repository

Rule Engine

Rule

Authoring

BPMS

CEP Architecture

25

• Consistent operational

rules applied to business

events

• Declarative rules and

implicit state management

• Event driven, non-linear,

closed-loop, agile business

processes

• Component failure (fine

grain) – outage (logical

/predictive)

Concept State

Rule Bases

BPMS

CEP Engine

Logical Events – Notifications, Consequences Actions

SOA

Business Applications

Fine-grain Business Events

System(s) of Record

Integration and Business Components

Flexible Workflows

ESB Event

Channel(s)

CEP High-level Architecture

Patterns

• Situation awareness is

about "knowing" the state of

the product, person,

document, or entity of interest

at any point in time.

• Sense and respond is about

detecting some significant

fact about the product,

person, document or entity of

interest, and responding

accordingly

• Track and trace is about

tracking the product, person,

document or entity of interest

over time and tracing

pertinent facts

Source: TIBCO Software

26

CEP Benefits

• Manage events, state transitions, and event correlation reducing code in the application layer

• Control logic

• Persistence logic

• Business Rules

• Drive business processes with correlated events

• Create operational efficiencies with the same events and drive longer-term strategic decision support

• Less complex rules with the event driven concepts

• Persistent business objects

• Known context of the event

27

CEP Roadmap and Methodology

• Target critical business events for process automation and decision optimization

• Inventory relevant events, rules and concepts

• Identify candidate business (sub)process to automate

• Project LoE(s) and Roadmap

• Integrate systems used in key business processes

• Instrument applications to emit events

• Define process activities

• Mine candidate rules – code and predictive analytics

• Model events, rules and concepts

• Iterate through business processes

28

Telco CEP Case Study

• Provisioning

• Track missing provisioning notifications and sends complex events to Billing Ops on

missing notifications

• Open Orders

• Used to track Orders that have not been closed due to a missing event. CEP detects

the missing event and auto closes the Order in the Payment Processing system.

• Pending Payments

• Used to process payments that are pended by the payment processing system. CEP

stores the payment data within the cache and closes the payment at a later via

SOA.

• Customer Coupon Offers

• CEP is used to monitor, alert and prevent Stores from going over a threshold of the

discount funds that they are allocated.

• Logistics Alerts

• CEP is used to track location and Product updates from logistics and to invoke GEH

to republish failed messages

• CEP Framework

• Created CEP developer guide and logging framework to log and search events in

Splunk

29

TIBCO BusinessEvents is a CEP Platform

• Platform Features • Stateful Rule Engine

• State for Temporal Rules

• Persistence Object Manager

• High Performance Rules Engine

• TIBCO integration platform • 150+ Adapters

• Channels approach

• Continuous queries

and Event Stream Patterns

• Decision Manager for Business

User Rules Authoring (can write

can upload rules from Excel!)

• Distributed Agents Architecture

for dynamic scalability

• Data Grid

• BE Views (Dashboard)

30

Source: TIBCO Software

Event Enabled Enterprise

31

3

1

Transformation

Projects

2009-2011

Business

Solutions

2011-2012

• Last minute addition • Concept to launch in 6 weeks • Decoupled architecture – no risk

Implementation

• Customers: Ensure timely activations

• Operations: Immediate visibility into order provisioning times

Customer Service

• Stores: Reduce inventory issues • Operations: Automate fall out of

shipping notices

Supply Chain

• Customers: Added security to account access

• Operations: Report/alert on suspicious access attempts

Security/CPNI

• Customers: More access to discounts • Revenue: Manage discount limits by

individual location

Retail Sales

• Customers: Use IVR to set up payment agreements

• Customer Service: Reduced call center volumes

Self-Service

Event

Enabled 2013

• Proven success in real-time, value-based activities – ready for prime-time!

• Sense. Model. Respond.

The Tipping Point

• Adapt and respond to real-time customer behaviors/experiences

• Example: Proactive retention offers

Fast Response

• Abandon one-size-fits-all customer limitations

• Enable event-driven decisions for best customer experience

Customer Flexibility

CEP Solution Architecture

32

CEP References

• http://it.toolbox.com/blogs/the-soa-blog/complex-

event-processing-reference-materials-48348

• http://it.toolbox.com/blogs/the-soa-blog/complex-

event-processing-patterns-message-routing-

48987

• http://complexevents.com/wp-

content/uploads/2008/02/1-a-short-history-of-cep-

part-1.pdf

• http://complexevents.com/wp-

content/uploads/2008/07/2-final-a-short-history-of-

cep-part-2.pdf

33

Questions

34

Daily unique content

about content

management, user

experience, portals

and other enterprise

information technology

solutions across a

variety of industries.

Perficient.com/SocialMedia

Facebook.com/Perficient

Twitter.com/Perficient

35

Thank you for your time

and attention today. Please visit us at Perficient.com

36