processing patterns for predictive business
DESCRIPTION
Tim Bass' keynote presentation at the 1st Workshop on Event Processing, held in NY, March 14, 2006.TRANSCRIPT
Processing Patterns for PredictiveBusinessTM
Event Processing Symposium March 14, 2006
Tim Bass, CISSP Principal Global Architect TIBCO Software Inc.
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.2
Our Agenda
Introduction
Event-Decision Architecture Traditional vs. State-of-the-Art Processing Architecture
Capstone Constraints and Requirements
Inference and Processing Architecture
Processing Patterns for PredictiveBusinessTM
Open Discussion
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.3
Introduction
Event-Decision Processing is Computationally Intensive
CEP requires a Number of Technologies: Distributed Computing, Publish/Subscribe and SOA
Hierarchical, Cooperative Inference Processing
High Speed, Real Time Processing with State Management
Event-Decision Architecture for Complex Situations and Events
There is no single “CEP Solution” or “CEP Product”
CEP needs a Common Vocabulary and Functional Architecture based on Mature, Industry-Standard Inference Models
Processing and Integration Patterns for CEP need to be Developed and Formalized
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.4
Resource Management
Data Fusion
Sensor Fusion
InformationFusion
Tracking
Data Mining
Correlation
Planning Complex Event Processing
ProcessingManagement
SensorManagement
Control
Estimation
Event StreamProcessing
A Vocabulary of Confusion (Work in Progress)
Adapted from: Steinberg, A., & Bowman, C., CRC Press, 2001
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.5
US Legislation - Monitoring Requirements
The Predictive Enterprise
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.6
PredictiveBusinessTM
Source: Ranadivé, V., The Power to Predict, 2006.
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.7
Example PredictiveBusinessTM Scenarios
Finance Program (Opportunistic) Trading and Execution Risk Management Pricing and Consumer Relationship Management Fraud and Intrusion Detection
Business Process Management Process Monitoring Exception Management and Outage Prediction Scheduling
Sensor Networks Reliability of Complex, Distributed Systems RFID Applications Manufacturing Floor – “Sense and Respond” Power Grid Monitoring Military
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.8
PredictiveBusinessTM & Complex Event Processing (CEP)
More CEP Scenarios:
Stock Trading
Automatic identification of buy/sell opportunities.
Compliance Checks
Sarbanes-Oxley detection.
Fraud Detection
Odd credit card purchases performed within a period.
CRM
Alert if three orders from the same platinum customer were rejected.
Insurance Underwriting
Identification of risk.
"Events in several forms, from simple events to complex events, will become very widely used in business applications during 2004 through 2008"
--- Gartner July 2003
Graphic Sources: TIBCO Software Inc & IBM
CEPSituationManager
Event Streams
Historical Data
Real-time Detection and Prediction
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.9
Our Agenda
Introduction
Event-Decision Architecture Traditional vs. State-of-the-Art Processing Architecture
Capstone Constraints and Requirements
Inference and Processing Architecture
Processing Patterns for PredictiveBusinessTM
Open Discussion
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.10
HTTP request / response
Structured messages
Screen Audit events
Message Audit events
Screen/ message Audit events
FraudDetectionRules
A Traditional Event-Driven Architecture (Fraud)
FraudDetectionRules
Queue Client/ServerChanne
lQueue Fraud
DetectionRules
EMSChanne
l
Queue FraudDetectionRules
Screen Based
Channel Fraud Event
Network TAPS
en
sor
Pre
pro
cessin
g
ServiceAPI
Queue FraudDetectionRules
HTTPChanne
l
Queue FraudDetectionRules
APIChanne
l
…1234Joe01021970…..
Fraud Event
Fraud Event
Fraud Event
Fraud Event
Structured messages
QueueUnix/ VTChannel
FraudDetectionRules
FraudEvent
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.11
Emerging Event-Decision Architecture
CustomerProfiles
Purpose-BuiltAnalytics
Distributed Multisensor Infrastructure
Internet/Extranet Sensors
Human Sensors
Edge/POCSensors
OperationsCenter
OtherReferences
Complex EventProcessors Sensors are Everywhere!
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.12
Capstone Constraints & Requirements
22
Constraints: Distributed, heterogeneous Internet and Intranet environments Purpose built systems and analytics, compartmentalization and specialization Data-at-rest (databases and warehouses) and data-in-motion (real time, event driven)
Infrastructure Requirements: Service-oriented architecture Event-driven, zero-latency, distributed message-oriented middleware Support for both standards-based interfaces and purpose-built (proprietary) interfaces Real-time event-decision processing Specialization, data warehousing, data mining, analytics Human interaction with computers and networks
Processing Requirements Layered knowledge / inference and analytics processing Complex event processing, state and temporal management, state estimation Progressive hierarchical inference – data, event, complex event, situation, impact, prediction
Adaptive control and resource management Enterprise processing model (architecture)
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.13
Event-Inference Hierarchy
22
Impact Assessment
Situational Assessment
Relationship of Events
Identify Events
Location, Times and Ratesof Events of Interest
Existence of PossibleEvent of Interest
Data/Event Cloud
Analysis of Situation & Plans
Contextual and Causal Analysis
Causal Analysis, BayesianBelief Networks, NNs,
Correlation, State Estimation,Classification
Use of DistributedSensors for Estimations
Raw Sensor Data(Passive and Active)
HIGH
LOW
MED
Adapted from: Waltz, E. & Llinas, J., Multisensor Data Fusion, 1990
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.14
Event-Decision High Level Architecture
22
EVENT CLOUD(DISTRIBUTED DATA SET)
KS
KS KS KS KSKS KS KS
KS KS KS KS KS KS
Adapted from: Engelmore, R. S., Morgan, A.J., & and Nii, H. P., Blackboard Systems, 1988 &Luckham, D., The Power of Events, 2002
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.15
Sensors• Systems that provide data and events to the inference models
and humans
Actuators • Systems that take action based on inference models and human
interactions
Knowledge Processors• Systems that take in data and events, process the data and
events, and output refined, correlated, or inferred data or events
HLA - Knowledge Sources
KS
KS
KS
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.16
Event-Decision Architecture
24
EVENT PRE-PROCESSING
EVENTSOURCES
EXTERNAL
. . .
LEVEL ONE
EVENTREFINEMENT
USERINTERFACE
COMPLEX EVENT PROCESSING (CEP)
DB MANAGEMENT
HistoricalData
Profiles &Patterns
DISTRIBUTED
LOCAL
EVENTSERVICES
.
.EVENT
PROFILES..
DATABASES
.
.OTHER DATA
LEVEL TWO
SITUATIONREFINEMENT
LEVEL THREE
IMPACTASSESSMENT
LEVEL FOUR
PROCESSREFINEMENT
Adapted from JDL: Steinberg, A., & Bowman, C., Handbook of Multisensor Data Fusion, CRC Press, 2001
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.17
Structured Processing for Event-Decision
Multi-level inference in a distributed event-decision architectures Level 5 – User Interface
Human visualization, interaction and situation management
Level 4 – Process Refinement Decide on control feedback, for example resource allocation, sensor and state
management, parametric and algorithm adjustment
Level 3 – Impact Assessment Impact threat assessment, i.e. assess intent on the basis of situation
development, recognition and prediction
Level 2 – Situation Refinement Identify situations based on sets of complex events, state estimation, etc.
Level 1 – Event Refinement Identify events & make initial decisions based on association and correlation
Level 0 – Event Preprocessing Cleansing of event-stream to produce semantically understandable data
Level of Inference
Low
Med
High
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.18
CEP Level 0 – Event Preprocessing
Cleanse/Refine/Normalize Data for Upstream Processing
Calibrate Raw Event Cloud: Web Server Farm Event Stream Example -
Group HTTP REQUESTS and RESPONSES Reduce and Extract Required Data from Transaction Format into Event for Upstream Processing
Intelligent Agent Fraud Detection Event Steam Example - Receive Event Stream from Purpose-Built FD Application Reduce and Extract Required Event from Event Stream Format for Upstream Processing
Reduces System Load by Preprocessing Events
Enables Upstream to Concentrate on Most Relevant Events
Focuses on Objects/Events
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.19
CEP Level 1 – Event Refinement
Problem: Which Events in the Event Stream Are “Interesting”?
Event Refinement Example (Association & Classification): Hypothesis Generation (HG)
Processing incoming events, data and reports Hypothesis: This Group of Events May Represent Fraud Output: Fraud Detection Scorecard or Matrix
Hypothesis Evaluation (HE) Evaluates Scorecard/Matrix for likelihood comparison Rank Evaluation: These Events have a Higher Likelihood of Fraud Output: Fills Scorecard/Matrix with relative likelihood estimation
Hypothesis Selection (HS) Evaluates Scorecard/Matrix for best fit into “badges of fraud” Evaluation: Provide an Estimate (Name) of the Fraudulent Activity Output: Assignment of fraudulent activity estimate to event
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.20
CEP Level 2 – Situation Refinement
What is the Context of the Identified Events?
Focuses on Relationships and States Among Events
Situation Refinement Event-Event Relationship Networks Temporal and State Relationships Geographic or Topological Proximity Environmental Context
Example: Brand currently used by phishing site in Internet increasing probability of fraud and identity theft
Event / Activity Correlation – Relational Networks
Pattern, Profile and Signature Recognition Processing
Question: Do “Complex Events” == “Situations”?
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.21
CEP Level 3 – Impact Assessment
Predict Intention of Subject (Fraudster example) Make changes to account identity information?
Transfer funds out of account?
Test for access and return at later time?
Estimate Capabilities of Fraudster Organized Gang or Individual Fraudster?
Expert or Novice?
Estimate Potential Losses if Successful
Identify Other Threat Opportunities
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.22
CEP Level 4 – Process Refinement
Evaluate Process Performance and Effectiveness Exception Detection, Response Efficiency and Mitigation
Knowledge Development
Identify Changes to System Parameters Adjust Event Stream Processing Variables
Fine Tune Filters, Algorithms and Correlators
Determine If Other Source Specific Resources are Required
Recommend Allocation and Direction of Resources
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.23
CEP - Database Management Examples
Reference Database User Profiles
Activity and Event Signatures and Profiles
Environmental Profiles
Inference Database Subject Identification
Situation and Threat Assessment
Knowledge Mining
Referential Mapping Database Examples Mapping Between IP Address and Domain
Mapping Between Known Anonymous Proxies
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.24
CEP Level 5 – User Interface / Interaction
Operational Visualization at all “Levels” Dynamic Graphical Representations of Situations
Supports the Decision Making Process of Analytics Personnel
Process and Resource Control Supports Resource Allocation and Process Refinement
Display Control & Personalization Different Operator Views Based on Job Function and Situation
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.25
Our Agenda
Introduction
Event-Decision Architecture Traditional vs. State-of-the-Art Processing Architecture
Capstone Constraints and Requirements
Inference and Processing Architecture
Processing Patterns for PredictiveBusinessTM
Open Discussion
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.26
Processing Patterns
BusinessContext
InferenceProcessingTechniques
Processing Patterns for PredictiveBusinessTM
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.27
Inference Algorithms for Event-Decision Processing
A sample of event-decision processing algorithms relevant to CEP:
Rule-Based Inference
Bayesian Belief Networks (Bayes Nets)
Dempster-Shafer’s Method
Adaptive Neural Networks
Cluster Analysis
State-Vector Estimation
Key Takeaway: Analytics for CEP exist in the art & science of mature multi-sensor data fusion processing - these analytics can be mapped to recurring business patterns.
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.28
Business Context Inference Processing Techniques
Classical InferenceBayesian Belief Networks
Hidden Markov Models Dempster-Shafer’s Method
Self-Organizing Feature MapsState-Vector Estimation
Adaptive Neural NetworksRule-Based Inference
Sensor OptimizationComplex DiagnosticsFraud Detection Intrusion DetectionNetwork ManagementCounterterrorismOpportunistic TradingCompliance MonitoringSupply Chain Optimization
Map Business Context to Classical MethodsNote: For Illustrative Purposes Only
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.29
Bayes Net: Identity Theft Detection / Phishing
UsesProxy
AlertService
Account
Lockout
ProfileMismatch
BrandPhishing
AlertSecurity
AlertCustomer
KnownFraud
IP
IdentityTheft
LoginSuccess
PhishingAlert
BrandMisuse
Source: Bass, T., TIBCO Software Inc., January 2006
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.30
Bayes Net: Simple Web-Click Behavior
Click Pg
Subtype
ClickElapse
d
Associate Session ID
StoresVisited
Click Pg
Type
Click toPurchase
SessionTime
# ItemsPurchased
IDBrowser
RecognizeSession
IDOS
TotalPurchase
Session IDCode
ClickPrice
Price
ClickCount
Source: Ambrosio, B., CleverSet Inc., December 2004
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.31
Recurring Pattern(s) for PredictiveBusinessTM
Bayesian Techniques for Complex Event Processing in: SPAM Filtering
Telecommunications Fraud
Other Behavior-Based Fraud & Intrusion Detection
Financial Risk Management
Credit Approval and Credit Limit Automation
Medical Diagnosis
Military ID, Command and Control
BNs dominate many other areas in Complex Event Processing Graphical representation of your domain knowledge
Both causality and probability reside in the models
Well established as a knowledge processing technique
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.32
Event-Decision Processing Characteristics
JDL Model Levels
Association Process
Estimation Process
Entity Estimate
Activity
(L4) Process Refinement
Planning (Control) (Action) Decision Making
(L3) Impact Assessment
Aggregation Plan Interaction
Effect (situation, given plan)
Impact Assessment
(L2) Situation Refinement
Aggregation Relational Aggregation (situation)
Situation Assessment
(L1) Event Refinement
Assignment Attribution Individual Event Event Processing
(L0) Event Preprocessing
Assignment Detection Sensor Output Sensor Processing
Adapted (this and the next slide) from: Steinberg, A., & Bowman, C., Handbook of Multisensor Data Fusion, CRC Press, 2001
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.33
Comparison of Event-Decision Models
JDL Model Levels
Waterfall Model
Boyd Loop
Sense & Respond
Intelligence Cycle
Activity
(L5) Visualization
Act Respond Disseminate Decision Execution
(L4) Process Refinement
Decision Making
Decide Decide Disseminate Decision Making
(L3) Impact Assessment
--- Orient Analyze Evaluate Impact Assessment
(L2) Situation Refinement
Situation Assessment
Orient Analyze Evaluate Situation Assessment
(L1) Event Refinement
Pattern Processing /
Feature Extraction
Orient Detect Collate Event Processing
(L0) Event Preprocessing
Sensor Processing
Orient Detect Collate Sensor Processing
--- Sensing Observe Sense Collect Sensor Acquisition
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.34
Key Takeaways
Event Processing can be a Computationally Intensive
CEP Requires a Number of Technologies: Distributed Computing, Publish/Subscribe and SOA Hierarchical, Cooperative Inference Processing High Speed, Real Time Rules Processing with State Management Event-Decision Architecture for Complex Events / Situations
CEP Community Needs Common Vocabulary and Functional Architecture based on Established Inference Models
Processing Patterns for CEP Need to be Developed based on using a Common Vocabulary and Functional Architecture
© 2006 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary.36
JDL Example: Inference ScoreCards
Event Stream
Raw
Dat
a
Level 0Pre-Processing
Fraud Events
Eve
nt
Str
eam
Level 1Event
Refinement
ScoreCard
Fraud Situations
Fra
ud
Eve
nts
Level 2Situation
Assessment
Business Impact
Fra
ud
Sit
uat
ion
s
Level 3Impact
Assessment
ScoreCard
ScoreCard
ScoreCard
Event Source
Tas
kLevel 4Process
RefinementScoreCard
Modified from: Steinberg, A., & Bowman, C., CRC Press, 2001