proactive eth talk
DESCRIPTION
Proactive event-driven computing talk in ETH, March 2012TRANSCRIPT
© 2011 IBM Corporation
IBM Haifa Research Lab
Proactive event-driven computing
Towards Proactive Event-Driven Computing Talk in ETH – March 2012
Opher Etzion ([email protected])
Proactive event-driven computing
© 2012 IBM Corporation2IBM Haifa Research Lab
The source code movie (spoiler)
The hero of the story is sent to the occupy theBody of a dead person during the last 8 minutes of
His life, trying to find out who put dirty bomb insideA train so he’ll be stopped from doing the next attack-
In an unexpected turn of events he succeeds toEliminate the attack and change the past
Proactive event-driven computing
© 2012 IBM Corporation3IBM Haifa Research Lab
The proactive event-driven principle
time
Proactive action
Forecast
Real-time decision
Detect
now
Proactive event-driven computing
© 2012 IBM Corporation4IBM Haifa Research Lab
Proactive traffic management system
Proactive event-driven computing
© 2012 IBM Corporation5IBM Haifa Research Lab
The proactive pattern
DetectForecast
Act(proactive)
Monitoring streams of events from sensor in highway and leading ways, from mobile devices, and from accidents reports
Monitoring streams of events from sensor in highway and leading ways, from mobile devices, and from accidents reports
Forecasting that at some point in 15 minute a traffic congestion of certain size will occur in probability of 0.6
Forecasting that at some point in 15 minute a traffic congestion of certain size will occur in probability of 0.6
Taking proactive actions in setting up entry and exit traffic lights durations and speed limit in highway segments
Taking proactive actions in setting up entry and exit traffic lights durations and speed limit in highway segments
Decide(RT)
Proactive event-driven computing
© 2012 IBM Corporation6IBM Haifa Research Lab
Proactive energy use for home consumers/producers
Proactive event-driven computing
© 2012 IBM Corporation7IBM Haifa Research Lab
The proactive pattern
DetectForecast
Act(proactive)
Monitoring sun, wind and demand on power grid Monitoring sun, wind and demand on power grid
Forecasting that in the morning hours the household will not produce any energy and the power grid’s price will be high
Forecasting that in the morning hours the household will not produce any energy and the power grid’s price will be high
Using RT optimization – schedule appliances use and apply through actuators
Using RT optimization – schedule appliances use and apply through actuators
Decide(RT)
Proactive event-driven computing
© 2012 IBM Corporation8IBM Haifa Research Lab
Proactive post-earthquake disaster management system
Proactive event-driven computing
© 2012 IBM Corporation9IBM Haifa Research Lab
The proactive pattern
DetectForecast
Act(proactive)
Monitoring earthquake, spread by sensors, and citizen reports
Monitoring earthquake, spread by sensors, and citizen reports
Forecasting that at some point in the next hour there is going to be a a potential damage in a certain location
Forecasting that at some point in the next hour there is going to be a a potential damage in a certain location
Taking proactive actions in notifying and performing actions like – close roads, reduce speed of trains, turn off gas and water supply…
Taking proactive actions in notifying and performing actions like – close roads, reduce speed of trains, turn off gas and water supply…
Decide(RT)
Proactive event-driven computing
© 2012 IBM Corporation10IBM Haifa Research Lab
The evolution towards proactive computingThe evolution towards proactive computing
Typically people employ computing in responsive way: the person makes decisions and the computer assists in data, knowledge, advice
The vision is to move to proactive computing: (Detect-Derive-Predict-Decide now-Do)
X
Recently, there is more employment of computers in reactive way: events drive decisions (Detect-Derive-Decide-Do)
The initiative remains in human hands;most persons are not proactive by nature
The initiative moves to the computer; reactions to events that already occurred
The initiative moves to the computer; actions to events before they occur
Proactive event-driven computing
© 2012 IBM Corporation11IBM Haifa Research Lab
Why it is difficult to create proactive solutions now ?
A way of thinking
Incompatible programming model of the moving parts, and gaps in each of them within the current product
Event Processing
Predictive analytics Optimization Decision models
Multiple skills are needed
Proactive event-driven computing
© 2012 IBM Corporation12IBM Haifa Research Lab
Proactive computing as cultural change The culture in many organizations and personal behavior advocates a routine behavior governed by fixed set of rules
Many people are deterred from ad-hoc behavior even if it has relative benefit in specific case and prefer statistical metrics .
Current analytics tools are geared towards improving the “fixed set of rules ”
Proactive thinking is different – it provides exception behavior to mitigate or eliminate problems when current rules will not work
Proactive event-driven computing
© 2012 IBM Corporation13IBM Haifa Research Lab
Scalable platform
Proactive event-driven computing
uncertain events, future
events, correctness
issues
Event processing foundations
adaptive real time
optimization for proactive decisions
human interaction in
proactive systems
Paradigm: methodology, seamless programming model
Integrative platform and validation
event recognition,
expertforecasting
models,goal driven supervised
learning
Event recognition and forecasting
Event-based optimization
Human computer interaction
The Proactive pillars
Proactive event-driven computing
© 2012 IBM Corporation14IBM Haifa Research Lab
Event-flow programming model: the EPN
Event Producer 1
Event Consumer 1
Event Consumer 2
Event Producer 2
Event Consumer 3
Agent 2
Channel
Agent 1
State
Agent 3
Proactive event-driven computing
© 2012 IBM Corporation15IBM Haifa Research Lab
EPA
EPA
EPA
Producer
Producer
PRA
State
Consumer
Actuator
e1
e2
d1
d2
d4A1
OK
State
Context
PRA can send events to
EPAs, e.g., “emergency generator fix”
PRA can be defined per
context segment, and receive
events only from EPAs in the same
context
EPAs may need to consult the current state of the
PRA
Actuator may respond immediately, or send acknowledgement via
an event
Messages to and from EPAs are
(potentially uncertain) events,
with present or future time
interval
EPA
e3d4
d3
dB
Enrichfrom dB
Proactive event-driven computing
© 2012 IBM Corporation16IBM Haifa Research Lab
Proactive Agent (PRA) PRA
Input: forecasted events + state information
Output: Action – recommendation, activation, command to actuator
Process: real-time decision making
Input: forecasted events + state information
Output: Action – recommendation, activation, command to actuator
Process: real-time decision making
Real time decision making
Spectrum from
Trivial: decision tree
Basic: basic conditions - MDP
Advanced: simulation based optimization, advanced modeling
Real time decision making
Spectrum from
Trivial: decision tree
Basic: basic conditions - MDP
Advanced: simulation based optimization, advanced modeling
Proactive event-driven computing
© 2012 IBM Corporation17IBM Haifa Research Lab
Enhancing the current event processing technology
Events may be uncertain: uncertainty about their occurrence ,occurrence time, and any of their attribute values; furthermore there may be uncertainty about relation between derived event and Situation, and propagation of uncertain values to derived events
Derived event may occur in the future)using predictive models – (
Running future time window in the presentFurthermore the semantics of derived event
Changes from virtual event to raw event
Applying event processing abstractions tostates – and use hybrid model
Proactive event-driven computing
© 2012 IBM Corporation18IBM Haifa Research Lab
Glo
bal
Dat
a V
olu
me
in E
xab
ytes
Sens
ors
(Inte
rnet
of T
hing
s)
Multiple sources: IDC,Cisco
100
90
80
70
60
50
40
30
20
10
Agg
rega
te U
ncer
tain
ty %
VoIP
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
2005 2010 2015
By 2015, 80% of all available data will be uncertain
Enterprise Data
Data quality solutions exist for enterprise data like customer, product, and address data, but
this is only a fraction of the total enterprise data.
By 2015 the number of networked devices will be double the entire global population. All
sensor data has uncertainty.
Social Media
(video, audio and text)
The total number of social media accounts exceeds the entire global
population. This data is highly uncertain in both its expression and content.
Proactive event-driven computing
© 2012 IBM Corporation19IBM Haifa Research Lab
The dimensions of “BIG DATA”Data has to be processed in higher Velocity
Data has high variability: poly-structured. Many sources: sensors, social media, multi-media…
Volumes of data are constantly growing
Veracity: Data has inherent uncertainty associated with it
Proactive event-driven computing
© 2012 IBM Corporation20IBM Haifa Research Lab
Uncertainty aspects
Meta-data representationof uncertainty
Removalof uncertainty
Propagation of uncertainty
Real-time decision underuncertainty
ByThresholds
Semantic propagation
ByRobust determination
Bayesian Nets
Monte-Carlo methods
Proactive event-driven computing
© 2012 IBM Corporation21IBM Haifa Research Lab
Adding canonic representation for uncertainty handling:
Uncertain whether an reported event has occurred (e.g. accident)
Uncertain whether an reported event has occurred (e.g. accident)
Uncertain what really happened. What is the type and magnitude of the accident (vehicles involved, casualties)
Uncertain what really happened. What is the type and magnitude of the accident (vehicles involved, casualties)
Uncertain when an event occurred (will occur): timing of forecasted congestion
Uncertain when an event occurred (will occur): timing of forecasted congestion
Uncertain where an event occurred (will occur): location of forecasted congestion
Uncertain where an event occurred (will occur): location of forecasted congestion
Uncertain about the level of causality between a car heading towards highway and a car getting into the highway
Uncertain about the level of causality between a car heading towards highway and a car getting into the highway
Uncertain about the accuracy of a sensor input: count of cars, velocity of cars…
Uncertain about the accuracy of a sensor input: count of cars, velocity of cars…
The pattern: more than 100 cars approach an area within 5 minutes after an accident derives a congestion forecasting
The pattern: more than 100 cars approach an area within 5 minutes after an accident derives a congestion forecasting
Uncertain about the validity of a forecasting pattern Uncertain about the validity of a forecasting pattern
Uncertain about the quality of the decision about traffic lights setting
Uncertain about the quality of the decision about traffic lights setting
Proactive event-driven computing
© 2012 IBM Corporation22IBM Haifa Research Lab
Real-time decision under uncertainty
Stochastic RT
optimization
Simulation-base RT
optimizationSimulation-
base RToptimization
Robust RTOptimization
Stochastic RTOptimization
Simulation-based RT optimization
Proactive event-driven computing
© 2012 IBM Corporation23IBM Haifa Research Lab
Learning patterns and causalities
EventPatterns
Pattern and causality acquisition
This is a direction to reduce the complexity of application development
There are challenges in doing it – since “detected situations ”are “inferred events” and may not be reflected in past events
Proactive event-driven computing
© 2012 IBM Corporation24IBM Haifa Research Lab
Summary
Proactive event driven computingis a new paradigm with potentialbig impact on society as well as future IT
There is an ecosystem of external collaborators mainly working on proposed EU project
The aim is to combine science and engineering to create a generic software platform