osdc complex event processing
DESCRIPTION
A presentation I did at OSDC in brisbane, 2009, on complex event processing. Includes a case study.TRANSCRIPT
![Page 1: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/1.jpg)
1
CEP – complex event processingOSDC, Brisbane, November 2009Michael NealeJBoss/Red Hat R&D
![Page 2: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/2.jpg)
2
Michael NealeR&D w. JBoss (specialise in drools, now
“cloud research”)
Open source history (user -> fulltime developer “acquired” by jboss 2005).
Thanks OSDC organisers !
me on the web:
www.michaelneale.net, twitter.com/michaelneale, michaelneale.blogspot.com
![Page 3: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/3.jpg)
3
Outline• Define CEP?
• Where is it used
• Events and patterns
• Sliding windows
• Temporal reasoning
• Defining an event “object”
• Libraries: Drools, Esper
• Sample scenario
![Page 4: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/4.jpg)
4
Definition• Treat “inputs” as “events” (time
based)
• Look for patterns/correlations to extract some meaning (“an inferred event/fact”)
• Act on these patterns (return a result, perform a calculation, send a message etc)
• Sometimes also called:
– Event Stream Processing
– Temporal correlation
![Page 5: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/5.jpg)
5
eg• Event: [Hand In Air]
• Pattern: Lots of [Hand In Air] (possibly waving, as if they didn't care), happening in close (time and space) quarters...
• Infer: A party
![Page 6: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/6.jpg)
6
Huge number of events, but only a few of real interest
Usually events are immutable
Usually queries/rules have to run in reactive mode
Strong temporal relationships between events
Individual events are usually not important
The composition and aggregation of events is important
![Page 7: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/7.jpg)
7
Where is it used? • Logistics: Package en-route
tracking/warning
• Stock market events/trades
– Provide real time alerts to end users
• Any where you have “needle in a haystack” problem, involving time based events
![Page 8: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/8.jpg)
8
Events and patterns• Events immutable** facts
(objects/records)
– Eg a reading from a sensor, a stock value, a “fact-oid”
• Patterns are logic constraints (predicate/higher order statements)
– Think like the “where” part of a SQL query
• Patterns search for particular events...
![Page 9: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/9.jpg)
9
Esper example:
select * from StockTickEvent.win:time(30 sec) where price > 20select avg(price) from StockTickEvent.win:time(30 sec) ...
Drools Fusion example:
when StockTick(price > 20) over window:time( 30s )
.. and with some temporal correlation:
$eventA : StockTick( this coincides[15s, 10s] $eventB )
Event instance of StockTick
![Page 10: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/10.jpg)
10
Patterns• Are used to detect specific events
(using constraints and higher order logic)
• Are used to detect co-incidences (over time periods)
• Allow you to be declarative (once again, think SQL, but involving time)
![Page 11: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/11.jpg)
11
Sliding time windows• Declare what “window” of time you
care about
• Needed to narrow down from huge number of events
• Help identify “co-incidences” which might be more then just co-incidences !
• Optional: maybe you care about ALL events, maybe just close together
![Page 12: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/12.jpg)
12
Temporal reasoning• A generalisation of “looking for co-
incidences” of interest
• Eg Hangover often follows from XXXX consumption, within certain time windows. Perhaps all of the time? (“forall”) Some of the time? (“exists”)
• Look for sets of events, related in time, to infer what is really “going on”
• Sets CEP apart from offline/batch processing
![Page 13: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/13.jpg)
13
• Event and Time semantics:
– Point in Time
– Interval-based
• Unified semantics for event correlation over time
• Temporal Constraints:
– Set of operators to express temporal relationship between events
Event semantics
![Page 14: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/14.jpg)
14
Event semanticsJF Allen defined a set of Time Event
semantics:
![Page 15: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/15.jpg)
15
Temporal operatorsAfter
Before
Coincides
During
Finishes
Finished By
Includes
Meets
Met By
Overlaps
Overlapped By
Starts
Started By
![Page 16: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/16.jpg)
16
Defining events• Generally classes defined in host
language (eg java, C#)
• Some extra metadata needed to define expiry, duration, timestamp etc (all optional).
• Expiry important for memory management
– Large amounts of data
– Why keep it in memory if not needed
– Very stateful
![Page 17: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/17.jpg)
17
Defining events// declaring existing class
import some.package.VoiceCall
declare VoiceCall
@role( event )
@timestamp( calltime )
@duration( duration )
end
// generating an event class
declare StockTick
@role( event )
symbol : String
price : double
end
![Page 18: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/18.jpg)
18
Drools Fusion• jboss.org/drools
• Submodule of Drools project
• Uses rule language to express CEP patterns
– Same “drl” language as normal rules
– Rules can take action directly, or be queries to return results
![Page 19: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/19.jpg)
19
rule "Sound the alarm in case temperature rises above threshold"
when
TemperatureThreshold( $max : max )
Number( doubleValue > $max ) from accumulate( SensorReading( $temp : temperature ) over window:time( 10m ), average( $temp ) )
then
// sound the alarm
![Page 20: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/20.jpg)
20
Esper• esper.codehaus.org
• Takes “temporal SQL” approach
• Return results for matches, a la SQL database (but its realtime events)
• SQL skill portability ?
![Page 21: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/21.jpg)
21
select fraud.accountNumber as accntNum, fraud.warning as warn, withdraw.amount as amount,
MAX(fraud.timestamp, withdraw.timestamp) as timestamp, 'withdrawlFraud' as desc
from FraudWarningEvent.win:time(30 min) as fraud,
WithdrawalEvent.win:time(30 sec) as withdraw
where fraud.accountNumber = withdraw.accountNumber
![Page 22: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/22.jpg)
22
Deployment• Usually a dedicated server/service
endpoint
– (but could be embedded in your app if JVM friendly)
• (eg wrapped around a REST servlet, or MQ)
• Sits there, gobbling up messages as fast as it can
![Page 23: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/23.jpg)
23
Concurrency...Need for “realtime” performance
Need to make use of threads/processes
Use of “rulebase partitioning” to separate less coupled data to allow this
So: User doesn't have to worry
![Page 24: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/24.jpg)
24
FedEx examplePresented at JavaOne, Adobe MAX etc..
FedEx Custom Critical:
1: En-Route Tracking Situational Awareness
2: Capacity Allocation Managementhttp://developers.sun.com/learning/javaoneonline/sessions/2009/pdf/TS-4475.pdf
![Page 25: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/25.jpg)
25
![Page 26: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/26.jpg)
26
![Page 27: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/27.jpg)
27
![Page 28: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/28.jpg)
28
![Page 29: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/29.jpg)
29
A flex front end
![Page 30: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/30.jpg)
30
![Page 31: Osdc Complex Event Processing](https://reader033.vdocuments.mx/reader033/viewer/2022061218/54b755484a795967588b4798/html5/thumbnails/31.jpg)
31
Thanks, Q&A