a complex event recognition architecture will fitzgerald kalamazoo college r. james firby i/net, inc
TRANSCRIPT
A Complex Event A Complex Event Recognition ArchitectureRecognition Architecture
Will FitzgeraldWill Fitzgerald
Kalamazoo CollegeKalamazoo College
R. James FirbyR. James Firby
I/NET, IncI/NET, Inc..
A Complex Event A Complex Event Recognition Recognition ArchitectureArchitecture
Protecting us from Protecting us from the Metal Horde!the Metal Horde!
Will FitzgeraldWill Fitzgerald
R. James FirbyR. James Firby
What is …What is …
A Complex Event?A Complex Event? Complex events are hierarchical, Complex events are hierarchical,
discrete, time-stamped structures discrete, time-stamped structures inferred from multi-channel, inferred from multi-channel, asynchronous signals.asynchronous signals.
What is …What is …
A Complex Event A Complex Event Recognition Architecture?Recognition Architecture?
A description or A description or implementation of typical implementation of typical patterns and recognition patterns and recognition algorithms for complex algorithms for complex events.events.
A real example…A real example…
Water Recovery Water Recovery System at NASA's System at NASA's Johnson Space Johnson Space Center Center Four complex Four complex
subsystems,subsystems, About 200 sensors About 200 sensors
and actuators, and actuators, Each subsystem Each subsystem
asynchronously asynchronously signals data.signals data.
Detecting “Safe Mode”Detecting “Safe Mode” When a problem is detected When a problem is detected
internally, the Water Recovery internally, the Water Recovery System attempts to go into “safe System attempts to go into “safe mode,” which occurs when the mode,” which occurs when the four subsystems are safed. four subsystems are safed.
““Safing” of the four subsystems Safing” of the four subsystems happen asynchronously. happen asynchronously.
““Safing” detection for each Safing” detection for each subsystem differs from one subsystem differs from one another.another.
On recognizing that the WRS On recognizing that the WRS has gone into safe mode, signal has gone into safe mode, signal an event that all subsystems an event that all subsystems have been safed. have been safed.
Another example…Another example…
To get directions to a To get directions to a location on the on-location on the on-board map, the user board map, the user says:says: ““Go here” andGo here” and Taps the display Taps the display
locationlocation
within 200 ms.within 200 ms.(CNN photo)
Parsing the worldParsing the world
Dynamic Predictive Memory Architecture Dynamic Predictive Memory Architecture (DPMA)(DPMA) KR and Semantic parsingKR and Semantic parsing Task execution and dialogue managementTask execution and dialogue management “ “complex, dynamic environments”complex, dynamic environments”
Do similar techniques apply to …Do similar techniques apply to … multi-channel, asynchronous sensors?multi-channel, asynchronous sensors? multi-modal interface input?multi-modal interface input?
A Complex Event Recognition A Complex Event Recognition ArchitectureArchitecture
What What assumptionsassumptions are reasonable to are reasonable to make about the form of input data?make about the form of input data?
What What useful general patternsuseful general patterns are are there in the data?there in the data?
What What recognition algorithms recognition algorithms do we do we need?need?
NLP AssumptionsNLP Assumptions
Input to Natural Language Processing Input to Natural Language Processing systems are typically assumed to be:systems are typically assumed to be: Discrete events of one type (“words”)Discrete events of one type (“words”) Single channelSingle channel Totally ordered by position; duration Totally ordered by position; duration
irrelevantirrelevant
11 22 33 44 55
timetime fliesflies likelike anan arrowarrow
More generally…More generally…
Events of various typesEvents of various types Over multiple channels and asynchronousOver multiple channels and asynchronous Duration of event often importantDuration of event often important ““Hierarchical” model still usefulHierarchical” model still useful
000000 005005 010010 015015 020020 025025 030030
putput thesethese herehere
click and drag click and drag taptap
Assumptions about EventsAssumptions about Events
Discrete: Discrete: Individually distinict, non-Individually distinict, non-continuous data (could be discretized).continuous data (could be discretized).
Time-stamped:Time-stamped: Event carries the start Event carries the start and end times (defining the event and end times (defining the event durationduration, which could be instanteneous)., which could be instanteneous).
Typed:Typed: Events form distinct types (e.g., Events form distinct types (e.g., wordswords vs. vs. tapstaps).).
Structured:Structured: Event may internal, Event may internal, hierarchical structure (“complex”). hierarchical structure (“complex”).
Standard Event PatternsStandard Event Patterns
Are there patterns of events which are Are there patterns of events which are particularly useful to identify?particularly useful to identify?
Are there recognition algorithms to identify those Are there recognition algorithms to identify those patterns?patterns?
Yes.Yes. ONE and BINDING ONE and BINDING IN-ORDER, ALL, ONE-OFIN-ORDER, ALL, ONE-OF Allen patternsAllen patterns WITHIN and WITHOUTWITHIN and WITHOUT
ONEONE and and BINDINGBINDING patterns patterns
ONEONE: The simple : The simple pattern of looking for pattern of looking for a single event (of a a single event (of a particular type).particular type).
BINDING:BINDING: ONEONE pattern plus collecting pattern plus collecting and constraining and constraining state.state.
Essentially “event-Essentially “event-driven” programming; driven” programming; the “stimulus” in S-R.the “stimulus” in S-R.
ON-CLICKON-CLICK A A ONEONE pattern if just pattern if just
looking for the ‘click’looking for the ‘click’ A A BINDINGBINDING pattern if pattern if
x,yx,y coordinates are coordinates are significant.significant.
IN-ORDERIN-ORDER patterns patterns
Events will occur “in order”Events will occur “in order” That is, saying two events, That is, saying two events, AA and and BB, occur in , occur in
order, the order, the start timestart time of B is of B is the the end timeend time of of A.A.
(IN-ORDER A B C D)(IN-ORDER A B C D) First an event of type A, then B, etc. First an event of type A, then B, etc.
IN-ORDERIN-ORDER as as NLPNLP
Combined with Combined with BINDINGBINDING and signaling of and signaling of subpatterns this is essentially a classic subpatterns this is essentially a classic natural language processing pattern.natural language processing pattern.
S S NP VP NP VP
NP NP DET N DET N
VP VP V NP V NP
The boy saw the girl.
[S [NP [DET the][N boy]] [VP [V saw] [NP [DET the] [N girl]]]]
ALL ALL PatternsPatterns
Events will all occur, but in any orderEvents will all occur, but in any order With this, we leave (our) standard NLP With this, we leave (our) standard NLP
approaches.approaches. For example, user will choose from all of the For example, user will choose from all of the
sets of options.sets of options. For example, all subsystems will be “safed”, For example, all subsystems will be “safed”,
but in any order.but in any order.
ALL patterns and ALL patterns and contradictioncontradiction
The problem: user or system “undoing” an The problem: user or system “undoing” an event that has already been seen event that has already been seen (interpreting events as state changes).(interpreting events as state changes).
Example: Class will start when all the Example: Class will start when all the students, Alice, Bob, Charles, Dominique, students, Alice, Bob, Charles, Dominique, have arrived.have arrived.
Consider this sequence for Consider this sequence for (ALL A B C D):(ALL A B C D):
1.1. Charles arrives.Charles arrives.
2.2. Alice and Bob arrive together.Alice and Bob arrive together.
3.3. Alice starts to sing.Alice starts to sing.
4.4. Charles leaves.Charles leaves.
5.5. Dominique arrives.Dominique arrives.
6.6. Charles arrives.Charles arrives.
Order is not relevant; Alice’s singing is not relevant; but Charles’s leaving undoes his earlier arrival.
ONE-OF PatternONE-OF Pattern
Look for any of a set of event formsLook for any of a set of event forms Example: Office hours begin as soon as Example: Office hours begin as soon as
one of the professors A,B,C or D arrives.one of the professors A,B,C or D arrives. (ONE-OF A B C D)(ONE-OF A B C D)
Time-based patternsTime-based patterns
Allen relationshipsAllen relationships WITHINWITHIN patterns patterns WITHOUTWITHOUT patterns patterns
Allen PatternsAllen Patterns
James Allen James Allen described the described the relationships between relationships between two intervals.two intervals.
Allen patterns look for Allen patterns look for temporal relationships temporal relationships between 2 events or between 2 events or an event and an an event and an interval.interval.
1.1. containscontains2.2. finishesfinishes3.3. startsstarts4.4. beforebefore5.5. meetsmeets6.6. overlapsoverlaps7.7. equalequal8.8. overlapped byoverlapped by9.9. afterafter10.10.met bymet by11.11.started bystarted by12.12.finished byfinished by13.13.during during
A containsB
A … overlaps B
WITHIN and WITHOUTWITHIN and WITHOUT
WITHIN WITHIN patterns patterns reflect that the reflect that the duration of an event is duration of an event is no longer than a no longer than a certain amount of certain amount of time.time.
E.g.E.g., an , an ALLALL pattern pattern wrapped in a wrapped in a WITHINWITHIN pattern.pattern.
WITHOUTWITHOUT patterns patterns reflect that an interval reflect that an interval of time will pass of time will pass without the without the occurrence of an occurrence of an event.event.
E.g.E.g., Sherlock , Sherlock Holmes’s Holmes’s “significance of the “significance of the barking dog.”barking dog.”
Pattern CombinationPattern Combination
““Go here” and a tap Go here” and a tap within 200 ms.within 200 ms.
(within
(all
(in-order go here)
(tap ?x ?y))
200 ms)(CNN photo)
Safe mode recognizerSafe mode recognizer
(define-recognizer (safing-complete)(define-recognizer (safing-complete)
(pattern (pattern
'(all '(all
(safing (system pbbwp) (status on))(safing (system pbbwp) (status on))
(safing (system ro) (status on))(safing (system ro) (status on))
(safing (system aes) (status on))(safing (system aes) (status on))
(safing (system pps) (status on))))(safing (system pps) (status on))))
(on-complete (st end)(on-complete (st end)
(signal-event '(all-safed) st end))) (signal-event '(all-safed) st end)))
Some details elided…
Parsing AlgorithmsParsing Algorithms
The parsing algorithms and recognizer The parsing algorithms and recognizer semantics are more fully described in the semantics are more fully described in the paper.paper.
Implementation DetailsImplementation Details
ConclusionsConclusions
Standard patterns of events.Standard patterns of events. Standard recognizers for these patterns.Standard recognizers for these patterns. Good for monitoring complex (internal) Good for monitoring complex (internal)
system state.system state. Useful for recognizing patterns of complex Useful for recognizing patterns of complex
events over multiple modes, over time.events over multiple modes, over time.
AcknowledgmentsAcknowledgments
Work done under NASA SBIR contract Work done under NASA SBIR contract NAS9-00122.NAS9-00122.
We would like to especially acknowledge We would like to especially acknowledge collaborators at NASA, including Debra collaborators at NASA, including Debra Schreckenghost, Pete Bonasso, Carrol Schreckenghost, Pete Bonasso, Carrol Thronesbery and others.Thronesbery and others.
Pulp Images from “Pulp of the Day”:Pulp Images from “Pulp of the Day”:groups.yahoo.com/group/pulpofthedaygroups.yahoo.com/group/pulpoftheday
Questions?Questions?