welcome welcome to the dagstuhl seminar on plan recognition please upload titles for the talks you...
TRANSCRIPT
![Page 1: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/1.jpg)
Welcome
Welcome to the Dagstuhl seminar on Plan Recognition
Please upload titles for the talks you want to give We would like everyone to have an opportunity to give
a short talk We have some panel ideas, but these are open
to reconsideration – contact me We will be scheduling incrementally
Scheduled through tomorrow… Schedules will be re-posted as updated… 1
![Page 2: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/2.jpg)
Panel ideas
Should there be a plan recognition competition?
Rational versus fallible agents Activity recognition, behavior recognition,
plan recognition, goal recognition Oh, my!
Full and partial observability Generative versus plan library approaches
2
![Page 3: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/3.jpg)
Schedule: Monday
AM: Welcome and survey PM:
Jerry Hobbs: discourse and plan recognition Short talks
George Ferguson Matthew Stone
Chris Baker: plan recognition and psychology Panel: a plan recognition competition?
Evening: get acquainted event
3
![Page 4: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/4.jpg)
Schedule: Tuesday
AM: Kathy Laskey: probabilistic methods for PR Short talks
Froduald Kabanza Francis Bisson Gita Sukthankar
PM: Tom Dietterich: learning and plan recognition Short talks
David Pattison Nate Blaylock
Panel: Rational versus fallible agents? 4
![Page 5: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/5.jpg)
Plan RecognitionHistorical Survey
Henry KautzUniversity of Rochester
Robert P. GoldmanSIFT, LLC
Dagstuhl, April 20115
Old school plan recognition…
![Page 6: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/6.jpg)
Outline
Dimensions of the plan recognition problem Historical survey of methods Challenges
6
![Page 7: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/7.jpg)
DIMENSIONS OF PLAN RECOGNITION
7
![Page 8: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/8.jpg)
Keyhole, intended and adversarial plan recognition
Keyhole Observer non-intrusively watches the agent Determine how an agent’s actions contribute to
achieving possible or stipulated goals Model
World Agent’s beliefs
8
![Page 9: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/9.jpg)
Keyhole, intended and adversarial plan recognition
Intended recognition Agent acts in order to signal his beliefs and
desires to other agents Speech acts – inform, request, …
Discourse conventions “The 3:15 train to Windsor?” “Gate 10” [Allen & Perrault]
Symbolic actions The Statue of Liberty 9/11?
The agent may require a model of the observer.9
![Page 10: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/10.jpg)
Keyhole, intended and adversarial plan recognition
Adversarial Agent acts in order manipulate the observer
Deception, bluffing, misdirection, etc. … Agent and observer will need sophisticated
models of each other’s inferences
10
![Page 11: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/11.jpg)
Ideal versus fallible agents
Mistaken beliefs John drives to Reagan, but flight leaves from
Dulles. The doctor bleeds the patient to cure disease.
Cognitive errors Distracted by the radio, John drives past the exit. Jill schedules a doctor’s appointment during her
office hours. Irrationality
John furiously blows his horn at the car in front of him.
11
![Page 12: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/12.jpg)
Output of plan recognition
Activity recognition Simply identify a known behavior pattern
Goals Recognize the objective, but not the specific
recipes used Plans Next action the agent will take? Best action to aid or counter the agent?
12
![Page 13: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/13.jpg)
Output of plan recognition: likelihood
Likelihood… Most likely interpretation? Distribution over plans and goals? The above have subtly different strengths and
weaknesses… Most critical plan or goal?
13
![Page 14: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/14.jpg)
Richness of plans
Are actions atomic? Or do they have parameters? Structure (e.g., cases)?
Do plans have structure and parameters? Coreference? The patient of the plan will be the destination of
step one and the patient of step two… Are there plan libraries at all?
14
![Page 15: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/15.jpg)
Other dimensions Reliable versus unreliable observations
“There’s a 80% chance John drove to Dulles.” Open versus closed worlds
Fixed plan library? Fixed set of goals? Fixed set of entities?
Metric versus non-metric time John enters a restaurant and leaves 1 hour later. John enters a restaurant and leaves 5 minutes later.
Single versus multiple ongoing plans “White knights”
Static versus evolving set of intentions Abandoning goals: I was going to drive to the store, but the weather was
too bad. Reacting to opportunities: I was going by the playroom on the way from
the laundry, so I picked up the toys.15
![Page 16: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/16.jpg)
Dimensions
Relation to agent
Model of agent
Goals and plans
Observ-ations
Infer Model (library)
Produce
Intended (possibly) Irrational
Static Noisy Activity Incomplete “The Answer”
Keyhole Partial knowledge
Partial Goal Best answer
Adversar-ial
Homo Econom-icus
Dynamic Complete Plan Complete Distribu-tion
Next action
16
![Page 17: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/17.jpg)
METHODS
17
![Page 18: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/18.jpg)
Earliest work
Generally in service of language understanding Often narrative understanding Understanding indirect speech acts
Allen & Perrault, “Analyzing Intention in Utterances,” AI, 1980
Rich vein of work using plan recognition in dialog understanding and IUI Will be hearing more from George Ferguson later today!
Methodologically: Mostly shared early enthusiasm for rule-based systems 18
![Page 19: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/19.jpg)
Hypothesize & Revise
The Plan Recognition Problem C. Schmidt, 1978 Related work from Yale AI Lab: Cullingford’s Script
Applier Mechanism, Wilensky’s PAM, etc., 1978 Charniak, Ms. Malaprop, 1978 – Frame-based and
used TMS
Based on psychological theories of human narrative understanding
Mention of objects suggest hypothesis
Pursue single hypothesis until matching fails
19
![Page 20: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/20.jpg)
Closed-world reasoning
A Formal Theory of Plan Recognition and its Implementation Henry Kautz, 1991
• Infers the minimum set(s) of independent plans that entail the observations
• Observations may be incomplete
• Infallible agent• Complete plan
library• Limited to pasta
preparation
20
![Page 21: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/21.jpg)
Parsing
Vilain 1990 --- use parsing results to characterize computational complexity of plan recognition There were earlier attempts to parse plans
Parsing techniques closely related to Closed-world reasoning (Built on Kautz and Allen) Find an explanation that covers all of the observations Parsing techniques deal poorly with partial ordering,
worse with interleaving Leads to:
Later work on stochastic parsing (Pynadath and Wellman) Attempts to exploit exotic parsing techniques (Geib)
21
![Page 22: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/22.jpg)
Abduction
Reason from effect to cause (C.S. Peirce) Explanation Diagnosis
People: Charniak Hobbs et al., TACITUS
Leads to interest in Bayes nets
22
![Page 23: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/23.jpg)
Bayes Nets DAG-structured
models of probability distributions
Came into the fore for diagnostic applications
Challenge: Static Bayes nets for complex domains can be extremely large
SprinklerRaining
Grasswet
23
![Page 24: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/24.jpg)
Bayes Nets Knowledge Based
Model Construction: Dynamically build Bayes nets showing how plans explain actions
Multiple goals Abstraction
hierarchies Equality reasoning for
coreference Poor treatment of
time
• “A Bayesian Theory of Plan Recognition,” Charniak and Goldman, AIJ, 1993.• “Interpretation as Abduction,” Hobbs, Stickel, Martin & Edwards, Proc. ACL,
1988.
“Jack went to the liquor store.”Was he shopping?
24
![Page 25: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/25.jpg)
More on Bayes net methods
Laskey and her colleagues have worked on military domains Further developed KBMC techniques (e.g. query
completeness); coreference, identity uncertainty Many related techniques
E.g., Hobbs et al. cost-based abduction ATMSes (d’Ambrosio, Provan, Charniak &
Goldman) Horn logic (Poole)
25
![Page 26: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/26.jpg)
Pending sets
A new model of plan recognition. Goldman, Geib, and Miller,1999“A probabilistic plan recognition algorithm based on plan tree grammars,”
Geib and Goldman, AIJ, 2009.
Explicitly models the agent’s “plan agenda” using Poole’s “probabilistic Horn abduction” rules
Bridge between Bayes net and HMM frameworks
Handles multiple concurrent interleaved plans & negative evidence
Number of different possible pending sets can grow exponentially
Happen(X,T+1) Pending(P,T), X in P, Pick(X,P,T+1).
Pending(P’,T+1) Pending(P,T), Leaves(L), Progress(L, P, P’, T+1).
26
![Page 27: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/27.jpg)
Version Space Algebra
A sound and fast goal recognizer Lesh & Etzioni, IJCAI 1995 Programming by Demonstration Using Version Space Algebra Lau,
Wolfman, Domingos, Weld. Related to later work on plan-recognition through planning
• Recognizes novel plans
• Complete observations
27
![Page 28: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/28.jpg)
CHALLENGES
28
![Page 29: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/29.jpg)
Evaluation
Ground truth Difficult to get labeled data Epistemic question --- do our proposed labelings
correspond to any real ground truth? Prediction tasks
Next action? Future action? Good choice of assistive action?
Countermeasure? Can prediction act as proxy for ground truth?
29
![Page 30: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/30.jpg)
Epistemic question
What is the status of the recipes that we postulate as explanations for actions?
Are they taken as being real in some sense? Corresponding to mental contents? Identified regularities that really exist in the world? Data structures that just exist for our convenience
30
![Page 31: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/31.jpg)
Computational difficulties
Computational complexity Theoretical results Practical results Challenges from domains
Some domains inherently ambiguous Adversarial reasoning Do we need game-theoretic reasoning
Cooperative as well as adversarial
31
![Page 32: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/32.jpg)
Plan libraries
Engineered? Learned? Something in between?
Learned ones often seem impoverished Engineering seems impossible!
32
![Page 33: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/33.jpg)
Learning
Structural learning Learn the contents of plan libraries (in one form or
another) Parameter learning
Adjust parameters of known libraries Both offer challenges related to those of
evaluation Plan recognition may be done in service of
learning, as well as the other way around. Infer goals to learn novel recipes 33
![Page 34: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/34.jpg)
Imperfections
Imperfect agents Imperfect information Imperfect reasoning Imperfect task performance Challenging for non-empirical algorithms
Imperfect observations Imperfect models
Including seemingly-irrelevant actions
34
![Page 35: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/35.jpg)
User models
In many domains, the behaviors exhibited are not just a function of the actions, goals and plans, but agent characteristics, as well.
Developing clean ways to combine agent-dependent and – independent information is a challenge going forward. Often per-agent training is unacceptable.
35
![Page 36: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/36.jpg)
Sensing In many cases it is difficult to sense the agents’
actions: Labeling actions in primitive sensor data
Vision Network packets Linguistic utterances
Hardware/software hybrid systems E.g., oil refinery --- user can go out and use a wrench un-
observed Conventional software
Even Horvitz et al. report difficulties “seeing” actions of Microsoft Office users
Mixed streams Individual actions in network packet streams
36
![Page 37: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/37.jpg)
Coreference and quantification
In some domains we don’t have object identity and permanence and the number of agents simply handed to us. Story understanding Military situation interpretation
Identity hypotheses enter into plan recognition
37
![Page 38: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/38.jpg)
Anomaly detection
Often appealed to as a solution for detecting some phenomenon that is difficult to model: Intrusion behavior in computer security Terrorist behavior in tracking and camera data Dementia-induced behavior in tracking elderly
subjects Accuracy requires deep understanding of the
models’ properties Stationarity (often violated in computer security) “Size” and “shape” of normal behaviors
As always, it’s hard to get something for nothing.38
![Page 39: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/39.jpg)
The Role of State
Many (but not all) plan recognition systems represent only the state of the planning agent. The state of the environment is modeled implicitly,
if at all.
39
![Page 40: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/40.jpg)
Groups
Teamwork Friendly: recognize teammates’ intentions to
coordinate and aid Hostile: recognize opponents’ intentions to hinder
and obstruct Role recognition
40
![Page 41: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/41.jpg)
Hypothesis retrieval
Some early work assumed that there were enough candidate hypotheses that retrieval could be an issue
41
![Page 42: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/42.jpg)
Predictive and explanatory inference
A lot of concern in early work about combining top-down and bottom-up inference
42
![Page 43: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/43.jpg)
Actions with weak diagnostic power
E.g., computer security We would desperately like to know the
attacker’s motivations But what do we do with
Get access to the target Gain administrator privileges on the target…
43
![Page 44: Welcome Welcome to the Dagstuhl seminar on Plan Recognition Please upload titles for the talks you want to give We would like everyone to have an opportunity](https://reader036.vdocuments.mx/reader036/viewer/2022081602/551749df550346b1338b47ec/html5/thumbnails/44.jpg)
COFFEE AND THEN HENRY’S TURN…
44