uncertainty, action, and interaction eric horvitz microsoft research may 2002 uncertainty, action,...
TRANSCRIPT
![Page 1: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/1.jpg)
Uncertainty, Action, Uncertainty, Action,
and Interactionand Interaction
Eric HorvitzEric HorvitzMicrosoft ResearchMicrosoft Research
May 2002May 2002
![Page 2: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/2.jpg)
Toward Mixed-Initiative User InterfacesToward Mixed-Initiative User Interfaces
Designs that assume Designs that assume from the ground upfrom the ground up that that user may guide, collaborate with automated user may guide, collaborate with automated service to achieve desired resultsservice to achieve desired results
User
Automation
![Page 3: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/3.jpg)
Principles of Mixed-Initiative InteractionPrinciples of Mixed-Initiative Interaction Endow system with ability to infer the likelihood Endow system with ability to infer the likelihood
of a user’s goals, intentionsof a user’s goals, intentions
Attempt to scope precision of action to match Attempt to scope precision of action to match goals and uncertainties goals and uncertainties
Determine the expected value of action given Determine the expected value of action given costs and benefits of actioncosts and benefits of action
Consider status of a user’s attention in timing of Consider status of a user’s attention in timing of actionaction
Allow for dialog at appropriate times to resolve Allow for dialog at appropriate times to resolve key ambiguitieskey ambiguities
![Page 4: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/4.jpg)
Provide efficient means for agent–user Provide efficient means for agent–user collaboration to refine guessescollaboration to refine guesses
Allow efficient Allow efficient directdirect invocation and invocation and terminationtermination
Seek innovative designs that maximize benefit Seek innovative designs that maximize benefit of service, minimize the cost of poor guessesof service, minimize the cost of poor guesses
Allow for natural assumptions of shared Allow for natural assumptions of shared memory of recent interactionsmemory of recent interactions
Continue to learn by observingContinue to learn by observing
Principles of Mixed-Initiative InteractionPrinciples of Mixed-Initiative Interaction
![Page 5: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/5.jpg)
Key goal: Provide the user with clear Key goal: Provide the user with clear advance toward goals advance toward goals
Automated, flexible scoping of Automated, flexible scoping of automated service to precision automated service to precision matching task uncertainty, context matching task uncertainty, context
Automated Scoping and Precision of Automated Scoping and Precision of ServiceService
Prefer automation to do less, but do it correctly
![Page 6: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/6.jpg)
Automated Reasoning about the Automated Reasoning about the Uncertainty of a User’s GoalsUncertainty of a User’s Goals
Automated reasoners must guess about a Automated reasoners must guess about a user’s user’s goalsgoals and and desiredesire for services for services
Good guesses can be quite valuableGood guesses can be quite valuable
……but guessing wrong can be costlybut guessing wrong can be costly
Even valuable automation can be distracting Even valuable automation can be distracting and steal user’s scarce attentional resourcesand steal user’s scarce attentional resources
![Page 7: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/7.jpg)
Minimizing Cost of Guessing WrongMinimizing Cost of Guessing Wrong
Seek design innovation: Advice / assistance Seek design innovation: Advice / assistance valuable when right, but errors with minimal valuable when right, but errors with minimal low costlow cost Natural gestures for declining service Natural gestures for declining service Avoid grabbing focus Avoid grabbing focus Alternate channel overlay: NASA Vista display managerAlternate channel overlay: NASA Vista display manager Nondistracting, simple guessing: Vellum gridpoint guessesNondistracting, simple guessing: Vellum gridpoint guesses
More graceful interaction with potentially More graceful interaction with potentially focused userfocused user
Better timing of services in sync with Better timing of services in sync with availability of attentionavailability of attention
![Page 8: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/8.jpg)
Probability, Utility, & Probability, Utility, & Mixed Initiative InteractionMixed Initiative Interaction
Perspective for designPerspective for designSpecific functions, layering of componentrySpecific functions, layering of componentry Foundations of intelligenceFoundations of intelligence
??*&(#))(@%+%%$#*%$#*&%*&(^*^
Infrastructure, fabric for UI innovationInfrastructure, fabric for UI innovation
![Page 9: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/9.jpg)
ProbabilitiesProbabilities
Infer likelihoods of key uncertainties, take ideal actions
Uncertainty and HCIUncertainty and HCI
• User queryUser query
• User activity User activity
• Content at focusContent at focus
• Data structures Data structures
• User locationUser location
• User profileUser profile
• Vision, speech, soundVision, speech, sound
*Utility-directed actionUtility-directed action
Meshing learning & reasoning with UI designMeshing learning & reasoning with UI design
![Page 10: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/10.jpg)
Beliefs & IntentionsBeliefs & Intentions What does a user believe? What are the user’s goals?What does a user believe? What are the user’s goals?
AttentionAttention What is the user’s workload? What is a user attending to? What is the user’s workload? What is a user attending to?
What What will will a user attend to? What a user attend to? What shouldshould a user attend to? a user attend to? PreferencesPreferences
What does the user like and dislike—and how much?What does the user like and dislike—and how much?
InitiativeInitiative What is the cost and benefit of interaction, interruption, What is the cost and benefit of interaction, interruption,
intervention?intervention? What is the right mix of user / system initiatives?What is the right mix of user / system initiatives?
Critical UncertaintiesCritical Uncertainties
![Page 11: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/11.jpg)
LumièreLumière Project Project
User’s ProfileUser’s ProfileUser’s GoalsUser’s Goals
User’s NeedsUser’s Needs
User ActivityUser Activity
Actions + Words Actions + Words Goals Goals
Joint work with J. Breese, D. Heckerman, K. Rommelse, D. Hovel, et al.
![Page 12: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/12.jpg)
Studies with Human Subjects Studies with Human Subjects
![Page 13: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/13.jpg)
ChallengesChallenges
Architectures for intelligent user Architectures for intelligent user interactioninteraction
Reasoning over timeReasoning over time Sensing activity from systems and Sensing activity from systems and
applicationsapplications Integration of probabilistic information Integration of probabilistic information
retrievalretrieval Models of a user’s competencies over Models of a user’s competencies over
timetime
![Page 14: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/14.jpg)
Big PictureBig Picture
ControlControlNewNew
Perceptions, Perceptions, InteractionsInteractions
EventsEvents UncertainUncertainInference aboutInference about
User, WorldUser, World
ComputationComputationof Ideal UI Actionof Ideal UI Action
Ideal Ideal ActionsActions
EventsEventsSynthesisSynthesis
LearningLearningModelsModels
![Page 15: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/15.jpg)
Inference about a User’s Time-Inference about a User’s Time-Dependent GoalsDependent Goals
TimeTime
Goalt-n
Ej,t-n Ei,t-n
Goalto
Ej,to Ei,to
Goalt-1
Ej,t-1 Ei,t-1
Profile
Profile
Profile
![Page 16: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/16.jpg)
Representing and Updating a Representing and Updating a Persistent “Competency Terrain”Persistent “Competency Terrain”
Skill Catogories
Co
mp
eten
cy
![Page 17: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/17.jpg)
Representing and Updating a Representing and Updating a Persistent “Competency Terrain”Persistent “Competency Terrain”
User’s Skills
Co
mp
eten
cy
![Page 18: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/18.jpg)
TimeTime
Toward a “peripheral nervous system” for Toward a “peripheral nervous system” for sensing user activity sensing user activity SDK with event abstraction languageSDK with event abstraction language Compiler for defining filters for user activityCompiler for defining filters for user activity
Sensing Context and ContentSensing Context and Content
![Page 19: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/19.jpg)
Atomic EventsAtomic Events Modeled EventsModeled Events
TimeTime
EventSource 1
EventSource 2
EventSource n
Eve Event-Specification
Language
Abstraction of Events
![Page 20: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/20.jpg)
Overall LumiOverall Lumièère Architecturere Architecture
BayesianBayesianInferenceInference
EventsEvents
TimeTime
• QueryQuery
• ActionsActionsEvent SynthesisEvent Synthesis
Control SystemControl System
![Page 21: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/21.jpg)
• Probability user Probability user
desires assistancedesires assistance
Lumiere Inference and ActionLumiere Inference and Action
![Page 22: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/22.jpg)
InitiativeInitiative
User vs. system initiativeUser vs. system initiative Allowing fluid collaboration via a mix of Allowing fluid collaboration via a mix of
initiativesinitiatives Toward Toward principles of mixed-initiative principles of mixed-initiative
interactioninteraction Projects: Projects: Lookout, DeepListener, QuartetLookout, DeepListener, Quartet
Reasoning about initiative is a Reasoning about initiative is a
high-payoff opportunity area high-payoff opportunity area
for HCI, Ubicomp, IUIfor HCI, Ubicomp, IUI
![Page 23: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/23.jpg)
Learning by watchingLearning by watching Costs-benefit analysis of initiativeCosts-benefit analysis of initiative Minimize disruption: Prefer Minimize disruption: Prefer doing less,doing less,
but doing it correctlybut doing it correctly
Initiative & Interaction: LookoutInitiative & Interaction: Lookout
?? Critical decision: Critical decision: Do nothing.Do nothing. Ask? Ask? Just do it?Just do it?
Joint work with Andy Jacobs
![Page 24: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/24.jpg)
Real-TimeProbabilistic Inference
Cost--Benefit Analysis
User Actions / Context
UI / Service
• Watch user’s behavior• Store cases, timing info• Learn model from data
Learning and Real-Time BehaviorLearning and Real-Time Behavior in Lookout in Lookout
![Page 25: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/25.jpg)
Lookout in Handsfree ModeLookout in Handsfree Mode
![Page 26: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/26.jpg)
Preferences and InitiativePreferences and Initiative
Expected utility as fundamental in Expected utility as fundamental in decisions about servicesdecisions about services
A: Computer takes action i
A: No action i
D: User desires action iD: User does not desire action i
Service
User’s Desire
u(A,D)
u(A,D)
u(A,D)
u(A,D)
Act
No act
Desired Undesired
![Page 27: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/27.jpg)
Action
No Action
P*
1.0
1.0
0.0
u(A,D)
u(A,D)
u(A,D)
u(A,D)
p(D|E)
eueu((AA) = ) = jj uu((AAii,,DDjj) ) pp((DDjj||EE))eueu((AA) = ) = jj uu((AAii,,DDjj) ) pp((DDjj||EE))eueu((AA) = ) = pp((DD||EE) ) uu((AA,,DD) + ) + pp((DD||EE) ) uu((AA,,DD) ) eueu((AA) = ) = pp((DD||EE) ) uu((AA,,DD) + ) + pp((DD||EE) ) uu((AA,,DD) ) eueu((AA) = ) = pp((DD||EE) ) uu((AA,,DD) + [1 - ) + [1 - pp((DD||EE)] )] uu((AA,,DD) ) eueu((AA) = ) = pp((DD||EE) ) uu((AA,,DD) + [1 - ) + [1 - pp((DD||EE)] )] uu((AA,,DD) )
eueu((AA) = ) = pp((DD||EE) ) uu((AA,,DD) + [1 - ) + [1 - pp((DD||EE)] )] uu((AA,,DD) ) eueu((AA) = ) = pp((DD||EE) ) uu((AA,,DD) + [1 - ) + [1 - pp((DD||EE)] )] uu((AA,,DD) )
Preferences and InitiativePreferences and Initiative
![Page 28: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/28.jpg)
1.0
1.0
0.0
u(A,D)
u(A,D)
u(A,D)
u(A,D)
p(D|E)
Action
No Action
P*
User rushed
Initiative and ContextInitiative and Context
Utility of outcomes as function of context,Utility of outcomes as function of context, uu((AA,,D,D,))
![Page 29: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/29.jpg)
1.0
1.0
0.0
u(A,D)
u(A,D)
u(A,D)
p(D|E)
Action
No Action
P*
No Action
u(A,D)
User rushed
Increase in Amountof Screen Real Estate
u(A,D)
Initiative and ContextInitiative and Context
Utility of outcomes as function of context,Utility of outcomes as function of context, uu((AA,,D,D,))
![Page 30: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/30.jpg)
1.0
1.0
0.0
u(A,D)
u(A,D)
u(A,D)
p(D|E)
Action
No Action
P*
No Action
u(A,D)
User rushed
Increase in Amountof Screen Real Estate
u(A,D)
Initiative and ContextInitiative and Context
Utility of outcomes as function of context,Utility of outcomes as function of context, uu((AA,,D,D,))
![Page 31: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/31.jpg)
1.0
1.0
0.0
u(A,D)
u(A,D)
u(A,D)
p(D|E)
Action
No Action
P*
No Action
u(A,D)
User rushed
Increase in Amountof Screen Real Estate
u(A,D)
Initiative and ContextInitiative and Context
Utility of outcomes as function of context,Utility of outcomes as function of context, uu((AA,,D,D,))
![Page 32: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/32.jpg)
Initiative and ContextInitiative and Context
1.0
1.0
0.0
u(A,D)
u(A,D)
u(A,D)
u(A,D)
p(D|E)
Action
No Action
P*
No Action
u(A,D)
u(A,D)
Increase in Amountof Screen Real Estate
u(A,D)
User rushed
Utility of outcomes as function of context,Utility of outcomes as function of context, uu((AA,,D,D,))
![Page 33: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/33.jpg)
1.0
1.0
0.0
u(A,D)
u(A,D)
u(A,D)
u(A,D)
p(D|E)
Action
P*
Ask
No Action
Expected value of engaging the user in dialogueExpected value of engaging the user in dialogue
Engaging in Dialog about InitiativeEngaging in Dialog about Initiative
![Page 34: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/34.jpg)
![Page 35: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/35.jpg)
Varying Precision of ServiceVarying Precision of Service
Consider contributions across Consider contributions across a spectrum of precision a spectrum of precision
Assume user will refine partial resultsAssume user will refine partial results Under uncertainty, trade off reduced precision for Under uncertainty, trade off reduced precision for
higher accuracyhigher accuracy
ApptDay
Week
![Page 36: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/36.jpg)
Timing of InitiativeTiming of Initiative
Timing is critical: consider patterns of attentionTiming is critical: consider patterns of attention Record length of message and dwell time before Record length of message and dwell time before
calendar invokedcalendar invoked Perform regressionPerform regression
0
2
4
6
8
10
0 500 1000 1500 2000 2500
Length of original message (bytes)Ob
serv
ed d
wel
l bef
ore
act
ion
(sec
)
![Page 37: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/37.jpg)
![Page 38: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/38.jpg)
Conversational ArchitecturesConversational Architectures ProjectProject
DeepListenerDeepListener Bayesian ReceptionistBayesian Receptionist QuartetQuartet
![Page 39: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/39.jpg)
Why do people find it more difficult and Why do people find it more difficult and frustrating to converse with a spoken dialog frustrating to converse with a spoken dialog
system than with a person?system than with a person?
QuestionQuestion
Several answersSeveral answers
• Poor recognition of wordsPoor recognition of words
• Meaning too difficult to captureMeaning too difficult to capture
• Lack of precise user modelsLack of precise user models
• Different social and personality Different social and personality dynamicsdynamics
Interpreting spoken language abounds with Interpreting spoken language abounds with uncertaintyuncertainty
![Page 40: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/40.jpg)
IntuitionsIntuitions Despite uncertainty in humanDespite uncertainty in human––human conversation human conversation
people manage to understand each other quite people manage to understand each other quite well.well.
People consider the source of their uncertainties People consider the source of their uncertainties and pursue actions to resolve confusions.and pursue actions to resolve confusions.
RecognitionRecognition LanguageLanguage Context, topic, meaningContext, topic, meaning Frank troubleshootingFrank troubleshooting
GoalGoal: Models and inference methods that seek : Models and inference methods that seek mutual understanding under uncertainty given mutual understanding under uncertainty given inescapably unreliableinescapably unreliable components. components.
![Page 41: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/41.jpg)
GroundingGrounding
People resolve uncertainties through a People resolve uncertainties through a process of process of groundinggrounding
Process by which participants establish Process by which participants establish and maintain the mutual belief that their and maintain the mutual belief that their utterances have been understood well utterances have been understood well enough for current purposesenough for current purposes
-Clark & Schaefer, 1987 -Clark & Schaefer, 1987
![Page 42: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/42.jpg)
DeepListenerDeepListener
Utility-directed clarification dialogUtility-directed clarification dialog Formal model of “understood well enough”Formal model of “understood well enough” Development environmentDevelopment environment Assessment toolsAssessment tools Focus: Spoken command and control Focus: Spoken command and control
systemssystems
![Page 43: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/43.jpg)
Stakes, Likelihoods, and Stakes, Likelihoods, and Clarification ActionsClarification Actions
Consider stakes of real-world action being Consider stakes of real-world action being consideredconsidered
Should I format your hard drive? Should I format your hard drive?
Should I try to schedule that?Should I try to schedule that?
Should I demolish the King Dome Should I demolish the King Dome nownow??
Consider uncertaintiesConsider uncertainties Consider expected utility of alternative “repair” Consider expected utility of alternative “repair”
actionsactions Costs and benefits of real-world action vs. alternative Costs and benefits of real-world action vs. alternative
dialog repair actionsdialog repair actions
![Page 44: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/44.jpg)
Infer likelihoods of alternative Infer likelihoods of alternative spoken spoken intentionsintentions Likelihoods of different Likelihoods of different spoken intentionsspoken intentions given acoustics given acoustics Optionally condition on goals inferred by user model Optionally condition on goals inferred by user model
external to the speech systemexternal to the speech system
Compute Compute clarificationclarification or real-world or real-world actions with highest expected utilityactions with highest expected utility
Fuse multiple attempts with Bayesian Fuse multiple attempts with Bayesian model that considers confidencesmodel that considers confidences Consider history of utterances within a sessionConsider history of utterances within a session No reason to start over at each turn! ..Leverage what was No reason to start over at each turn! ..Leverage what was
heard beforeheard before
ApproachApproach
![Page 45: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/45.jpg)
Decision ModelDecision Model
Speaker’s Goal(t-1)
User’s SpokenIntention(t-1)
Dialog or Domain-Level
Action(t-1)
ASR Candidate 1Confidence(t-1)
Utility(t-1)
User Actions(t-1)Content at Focus (t-1)
ASR Candidate nConfidence(t-1)
Context
. . .
ASR ReliabilityIndicator(t-1)
External
User Model
![Page 46: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/46.jpg)
Speaker’s Goal(t-1)
User’s SpokenIntention(t-1)
Dialog or Domain-Level
Action(t-1)
ASR Candidate 1Confidence(t-1)
Utility(t-1)
User Actions(t-1)Content at Focus (t-1)
ASR Candidate nConfidence(t-1)
Context
...
Speaker’s Goal(t)
User’s SpokenIntention(t)
Dialog or Domain-Level
Action(t)
ASR Candidate 1Confidence(t)
Utility(t)
User Actions(t)Content at
Focus (t)
ASR Candidate nConfidence(t)
Context
...
ASR ReliabilityIndicator(t-1)
ASR ReliabilityIndicator(t-1)
Dynamic Model for Reasoning Over Dynamic Model for Reasoning Over Multiple TurnsMultiple Turns
![Page 47: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/47.jpg)
Dialog Actions under ConsiderationDialog Actions under Consideration
Perform Perform real-world actionreal-world action (e.g., (e.g., implode the King implode the King Dome nowDome now))
Ask for repetitionAsk for repetition to clarify to clarify
Note hesitationNote hesitation or reflection and try again or reflection and try again
Note potential Note potential overhearingoverhearing of noise and inquireof noise and inquire
Note inattentionNote inattention of user and try to acquire user’s of user and try to acquire user’s attentionattention
Don’t perform action and just Don’t perform action and just go awaygo away
Note problem with conversational interaction and Note problem with conversational interaction and attempt to attempt to troubleshoottroubleshoot
Example: DeepListener for handling Example: DeepListener for handling
confirmation, negationconfirmation, negation
![Page 48: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/48.jpg)
DeepListener: SDK and Real-Time DeepListener: SDK and Real-Time Clarification Dialog SystemClarification Dialog System
Dynamic Bayesian network modeling and Dynamic Bayesian network modeling and inference inference
MS command and control speech systemMS command and control speech system Backchannel animations: MS AgentBackchannel animations: MS Agent
![Page 49: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/49.jpg)
DeepListener: SDK and Real-Time DeepListener: SDK and Real-Time Clarification Dialog SystemClarification Dialog System
![Page 50: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/50.jpg)
Accruing Evidence Over Repeated UtterancesAccruing Evidence Over Repeated Utterances
![Page 51: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/51.jpg)
Beliefs and Actions for Clarification DialogBeliefs and Actions for Clarification Dialog
Clarification
![Page 52: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/52.jpg)
Beliefs about Spoken IntentionBeliefs about Spoken Intention
yes
no overheard
yes
no overheard
Inferred beliefs
![Page 53: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/53.jpg)
Expected Utility of Alternate ActionsExpected Utility of Alternate Actions
Repeat
Again
EngageNoise
Tshoot Disengage
Attention
Repeat
Again
EngageNoise
Tshoot Disengage
Attention
Expected utility of actions
![Page 54: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/54.jpg)
Preference elicitationPreference elicitation For developers (assessment “at factory”)For developers (assessment “at factory”) For users! Prototypical patterns, assessment For users! Prototypical patterns, assessment
wizard, direct detailed assessmentwizard, direct detailed assessment
Toward a more general SDK for command Toward a more general SDK for command and control dialog (e.g., telephony systemsand control dialog (e.g., telephony systems
Preference Assessment and EncodingPreference Assessment and Encoding
![Page 55: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/55.jpg)
Assessing Preferences on OutcomesAssessing Preferences on Outcomes
![Page 56: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/56.jpg)
Speaker’s Goal(t-1)
User’s SpokenIntention(t-1)
Dialog or Domain-Level
Action(t-1)
ASR Candidate 1Confidence(t-1)
Utility(t-1)
ASR Candidate nConfidence(t-1)
External UserModel(t-1)
.. .
Speaker’s Goal(t)
User’s SpokenIntention(t)
ASR Candidate 1Confidence(t)
ASR Candidate nConfidence(t)
ExternalUser Model(t)
.. .
Dialog or Domain-Level
Action(t)
Utility(t)
ASR ReliabilityIndicator(t-1)
ASR ReliabilityIndicator(t)
Troubleshooting Conversation Failures Troubleshooting Conversation Failures Over Multiple StepsOver Multiple Steps
Expected utility of taking action to repair listening situationas function of details of multi-turn dialog “history”
![Page 57: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/57.jpg)
Assessing Frustration with Assessing Frustration with Number of StepsNumber of Steps
![Page 58: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/58.jpg)
yesTshoot
reflect
Repeat
overheard
Engage
no
When Troubleshooting is the When Troubleshooting is the Best ActionBest Action
![Page 59: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/59.jpg)
yes
Tshoot
reflect
Repeat
When Troubleshooting is the When Troubleshooting is the Best ActionBest Action
![Page 60: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/60.jpg)
Considering Attentional IssuesConsidering Attentional Issues
![Page 61: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/61.jpg)
Toward Continuous Listening SystemsToward Continuous Listening Systems
Beyond cumbersome “push-to-talk” Beyond cumbersome “push-to-talk” systemssystems
Discriminating target of speechDiscriminating target of speech Understanding conversation Understanding conversation
maintenance statusmaintenance status Allowing user barge-inAllowing user barge-in
![Page 62: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/62.jpg)
DeepListener StatusDeepListener Status
Application development environmentApplication development environment Mobile telephonyMobile telephony Desktop applicationsDesktop applications Managing “subdialog”Managing “subdialog”
Evaluation: Tradeoffs—steps vs. stakesEvaluation: Tradeoffs—steps vs. stakes Learning by watchingLearning by watching
![Page 63: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/63.jpg)
44 S is proposing activity to L L is considering S’s proposal of
ConversationConversation
33 S is signaling that p for L L is recognizing that p from S
IntentionIntention
22 S is presenting signal to L L is identifying signal from S
SignalSignal
11 S is executing behavior for L L is attending to behavior from S
ChannelChannel
QuartetQuartet:: Multilevel Grounding & Extended Sensing Multilevel Grounding & Extended Sensing
LevelLevel
Maintenance ModuleMaintenance Module
![Page 64: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/64.jpg)
Maintenance ModuleMaintenance Module
User’s ResponseLatency
Used Nameof System
Stressed Nameof System
Repeat SimilarLogical Form
Eye Gaze onSystem
Time Since LastUser Speech
MaintenanceStatus (t)
MaintenanceStatus (t-1)
SignalAccuracy (t-1)
Time Since LastSystem Action
Others Presentin Room
Calendar ShowsMeeting
Telephone in Use
Overall Parse Fit
UtteranceComplexity
Syntactic SketchScore
NumNon-Terminals
Num PhrasalHeads
Final UtteranceConfidence
SignalAccuracy (t)
InterferenceEvent
Num HypothesesPer Word
Energy Floor
Signal Identified
User’s Focus ofAttention (t-1) User’s Focus of
Attention (t)
General ASRQuality
Is Trained forUser
Microphone Type
ThresholdSetting
Custom GrammarFor Domain
Channel Level
Signal Level
![Page 65: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/65.jpg)
Linguistic EvidenceLinguistic Evidence
Nature, correctness of NL parse Nature, correctness of NL parse as evidence sourceas evidence source
![Page 66: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/66.jpg)
Visual EvidenceVisual Evidence
Evidence for user attentionEvidence for user attention
![Page 67: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/67.jpg)
Visual EvidenceVisual Evidence
Evidence for user attentionEvidence for user attention
![Page 68: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/68.jpg)
Acoustical EvidenceAcoustical Evidence
![Page 69: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/69.jpg)
Vision and GroundingVision and Grounding
![Page 70: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/70.jpg)
Vision and GroundingVision and Grounding
![Page 71: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/71.jpg)
Considering Visual and Considering Visual and Linguistic CluesLinguistic Clues
“If you could take me to the next slide that would be great.”
![Page 72: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/72.jpg)
“I can look away and talk about the computer.”
Considering Visual and Considering Visual and Linguistic CluesLinguistic Clues
![Page 73: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/73.jpg)
… and even talk about going to the next page.”
Considering Visual and Considering Visual and Linguistic CluesLinguistic Clues
![Page 74: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/74.jpg)
Bayesian ReceptionistBayesian Receptionist
![Page 75: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/75.jpg)
Continue to Continue to gather gather information information
User’s Goal
Goal 1 Goal n
Subgoal 11
Subgoal 1x1 Subgoal 1xy
Subgoal 1x
Level 0
Level 1
Level 2
Return to previous level Return to previous level of analysisof analysis
Progress to next level of Progress to next level of precision without precision without confirmationconfirmation
Progress to next level Progress to next level of precision after of precision after confirmationconfirmation
EVI
EVI
EVI
Take action in worldTake action in world
Initial utterance, observations
World action
Open request for information
Richer Models of InitiativeRicher Models of InitiativeConversational Architectures ProjectConversational Architectures Project
Joint work with Tim Paek
![Page 76: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/76.jpg)
Goal 1 Goal nLevel 0
User’s Goal
Subgoal 11
Subgoal 1x1 Subgoal 1xy
Subgoal 1x
Level 1
Level 3
VOI
VOI
VOI
Bayesian Models and DialogBayesian Models and Dialog
VOI
![Page 77: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/77.jpg)
Goal 1 Goal nLevel 0
User’s Goal
Subgoal 11
Subgoal 1x1 Subgoal 1xy
Subgoal 1x
Level 1
Level 3
VOI
VOI
VOI
Bayesian Models and DialogBayesian Models and Dialog
VOI
![Page 78: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/78.jpg)
User’s Goal
Goal 1 Goal n
Subgoal 11
Subgoal 1x1 Subgoal 1xy
Subgoal 1x
Level 1
Level 3
VOI
VOI
VOI
Bayesian Models and DialogBayesian Models and Dialog
Level 0
![Page 79: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/79.jpg)
![Page 80: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/80.jpg)
![Page 81: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/81.jpg)
![Page 82: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/82.jpg)
![Page 83: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/83.jpg)
![Page 84: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/84.jpg)
![Page 85: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/85.jpg)
![Page 86: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/86.jpg)
![Page 87: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/87.jpg)
![Page 88: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/88.jpg)
![Page 89: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/89.jpg)
![Page 90: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/90.jpg)
![Page 91: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/91.jpg)
DARPA ISAT Study:DARPA ISAT Study: 2000-2001 2000-2001
Foundations of Augmented CognitionFoundations of Augmented Cognition
(E. Horvitz and M. Pavel, co-chairs)(E. Horvitz and M. Pavel, co-chairs)
Multiple meetings culminating at NAS, Wash Multiple meetings culminating at NAS, Wash DC (August 2001)DC (August 2001)
Augmented Cognition programAugmented Cognition program
![Page 92: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/92.jpg)
Toward Augmented CognitionToward Augmented Cognition
Divided attentionDivided attention Visual search Visual search Concept attainmentConcept attainment MemoryMemory VisualizationVisualization Judgment & decision makingJudgment & decision making Action under limited time and informationAction under limited time and information ComprehensionComprehension Training & educationTraining & education
Apply computation to support / augment human cognition
Results in Cognitive PsychologyResults in Cognitive Psychology
Characterization of limitations in human cognitionCharacterization of limitations in human cognition
Human capabilities static, computational capabilities Human capabilities static, computational capabilities have grown rapidlyhave grown rapidly
![Page 93: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/93.jpg)
Human abilitiesHuman abilities
Computational Computational resources, prowessresources, prowess
Augmented Augmented CognitionCognition
TodayToday
Toward Augmented CognitionToward Augmented Cognition
![Page 94: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/94.jpg)
ExistingExisting
Psychological Psychological
Results on Results on
Cognitive LimitationsCognitive Limitations
TargetTarget
Cognitive TasksCognitive Tasks
MemoryMemory ConceptConcept
AttainmentAttainmentDivided Divided
AttentionAttention
Ab
ilities & efficien
ciesA
bilities &
efficiencies
* * New Cog. Psych. New Cog. Psych. Research Research
* HCI, Aug. Cognition * HCI, Aug. Cognition ResearchResearch
??
HCI and Augmented CognitionHCI and Augmented Cognition
Designing services with knowledge of Designing services with knowledge of cognitive limitationscognitive limitations
Value of probabilistic reasoning for Value of probabilistic reasoning for assessing goals, states, contextassessing goals, states, context
Value of probabilistic reasoning for Value of probabilistic reasoning for assessing goals, states, contextassessing goals, states, context
![Page 95: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/95.jpg)
Key ArenasKey Arenas
AttentionAttention AttentionAttention MemoryMemoryMemoryMemory
Judgment & Decision makingJudgment & Decision makingJudgment & Decision makingJudgment & Decision making
Visualization Visualization & Display& Display
Visualization Visualization & Display& Display
Learning Learning & Training& Training
Learning Learning & Training& Training
Language Language & Interaction& InteractionLanguage Language
& Interaction& Interaction
Neurobiological issuesNeurobiological issuesNeurobiological issuesNeurobiological issues
![Page 96: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/96.jpg)
Meeting FociMeeting Foci
AttentionAttention AttentionAttention MemoryMemoryMemoryMemory
Judgment & Decision makingJudgment & Decision makingJudgment & Decision makingJudgment & Decision making
Visualization Visualization & Display& Display
Visualization Visualization & Display& Display
Learning Learning & Training& Training
Learning Learning & Training& Training
Language Language & Interaction& InteractionLanguage Language
& Interaction& Interaction
Neurobiological issuesNeurobiological issuesNeurobiological issuesNeurobiological issues
Advances in Computation:Advances in Computation:Sensing, Learning, Sensing, Learning,
Reasoning & ApplicationsReasoning & Applications
Advances in Computation:Advances in Computation:Sensing, Learning, Sensing, Learning,
Reasoning & ApplicationsReasoning & Applications
Perspective onPerspective on
ApplicationsApplications
Perspective onPerspective on
ApplicationsApplications
![Page 97: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/97.jpg)
Recent Sample Results from the LabRecent Sample Results from the Lab
UVA/CMUUVA/CMU: Memory, : Memory, spatialization, and contextspatialization, and context
MSRMSR: Alerting, performance, and : Alerting, performance, and memorymemory
![Page 98: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/98.jpg)
UVA/CMU: UVA/CMU: Memory Efficiency and SpatializationMemory Efficiency and Spatialization
(D. Proffitt, R. Pausch, et al.)(D. Proffitt, R. Pausch, et al.)
Significant memory boosts for Significant memory boosts for word-pair trials for single vs. word-pair trials for single vs. multiple screen learningmultiple screen learning
fMRI analysis of differential fMRI analysis of differential activityactivity
![Page 99: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/99.jpg)
InfocockpitInfocockpit
![Page 100: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/100.jpg)
Experiment: Word-Pair RecallExperiment: Word-Pair Recall
TaskTask 10 cue words and 10 target words. 3 lists. Cue words 10 cue words and 10 target words. 3 lists. Cue words
same for all lists. Target words varied from list to list. same for all lists. Target words varied from list to list. Next day: Recall pairs (not told ahead of time)Next day: Recall pairs (not told ahead of time)
Two conditionsTwo conditions Infocockpit condition Infocockpit condition 3 monitors, 3 large projection screens w/ contextual 3 monitors, 3 large projection screens w/ contextual
imageimage & surround sound. Word lists displayed on & surround sound. Word lists displayed on different monitors.different monitors.
Control Control Standard desktop computer.Standard desktop computer.
![Page 101: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/101.jpg)
Recall EnhancementRecall Enhancement
![Page 102: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/102.jpg)
Difference fMRIDifference fMRI
![Page 103: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/103.jpg)
MSR: Studies of Divided Attention, MSR: Studies of Divided Attention, Alerting, NotificationAlerting, Notification
(M. Czerwinski, E. Cutrell, E. Horvitz)(M. Czerwinski, E. Cutrell, E. Horvitz)
![Page 104: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/104.jpg)
Psychological Studies of AttentionPsychological Studies of Attentione.g., Interruption & recoverye.g., Interruption & recovery
(With Mary Czerwinski & Ed Cutrell)(With Mary Czerwinski & Ed Cutrell)
![Page 105: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/105.jpg)
![Page 106: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/106.jpg)
Differing Costs of DisruptionDiffering Costs of Disruption
High-level characterization of taskHigh-level characterization of task
Planning Execution Evaluation0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Interrupted Phase
![Page 107: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/107.jpg)
Relevant vs. Irrelevant AlertsRelevant vs. Irrelevant Alerts
Total Resume0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Task Timing
Relevant
Irrelevant
![Page 108: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/108.jpg)
Probing Deeper Structure of DisruptionProbing Deeper Structure of Disruption
![Page 109: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/109.jpg)
Probing “reorientation cost”Probing “reorientation cost” Visual reorientation (e.g., re-acquire position in list Visual reorientation (e.g., re-acquire position in list
being searched)being searched) Conceptual reorientation (e.g., re-acquire context, Conceptual reorientation (e.g., re-acquire context,
goal)goal)
??
Probing Deeper Structure of DisruptionProbing Deeper Structure of Disruption
![Page 110: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/110.jpg)
ChallengeChallenge: Identify target book title in a 3 : Identify target book title in a 3 page list of titles with and without cursor page list of titles with and without cursor markermarker
Two targetsTwo targets .5 cases: Verbatim title.5 cases: Verbatim title .5 cases: Gist: .5 cases: Gist: ee.g., .g., ““A book about Ramses II and A book about Ramses II and
the Nilethe Nile.”.” Remind meRemind me button button DesignDesign
2 (title v. gist search) 2 (title v. gist search) x 2 (marker v. no marker) x 2 (marker v. no marker) x 2 (notification vs. no notification trial) x 2 (notification vs. no notification trial) x 8 (replications per condition) x 8 (replications per condition) [[16 participants, 16 participants, 64 trials per session]64 trials per session]
Probing Deeper Structure of DisruptionProbing Deeper Structure of Disruption
![Page 111: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/111.jpg)
Notifications Notifications Mimicked sound, onset of MSN Messenger 2.0Mimicked sound, onset of MSN Messenger 2.0 Simple multiplication, division problemsSimple multiplication, division problems
Dependent variables Dependent variables Total task timeTotal task time Time to switch to a notificationTime to switch to a notification Number of reminders requested Number of reminders requested Time spent on a notificationTime spent on a notification
Probing Deeper Structure of DisruptionProbing Deeper Structure of Disruption
![Page 112: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/112.jpg)
Cognitive reorientation dominates spatial Cognitive reorientation dominates spatial reorientation.reorientation.
Reminders requested more for notifications Reminders requested more for notifications under higher cognitive loadunder higher cognitive load
Significant relationship between need for Significant relationship between need for reminder and reminder and timingtiming of the interruption of the interruption Participants more likely to request a reminder when Participants more likely to request a reminder when
disrupted early in taskdisrupted early in task
Sample Results on Sample Results on Disruption and ReorientationDisruption and Reorientation
![Page 113: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/113.jpg)
![Page 114: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/114.jpg)
Augmented Cognition: OpportunitiesAugmented Cognition: Opportunities
AttentionAttention Integrate consideration of results on Integrate consideration of results on
divided attention and disruption in divided attention and disruption in monitoring systems.monitoring systems.
e.g.,e.g., Control if, when, and how information about Control if, when, and how information about a monitored system or situation is presented to a monitored system or situation is presented to operators focusing on another, more central task.operators focusing on another, more central task.
Employ results from visual search and Employ results from visual search and
attention in rendering policiesattention in rendering policies
e.g., Develop automated display layout and e.g., Develop automated display layout and optimization considering time-criticality, content.optimization considering time-criticality, content.
![Page 115: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/115.jpg)
Memory and LearningMemory and Learning Harness results on memory and contextual Harness results on memory and contextual
cuescuese.g.,e.g., Automated sensing / generation of cues Automated sensing / generation of cues during learning, coupled with re-generation of during learning, coupled with re-generation of multiple contextual clues during remembering.multiple contextual clues during remembering.
Harness results on ideal spacing for enhanced Harness results on ideal spacing for enhanced learninglearning
e.g.,e.g., Infer ideal time for reinforcement with Infer ideal time for reinforcement with repetition in a training system.repetition in a training system.
Augmented Cognition: OpportunitiesAugmented Cognition: Opportunities
![Page 116: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/116.jpg)
Judgment and Decision MakingJudgment and Decision Making Consider errors of framing and well-Consider errors of framing and well-
characterized biases of judgment, action characterized biases of judgment, action under uncertainty.under uncertainty.
e.g., Guide visual representation of actions, e.g., Guide visual representation of actions, alternatives, and outcomes in a manner that alternatives, and outcomes in a manner that debiases stereotypical errors of judgment in debiases stereotypical errors of judgment in decision making under uncertainty.decision making under uncertainty.
Augmented Cognition: OpportunitiesAugmented Cognition: Opportunities
![Page 117: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/117.jpg)
Potential ApplicationsPotential Applications
Information filtering, triage, and alertingInformation filtering, triage, and alerting Mixed-initiative agent-operator interactionMixed-initiative agent-operator interaction Context-sensitive computingContext-sensitive computing Intelligent remindingIntelligent reminding Managing attention and disruptionManaging attention and disruption Automated visualization in control systemsAutomated visualization in control systems Learning and training systemsLearning and training systems Human-errorHuman-error——aware systemsaware systems Automated sensor fusion and decision Automated sensor fusion and decision
supportsupport Judgment de-biasing systemsJudgment de-biasing systems
![Page 118: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/118.jpg)
Presentations on Key ArenasPresentations on Key Arenas
AttentionAttention (Sperling) (Sperling)
Memory Memory (Landauer)(Landauer)
Judgment & Decision makingJudgment & Decision making (Fischhoff) (Fischhoff)
Visualization & DisplayVisualization & Display (Ellis) (Ellis)
Language & InteractionLanguage & Interaction (Oviatt) (Oviatt)
Learning & TrainingLearning & Training (Carroll) (Carroll)
Neurobiological issuesNeurobiological issues (Levy) (Levy)
![Page 119: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/119.jpg)
Breakout GroupsBreakout Groups Key models & findings on limitations from Key models & findings on limitations from
psychologypsychology
Opportunities for real-world applications Opportunities for real-world applications
Challenges per psychological understandingChallenges per psychological understanding
Challenges per computation, display, HCIChallenges per computation, display, HCI
Take an optimistic perspectiveTake an optimistic perspective
Take a pessimistic perspectiveTake a pessimistic perspective
Evaluation metricsEvaluation metrics
Attention Attention Attention Attention MemoryMemoryMemoryMemory
Judgment & Decision makingJudgment & Decision makingJudgment & Decision makingJudgment & Decision making
Visualization Visualization & Display& Display
Visualization Visualization & Display& Display
Learning Learning & Training & Training
Learning Learning & Training & Training
Language Language & Interaction& InteractionLanguage Language
& Interaction& Interaction
Neurobiological issuesNeurobiological issuesNeurobiological issuesNeurobiological issues
![Page 120: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/120.jpg)
AttentionAttention AttentionAttention
Research challenges Most results (e.g., AOC) focus on short durations (< 500 msec)
Need to understand attention over longer time scales
Links among low-level components, conceptual issues with scaling up to perception, attention, action for larger time scale strategies
Attention and disruption in realistic environments
Opportunities Handling divided attention, attention-guided access, attention-centric rendering and visualization, context-sensitive “attentional security” for learning & performance
Potential Applications Context-sensitive alerting systems, automated monitoring & guiding of attention, attending to anomalies, ideal mixed-initiative interaction
![Page 121: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/121.jpg)
Memory & LearningMemory & LearningMemory & LearningMemory & Learning
Research challenges Basic research, e.g., understanding complex interactions among component processes of memory, links to attention, etc.
Understanding state of knowledge, best strategy per this state
User interfaces, user interaction in real-world tasks and systems
Opportunities Significant results have not seen application!
Specific policies that promote forgetting and impair performance during training actually enhance long-term retention!
e.g., spacing, variation, contextual interference, intermittent feedback
Can make training qualitatively a more difficult experience
Potential Applications Context-sensitive reminder systems
Enhanced situation monitoring, training
![Page 122: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/122.jpg)
Judgment & Decision makingJudgment & Decision makingJudgment & Decision makingJudgment & Decision making
Research challenges Understand fixation on representations, premature closure vs. paralysis of analysis, propagating implications of key changes in assumptions, considering reliability of sources, influence of stress and fatigue; cognitive processes of situation assessment
Dynamics of belief about processes, uncertainty about spatial relations given streams of information
Decisions, judgment in gaming setting with opponent
Opportunities Harness well-characterized results on biases of judgment
Use of representations for communicating about, understanding situations; sensitivity of variables; value of different kinds of modeling effort
Potential Applications Decision making companions that expand considerations, pose reformulatations, debias, assist in real-time, training
Rendering for ideal fusion of information, judgment, action
![Page 123: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/123.jpg)
Visualization & DisplayVisualization & DisplayVisualization & DisplayVisualization & Display
Research challenges Understanding how people integrate information from multiple components of scene into usable concepts
Basic research needed on how to map distinct findings about visual attention, concept attainment to control of display, layout for tasks
Opportunities Map cognitively deliberate serial analyses into fast perceptual recognition Harness knowledge about emergent features, gestalt principles, redundant coding, popout effects (e.g., use of line orientation, length, width, size, number, terminators, intersection, closure, color, intensity, flicker, direction of motion, depth cues, lighting
Exploit findings on timing, animation, scene complexity
Harness knowledge about biases in representation
Potential Applications Displays that adapt to task at hand, employing perceptual and cognitive principles to highlight most important information
Cognitively efficient representations of spatial relationships, dynamics, uncertainty, reliability
![Page 124: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/124.jpg)
Language & InteractionLanguage & InteractionLanguage & InteractionLanguage & Interaction
Research challenges Handling disfluency speech, variations such as hyperarticulation, speech under stress, in mobile settings
Handling signal variability
Handling ambiguity
New languages, principles for mixed-initiative interaction
Opportunities Ideal designs for multimodal input, based on stereotypic patterns of gesture and language
Error reduction, speech interaction with minimal cognitive load
Potential Applications Natural interfaces that allow variable, noisy gesture and language
Automated detection of misunderstandings in communications
Enhanced multimodal interfaces for efficient interaction
![Page 125: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/125.jpg)
Some Findings and Some Findings and RecommendationsRecommendations
Perspective, focus of Augmented Cognition as Perspective, focus of Augmented Cognition as distinguished from much of psychologically-oriented HCI distinguished from much of psychologically-oriented HCI
Interest, enthusiasm among participantsInterest, enthusiasm among participants
Mathematical psychologists traditionally have been Mathematical psychologists traditionally have been focused mainly on basic mechanisms as disjoint from real-focused mainly on basic mechanisms as disjoint from real-world applications, usage.world applications, usage.
Many current psychological models and research results Many current psychological models and research results are not necessarily well-adapted, matched for augmented are not necessarily well-adapted, matched for augmented cognition work.cognition work.
Need to map some basic results at msec time scale, Need to map some basic results at msec time scale, micromechanisms to broader strategic, planning level of micromechanisms to broader strategic, planning level of perception, analysis, decision making, learningperception, analysis, decision making, learning
![Page 126: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/126.jpg)
New research in psychology will be valuable in support of New research in psychology will be valuable in support of Augmented Cognition efforts—some focused in the Augmented Cognition efforts—some focused in the context of real-world taskscontext of real-world tasks
Criticality of interdisciplinary workCriticality of interdisciplinary work––but potential difficulties but potential difficulties with building interdisciplinary teams for augmented with building interdisciplinary teams for augmented cognitioncognition
Some low hanging fruit, but also some very difficult Some low hanging fruit, but also some very difficult challenges in science and technology in this arenachallenges in science and technology in this arena
Even applications of some simple well-characterized Even applications of some simple well-characterized limitations could go a long way in enhancing performancelimitations could go a long way in enhancing performance
Potential value of composing interdisciplinary academic Potential value of composing interdisciplinary academic advisory group in support of potential DARPA program advisory group in support of potential DARPA program
Some Findings and Some Findings and RecommendationsRecommendations
![Page 127: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/127.jpg)
The timing is right for building practical The timing is right for building practical systems via pursuit of a deeper science of systems via pursuit of a deeper science of
humanhuman––computer symbiosiscomputer symbiosis
But…But…
Research and innovation will face Research and innovation will face significant challenges and impediments, significant challenges and impediments,
both technically and professionally.both technically and professionally.
![Page 128: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/128.jpg)
Assuming current predictions Assuming current predictions of future importance as same of future importance as same
as the actual future importanceas the actual future importance
![Page 129: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/129.jpg)
All events
![Page 130: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/130.jpg)
All events
Events assessed as important
![Page 131: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/131.jpg)
All events
Events assessed as important
Events forgotten
![Page 132: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/132.jpg)
All events
Events assessed as important
Events forgotten
Sensed
![Page 133: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/133.jpg)
SummarySummary
Key uncertainties and human-computer Key uncertainties and human-computer interaction…interaction…
Beliefs & intentionsBeliefs & intentions PreferencesPreferences InitiativeInitiative AttentionAttention
Representations & machinery for Representations & machinery for reasoning under uncertainty provide a reasoning under uncertainty provide a
rich fabric for developing valuable rich fabric for developing valuable software services & experiences.software services & experiences.
![Page 134: Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May 2002 Uncertainty, Action, and Interaction Eric Horvitz Microsoft Research May](https://reader038.vdocuments.mx/reader038/viewer/2022110402/56649e4f5503460f94b473cc/html5/thumbnails/134.jpg)