Cognitive Design
Nishida Lab
Yoshimasa Ohmoto
Conversational Informatics, June 15th, 2016
What is Cognitive Design?
◎Purpose ◎Designing the interaction with artifacts based on a physio-
cognitive approach.
◎“Remaking products, services and organizations to fit and enhance the human mind”
◎Communication◎The process by which people exchange information or
express their thoughts and feelings [Longman]
◎The word “communication” will be used in a very broad sense to include all of the procedures by which one mind may affect another. [Shannon and Weaver]
◎From the perspective of cognitive design◎Most important points are “exchanging intentions” and “sharing
thoughts and feelings”
Communication model of Shannon and Weaver
◎Send messages from Sender to Receiver
◎This model is very simple but very important
◎This model includes exchange and share of the information (intention in communication).
◎It is different between information in source and that in destination because of the noise.
◎The intersection of the information is sharing part.
[Claude E. Shannon and Warren Weaver. “The mathematical theory of communication” -- University of Illinois Press, 1949]
Sender Receiver
transmitter receiverInfo. source
destination
noise
cannel
Frameworks to interpret social signals
◎A framework coming from linguistics◎This framework approaches social interaction from the viewpoint
of dialogue understanding.◎Vocal prosody and gesture are treated as annotations of linguistic
information. [Cassell, 2000]◎This framework has proven useful for computer graphics and language
production systems.
◎ It has been difficult to apply to dialogue interpretation, perception, and unconscious behaviors.
◎A framework coming from cognitive psychology◎This framework focuses on emotion.◎The key idea is that people perceive others’ emotions through
stereotyped displays of facial expression, tone of voice etc. [Picard, 1997]◎The simplicity and perceptual grounding of this theory has recently given
rise to considerable interest in the scientific and engineering literature.
◎ It has been difficult to measure these displays and even more difficult to use them to build practical applications.
A framework coming from cognitive design
◎A person’s attitude or intention is conveyed through unconscious behavior (social signals).
◎such as changes in the amplitude and frequency of prosodic and gestural activities.
◎The relationship between interaction features are communicated rather than content of the dialogue or emotion.
◎Total information from auditory perception and visual perception is synthetically interpreted to find the relationship.
◎This is “multi-modal” interaction.
◎There are many signals but a few signals can be used.
Influence of social signals
◎Explicit signals (verbal expressions) unilaterallycommunicate final and conclusive information.
◎Social signals bilaterally influence to thoughts and behavior.
◎Most of the social signals are unconscious and reliable.
◎For example,
◎Mimicry (a kind of social signals) triggers a positive response among all of the communication partners and own. [Chartrand, Maddux & Lakin, 2005]
◎People think they understand after nodding behavior. [Brinol & Petty, 2003]
An example of influence
◎You tube video
Car
Driving
Tree
Absorbing CO2
Human
Walking
Pretty dog
Following the person
Cognition
??
Car
Driving
Tree
Absorbing CO2
Human
Walking
Pretty dog
Following the person
Information
Environment
Emotion
The largest object
Shape
Follow the person
Near the car
Car
Driving
Tree
Absorbing CO2
Human
Walking
Pretty dog
Following the person
Small hatchback car
On the way home
Road side trees
Absorbing CO2
This is “me”
Walking across the road
Cool dog
Following but not a pet
Different cognition!
Small hatchback car
On the way home
Road side trees
Absorbing CO2
It is “me”
Walking across the road
Cool dog
Following but not a pet
Information
Environment
Projection
Internal, objective
External, subjective
External, objective
Emotion
Internal, subjective
objective
subjective
external
internal
Information
Emotion
EnvironmentProjection
Speculating Intention
◎Intention◎We cannot objectively find “intention.”◎We speculate depending on situations.
◎People cannot report intentions precisely.
◎People’s memories are easily rewritten. [*]
◎Hypothesis◎Intentions in interaction are
extemporarily shaped based on the underlying and ambiguous wish.
◎The external stimulus◎partner’s behavior, new information etc.
◎The internal pressure◎reflection of own activity, own preferences
etc.
[*] Edelson, M. and Sharot, T. and Dolan, R.J. and Dudai, Y.: Following the Crowd: Brain
Substrates of Long-Term Memory Conformity. Science, vol. 333, num. 6038, pp 108-111, 2011.
Bubbling intention
Analysis of the behaviors of a good facilitator
◎Discussion is critical to build consensus in our social activities. ◎The facilitator can conduct smooth and effective discussions by
mediating between participants in the discussion. [Reagan-Cirincione 1994]
◎This is a kind of speculating intentions.
◎We investigated how experienced facilitators control the discussions effectively and smoothly.◎We conducted experiments in face-to-face discussion for
statistical analysis to find the factors which caused good facilitating behaviors.
◎by context, agreement or disagreement, and nonverbal and paralinguistic data
◎Discriminant analyses were applied to the data.◎The facilitator paid attention to not only the context of the
discussion but also the individual behavior◎The participants conveyed their requests to the facilitator by
using nonverbal behavior.
Problems
◎The facilitator mediates between participants.
◎The facilitator has to support the group's social and cognitive processes, allowing participants to focus their attention on more substantive issues and ultimately reach the most appropriate solution to a problem.
◎It is not known that the good facilitator decides the facilitating behavior based on what kinds of information.
◎Most of previous works did not directly investigate what information the facilitator pays attention to.
Outline of the experiment
◎To obtain the participants' behavioral data in face-to-face discussion
◎Five participants formed a group
◎four undergraduate students ("discussers“) and one facilitator
◎The discussers were acquainted with each other.
◎A total of five groups participated (a facilitator and 20 students)
◎"the plan of overnight trip in this summer.“
Experimental settings
◎sitting on seats in a circle
◎There were no table and they could not take notes
◎Obtained data
◎head orientation◎By a motion capture system
◎Voice◎By throat microphones
◎Video◎By four HD-video cameras
Video analysis: facilitating behavior
◎We could classify facilitating behavior to four types in discussions.
◎Diverging◎the facilitator interposed someone to encourage the
divergent discussion
◎Converging◎the facilitator interposed someone to encourage the
convergent discussion.
◎Conflicting person◎the facilitator asked a discusser who had conflicting opinion
against the last speaker.
◎Objectifying◎the facilitator asked the last speaker for objectifying his/her
opinion.
Results: The discriminant analyses
◎Linear discriminant analyses were applied to classify the facilitating behaviors◎each facilitating behavior type and the others.
◎We could selected which facilitating behavior was appropriate by the discriminant functions◎The concordance rate between the selected behavior
and the actual behavior was 78.6%.
◎the explanatory variables could correctly classify the facilitating behaviors.
No-good facilitator
◎The feature of the facilitation by the no-good facilitator
◎The number of the facilitating behavior was smaller (4.3 times / 10min.) than that of the good facilitator (7.1 times / 10min.).◎No-good could not decide which facilitating behavior had to
be selected.
◎No-good could not interpose at the appropriate time. ◎He could not understand the context of the discussion.
◎No-good often changed the discussion topic◎He could not control the discussion
◎These were some of the reasons why no-good could not conduct the smooth and effective discussion.
Conclusions
◎The statistical analysis to find the factors which caused good facilitating behaviors
◎We could suggest some factors to which a good facilitator used to conduct smooth discussion.◎Whether the discussion was diverging or converging.
◎Which discussers had a similar opinion.
◎the total time of speaking
◎the total time which discussers paid attention to the facilitator
◎The facilitator controlled the discussions by the facilitating behaviors.
◎We achieved 78.4% accuracy in classifying four types of the facilitating behaviors.
Behavior model based on bilateral speculation
◎The facilitation is basically one-way speculation of intention.◎Social signals bilaterally influence to thoughts and
behavior.
24
[Ohmori. 2010]
◎The behavior model has nested structure.◎Lv0: behaving based on my own
intention
◎Lv1: behaving based on partner’s intention
◎Lv2: behaving based on consideration that the partner speculates my own intention
◎Lv3: ・・・・・・
Induction of Intentional Stance in HAI
◎We want to develop an embodied conversational agent that is regarded as a social partner, not just multimodal interface. ◎The interaction behavior is very different.
◎The mental stance of people when they interact with agents is usually different from when they interact with humans. ◎Social agency theory proposes that more social cues lead to more
social interaction
◎But a result of a previous study was the exact opposite.
◎The purpose of this study is to investigate how to induce the intentional stance in human-agent interaction.
The agent has information for me.
Hello, Dylan! How are you doing?
The mental stances
When the guardian robot stands by a gate,
a person who try to pass the gate is caught by the robot
The robot think “I do not permit the passage because this gate is now broken.”
When the sensors of the robot detect a person, the robot catches me.
The actuator and the computer controls the robot body.
Intentional stance
Physical stance
Design stance
We propose a method to induce the intentional stance, implement the method in an
agent and experimentally investigate the effect of inducing the intentional stance.
Methods for Presenting Goal-oriented Behavior
◎We expected that making a person speculate the agent's subjective perspectives is important.
◎We implemented two methods for presenting goal-oriented actions:◎showing a trial-and-error progression towards a goal using
multimodal behavior (trial-and-error agent)◎People construct an imperfect behavior estimation model
◎because it is difficult to precisely interpret multimodal behavior in terms of estimating the behavioral goal.
◎displaying the agent’s behavioral intention using text (text display agent)◎This method encourages the construction of an complete behavior
estimation model.
◎Both of them presented goal-oriented behavior.
◎We investigate how effective they are at inducing the intentional stance.
Outline of architecture
◎The agents decide their behavior through three layers.
◎The goal layer shows the task goal. ◎The first goal is predefined.◎The goal is changed depending on
the others’ behavior.
◎The behavior category layer shows the category of possible behavior. ◎Each category is a sub-goal of a
concrete behavior.
◎The concrete behavior layer shows the concrete behavior produced by an agent. ◎Each category contains some of the
concrete behavior.
Goal layerchasing long time,
equalizing the time of a tagger,Equalizing the number of a tagger
Behavior category layer”chasing”, ”provocation”,
”dissatisfaction”, ”escape” and ”appeal”.
Concrete behavior layer
Etc.
Waving hand near the player
Standing near
Middle distance
Large distance
Sometimes looking around
Stay behind an obstacle
Presentation method
◎Trial-and-error agent
◎The agent produces a concrete behavior with higher expression strength than before.
◎The changes for achieving goal are made in a trial-and-error fashion.
◎Text display agent
◎The agent produces patterns of text corresponding to the behavior category.
◎If only presenting goal-oriented behavior is important, people should take the intentional stance.
Jumping and waving hand
Small distance
Waving hand near the player
Standing near
Middle distance
Large distance
Provocation Chase
Crapping hands
Sometimes looking around
Stay behind an obstacle
Appeal
Lv 1
Lv 3
Lv 2
Success
Fail
Provocation Chase Appeal
“Try to catch me!”
Following near
“Hey, come on!”
Provoking
“Escape or tagged”
Chasing carefully
“I’m stay here!”
Looking around
Hiding in the house
Fail
Outline of an Experiment
◎To investigate the effect of inducing intentional stance, we conducted an experiment using the two agents:◎a ”trial-and-error agent” and a ”text display agent”◎The behavior planning of the task and the expressions were
automatically controlled.
◎We used a virtual reality ”customized tag game” as an interaction task.◎Two human participants and one agent joined the game.◎Participants could freely communicate and perform the game.◎They could add new rules, cooperate with others, ignore the agent, an
so on.
◎We compared two types of the experimental results.◎we analyzed the number of communicative actions towards the
agent throughout the experiment. ◎we asked the participants to answer questionnaires after the
experiment.
Experimental settings
◎The player’s virtual avatar could be controlled by their body motions using a motion capture system. ◎The participants could easily interact using body motions.
◎”The agent can recognize your speech. But the agent does not respond to your speech and actions except when they need to respond for performing the task.”
Results: Analysis of the number of communicative actions towards the agent
◎The purpose is to investigate whether performing goal-oriented behavior influenced the communication behavior. ◎We divided the time series of the experiment evenly into four periods.
◎We compared the number of communicative actions in the 2nd and 4th periods.
◎The number in the 4th period was significantly less than that in the 2nd period only for TD-agent (p = 0.0003). ◎All of the participants decreased the number of com. actions.
◎They took the design stance towards this agent in the end.
◎The number in the 2nd period for TD-agent was more than that for TaE-agent (but there was no significant difference).
The design stance interaction The intentional stance interaction
Results: Questionnaire analysis
◎ The purpose is to investigate how the presentation method influenced participants’ final subjective impressions in the end.
◎ We performed a Mann-Whitney U test on the data in the questionnaire.
• How human-like do you feel that the agent is?
– relatively higher but no significant difference.
• How strongly do you think the agent has a definite goal?
– The TaE-agent had significantly more definite goals than TD-agent (p = 0.039).
• How smart is the agent?– The TaE-agent was significantly smarter than
TD-agent (p = 0.015).
• How well does the agent understand your intentions?
– The TaE-agent understood their intentions better than TD-agent (p = 0.054, marginally significant difference).
– But the average scores were not high.
• How much did you enjoy the game?– The participants enjoyed the game
significantly more with TaE-agent than with TD-agent (p = 0.025)
– The scores for both agent were fairly high.
Discussions
◎To sum up the experimental results: ◎the trial-and-error agent could not quickly induce
participant’s intentional stance
◎but, when induced once, the participant maintained intentional stance long time.
◎We suggest that the process of constructing a behavior model influences the mental stance of the participants.◎An obvious presentation of the inner state is not always
effective.
◎Therefore, for the text display agent, the participants did not maintain the intentional stance.
◎On the other hand, two participants could not understand the trial-and-error behavior.◎This is future work.
Conclusion
◎To establish social partner relationships between humans and agents, we tried to induce intentional stance by presenting goal-oriented behavior.
◎We implemented two methods.
◎We conducted an experiment to investigate the effect of inducing intentional stance.
◎Trial-and-error agent and text display agent
◎We suggest that both of the goal-oriented behaviorand the continuous model estimation are needed to induce and maintain the intentional stance.
Dynamically estimating emphasizing points
◎We often make a decision interactively as a group with our friends and advisers.◎We dynamically and interactively change the factors
that we emphasize (termed “emphasizing points”).
◎E.g. travel planning: Country? Budget? Members?
◎It is difficult even for human◎to directly estimate changing emphasizing points.
◎to estimate internal states of communication partners.
◎I think “intention” is constructed throughinteractions with humans, artifacts and environments.
We tried to estimate the emphasizing points through
decision-making interaction.
Method
◎We dynamically estimating emphasizing points for decision-making.◎using physiological responses and verbal and
nonverbal information.◎Keywords, nodding, SCR, heart rate, skin temperature
◎Estimate based on active demands and passive responses◎Active demands◎Keywords which are related to demands
◎Comparative results of proposals
◎Passive responses◎Keywords which are related to preferences
◎Nonverbal reactions
◎Physiological indices
HAI demo
Evaluation Experiment
◎Objectives
◎To confirm accuracy of our estimation method
◎To confirm that estimation of emphasizing points were useful to understand causes of decision-making
◎Measured data
◎Physiological indices◎SCR
◎Electrocardiograph (LF/HF)
◎Skin temperature of a finger
◎Videos
◎Questioner to confirm the accuracy and degree of satisfaction
Experimental setting
◎Each participant interacted with a WOZ controlled agent to design two types of mobile robots.
◎In a session: one participant and one experimenter (WOZ operator)
◎Total: 7 participants (average 20.6 years old)
To confirm accuracy of the estimation
◎Participants answered top three emphasizing points out of 23 factors.
◎We compared the answers with the results by each method (t-test)
average
S.D.
Ours Gradual
提案ours gradual
To confirm that estimation were useful
◎Participants answered their degree of satisfaction (-3~+3) (Wilcoxon signed-rank test)
◎Which method provided satisfied proposals (-3~+3) (sign-test)
average
S.D.
Ours Gradual
提案ours gradual
average
S.D.
Ours > gradual
(提案>既存)Ours > gradual
Conclusion
◎We proposed one of the methods to estimate emphasizing points based on human behavior.
◎We confirmed
◎Proposed method could achieve relatively accurate estimation of emphasizing points.
◎By using proposed method, participants were more satisfied in decision-making task.
Extended Methods to DEEP for Group Decision-making
◎We often make a decision interactively as a group◎We want to make an Embodied Conversational Agent to
join the group decision making.
◎The purpose of this study is to extend DEEP to estimating the emphasizing points of a group.
◎We observed and investigated interaction processes in group decision making with friends and advisers.
◎We proposed two extended methods corresponding to the different interaction processes.
◎We then conducted an experiment to evaluate the methods using ECAs.
◎In conclusion, these proposed methods accurately estimate proposals and satisfy participants in the appropriate group.
Background
◎In many cases of group decision-making, people often have conflicting opinions.
◎People consider not only their demands but also the effects on their relationships.
◎We extend the our proposed method to estimating the emphasizing points of groups (“group-DEEP”).
◎Based on typical verbal and nonverbal information and physiological indices◎Since we can only use basic and general representations of
verbal and nonverbal information, to cover the loss, we use physiological indices
Interaction process: Expressing Opinion group
◎First, the members clearly expressed which proposal was better and why.
◎In the case of conflicting opinions, the members never changed their own opinions
◎They only changed when their partner proved them wrong.
◎The emphasizing points of the group only contained those held by all members.
◎We expect that the emphasizing points of the EO group can be estimated by focusing on clearly accepted opinions.
Interaction process: Avoiding Conflict group
◎The members often used ambiguous words (e.g. relatively, so-so, not so bad …) to express their opinion.◎They repeatedly confirmed their partner’s emphasizing points
and tried to establish consensus.
◎They often retracted their opinions during the group conversation if their partner did not emphasize that point.◎In addition, if the partner did not challenge their opinion, they
regarded their opinion as accepted.
◎The AC group’s emphasizing points contained all not refused opinions.
◎We expect that the emphasizing points of the AC group can be estimated by focusing on the responses to a partner’s opinion.
Two methods of group-DEEP
◎Union-base◎The union-based method focuses on the responses to the other members’
opinions.◎We expect that this method will be used by the AC group.◎Estimated emphasizing points contain as many emphasizing points as
possible as identified by members.
◎ Intersection-base◎The intersection-based method focuses on clearly accepted opinions. ◎We expect that this method will be used by the EO group.◎Estimated emphasizing points only contain the emphasizing points shared
by all the members.
Positive opinions
Union-base
Intersection-base
Evaluation experiment
◎Objective
◎to investigate whether we should change the estimation method depending on the interaction style.◎union-based method or intersection-based method
◎avoiding conflicts or expressing opinions
◎Participants
◎16 Japanese college students (all female).◎The participants were divided into eight pairs.
◎Each participant interacted with two agent which was implemented different estimation method◎Union-base and Intersection-base.
◎Task
◎Choosing a present
Experimental setting
◎Each participant interacted with a WOZ controlled agent.
◎The ECA capture keywords, nodding motion and physiological indices (electrocardiogram (LF/HF) and SCR)
displaying an ECA
skin conductance
response (SCR):electrodes placed on two
fingers of a left hand
electrocardiogram:measured by connecting
electric poles with paste
to the participant’s left
and right sides and to
both ears for grounding
and reference.
Results of accuracy of ECA’s final proposal
◎All the participants chose their best proposal out of 40 prepared proposals at the end of both sessions.
◎We calculated the concordance rates between the proposals chosen by the participant and the proposals estimated by each ECA.
SS df MS F pGroup 0.031 1 0.031 0.14 0.72error 3.2 14 0.23Method 0.28 1 0.28 1.5 0.25error 2.7 14 0.19interaction 1.5 1 1.5 8.0 0.014*Total 7.7 31
SS MS F p
Group (union-base) 0.56 0.56 2.7 0.11
Group (intersection-base) 1.0 1.0 4.8 0.038*
error 0.21
Method (AC group) 0.25 0.25 1.4 0.27
Method (EO group) 1.6 1.6 8.1 0.013*
error 0.19
Results of participant satisfaction with HAI
◎The participants answered rating questions regarding the level of satisfaction with HAI using a seven-point scale.
◎We then calculated averages in each group and each method.
SS df MS F pGroup 0.78 1 0.031 0.14 0.72error 28 14 0.23Method 0.031 1 0.28 1.5 0.25error 20 14 1.5interaction 9.0 1 9.0 6.2 0.026*Total 59 31
SS MS F pGroup (union-base) 2.3 2.3 1.3 0.27Group (intersection-base) 7.6 7.6 4.3 0.047*error 0.21Method (AC group) 5.1 5.1 3.5 0.084+Method (EO group) 4.0 4.0 2.7 0.12error 1.5
Conclusion
◎The purpose is to extend our proposed method to estimating the emphasizing points of a group.
◎We conducted an experiment and confirmed that the interaction process differed between the group that tried to avoid conflict and those that tried to express their opinions.
◎From the results of the experiment, we propose two extended methods: the union-based method and the intersection-based method.
◎We also conducted an experiment to evaluate the methods using ECAs.
◎As a result, we suggest that the methods accurately estimated proposals and satisfied participants in the appropriate group.
Behavior model based on mutual speculation
◎We basically consider one-way speculation of intention.
◎We especially focused on internal pressure.
54
◎The behavior model of bubbling intention needs mutual speculation.
◎We also have to consider external stimuli.
◎The external stimuli are mainly produced communication partner(s).
The effect of convergent interaction using subjective opinions
◎ DEEP encouraged decision-making by awakening the intrinsicemphasizing points.
◎ Extrinsic subjective interpretations, such as friend’s opinion and word-of-mouth advertising, also encourage decision-making◎ because they provide case examples to interpret the factors we have
to consider and emphasize to reach an appropriate decision.
◎ In this study, we investigated the effect of extrinsic subjective interpretations of the adviser in interactive decision-making.
◎We conducted an experiment that compared the results of interactive decision-making with two types of ECAs◎ A facilitative agent: who provided subjective opinions to realize
divergent and convergent processes in decision-making◎ An estimation agent: who only provided proposals that reflected the
emphasizing points of each participant
◎ As a result, we can confirm that the facilitative agent could improve the interaction◎ the participant’s satisfaction of interaction with the ECA, the
naturalness of ECA’s interaction, and the impression of decision-making process.
Background
◎We have to materialize ambiguous demands in decision-making.
◎Subjective information provide case examples to interpret the factors we have to consider and emphasize to reach an appropriate decision.
Extrinsic stimulus
Intrinsic consideration
Well…”green dress”
Red is also good.The sleeveless dress doesn’t
suit you.
I like green dress.
But I already have long-
sleeved dress.
Red is also good.The sleeveless dress
doesn’t suit you.
Green!
Important?
Important?
Important?Important!
Not
focused
Not
important
Important!Important?
NO (first)
Method
◎To “facilitate” decision-making by controlling divergent and convergent process.Switching rules between the divergent and convergent
• There are more than three emphasizing points, with a degree of emphasis of more than one.
• The degree of emphasis does not change during the interaction.
or• The user offers a convergent opinion.
Divergent• The agent provides a small nod once in reaction to the
user’s utterance.• The frequency of providing a new proposal is low.• The agent provides a new proposal after she explains
three emphasizing points.• The furthest proposal from the previous one is selected
as a new proposal. • The degree of emphasis decreases if the emphasizing
point is not explained in the previous proposal.
Convergent• The agent provides two nods in reaction to the user’s
utterance.• The frequency of providing a new proposal is high.• The agent provides a new proposal after she explains one
emphasizing point, which is a recommendation.• The nearest proposal to the previous one is selected as a
new proposal.• The degree of emphasis decreases only when the
emphasizing point is refused in the previous proposal.
YES (last)
Selection method of a next proposal
Current proposal
Agent’s
preferences
Divergent Convergent
Next
proposal
farthest
Nearest
Next
proposal
Agent Control
Estimation of emphasizing points
Switch or not between divergent
and convergent
Next proposal
Agent behavior
ResultsEvaluating estimation
process
Results
Select next proposal
Next proposal
Agent behavior
Results
Select next proposal
Facilitative agent Estimation agent
Divergent
Social signals
Social signals
Estimation of emphasizing points
Evaluation experiment
◎ Task: gift-wrapping (30 factors)◎ The participants did not know what was appropriate gift-wrapping.◎ The participants would take advice from the agent because they tried
to predict what the receiver of the gift would like.
◎ Participants: 20 Japanese college students (all female)◎ The reason why the participants were females was that there was
motivation gap for the gift-wrapping task between males and females.
Example of the experiment
Result of reaction latency analysis
◎To investigate whether participants attentively listening proposals by the ECA, we extracted a reaction latency for each participant.
◎There is a significant difference in the second half of the interaction. (p=0.029)◎There is also a significant difference between the reaction
latency in the estimation agent group in the first half and that in the second half.
When the participantinteracted with the estimation agent, she carefully thought about the proposal in the second half of the interaction.
→ participants did not care the interaction with the agent
Result of analysis of emphasizing point changes
◎To investigate whether the control of divergent and convergent processes influences participants’ emphasizing points, the participants chose emphasizing points that they changed during the interaction.
◎The number in the facilitative agent group was significantly higher than that in the estimation agent group (t=-2.63, p<0.05).
Because participants made their decision only based on intrinsic emphasizing points, they could not recognize changes to the emphasizing points.
→ subjective opinion influence the recognition of search space
Results of questionnaires
◎To investigate the impression of the decision-making process with ECAs, the participants answered three rating questions on the ECA’s behavior using a seven-point scale.
◎We conducted Wilcoxon signed-rank tests on each questionnaire result.
Conclusion
◎We investigated the effect of the subjective information by the agent in interactive decision-making.
◎We conducted an experiment that compared the results of interactive decision-making with two agents◎A facilitative agent: who provided subjective opinions that
were external stimuli for decision-making.◎An estimation agent: who only provided proposals that
reflected the emphasizing points of each participant.
◎As a result, ◎we evaluated a facilitative agent who provided subjective
opinions could encourage interaction in decision-making.◎we found that the facilitative agent led to higher scores for
participant satisfaction regarding agent’s interactions, the naturalness of agent’s interaction, and positive impressions of the decision-making process.
The Effects of Extended Estimation on Affective Attitudes in a Series of Tasks
◎Agent characters that interact with users often appear on several situations. ◎These characters are regarded as multi-modal interfaces, but are not
regarded as social partners.◎-> Our aim is to develop an agent that could be regarded as a
communicative partner like human in continuous interaction.
◎Active affective attitude is needed.
◎In previous studies, we proposed methods to support a decision-making process◎Dynamic Estimation of Emphasizing Points (DEEP)◎“Emphasizing points”: we have to consider and emphasize them to reach
an appropriate decision.
You are tired because you had hard work
yesterday. I provide easy training today.
OK! Today’s training menu
is a little easy.
Purpose
◎This study aimed to propose a method to induce an user’s active attitude in decision-making process in a series of interactions. ◎If the agent cannot induce the attitude, the users do not
demonstrate and share their own preferences, mental attitudes, and inner states with the agent.
◎We used historical estimated emphasizing points within a series of tasks to estimate emphasizing points within a new but similar task. (“extended estimation”)
◎We expected the provision of consistent estimation, using accumulated data on interactions, to induce a positive human attitude toward the agent and the interaction.
TaskTask TaskTask Task Task
Stored historyStructured knowledge
Estimation
Basic method to estimate emphasizing points
"facilitative DEEP" (fDEEP)
Emphasizing factors
Candidates DB
Input
Verbal
Nonverbal
Physiologic
al
Proposal
s
Estimation of emphasizing
points
Proposal generationIntroduction planning
Output
User’s demands
Agent’s
Opinions
Extended estimation through maintenance of emphasizing points within a series of
interactions
Emphasizing factors
Candidates DB
Input
Verbal
Nonverbal
Physiologic
al
Proposal
s
Estimation of emphasizing
points
Emphasizin
g factor DB
Stored
and
structured
Relational
network
Convert
and
compleme
nt"facilitative DEEP with extended estimation" (feeDEEP)
Output
User’s demands
Agent’s
Opinions
Proposal generationIntroduction planning
Outline of the human-agent interaction
Repeating and conversing
Asking emphasizing points for the task
User’s utterances and reactions
Proposals including user’s demands and agent’s
opinion
Emphasizing pointsB
uild
ing a co
nsen
sus b
etween
h
um
an an
d agen
tin
an
interactio
nal series o
f tasks
Common ground
Proposals including common ground factors
User’s utterances and reactions
Emphasizing factors
Experiment
◎The purpose was to investigate how feeDEEP affects the efficiency of the decision-making process and impressions related to agent behavior in a series of tasks
◎The participants were asked to coordinate a new living space.◎The primary task included three tasks.◎furniture selection, electronics selection, and living space planning.
◎The emphasizing points contained 16 factors.◎relaxing, natural, for work, clean, leisure, high spec and so on
The feeDEEP agent provided proposals using maintained emphasizing factors in planning.
Selection Planning Proposals using maintained
factors
furniture
electronics
planningsimple relaxing
natural
leisure
work space
clean
high spec
relaxing
feeDEEP
fDEEP
Converting by
the predefined rules
Relaxingand leisure
Naturaland relaxing
Integration of the emphasizing points through a series of interactions
Experimental settings
◎Participants
◎11 participants (8 males and 3 females) interacted with the feeDEEP agent (feeDEEP group)
◎10 participants (8 males and 2 females) interacted with the fDEEP agent (fDEEP group)
Desktop
PC
Note PC
Polymat
e
Agent
Control PC
USB
camera
USB
camera
Experimente
r
Participant
60-inch Monitor
Microphone
The agent and the task were displayed on the monitor.
Skin Conductance Responses and electrocardiogram were recorded. These are reflected mental
states to the agent proposals in real-time.
The experimenter manually inputted verbal reactions. The inputted words were listed
in advance.
Results of interaction behavior
◎To investigate whether the extended estimation contributed to effective decision-making, we counted the number of proposals from the agent in the 2nd selection task and the planning task.
◎We performed a paired t-test on the data of each group.◎There was a significant difference in the feeDEEP group (p < 0.05).
◎There was no significant difference in the fDEEP group (p = 0.26).
◎We performed an unpaired t-test on the data in the planning. ◎The value for the feeDEEP group was significantly less than that for the
fDEEP group (p = 0.0078).
These results indicate that the feeDEEP agent achieved more
effective decision-making support in the planning task
than the fDEEP agent.
4.9
2.6
5.3
4.5
0
1
2
3
4
5
6
7
8
second selection planning
The feeDEEP-group The fDEEP-group
*
*
Results of the questionnaire
◎The purpose of this analysis was to investigate how the extended estimation influenced the participants' subjective impressions related to the agent's behavior. ◎The participants answered six questions using a seven-point scale.
◎We performed a Mann-Whitney U test on the data from the questionnaire.
5.7
4.8
5.3
5.7 5.6
3.5
5.6
3.4
5.34.9 4.9
5.3
1
2
3
4
5
6
7
How satisfied are you
with the final products?
How human-like do you
feel that the agent's
behavior was?
How natural do you feel
that the agent's
interaction was?
How satisfied are you
with the interaction
process?
How would you rate the
agent's level of effort?
How frequently did you
accept the agent's
proposals?
feeDEEP fDEEP
* *+
Discussions
◎An important finding was that the rate of participant acceptance of the feeDEEP agent's proposals was significantly low despite a decrease in the number of interactions.
◎ This was possible because the agent provided a consistent estimation for each participant due to the extended estimation.
◎We suggest that the feeDEEP agent induced an active attitudetoward the decision-making interaction.
◎The attitudes indicate that they regarded the agent as communicative.
◎To create a system that is user-centric, it is necessary for the user to maintain an active attitude toward the decision-making interaction in order to accomplish their goals.
Future work
◎In our previous work, we analyzed physiological indices (SCR and LF/HF values) that were obtained experimentally. ◎However, we could not include an analysis of these indices in
this paper.
◎We are now analyzing these data in detail to investigate the underlying reasons for these experimental results.
12
13
14
15
16
17
18
19
20
1 45
89
13
31
77
22
12
65
309
35
33
97
441
48
55
29
57
36
17
66
17
05
74
97
93
83
78
81
925
96
91
01
31
05
71
10
1
fDEEP group
12
13
14
15
16
17
18
19
20
1 50
99
14
8
19
7
24
6
29
5
34
4
39
3
44
2
49
1
54
0
58
9
63
8
687
73
6
78
5
83
4
88
3
932
98
1
10
30
10
79
11
28
1177
12
26
12
75
13
24
feeDEEP group
Conclusion
◎We investigated the effects of the consistent estimation of a human's preferences on the human's affective impressions.
◎We conducted an experiment to evaluate the effect of the method using two agents;◎a feeDEEP agent, which was proposed in this study
◎a fDEEP agent, which was proposed in our previous work.
◎The results showed that ◎ feeDEEP agent could reduce the number of interactions in the decision-
making process
◎ it also improved some of the affective impressions related to the agent's character.
◎ In addition, we found that the rate of participants' acceptance of the feeDEEP agent's proposals was significantly low. ◎This was possible because the agent provided a consistent estimation of
emphasizing points for each participant.
◎The participants' attitude indicates that they regarded the agent as being communicative.
(Again) Speculating Intention
◎speculating intentions ≠ understanding the parson’s thought
◎Hypothesis
◎Intentions in interaction are extemporarily shaped based on the underlying and ambiguous wish.
◎The external stimulus◎partner’s behavior, new information and so on
◎The internal pressure◎reflection of own activity and so on
Bubbling intention