Expressive Emotional Expressive Emotional ECAECA

✔ Catherine PelachaudCatherine Pelachaud✔ Christopher PetersChristopher Peters✔ Maurizio ManciniMaurizio Mancini

Expressive Emotional Expressive Emotional ECAECA

✔ Emotion behavior modeling:Emotion behavior modeling:✔ SignalsSignals✔ Temporal course of signalsTemporal course of signals✔ Expressive qualitiesExpressive qualities

✔ Often in ECA systems: Often in ECA systems: ✔ only modeling of signals is consideredonly modeling of signals is considered✔ 6 ‘universal’ expressions6 ‘universal’ expressions

Large variety of facial Large variety of facial expressions (Kollias et expressions (Kollias et

al)al)✔ Archetypal expressions modeling using Archetypal expressions modeling using

FAPs MPEG-4FAPs MPEG-4✔ Estimation of the range of variations of Estimation of the range of variations of

FAPs in archetypal expressionsFAPs in archetypal expressions✔ Intermediate expressions: rule-based Intermediate expressions: rule-based

computational model based oncomputational model based on✔ Whissell’s representation:Whissell’s representation:

✔ Activation, evaluation, Activation, evaluation, ✔ Plutchik’s angular measurePlutchik’s angular measure

Laban Movement AnalysisLaban Movement Analysis

✔ 5 major components:5 major components:✔ BodyBody: part of the body in used: part of the body in used✔ SpaceSpace: description of the directions and : description of the directions and

paths of motionpaths of motion✔ ShapeShape: changing forms of the body: changing forms of the body✔ EffortEffort: dynamics of the body moves: dynamics of the body moves✔ RelationshipRelationship: modes of interaction with : modes of interaction with

oneself, others and the environment oneself, others and the environment (facing, contact…)(facing, contact…)

✔ Textual and symbolic language of Textual and symbolic language of movement descriptionmovement description

EMOTE (Badler EMOTE (Badler et al)et al)

✔ EMOTE model for Effort and EMOTE model for Effort and ShapeShape

✔ Based on Movement Observation Based on Movement Observation ScienceScience

✔ Laban Movement AnalysisLaban Movement Analysis✔ Computational model of Effort Computational model of Effort

and Shape componentsand Shape components

EffortEffort

✔ Four motion factors: space, Four motion factors: space, weight, time, flowweight, time, flow✔ SpaceSpace✔ WeightWeight✔ TimeTime✔ FlowFlow

✔ Each factor ranges from:Each factor ranges from:✔ IndulgingIndulging in the quality in the quality✔ FightingFighting against the quality against the quality

EffortEffort✔ SpaceSpace: indirect : indirect direct direct

✔ Waving away bugs Waving away bugs pointing to a pointing to a particular spotparticular spot

✔ WeightWeight: light : light strong strong✔ Feather movement Feather movement punching punching

✔ TimeTime: sustained : sustained sudden sudden✔ Stretching Stretching grabbing a falling object grabbing a falling object

✔ FlowFlow: free : free bound bound✔ Waving widely Waving widely moving in slow motion moving in slow motion

ShapeShape

✔ Three distinct qualities of change Three distinct qualities of change in the form of movement:in the form of movement:✔ Shape flowShape flow: mover’s attitude toward : mover’s attitude toward

the changing relationship among the changing relationship among body partsbody parts

✔ Directional movementDirectional movement: mover’s : mover’s intent to bridge the action to a point intent to bridge the action to a point in the environmentin the environment

✔ ShapingShaping: mover’s carving or molding : mover’s carving or molding attitude with the environmentattitude with the environment

ShapeShape

✔ Three dimensions:Three dimensions:✔ HorizontalHorizontal: spreading : spreading enclosing enclosing

✔ Opening arms to embrace Opening arms to embrace clasping clasping someone in a hugsomeone in a hug

✔ VerticalVertical: rising : rising sinking sinking✔ Reaching for something in a high shelf Reaching for something in a high shelf

stamping the floor with indignationstamping the floor with indignation✔ SagittalSagittal: advancing : advancing retreating retreating

✔ Reaching out to shake hand Reaching out to shake hand Avoiding a Avoiding a punchpunch

Expressive Expressive Communicative Communicative

Gestures (Hartmann Gestures (Hartmann et al)et al)

✔ Expressivity parameters:Expressivity parameters:✔ Spatial: amplitude of movementSpatial: amplitude of movement✔ Temporal: duration of movementTemporal: duration of movement✔ Power: dynamic property of movementPower: dynamic property of movement✔ Fluidity: smoothness and continuity of movementFluidity: smoothness and continuity of movement✔ Repetitiveness: tendency to rhythmic repeatsRepetitiveness: tendency to rhythmic repeats✔ Overall Activation: quantity of movement across modalitiesOverall Activation: quantity of movement across modalities

✔ Expressivity modeling: modal and intermodalExpressivity modeling: modal and intermodal

Emotional Emotional ModelingModeling

• Carmen's Bright IDEAS ( Stacy Marsella, USC)

• Interactive drama

• Agent exhibits gestures based on their emotional states and personality traits

FantasyA (Ana FantasyA (Ana Paiva, Inesc)Paiva, Inesc)

• Role-playing game

• The emotion detected from user drives the behavior of one of the agents in the video game

SenToy (Ana Paiva, SenToy (Ana Paiva, Inesc)Inesc)

• SenToy is a tangible interface device, used to influence emotional behaviour in FantasyA

• SAFIRA: EU project

Behavior ModelingBehavior Modeling

✔ Facial expression:Facial expression:✔ Use facial analysis techniques/motion Use facial analysis techniques/motion

capture to look at:capture to look at:✔ Value of facial animation parameters Value of facial animation parameters

(muscular contraction, FAP…)(muscular contraction, FAP…)✔ Pattern of movement for a given parameterPattern of movement for a given parameter✔ Relation of facial animation parameters Relation of facial animation parameters

within a same expressionwithin a same expression✔ Coarticulation of movements between Coarticulation of movements between

expressionsexpressions

MocapMocap

MocapMocap

FAPs – MPEG-4FAPs – MPEG-4

ExampleExample

Gesture Gesture

✔ Gesture specification often based on sign Gesture specification often based on sign language gesture specification:language gesture specification:✔ Hand descriptionHand description✔ Palm orientationPalm orientation✔ Wrist position Wrist position ✔ Movement: wrist, elbow, finger…Movement: wrist, elbow, finger…

✔ Parameters characterizing quality of Parameters characterizing quality of movement: Laban, sub-set…?movement: Laban, sub-set…?

✔ Parameters act on overall animation Parameters act on overall animation (Emote) or on gesture phases (Emote) or on gesture phases (preparation, stroke, …)?(preparation, stroke, …)?

Behavior Behavior

✔ Gaze: Gaze: ✔ Direction, gaze length, mutual gaze, ratio of Direction, gaze length, mutual gaze, ratio of

gaze avoidance/gaze look atgaze avoidance/gaze look at✔ Body:Body:

✔ Body posture, overall movement qualityBody posture, overall movement quality✔ Voice: Voice:

✔ Paralinguistic parametersParalinguistic parameters✔ Text Text

Signals Signals analysis/generationanalysis/generation

✔ Use analysis method to:Use analysis method to:✔ Get information on parametersGet information on parameters✔ Multimodal integrationMultimodal integration✔ Signals synchrony across modalitiesSignals synchrony across modalities✔ Build computational model for generationBuild computational model for generation

✔ Get the expressive quality of behaviorGet the expressive quality of behavior✔ Signals generation should not be done on Signals generation should not be done on

one frame only but on a temporal one frame only but on a temporal intervalinterval

Conclusion Conclusion

✔ Emotion is shown through:Emotion is shown through:✔ SignalsSignals✔ Their temporal courseTheir temporal course

✔ Which parameters to look at? Which are Which parameters to look at? Which are the perceptually relevant ones?the perceptually relevant ones?

✔ How to describe:How to describe:✔ The signals?The signals?✔ Their evolution through time?Their evolution through time?✔ Correlation between modalities?Correlation between modalities?

Conclusion Conclusion

✔ Relation between intensity of emotion Relation between intensity of emotion and behaviors?and behaviors?✔ Selection of behaviorsSelection of behaviors✔ Quality of behaviorsQuality of behaviors✔ Duration of behaviorsDuration of behaviors✔ Behaviors across modalitiesBehaviors across modalities