(Socio)-affective computing and its potential role in collaboration

Guillaume Chanel

Swiss Center for Affective Sciences

Computer Science Department

University of Geneva

1

Affective computing

2

Spike Jonze, “Her”, 2013

Joaquin Phoenix, Scarlett Johansson

Affective computing

3

Emotional cues and signals

Senses

Sensors

User

Emotion synthesis

Ouput devices

Machine

Emotion assessment

Decision of reaction

Facial expressions, voice intonation, physiology, …

Camera, microphone, electrocardiogram, …

Emotion modeling

Physiological emotion recognition – Why?

1. Contains discriminative information

2. Mostly involuntary signals -> less sensitive to deception

3. Continuous measures of affect (almost no interruption)

4. Fast: as soon as 200 ms for neuro-physiological signals

4

Physiological emotion recognition - invasive

5

Physiological emotion recognition – invasive or not ?

6

E4 wristband

- Heart rate - Temperature - Electordermal activity

Emotiv

EarEEG1

1 Looney, D., Kidmose, P., Park, C., Ungstrup, M., Rank, M., Rosenkranz, K., & Mandic, D. (2012). The in-the-ear recording concept: user-centered and wearable brain monitoring. IEEE Pulse

Physiological emotion recognition – invasive or not ?

7

Poh, M.-Z., McDuff, D. J., & Picard, R. W. (2010). Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics Express, 18(10), 10762–74.

Plethysmograph blood volume pulse Face volume pulse

Plethysmograph heart rate Face heart rate

Physiological Features

8

• Standard features: mean, standard deviation, …

• Energy features: • Energy 0.05-0.15Hz:

parasympathetic and sympathetic

• Energy 0.15-1Hz: parasympathetic activity

• Multiscale entropy: measure of complexity

TEAP – Toolbox for Emotional feAture extraction from Physiological signals

9

• Load and display signals

• Compute features from: • Electro-dermal activity

• Skin temperature

• Blood volume pulse

• Electrocardiography

• Electromyography

• EEG

• Respiration

Open-source

http://github.com/Gijom/TEAP

http://www.teap.science

Soleymani, M., Villaro-Dixon, F., Pun, T., & Chanel, G. (2017). Toolbox for Emotional feAture extraction from Physiological signals (TEAP). Frontiers in ICT, 4.

Emotion classification – Machine learning

10 1st scale entropy HR

Ener

gy E

EG

Calm Excited

Applications: Engaging Gaming

11 Mihaly Csikszentmihályi (1990). Flow: The Psychology of Optimal Experience. Harper & Row

Applications: Engaging Gaming

12

Estim. True

Easy Medium Hard

Easy 80% 10% 10%

Medium 37% 33% 30%

Hard 21% 19% 60%

Estim. True

Easy Medium Hard

Easy 57% 43% 0%

Medium 21% 50% 29%

Hard 19% 19% 62%

Estim. True

Easy Medium Hard

Easy 82% 14% 4%

Medium 29% 39% 32%

Hard 4% 27% 69%

Peripheral physiological signals EEG signals

Fusion of both set of features

Accuracy: 58% Accuracy: 56%

Accuracy: 63%

Chanel, G., Rebetez, C., Bétrancourt, M., Pun, T., & Betrancourt, M. (2011). Emotion Assessment From Physiological Signals for Adaptation of Game Difficulty. IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans, 41(6), 1052–1063.

13

Neuchatel’s history museum, “Emotions”, 2015

Swiss Center for Affective Sciences, Phasing out, end of april 2017

Socio-affective computing

14

Physiological cues and signals

Senses

User Ouput devices

Machine(s)

Interaction assessment

Symbolic conversion / adaptation

Other user(s)

Social presence, performance,

conflict detection

Senses

Physiological cues and signals

Chanel, G., & Muhl, C. (2015). Connecting Brains and Bodies: Applying Physiological Computing to Support Social Interaction. Interacting with Computers.

Social interaction modeling

Collaboration assessment by measuring coupling

15

Coupling measures the synchrony, interdependency and the co-occurrence of states existing between two systems.

conflicting interactions

Empathy

Social presence Grounding: sharing a

common knowledge Collaborative performance

Collaborative performance

Physiological coupling (correlation and coherence)

Eye-movement coupling (Cross-recurrence plot)

time

Hea

rt r

ate

Social gaming

Analyzed variables: • Cooperative vs. competitive (+ home

vs. laboratory)

• Social presence: • Psychological involvement: how we are

aligned emotionally (emotion contagion, empathy, etc.)

• behavioral involvement: how our behaviors are inter-dependent

16

Orb

. Oc.

co

up

ling

Results - social gaming

17

Covariates Dep. Var.

psychological involvement (Social empathy)

Orb. Occ. (smile)

Respiration

Behavioral involvement IBI

Competitive mode leads to a higher coupling of facial activity Coupling is correlated with social presence (psychological and behavioral involvement)

Chanel, G., Kivikangas, J. M., & Ravaja, N. (2012). Physiological compliance for social gaming analysis: cooperative versus competitive play. Interacting with Computers, 24(4), 306–3016.

Emotion Awareness Tools for Mediated Interaction (EATMINT)

18

Can we provide socio-emotional statistics to collaborators which will help them to better collaborate?

Online collaborative

software

19

Eye-tracking Eye-tracking

Speech Speech

Face video Face video

Physiological data(Peripheral)

EATMINT: data acquisition

http://eatmint.unige.ch

Eye-movement

coupling Convergence

Mutual understanding

Action synchrony

Looking at the same place at

approx. the same time

Physiological coupling

Emotion management

Heart rate coherence

Amount of communication of owns

and other’s emotions

Effort to understand own

and other’s emotions

RR-FS R2=0.21 BRT R2=0.18

RR-FS R2=0.38 BRT R2=0.24

20

Chanel, G., Bétrancourt, M., Pun, T., Cereghetti, D., & Molinari, G. (2013). Assessment of computer-supported collaborative processes using interpersonal physiological and eye-movement coupling. In Proceedings - 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction.

Facial coupling

BRT R2=0.24

Ground truth definition: each member of a dyad had to annotate:

- 20 emotional moments;

- 20 non-emotional moments.

21

Multi-person emotion assessment method

Features extraction EDA: number of peaks, mean, % of decay

Heart rate: mean, energy in low / high freq. bands

Random forest classifier One participant

Random forest classifier Two participants

Performance one participant Performance two participants ?

Adding information about partners reaction’s improves accuracy Possibly because people in a group tend to feel emotions at the same time: - emotion contagion and imitation; - conflicts…

22

t=3.1, p=0.003

One participant Two participants

23

Value Importance of success

Control Ability to succeed

Protocol validation

24

Manipulating control and value successfully elicited different emotions

control

value

Satisfaction

Boredom

Despair

Frustration

Shame

Joy

Emotions and collaborative processes

25

Emotion Grp perf. Perceived collaboration quality

Transactivity Grounding Argumentation Consensus

Satisfaction + +

Boredom - -

Joy + +

Frustration -

Despair -

Shame

Gratitude + + + +

Positive emotions are associated with a better collaboration TBD: study how the dynamic of emotions and behaviors predicts collaborative processes

Other projects on multi-user modeling

• Seconds that matter: Managing First Impressions for a more Engaging Virtual Agent • Remote physiological monitoring

• Detection of impressions from multiple cues: what does my colleague think of me?

• Emotions and affective computing in mediation • Investigating the emotional impact of mediation

• Developing emotional indices for the mediator

26

Toward the arts

27

Main research question: Is it possible to find a relationship between movie aesthetic highlights and spectators reactions?

Synchrony by dynamic time wrapping

28

Performance of highlight detection

29

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

H1 H2 H3 H4 H5 H

Are

a u

nd

er c

urv

e

GSR

Acceleration

Fusion

form Content

Kostoulas, T., Chanel, G., Muszynski, M., Lombardo, P., & Pun, T. (2015). Dynamic Time Wraping of Multimodal Signals for Detecting Highlights in Movies. In Workshop on modeling interpersonal synchrony and influence interpersonal, International Conference on Multimodal Interaction. Seattle, USA.

Take home messages

• Physiology can be measured with limited obtrusiveness.

• Socio-affective processes can be modeled and assessed by the combination of several cues

• These models can be used to:

• personalize human-machine interactions;

• reshape human-human interactions.

30

Gaëlle Molinari Unidistance

Sunny Avry Unidistance

Patrizia Lombardo UNIGE

Collaborators Team

Theodoros Kostoulas Thierry Pun UNIGE

Michal Muszynski

Chen Wang Teresa Koster