Social Signal ProcessingUnderstanding Nonverbal Behaviour

in Social Interactions

Alessandro VinciarelliUniversity of Glasgow - Idiap Research Institute

Institute of Neuroscience and Psychologyhttp://www.dcs.gla.ac.uk/vincia

vincia@dcs.gla.ac.uk

Outline

•Social  Signal  Processing

•The  Conflict  Example

•Conclusions

Outline

•Social  Signal  Processing  

•The  Conflict  Example

•Conclusions

Nonverbal Communication (I)

3

Vinciarelli, Pantic and Bourlard, “Social Signal Processing: Survey of an Emerging Domain”, Journal of Image and Vision Computing, 27(12):1743-1759, 2009

Nonverbal Communication (I)

3

Height

ForwardPosture Forward

Posture

GesturesInterpersonalDistance

NonverbalBehavioral

Cues

VocalBehavior

MutualGaze

Vinciarelli, Pantic and Bourlard, “Social Signal Processing: Survey of an Emerging Domain”, Journal of Image and Vision Computing, 27(12):1743-1759, 2009

Nonverbal Communication (I)

3

SocialSignals

Height

ForwardPosture Forward

Posture

GesturesInterpersonalDistance

NonverbalBehavioral

Cues

VocalBehavior

MutualGaze

Vinciarelli, Pantic and Bourlard, “Social Signal Processing: Survey of an Emerging Domain”, Journal of Image and Vision Computing, 27(12):1743-1759, 2009

Nonverbal Communication (II)

5

Richmond and McCroskey, “Nonverbal Behaviors in Interpersonal Relations”, Allyn and Bacon, 1995

Nonverbal Communication (II)

5

AttractivenessClothes, etc.

ExpressionsGaze, etc.

Vocalisations,Prosody, etc.

Self-touching,Orientation

Distances, Seating, etc.

Cues

Richmond and McCroskey, “Nonverbal Behaviors in Interpersonal Relations”, Allyn and Bacon, 1995

Nonverbal Communication (II)

5

AttractivenessClothes, etc.

ExpressionsGaze, etc.

Vocalisations,Prosody, etc.

Self-touching,Orientation

Distances, Seating, etc.

CuesPhysical

Appearance

FaceBehavior

VoiceBehavior

Gestures andPostures

Space andEnvironment

Codes

Richmond and McCroskey, “Nonverbal Behaviors in Interpersonal Relations”, Allyn and Bacon, 1995

Nonverbal Communication (II)

5

AttractivenessClothes, etc.

ExpressionsGaze, etc.

Vocalisations,Prosody, etc.

Self-touching,Orientation

Distances, Seating, etc.

CuesPhysical

Appearance

FaceBehavior

VoiceBehavior

Gestures andPostures

Space andEnvironment

Codes

FormingImpressions

ExpressionEmotion

RelationalMessages

DeceptionManagement

Power Persuasion

Functions

Richmond and McCroskey, “Nonverbal Behaviors in Interpersonal Relations”, Allyn and Bacon, 1995

Nonverbal Communication (II)

5

AttractivenessClothes, etc.

ExpressionsGaze, etc.

Vocalisations,Prosody, etc.

Self-touching,Orientation

Distances, Seating, etc.

CuesPhysical

Appearance

FaceBehavior

VoiceBehavior

Gestures andPostures

Space andEnvironment

Codes

FormingImpressions

ExpressionEmotion

RelationalMessages

DeceptionManagement

Power Persuasion

Functions

Richmond and McCroskey, “Nonverbal Behaviors in Interpersonal Relations”, Allyn and Bacon, 1995

SSP: Synthesis

6

AttractivenessClothes, etc.

ExpressionsGaze, etc.

Vocalisations,Prosody, etc.

Self-touching,Orientation

Distances, Seating, etc.

CuesPhysical

Appearance

FaceBehavior

VoiceBehavior

Gestures andPostures

Space andEnvironment

Codes

FormingImpressions

ExpressionEmotion

RelationalMessages

DeceptionManagement

Power Persuasion

Functions

Vinciarelli, Pantic, Heylen, Pelachaud, Poggi, D’Errico, Schroeder, “Bridiging the Gap Between Social Animal and Unsocial Machine: A Survey of SSP”, IEEE

Transactions on Affective Computing, 3(1):69-87,2012

SSP: Analysis

7

AttractivenessClothes, etc.

ExpressionsGaze, etc.

Vocalisations,Prosody, etc.

Self-touching,Orientation

Distances, Seating, etc.

CuesPhysical

Appearance

FaceBehavior

VoiceBehavior

Gestures andPostures

Space andEnvironment

Codes

FormingImpressions

ExpressionEmotion

RelationalMessages

DeceptionManagement

Power Persuasion

Functions

Vinciarelli, Pantic, Heylen, Pelachaud, Poggi, D’Errico, Schroeder, “Bridiging the Gap Between Social Animal and Unsocial Machine: A Survey of SSP”, IEEE

Transactions on Affective Computing, 3(1):69-87,2012

Outline

•Social  Signal  Processing

•The  Conflict  Example

•Conclusions

9

Conflict

“[Conflict is a] mode of interaction [where]the attainment of the goal by one partyprecludes its attainment by the others.”

Judd, “Cognitive Effects of Attitude Conflict Resolution”, Journal of Conflict Resolution, 22(3):483-498, 1978.

10

Conflict

“[Conflict is a] mode of interaction [where]the attainment of the goal by one partyprecludes its attainment by the others.”

Judd, “Cognitive Effects of Attitude Conflict Resolution”, Journal of Conflict Resolution, 22(3):483-498, 1978.

11

Conflict

“[Conflict is a] mode of interaction [where]the attainment of the goal by one partyprecludes its attainment by the others.”

Judd, “Cognitive Effects of Attitude Conflict Resolution”, Journal of Conflict Resolution, 22(3):483-498, 1978.

Political Debates

Vinciarelli, Dielmann, Favre, Salamin, “Canal9: a database of political debates for analysis of social interactions ”, Proc. of Social Signal Processing Workshop, 2009

12

Political Debates

Vinciarelli, Dielmann, Favre, Salamin, “Canal9: a database of political debates for analysis of social interactions ”, Proc. of Social Signal Processing Workshop, 2009

12

SSPNet Conflict Corpus

Vinciarelli et al., “Collecting Data for Socially Intelligent Surveillance and Monitoring Approaches: The Case of Conflict in Competitive Conversations”, Proc. of IEEE Intl. Symposium on Communications, Control and Signal Processing, pp. 1-4, 2012.

13

Source Canal9Number of Clips 1430Clip Length 30 sec.Total Length 11 h 55 mSubjects 135Subjects per Clip At least 2Assessors 10/clip (MTurk)Questionnaire Items 15Total Items 214,500

Measuring Conflict Perception

14

The atmosphere is relaxedPeople wait for their turn before speakingOne or more people talk fastOne or more people fidgetPeople argueOne or more people raise their voiceOne or more people shake their heads and nodPeople show mutual respectPeople interrupt one anotherOne or more people gesture with their handsOne or more people are aggressiveThe ambience is tenseOne or more people compete to talkPeople are actively engagedOne or more people frown

Measuring Conflict Perception

15

The atmosphere is relaxedPeople wait for their turn before speakingOne or more people talk fastOne or more people fidgetPeople argueOne or more people raise their voiceOne or more people shake their heads and nodPeople show mutual respectPeople interrupt one anotherOne or more people gesture with their handsOne or more people are aggressiveThe ambience is tenseOne or more people compete to talkPeople are actively engagedOne or more people frown

Measuring Conflict Perception

15

The atmosphere is relaxedPeople wait for their turn before speakingOne or more people talk fastOne or more people fidgetPeople argueOne or more people raise their voiceOne or more people shake their heads and nodPeople show mutual respectPeople interrupt one anotherOne or more people gesture with their handsOne or more people are aggressiveThe ambience is tenseOne or more people compete to talkPeople are actively engagedOne or more people frown

I

Measuring Conflict Perception

15

The atmosphere is relaxedPeople wait for their turn before speakingOne or more people talk fastOne or more people fidgetPeople argueOne or more people raise their voiceOne or more people shake their heads and nodPeople show mutual respectPeople interrupt one anotherOne or more people gesture with their handsOne or more people are aggressiveThe ambience is tenseOne or more people compete to talkPeople are actively engagedOne or more people frown

I

P

16

Inference vs Physical

-10

-7.5

-5

-2.5

0

2.5

5

7.5

10

-16 -11.2 -6.4 -1.6 3.2 8

Infe

renc

e Sc

ore

Physical Score

r=0.95

Vinciarelli et al., “Collecting Data for Socially Intelligent Surveillance and Monitoring Approaches: The Case of Conflict in Competitive Conversations”, Proc. of IEEE Intl. Symposium on Communications, Control and Signal Processing, pp. 1-4, 2012.

17

Examples (Low)

Vinciarelli et al., “Collecting Data for Socially Intelligent Surveillance and Monitoring Approaches: The Case of Conflict in Competitive Conversations”, Proc. of IEEE Intl. Symposium on Communications, Control and Signal Processing, pp. 1-4, 2012.

17

Examples (Low)

Vinciarelli et al., “Collecting Data for Socially Intelligent Surveillance and Monitoring Approaches: The Case of Conflict in Competitive Conversations”, Proc. of IEEE Intl. Symposium on Communications, Control and Signal Processing, pp. 1-4, 2012.

18

Examples (Medium)

Vinciarelli et al., “Collecting Data for Socially Intelligent Surveillance and Monitoring Approaches: The Case of Conflict in Competitive Conversations”, Proc. of IEEE Intl. Symposium on Communications, Control and Signal Processing, pp. 1-4, 2012.

18

Examples (Medium)

Vinciarelli et al., “Collecting Data for Socially Intelligent Surveillance and Monitoring Approaches: The Case of Conflict in Competitive Conversations”, Proc. of IEEE Intl. Symposium on Communications, Control and Signal Processing, pp. 1-4, 2012.

19

Examples (High)

Vinciarelli et al., “Collecting Data for Socially Intelligent Surveillance and Monitoring Approaches: The Case of Conflict in Competitive Conversations”, Proc. of IEEE Intl. Symposium on Communications, Control and Signal Processing, pp. 1-4, 2012.

19

Examples (High)

Vinciarelli et al., “Collecting Data for Socially Intelligent Surveillance and Monitoring Approaches: The Case of Conflict in Competitive Conversations”, Proc. of IEEE Intl. Symposium on Communications, Control and Signal Processing, pp. 1-4, 2012.

Speech Features

20

Speech Features

20

t

Speech Features

20

t

t

Speech Features

20

t

t

t

Speech Features

20

t

t

t

t

Speech Features

20

t

t

t

t

t

21

Results

Kim, Filippone, Valente, Vinciarelli, “Predicting the Conflict Level in Television Political Debates: an Approach Based on Crowdsourcing, Nonverbal Communication

and Gaussian Processes”, Proc. of ACM Intl. Conf. on Multimedia, 793-796, 2012.

0.7

0.73

0.76

0.79

0.82

BLR GPR GPR ARD SVR LIN SVR RBF

Manual Automatic Auto w.o.s

22

ARD Results (Automatic w.o.s.)C

oeffi

cien

t Val

ue

Features

Clip

Pitc

h

Clip

Inte

nsity

Std.

Minimum

Min

imum

Turn

Std

Inte

nsity

Min

imum

Turn

Mea

n Pi

tch

Turn

Std

Pitc

h

Turn

Mea

n In

tens

ity

Ove

rlap

ping

Inte

nsity

Ove

rlap

ping

Dur

atio

n

Spea

king

Dur

atio

n

Minimum

TotalM

inim

um

Minimum Q - 99%

Max

imum

Outline

•Social  Signal  Processing

•The  Conflict  Example

•Conclusions

24

The SSPNet Portal

More information available at:

http://www.sspnet.eu

•Around 250 hours of annotated material•More than 150 Presentation Recordings•More than 20 software packages

A.Vinciarelli, M.Pantic, “A Web Portal for Social Signal Processing”, IEEE Signal Processing Magazine, 27(4):142-144, 2010

25

Conclusions

25

Conclusions

• Nonverbal communication is a physical, machine detectable evidence of social and psychological phenomena

25

Conclusions

• Nonverbal communication is a physical, machine detectable evidence of social and psychological phenomena

• Interdisciplinary approaches including both human and computing sciences can perform better

25

Conclusions

• Nonverbal communication is a physical, machine detectable evidence of social and psychological phenomena

• Real world applications are the next frontier in terms of data, problems and methodological issues

• Interdisciplinary approaches including both human and computing sciences can perform better

26

Thank you!

Many thanks to: •Gelareh Mohammadi (Idiap Research Institute / EPFL)•Maurizio Filippone (University of Glasgow)•Antonio Origlia (University of Napoli Federico II)•Fabio Valente (idiap Research Institute)•Samuel Kim (Yonsei University)•Bjoern Schuller (TU Munich)