© 2015 IBM Corporation

Affective Computing

Michal Shmueli-Scheuer, David Konopnicki – IBM Research, Haifa Research Lab

May 28, 2015

© 2013 IBM Corporation

The ABC that makes up a Human Being

2

Affect(feel)

Behavior(act)

Cognition(think)

What is Affective Computing

Affective computing is the study and development of systems and devices that

can recognize, interpret, process, and simulate human affects (Wikipedia)

3

A Computer that has EQ

• monitors (one's own?) and other people's emotions

• uses emotional information to guide thinking and behavior

4

Crowd Sensing Better Selling

Product Test Affective CRM

Emotions can be measured

Some Scenarios

- “we had an accident. Please come fast.”Sensing the stress in his voice, the computer replied with a calm, soothing voice- “Sir, please calm down, we are on our way”and dispatched emergency forces to the scene

Real-time emotion sensing from voice

Affect-aware Persona adaptation

Affect-aware Customized response

- “we had an accident. Please come fast.”Sensing the stress in his voice, the computer replied with a calm, soothing voice- “Sir, please calm down, we are on our way”and dispatched emergency forces to the scene

- “… those refrigerators have been beenvoted ‘Best Design’ of 2014” said the computerThe customer frowned.“… and they are environmentally friendly”…

Real-time emotion sensing from video

Affect-aware Dynamic Argument adaptation

- “… those refrigerators have been beenvoted ‘Best Design’ of 2014” said the computerThe customer frowned.“… and they are environmentally friendly”…

Scenarios

Emotions in CRM Conversational agents Social Media Analytics Emotions in Organizations

Emotions in Email, IM

Media and Entertainment: focus

groups

Retail: store behavior Digital analytics Consumer Products: market research, focus

groups

Automotive: mood recognition

Education: attention management

Healthcare Deception prevention

© 2013 IBM Corporation© 2015 IBM Corporation

Key roles of Affective Computing

Making cognitive agents convincing

– Monitoring users’ emotions and being adaptive to them

– Conveying affect in the communication interfaces

Helping users become aware of their own emotions or

emotions of others

– Supporting better decision making

Making cognitive agents easier to understand and

follow through regulation of the information flow

– Adapting the flow to the user’s attention level and

emotional state.

12

© 2013 IBM Corporation© 2015 IBM Corporation

Analysis Process

Extraction

•Raw emotion recognizers from text, speech, face/body & sensors

Interpretation

•In-context emotional state modeling

Analysis

•Use of user models in predictive and prescriptive scenarios

Ouput

•Expressive Communication with humans

13

Text

Speech

Video

Wearable

Predictive

Prescriptive

ExpressiveSpeech

Avatars

Application

ContextModifications

ExpressiveText

Fusion

Primary

Secondaryand

Social

Context

User Model

© 2013 IBM Corporation© 2015 IBM Corporation

What we work on

14

Virtual Agents with Emotional Intelligence

• Develop virtual agents that have emotional intelligence.

• Improve customer satisfaction during a conversation by communicating in an empathetic manner.

Personal Emotion Measurement in Mobile using Video

• Large-scale measurement of emotional response to content presentation and content optimization

Emotion Speller For Email

• Personal assistant for managing emotions in incoming and outgoing email

Analysis of Emotional Text in Enterprise

• Sociological analysis of emotions expressed in enterprise social text in order to evaluate mood and etiquette

© 2013 IBM Corporation© 2015 IBM Corporation

Emotions in email

In-coming email

© 2013 IBM Corporation© 2015 IBM Corporation

Emotions in email

Out-going email

© 2013 IBM Corporation17

Modern workplaces involve a lot of emotions

Emotions play an integral role in employee motivation, personal

commitment and social interactions at work

On the other hand, emotional labor inhibits workers from expressing

authentic emotions

Potential to improve the effectiveness and overall health of large organizations

Large scale organizational data and analyzes multiple data sources with six Ekman’s emotions

Demographics & temporal effects

Deep analytics:– analyzed the evolution of users’ emotion along time and their

characteristics– groups analysis– effect of external events on the happiness of employees

Contributions

Emotions in internal social network communications

© 2013 IBM Corporation© 2015 IBM Corporation

Emotions in internal social network communications

Time of day:

Happy and Surprised

0.33

0.35

0.37

0.39

0.41

0.43

0.45

0.47

0.49

0.51

0.53

Jan-1

2

Mar

-12

May

-12

Jul-1

2

Sep-1

2

Nov-12

Jan-1

3

Mar

-13

May

-13

Jul-1

3

Sep-1

3

Nov-13

Jan-1

4

Mar

-14

May

-14

Jul-1

4

Ave

rage

em

oti

on

inte

nsi

ty

175

180

185

190

195

200

205

210

Ave

rage

sto

ck v

alu

e

Happiness

Stock

0.09

0.095

0.1

0.105

0.11

0.115

0.12

8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Hapiness

0.29

0.295

0.3

0.305

0.31

0.315

0.32

0.325

0.33

0.335

8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Emotion display rules

© 2013 IBM Corporation

Cognitive Virtual Agents

Many enterprises use self-service agents that automatically deliver

answer to customers’ queries across a variety of interaction channels.

These agents have no emotional intelligence:

They can’t sense emotions

They can’t express themselves in an emotional manner (e.g.

express empathy)

© 2013 IBM Corporation© 2015 IBM Corporation

Personal Emotion Measurement in Mobile using Video

Demographics and

psychology questionnaire

Content Presentation

and User Capture

User Emotion Analysis

ReportingContent

optimization

© 2013 IBM Corporation© 2015 IBM Corporation

Thank You!

21