csci 534: affective computinggratch/csci534/lecture2021-01.pdfcsci 534(affective computing)...

CSCI 534(Affective Computing) – Lecture by Jonathan Gratch

CSCI 534: Affective Computing:An interdisciplinary approach

Lecture 1: 20 Jan, 2020Prof Jonathan Gratch

https://people.ict.usc.edu/gratch/public_html/CSCI534/CSCI534-Syllabus.pdf


Outline

▪ What is affective computing?

▪ Is emotion a good thing? (for computers?)

– Review history of thinking on emotion

– Preview some applications

▪ Structure and Goals of Course

▪ Class projects and resources

▪ Will stay to talk after lecture



General comments

▪ Interdisciplinary focus

– Emphasis on concepts, not code

– You will learn more social science than you may think you need

▪ Project focus– You will work in teams on project to be presented at end of semester

– Typically 4 students per team

▪ Interactive and Participatory Style– Turn on video: This course is all about affective signals

– Please interrupt me. I prefer when people ask questions or disagree

with what I’m saying

– I will ask and expect you to participate

– If you don’t understand, show me a sign…



Note on participation in the Zoom era

▪ I frequently have in class experiments

– look at some stimuli and vote for best answer or record face▪ These recordings may play a role in homework

▪ Will explore new options because of zoom

– e.g., Later today: send Quatrics link. Be prepared to enter your

student ID. Then use zoom poll function

▪ Some experiments you have chance to make money

– Need to figure out how to pay you. Gift card? Vemo? Send suggestions

▪ If you are taking this in Europe/Asia– Participate in “in class” polls while watching the lecture

– I may figure out some other things



Who are you?



Can, how (and should) machines have

emotional intelligence?

And how does this change what it means to be human?


7CSCI 534(Affective Computing) – Lecture by Jonathan Gratch

Haslam (2006)

Human

Uniqueness

What is it to be human?



Human

Uniqueness




Human

Uniqueness


▪ Identity threats lead people to elevate alternative important traits

▪ What is one skill or attribute that, in your opinion, AI will likely never

be able to achieve or replicate?

▪ Humans are better for entertainment, creative processes, storytelling, and having

emotions. AI will never surpass the core characteristics of the human race

▪ Human emotion is probably the biggest obstacle that would be nearly impossible

to program

▪ AI will never be able to replace the emotional part of humans.



Human

Uniqueness

Human

Uniqueness




Human

Uniqueness




Are we in an era of Emotion? (see optional reading)

▪ Across the sciences, there is growing interest in the

importance of human emotion

NIMH increase in funding on research on affect since 1985

Budget

am

ount

(US

D)

Budget

%



Are we in an era of Emotion?

▪ Across the sciences, there is growing interest in the


Growth of publications on affect

# P

ublic

ations

% P

ublic

ations




▪ In the humanities, there is growing interest in the





▪ In Law, there is growing interests in emotion




▪ And in industry

Amazon RekognitionEasily add intelligent image and video analysis to

your applications

“The affective computing market was valued at US$6.9 billion in 2017

and is expected to grow at a CAGR of 42.63% to reach a market size

of US$41 billion by the year 2022.”



AI just at dawn of an era of emotional intelligence

Student frustration

Road rage

Regulating emotion

Negotiation tactics

Teaching emotion skills

Autism therapy Sales, service, marketing

Medical adherence

Manipulating emotion

Creative Arts

Computer artists Computer composers Computer chefs


And this is raising some concern

The AI Now Institute is an interdisciplinary research institute dedicated to

understanding the social implications of AI technologies



Artificial intelligence good at generating emotion


▪ “We are ourselves creating our own successors. Man

will become to the machine what the horse and dog are

to man.”

─ Samuel Butler, 1863

▪ “Men have become the tools of their tools”

─ Henry David Thoreau,1854



How will you feel when they achieve emotional

intelligence?

You know, I can feel the fear that you carry around and I wish there was... something I could do to help you let go of it

- Her (2013)


What is “Affect”

▪ In the field of Affective Computing, “affect” refers to emotion and

“related phenomena”:

– Emotions (e.g., angry, sad, joyful, fearful)

– Moods (e.g., cheerful, gloomy, irritable, listless, depressed, buoyant)

– Interpersonal stances (e.g., distant, cold, warm, supportive)

– Preferences/Attitudes/Sentiment (e.g., liking, loving, hating)

– Personality (e.g., nervous, anxious, reckless, morose)

– Culture (e.g., Individualistic vs. Collectivist; engineering vs. social sciences)



What is “Affective Computing”?

An Interdisciplinary field of research

▪ Research and develop systems that recognize, interpret, stimulate

and simulate human affect including:

– How affect influences human-computer and human-robot interaction

– How affective sensing informs machine-understanding of people

– How to make computers more human-like

– The ethics of “giving” machines emotional capabilities

▪ Covers but is not limited to the topics involving:

– Sensing and analysis (i.e., recognition of facial expressions)

– Psychology and behavior as they relate to affective computing

– Behavior generation and user interaction



Why mess up an intelligent machine by adding emotions?

▪ Affective computing presumes there is a benefit to giving

computers “emotional intelligence”

Typical view of AI Typical humanWho do you want

making decisions?

”I have a gut, and my gut tells me

more sometimes than anybody else’s

brain can ever tell me.”



Would an “emotional” machine be better?

▪ Are emotions good for people?

▪ Very old (and ongoing) debate



Theme: A war between emotion and reason

▪ Plato: The Allegory of the Chariot

▪

Biological impulses

Moral Impulses

crooked lumbering animal, … the mate of

insolence and pride, shag-eared and

deaf, hardly yielding to whip and spur.”

lover of honor, modesty and temperance,

…follower of true glory; he needs no touch

of the whip, but is guided by word and

admonition only

Reason



Theme: A war between emotion and reason

▪ Freud

Id

Super ego

Subconscious.

Primitive instinctual component of

personality

Internalized moral compass. incorporates

the values and morals of society which are

learned from one's parents and others

EgoDevelops to mediate between the

id and the real world. The

decision-making component of

personality.



Preview: Dual Process theories of emotion

▪ Argues people have 2 modes of thinking

– “Rational”

– “Emotional”

▪ Often recommend contradictory decisions

▪ In course, we’ll talk about how technology can

help one side or other “win the war”

Cognition (System 2)

• Sequential

• Rule-based

• Rational

Emotion (System 1)

• Parallel

• Associative

• Intuitive





• Sequential

• Rule-based

• Rational

Emotion (System 1)

• Parallel

• Associative

• Intuitive

Down-regulate emotion:

Use emotion sensing to detect

frustration and adjust teaching to

avoid negative student emotions





• Sequential

• Rule-based

• Rational

Emotion (System 1)

• Parallel

• Associative

• Intuitive

Up-regulate emotion

VR as an “Empathy

Engine”

Idea: Encourage people

to draw on emotions

when making decisions



If emotion a war, who should win?



Emotion a benefit or curse? A very old debate

▪ Emotions long viewed as problematic

“He only employs his passion who can make no use of his

reason” ― Cicero



Emotion a curse

▪ Emotions long viewed as problematic

“He only employs his passion who can make no use of his

reason” ― Cicero

“Passionate persons are like men that stand upon their

heads; they see all things the wrong way ” ― Plato

“The passions are like those demons … Our only safety

consists in keeping them asleep. If they wake, we are lost.”

― Goethe

“Our headstrong passions shut the door of our souls against

God” ― Confucius



Example: How could emotions make us stupid?



In class experiment

Imagine you lead LA County Public Health

You must decide between 2 plans to vaccinate a group of 60,000

high-risk people for COVID-19. Assume the exact scientific estimate

of the consequences of the two programs are as follows:

Don’t think too hard about answer.

Go with your first instinct



In class experiment





Program A: 20,000 survive

Program B: 1/3 prob. 60,000 survive

2/3 prob. no one survives

Based on Tversky and Kahneman’s 1981 Asian Disease Scenario

1/3* 60,000 + 2/3 * 0 =

20,000 + 0 =

20,000

=



In class experiment








Program C: 40,000 die

Program D: 1/3 prob. no one dies

2/3 prob. 60,000 die


= =



In class experiment








83% of class chose Program A

Program C: 40,000 die

Program D: 1/3 prob. no one dies

2/3 prob. 60,000 die

65% of class chose program D




▪ A and B (C and D) have same expected utility

▪ People don’t decide using expected utility (only AI does)

▪ People imagine different outcomes and decide based on how these make us feel

▪ Certain/concrete outcomes evoke stronger feelings

– Expected outcome identical: 200 = 1/3 * 600

Explanation (Why so many pick A?)

= =



▪ People don’t calculate expected utility

▪ We imagine different outcomes and decide based on how these make us feel

▪ Certain/concrete outcomes evoke stronger feelings

▪ These feelings sensitive to salient aspects of situation

– “Framing effect”: people can focus on gains or losses

Explanation (Why A and C different?)

=

=

200

Saved

400

Die

600

People

total



Preview: Emotion distorts reason

▪ Emotion shapes decision-making in ways that seem

clearly irrational

▪ We can mathematically model this decision “bias”



Emotion a benefit?

▪ But emotion also argued to be indispensable for good

decision making (and this view growing over time)

“The passions are the voice of the body” ― Rousseau

““Nothing ever becomes real 'til it is experienced.” ― Keats

“The passions are the winds which fill the sails of the vessel;

they sink it at times, but without them it would be impossible

to make way” ― Voltaire

“No one cares how much you know, until they know how

much you care” ― Teddy Roosevelt



Can emotions make us smart? Example

Wilson, T. D., Lisle, D., Schooler, J. W., Hodges, S. D., Klaaren, K. J., & LaFleur, S. J.

(1993). Introspecting about reasons can reduce post-choice satisfaction. Personality and

Social Psychology Bulletin, 19, 331–339.

Task: Pick a poster to take home

• One group just grabbed one they

feel good about

• Other group asked to think carefully

and write down their reasons for

choosing

• Got to reconsider 6-months later



Emotion a benefit: decision-theory perspective

“Reason is, and ought to be, only the slave of the passions”David Hume, 1711-1776

• Reason essential for accomplishing our goals

• But without emotion, we wouldn’t have goals• Without passions we would lack all motivation, all

impulse or drive to act, or even to reason

• Emotion = Utility

The Ideal



Emotion a benefit: evolutionary perspective

▪ Emotions must be adaptive or they wouldn’t have evolved



Darwin’s principles

▪ The principle of “serviceable habits”– Certain expressions confer evolutionary advantage

– Surprise is useful “so the field of vision may be increased,

and the eyeballs moved easily in any direction” p.281

– Some expressions may no longer be relevant for civilized man

– Sneering in rage is a sign that our ancestors fought with our teeth

whereas “civilized” humans do not



Darwin’s principles

▪ The principle of antithesis – Some expressions of emotion look the way they do because they are

the opposite of a “serviceable” expression

– Dogs “playful” expression evolved because this minimizes confusion

with their signal of anger



Emotion a benefit: neuroscience perspective

Describes people brain damage to the ventromedial prefrontal cortex

▪ Intelligence intact (measured by standard IQ tests)

▪ But seem to lack emotion

Phineas Gage



Emotion a benefit: neuroscience perspective

Describes people brain damage to the ventromedial prefrontal cortex

▪ Intelligence intact (measured by standard IQ tests)

▪ But seem to lack emotion

These people seem incapable of making

good decisions

Argues emotion indispensable for

“real-world” decision-making



Emotions and Artificial Intelligence

Early AI research discussed the importance of emotion



Emotions and Artificial Intelligence

Psychologist Ulric Neisser argued that computers

couldn’t capture human-level intelligence because

they lacked bodies and emotion

Herb Simon (a founding-father of AI) responded

that intelligent machines must have mechanisms

akin to emotion

"Motivational and emotional controls of

cognition", Psychological Review, vol. 74, 1967

Other early AI researchers also emphasized importance of

emotions for machines (Johnson-Laird; Minsky)


http://diva.library.cmu.edu/webapp/simon/item.jsp?q=/box00065/fld04976/bdl0001/doc0001/


But somehow, emotion was left aside

▪ Despite early interest in emotions, AI turned to focus on rational and dis-embodied cognition– Chess, not “Life”

– Maybe emotion was too hard?

▪ Emphasized – Logic & Rationality over emotion

– Non-social phenomena

– Thinking over acting

▪ Ignored emotions

▪ Ignored the body

▪ But this starting to change



Emergence of Affective Computing: 1995

▪ Roz Picard introduced the term “Affective Computing” in 1995

– Her book with that title published in 1997

▪ Originally defined as

– “computing that relates to, arises from, and deliberately

influences emotion,”



Today

A vibrant community of practice

▪ Represented by a professional society

▪ International Conference (started in 2005)

▪ International Journal, impact factor 6.29 (AIJ impact 3.33)



Some typical recent research topics (from TAC)

▪ Emotion Recognition in:

– Speech:▪ Emotion in natural speech

▪ Depression detection

– Text▪ Opinions in twitter; blogs

▪ Emotocons

– Face▪ Understanding impact of aging

▪ Recognizing expressions with thermal

– Physiology▪ Inferring response to music via EEG

▪ Detecting stress from skin conductance

▪ Synthesis– Emotional speech

– Emotional facial expressions

▪ Games/Entertainment computing

– Responses to victory and defeat

– Affective music player

– Boredom detection

▪ Modeling– Modeling emotional influences on decision-

making

– Modeling factors that elicit emotions

▪ Applications

– Health – detection and shaping

– Games/entertainment – detection

and shaping; synthesis/realism

– Education – detections; shaping

▪ Behavioral science



Current thinking: Emotions good

▪ Emotions serve important functions, not just for

humans but any intelligent entity

▪ Some of these functions missing from traditional

AI/rational models– Thus, and analysis of the function emotions serves in people can

improve artificial intelligences

▪ And, regardless of their function for machines,

computers have to deal with emotional humans– Thus, they would benefit from recognizing and understanding these

processes



Although thinking may be shifting?



So what is the function emotion?



Marsella & Gratch, Journal of Cognitive Systems Research 2009

Example: Emotion “in the wild”



What is function emotion?

▪ Physical: Shapes the body– Action preparation: energizes body, changes physical orientation

▪ Cognitive: Shapes the mind– Rapid, continually adjusting assessment of significant events

– Interruption of behaviors and changing of goals

▪ Social: Shapes the minds of others– Signaling: broadcast information about mental state

– Coordination: orient and coordinate group response



Emotion prepares the body for the circumstances that

elicited the emotion

▪ Orienting response:

– Opens eyes wider, allowing in more light to

perceive information

– Mouth opens and breath taken, pulling more

oxygen to the body

▪ Defensive response

– Eyes lower, defensive posture

– When angry, blood flows to the extremities:

preparing for physical action

Surprise



Emotion shifts cognition to a mode appropriate for the

circumstances that elicited the emotion

▪ Emotions change perception & decision-

making (e.g., Anger)

– Quicker to perceive threats (DeSteno et al 2000/2004)

– Underestimate risk (Lerner & Keltner 2000/2001)

– Use quick/heuristic reasoning

(Bodenhausen et al 1994)

– Blame others/outgroups

(Keltner et al 93; Mackie et al 00)

– Remember anger-evoking past events (Bower, 91)



Seeing emotions in others shapes our own cognition

to prepares to interact with that person

▪ Emotions impact social interaction

– Distress elicits helping (Eisenberg et al 89)

– Anger elicits fear (Dimberg&Ohman96)

– Negotiators concede more to angry partner (van

Kleef et al. 2007)

– Emotion communicates information to other

social actors (Darwin; Parkinson01)



Emotion is also information: Mind reading

Well, Guy Cuny is the editor of the technology website, news wireless...



Some practical applications of affective computing



Facial expression

recognition

Detect affect in

speech

Detect affect in

posture & gesture

Sentiment analysis

Facial expression

synthesis

Posture and

gesture synthesis

Automatic

contingent

empathetic

feedback

High-level

understanding

and decision-

makingEstimate prevalence

of depression

Example: Mental Health



69

Example: Skills Training

Teach us to use and control our emotions in a negotiation

David DeVault, Johnathan Mell and Jonathan Gratch. Toward Natural Turn-Taking in a Virtual Human Negotiation Agent. AAAI Spring

Symposium on Turn-taking and Coordination in Human-Machine Interaction. 2015, AAAI Press: Stanford, CA.



70

Example: Psychology

Understand how emotions shape social decisions

Stratou, Hoegen, Lucas and Gratch. Emotional Signaling in a Social Dilemma: an Automatic Analysis. 6th International

Conference on Affective Computing and Intelligent Interaction. Xi’an, China, 2015

Synchronized

video

Personality

Task events

Automated Facial

Expression



71

Example: Marketing



72

Example: Marketing



73

Example: Tutoring systems (e.g., D’Mello)



74

Example: Entertainment and Games

Understand how to make emotionally compelling interactions

Hartholt, Gratch, Leuski, et al., At the Virtual Frontier: Introducing Gunslinger; a Multi-Character; Mixed-Reality; Story-Driven Experience. 9th International

Conference on Intelligent Virtual Agents, Amsterdam. 2009



75

Example: Entertainment and Games

• How would a machine know

what to feel?

• How should it display those

expressions?

• How should it change its

decisions?

• How does it sense our

intentions and emotions?

• How does it react to our

touch?

• Do we feel sad for it?

• Does it change how we treat

other people?

• Does it matter if its organic or not?



Summary of Course Goals

▪ Develop a basic understanding of emotion theory

and findings from social sciences

▪ Understand functions of emotions in context of

autonomous systems and human-machine

interaction

▪ Understand state-of-art in affective computing

techniques

▪ Understand methodological issues in collecting and

analyzing data on human emotions



Grades

▪ 10% Class participation

▪ 40% Homework

▪ 50% Group Project

15% Mid-term project presentation

15% Final project presentation

20% Project writeup

TA: Su Lei; Office hours by appointment



Syllabus



Student projects

▪ Big focus of course on group project– Teams typically involve 4 students

– Expect to finalize teams/tentative-project idea by Feb 22



Student project examples




▪ Looked at 2 questions– Does felt emotion change tendency to lie? (predicted that induced

anger would increase number of lies)

– Can a machine detect a person is lying from facial expressions



▪ General Toolkits

▪ Virtual Human Toolkit: contains a number of sensing, language

and synthesis tools. Allows one to construct interactive

affectively-aware digital characters

.

Resources


https://vhtoolkit.ict.usc.edu/

http://people.ict.usc.edu/~gratch/CSCI534/Tools-and-projects-2020.pdf


OpenFace

High-quality freely-available

facial action unit recognizer

▪ https://github.com/TadasBa

ltrusaitis/OpenFace

Resources Facial Expression Analysis


https://github.com/TadasBaltrusaitis/OpenFace



COVAREP

Freely-available voice analysis toolkit

▪ http://covarep.github.io/covarep/

OpenEAR

Another freely-available voice analysis tool

▪ https://sourceforge.net/projects/openart/

Resources Voice Analysis


http://covarep.github.io/covarep/

https://sourceforge.net/projects/openart/



LIWC

Calculates the degree to which people use different categories of

words associated with emotion and other mental states

▪ http://liwc.wpengine.com/

Text Analysis, Crawling and Interpretation Tool

Freely-available tool for classifying social media text

▪ http://tacit.usc.edu/index.html

Sentiment analysis tools

Large collection of tools maintained by Rada Mihalcea

▪ https://web.eecs.umich.edu/~mihalcea/downloads.html

BERT, word2vec, ….

Resources Text Analysis


http://liwc.wpengine.com/

http://tacit.usc.edu/index.html

https://web.eecs.umich.edu/~mihalcea/downloads.html

Tools-and-projects.pdf


LIWC

Calculates the degree to which people use different categories of

words associated with emotion and other mental states

▪ http://liwc.wpengine.com/

Text Analysis, Crawling and Interpretation Tool

Freely-available tool for classifying social media text

▪ http://tacit.usc.edu/index.html

Sentiment analysis tools

Large collection of tools maintained by Rada Mihalcea

▪ https://web.eecs.umich.edu/~mihalcea/downloads.html

Resources Text Analysis


http://liwc.wpengine.com/

http://tacit.usc.edu/index.html



▪ Affect generation

▪ SmartBody – character animation system

▪ NVBG – Nonverbal Behavior Generation System

Resources




Resources Agent Architectures




Resources Experimental game environments




▪ Databases

▪ Face expressions

▪ Depression: DAIC-WOZ

▪ Rapport

▪ Economic games (Poker, Prisoner’s Dilemma,

Negotiation)

▪ Twitter

Resources




Final comment

▪ This class emphasizes Western thought on emotion

▪ But views of emotion differ by culture



Final comment

▪ This class emphasizes Western thought on emotion

▪ But views of emotion differ by culture

– No word for emotion in Sanskrit

– Strong dichotomy between emotion and cognition missing from Indian

Philosophy

– Strong dichotomy between individual and social emotions missing

from East Asian philosophy


csci 534: affective computinggratch/csci534/lecture2021-01.pdfcsci 534(affective computing)...

Documents