lecture 5: physiological and brain computingict.usc.edu/~gratch/csci534/lecture2015-11.pdf ·...

Lecture 5: Physiological and Brain Computing

Outline

Review of ACII

Discuss EMA and Battleship

Brain and Body Computing

– Previously learned about physiological measures

Guess Lecture by Sarah Townsend, Business School

This one type of body measurement

– Talk more generally about brain/body measures

– Talk about different ways computers can induce and use these signals

– Focus on EEG

Embodied Cognition (how body effects mind)

Workshops

– 4th Workshop on Affective Brain Computer Interaction (aBCI 2015) development of human-computer interfaces able to react and adapt to users' emotions and related

cognitive states as measured from neurophysiological signals

– Affective Social Multimedia Computing analysis of affective signals in social multimedia (e.g., twitter, wechat, weibo, youtube, facebook, etc)

– Workshop on ENgagement in HumAN Computer IntEraction focuses on engagement modeling and recognition in human-human and human-machine interaction

– Workshop on Automatic Sentiment Analysis in the Wild analysis of human sentiment, and empathic and social behaviour observed in multi-party games,

user-centric healthcare and online services, automatic market research analysis, etc

– Workshop on Affective Touch Touch is important in social interactions, such as hugging, and can elicit strong affective responses.

Think for example of using touch for social communication with virtual agents or social robots,

Workshop on Sentiment Analysis in the Wild

Conference Themes

Special sessions– Laughter

– Affect in Games

Main tracks– Affect in human-machine

interaction

– Modeling emotion and cognition

– Affect in Psychophysiology

– Affect in Speech

– Affect in Social Communication

– Affect in Health

– Multimodal Perception

– Machine learning

– Affect synthesis

– Video perception

Best paper

Zakia Hammal, Jeffrey Cohn, Carrie Heike and Matthew Speltz. What

Can Head and Facial Movements Convey about Positive and

Negative Affect?

The Appraisal Equivalence Hypothesis

The projects or efforts depicted were or are sponsored by the U.S. Army Research,

Development, and Engineering Command (RDECOM). The content or information

presented does not necessarily reflect the position or the policy of the Government, and no

official endorsement should be inferred.

Jonathan Gratch, Stacy Marsella and Lin Chen

University of Southern California and Northeastern University

Is it really possible to make domain-independent

computational models of emotion?

10

Richard Lazarus Robert Zajonc

11

Good Old-fashioned AI

Classical AI rests on the foundation that general intelligence

is symbol manipulation

e.g. Alan Newell’s (‘80) Physical Symbol System Hypothesis

– Necessary and sufficient condition for a system to exhibit general

intelligence is that it be a symbol system

And this applies to human intelligence as well

– “symbol systems are the appropriate class within which to seek the

phenomena of mind” (Newell 1980)

And this a powerful and useful concept

– The foundation of domain independence

– AI systems involve general reasoning methods that operate over

symbols, regardless of what those symbols denote

Thinking = Rule-governed manipulation

of symbolic representations

In humans, symbols

are instantiated in the

brain

The same symbols

can be instantiated in

a computer

12

But does this apply to emotion?

This hypothesis has come under sustained attack (cf Nilsson 2007)

– Embodiment: Intelligence symbols to be connected to the world via a

physical body that senses, acts and experiences (Niedenthal, Damasio)

– Non-symbolic: Intelligence involves analog signal processing

– Nonconscious: Intelligent behavior is really mindless, reflexive,

chemical activity

These are the hallmarks of emotion

Intelligence requires more than a brain in a vat

13

System 1 System 2Integration

Intelligence

Counter claim: dual-process theories of minde.g., Kahneman

Symbol SystemAffect System Symbol System

• slow

• conscious

• reflective

• forward-looking

• self-regulatory

• effortful

• exhaustible

Affect System

• fast

• unconscious

• reflexive

• myopic

• effortless

14

Emotions are not symbol systems?

Emotion arises from non-propositional content (e.g., vividness)(Nisbett and Ross 1980)

Jack sustained fatal injuries in a car crash

Jack was killed by a semi trailer that rolled over on his car and

crushed his skull

Niedenthal et al

– Just as people appear to know that CARS possess the features

engines and tires, they know that ANGER involves the experience of a

thwarted goal, and a willingness to strike

– But this is not what we mean by emotion

15

What does this say for the enterprise of creating

computational models of emotion?

Modeling?

Depends on Newell’s Physical Symbol System Hypothesis

16

Appraisal

Desirability

Expectedness

Controllability

Causal Attribution

Bodily Response

ExpressionAction

Tendencies

PhysiologicalResponse

Environment Mental State(beliefs, goals)

Theoretical Perspective: Appraisal Theory

Scherer, Klaus R., Angela Ed Schorr, and Tom Ed Johnstone. Appraisal processes in emotion: Theory, methods, research. Oxford University Press, 2001.

Emotion reflects the “person-environment relationship”

17

Appraisal

Desirability

Expectedness

Controllability

Causal Attribution

Bodily Response

ExpressionAction

Tendencies

PhysiologicalResponse

Environment Mental State(beliefs, goals)

Theoretical framework: Appraisal Theory

Coping

Problem-focused

coping

Emotion-focused

coping

18

RetakeCause: self

Intend: yes

Prob.: 50%

Get PhDUtility: 50Prob: 50%Intend: True

Past Present Future

Get PhD

Utility: 50 Prob.: 100%Belief: FalseFail Exam

Cause: Stupid

Teacher

Intend: yes

Prob: 100%

FacilitatesInhibits

Appraisal Theory in EMAUses good old-fashioned AI to derive appraisals

Working memory of beliefs, desires, intentions

HOPE (25)Desirability: 50

Likelihood: 50%

Causal Attribution: self

Coping Potential: Moderate

ANGER (50)Desirability: -50

Likelihood: 100%

Causal Attribution: Other

Coping Potential: moderate

Appraisal = Symbolic inference

19

Dimensional

Appraisal

ARElliott

ARElliott

EMNeal Reilly

EMNeal Reilly

FLAME

El Nasr

FLAME

El Nasr

EMILEGratch

EMILEGratch

CBIMarsella

CBIMarsella

EMAGratch/Marsella

EMAGratch/Marsella

FearNot!Dias

FearNot!Dias

PEACTIDMMarinier

PEACTIDMMarinier

CATHEXISVelásquez

CATHEXISVelásquez

ArmonyArmony

Scheultz&

Sloman’01

Scheultz&

Sloman’01

ALMA

Gebhard

ALMA

Gebhard

WASABIBecker-Asano

WASABIBecker-Asano

ACRES

Swagerman

ACRES

Swagerman

WILLMoffat

WILLMoffat

Anatomical

RationalNML1Beaudoin

NML1Beaudoin

MINDER 1Wright

MINDER 1Wright

Alvila-Garcia,

Canamero

Alvila-Garcia,

Canamero

FrijdaFrijda

OCCOCC

LazarusLazarus

SchererScherer

MehrabianMehrabian

DamasioDamasio

LeDouxLeDoux

SlomanSloman

ParleEBui

ParleEBui

THESPIANSi et al.

THESPIANSi et al.

Gmytrasiewicz

Lisetti ‘00

Gmytrasiewicz

Lisetti ‘00

TABASCOStaller&Petta

TABASCOStaller&Petta

ActAffActRank

ActAffActRank

Most models rely on appraisal theory

20

EMA as a symbol system

EMA is domain independent

– A domain is represented as propositions

– EMA leverages general reasoning methods that operate over

symbols, regardless of what those symbols denote

Emotion Symbol System Hypothesis:

(Appraisal Equivalence Hypothesis)

Emotion should be determined by the deep symbolic structure (goals,

actions, causal threats..)

Competing hypothesis (Embodiment hypothesis)

Emotion should be determined by surface characteristics (e.g., vividness)

21

Test of the Symbol System approach to Emotion

Create two domains that share identical deep

propositional structure but differ considerably in terms

of surface characteristics

Implement the deep structure in EMA

See if EMA predicts emotional responses in these two

domains

22

First Domain: Battleship

23

Second domain: Mousewars

24

These are the same game: Deep structure

WIN

LO

SE

25

These are the same game: Deep structure

Sunk!

Sunk!

Sunk!

I win

You win

26

Battleship setup

Confederate Participant

27

Mousewars setup

28

Failure!

Success!

State

Being Ahead

Being Behind

Being Even

Win

nin

g

Losing

Automatically measure smiles (Mousewars only)

Self-reported

• Appraisal

• Emotion

• Coping

Measures of Emotion

29

EMA Predictions

Emotions influence by probability of goal attainment

Emotions influenced by Utility of goal attainment

Should hold independent of surface form

Winning

Losing

30

Results

Emotion intensity

reflects probability

of goal attainment

– Up as win approaches

– Down as win recedes

Intensity reflects

goal importance

– Strongest changes for

those a priori most

invested in winning

Identical pattern for

Mousewars

Ho

pe

Jo

y

Ho

pe

Jo

y

31

Results

Control influenced

by probability and

initial utility

Subjective

probability reflects

objective change but

also initial utility

Same pattern for

mousewars

Battleship Mousewars

32

Face Results (Mousewars only)

Perceptions of winning determine expressed joyE

xpre

ssed J

oy

*

*

33

Limitations

Supportive but not definitive

– Many ways to manipulate surface form:

Vividness

Incidental emotion

– Many ways to manipulate appraisals

Trajectory

Control

Loss vs. Gain

What if we had failed?

– Wouldn’t necessarily rule out symbol-system hypothesis

– Surface changes could have evoked different goals, actions

34

Coin-flip game

Probably less sense of control

– In battleship, can place ships (sense of control)

But not obviously reflected in mousewars/battship distinction

– In mousewars/battleship, perceived control whereas coin-flip is

perceived random

Physiological and Brain Computing

Psychological constructse.g., frustration

Physiological measures

e.g., skin conductance

One possible application

Somewhat more serious application




Psychological constructs

e.g., frustration

1 23

4




1

Measures

fMRI – functional magnetic

resonance imaging

Measures


resonance imaging

EEG – electro encephalogram

Measures


resonance imaging


fNIRS - functional near-infrared

spectroscopy

Measures


resonance imaging


fNIRS - functional near-infared

spectroscopy

Hormone levels in blood, saliva or

urine

Measures


resonance imaging



spectroscopy


urine

EMG - electromyography

Measures


resonance imaging



spectroscopy


urine


GSR/EDA – skin conductance

HRV – heart-rate variability

BP – blood pressure

TPR – total peripheral resistance

Measures


resonance imaging



spectroscopy


urine


GSR/EDA – skin conductance

HRV – heart-rate variability

BP – blood pressure

TPR – total peripheral resistance

Disease history

White blood cell counts

Measures

Properties of measures: resolution

Resolution– Spatial: how specific is the part of the body/brain measured

fMRI has high spatial resolution

EEG / fNIRS have low spatial resolution

– Temporal: what is the “frame rate” EEG has high temporal resolution

fMRI has low temporal resolution (depends on metabolic processes)

“effective” resolution– Also have to consider properties of system being measured

Immunological changes: days or weeks

Endocrine changes: minutes

Visceral changes: seconds

Neural changes: milliseconds

– Slower systems tend to be less localized (GSR same at hands and feet)

Measurement resolution

Temporal Resolution

Spatial R

esolu

tion

High

Low

Low HighWhite

blood cellsCortisol

Impedance

Cardio.

fMRI

(metabolic)

EEG

(electrical)EMG

(electrical)

How do we use these measures

e.g., EMG tells us state information– Brows are furrowed

Affect tends to be short term response to stimuli– Emotion stimuli specific and momentary

– Mood more generalized and tends to last minutes

Measures most useful to index response– i.e., state change

Properties of measures: probing or monitoring

Evoked (Probing / Event-related)– System produces stimulus and measures immediate response

e.g, Flash an angry face

– Analysis: often average over repeated presentations to control noise e.g., IAPS: present multiple positive and negative images

Induced (Monitoring / Endogenous)– System monitors changes in user state

– Changes considered “endogenously” produced – i.e. Reflects some

mental processing

– Could include indirect responses to computer stimuli: e.g., frustration

in response to computer delivered exercise

– Analysis: apply frequency transformation and look for oscillations

Properties of measures: probing or monitoring

One example: Gene expression (Steven Cole UCLA)

Affect impacts your genes

– Genes determined by heredity

– Gene expression determined by affect and environment?

– DNA analysis identified 209 genes that were differentially expressed in

high- versus low-lonely individuals

up-regulation of genes involved in immune activation, and cell proliferation

down-regulation of genes supporting lymphocyte and interferon response

Correlation or causation?

Mindfulness-based stress reduction

training reduces loneliness and pro-

inflammatory gene expression in older

adults: a small randomized controlled trial

8 week training program

http://www.sciencedirect.com/science/article/pii/S0889159112001894 2

One example: Gene expression (Steven Cole UCLA)

Isolated for 21 days Controls

More practical for this class: EEG

Good

temporal

resolution

Bad

spatial

resolution

Illustration EEG

Noisy Signal

EEG Problems

Noisy Signal

EEG Problems

Normal Effect of eye blinks Effect of eye movement

EEG Measurement approaches

Evoked: Time-domain correlates (ERP)– Derived by averaging multiple traces following stimulation events of

same condition

– Example of “evoked” (probing) approach

– Can look for potentials in different parts of head


P300

Time-domain correlates (ERP)– Early ERPs: reflect automatic evaluation

P1/N1: reflect initial perception and automatic evaluation of stimuli

– Late ERPs reflect higher-level processes.

P200: sensation-seeking behavior

P300 unexpected stimuli

N400 processing difficulty (e.g., difficulty in

understanding meaning of a word)

P600: follows syntax violation in language

Problem: requires multiple probes

Not practical in many apps.

Example: P300

Present series of stimuli– Beep, Beep, Beep, Beep, BOOP, Beep, Beep,….

– P300 bigger when stimuli important to subject (increased utility) e.g., loose $5 every time BOOP occurs

– P300 bigger when unexpected stimuli less common (lower probability)

– Thus, related to two common appraisal variables

Evokes P300


Alternative: brain rhythms in frequency domainApply frequency transformation and look for oscillations

FERRARI, Rosana; ARCE, Aldo Ivan Cespedes; MELO, Mariza Pires de and COSTA, Ernane Jose Xavier. Noninvasive method to assess the electrical brain activity

from rats. Cienc. Rural [online]. 2013, vol.43, n.10


Alternative: brain rhythms in frequency domainApply frequency transformation and look for oscillations

– Delta rhythm: associated w/ hunger and drug craving (reflects workings of

brain reward system). Also correlates w/ P300 ERP. May be associated with

detection of emotionally salient stimuli

– Theta rhythm: indexes working memory / memory demands;

Frontal-medial theta associated w/ positive valence

– Alpha rhythm: associated with sensory processing (e.g., music, films)

Frontal alpha symmetries vary as function of valence.

Rightward lateralization associated w/ positive or approach-related emotions

(contrasting w/ negative withdrawal-related emotions)

– Beta rhythm: associated w/ sensory-motor system: increased activity

associated w/ positive emotions

– Gamma rhythm: associated w/ attention, memory and consciousness.

Correlated w/ positive valence.

Posterior increases associated with highly arousing visual stimuli.

Gamma over somatosensory cortex associated w/t pain





e.g., frustration

2

State Predictions


e.g., frustration

2


Any affective brain state that can be inferred with reliability

– Emotion, mood, appraisals

Can distinguish by temporal characteristics

– Tonic state: a longer-term steady state (e.g., mood) E.g. average skin conductance over a 5minute speech

– Phasic state: a short term state or transition (e.g., emotion) E.g., skin conductance 5 seconds following presentation of disgusting picture


Any affective brain state that can be inferred with reliability

– Emotion, mood, appraisals

Can distinguish by temporal characteristics

– Tonic state: a longer-term steady state (e.g., mood) E.g. average skin conductance over a 5minute speech

– Phasic state: a short term state or transition (e.g., emotion) E.g., skin conductance 5 seconds following presentation of disgusting picture

Can distinguish if “deliberate” or not– Active (explicit): states “created” intentionally by user

Remember a stressful event

Try to relax

– Passive (implicit): states occurring w/o conscious control Subliminal presentation of angry face





e.g., frustration3

Mappings

Predictors of psychological state

Physiological

measures


e.g., skin conductance e.g., frustration

Properties of mappings

Specificity: characteristic of f(measure) = construct

– One-to-one: amygdala activity represents appraisal of relevance

– Many-to-one: cortical activity, BP and heart rate together predict effort

– One-to-many: skin conductance indicates positive or negative emotions

– Many-to-many

Context-dependence: invariance of measure across situations

– Context-independent mapping would hold across lab and “real world” and

across social and task contexts

– Context-dependent would hold only in certain situations (e.g., smiles might

predict true feelings in presence of friends by not strangers)

Sensitivity: correlation between measure and construct

Some applications




e.g., frustration

(Response to stimuli)(Endogenous)

(Intention to control -- e.g. imagined arm movements)

(No intentional to control)

Affect

monitoring

Affect

Regulation

Affect

probing

Affect

Communication




Affect

monitoring

Affect

Regulation

Affect

probing

Affect

Communication

Examples

Neuro-feedback Systems

(e.g., depression treatment)

• Teach positive emotion regulation

techniques

• Use EEG to measure success

• Analyze EEG in frequency

domain

• Look for alpha rhythm

asymmetries

• Ring bell when user achieves

desired brain state

• Proved clinically effective

Hammond, D. Corydon. "Neurofeedback

treatment of depression and anxiety."Journal of

Adult Development 12.2-3 (2005): 131-137.



Affect

monitoring

Affect

Regulation

Affect

probing

Affect

Communication

Examples

Sensing user boredom or frustration

• Could use to adjust difficulty in

computer game to maintain user

satisfaction

• Could use as feedback in

tutoring system to adjust learning

level or pedagogical feedback

(Response to stimuli)


Affect

monitoring

Affect

Regulation

Affect

probing

Affect

Communication

Examples

Assess affective responses across multiple “probes” (e.g., music or video clips

• Monitor the user response to media

• Tag media with affect it produced

• Selectively offer or automatically play

back media items known to induce a

certain affective state in the user

• (e.g., Koelstra et al., 2012; Soleymani

et al., 2011).

Use audio probe and measure if

attending to it as measure of

workload

(Response to stimuli)



Affect

monitoring

Affect

Regulation

Affect

probing

Affect

Communication

Examples

Standard approach for non-affective

BCI

• e.g., P300 speller: relies on P300

response to letter “probes”

Slide from http://sccn.ucsd.edu/wiki/Introduction_To_Modern_Brain-Computer_Interface_Design

Summarize

Emotions shape human decision-making– Can try to account for this in a general sense in system design

Emotions reflected in physiological signals– Can try to make use of this to recognize and react to moment-to-

moment emotions

Many approaches proposed– Measuring different physiological systems

– Exploiting different measurement strategies (active probing vs. passive

monitoring)

– Examining different characteristics of signals (tonic, phasic, …)

Work in progress

lecture 5: physiological and brain computingict.usc.edu/~gratch/csci534/lecture2015-11.pdf ·...

Documents