using electrodermal activity to recognize ease of...
TRANSCRIPT
![Page 1: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/1.jpg)
Using Electrodermal Activity
to Recognize Ease of
Engagement in Children
during Social Interactions
Javier Hernandez Ivan Riobo
Agata Rozga
Gregory D. Abowd
Rosalind W. Picard
Expeditions in
Computing
![Page 2: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/2.jpg)
![Page 3: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/3.jpg)
3
Can we characterize qualitative
aspects of children’s social
engagement with wearable biosensors?
![Page 4: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/4.jpg)
Contents • Background
• Data Collection
• Response Characterization
• Engagement Recognition
• Conclusions
4
![Page 5: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/5.jpg)
Background
![Page 6: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/6.jpg)
Engagement Measurement
6
Self-reports
Fast
Direct
Subjective
Recall problems
Disruptive
Not adequate for children
![Page 7: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/7.jpg)
Engagement Measurement
7
Self-reports
Fast
Direct
Subjective
Recall problems
Disruptive
Not adequate for children
Continuous
Widely used for faces
Time intensive
Costly
Training
Coders
![Page 8: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/8.jpg)
Engagement Measurement
8
Self-reports Physiology
Less disruptive
Objective
Difficult to analyze
Instrumentation
Unfamiliar
Fast
Direct
Subjective
Recall problems
Disruptive
Not adequate for children
Continuous
Widely used for faces
Time intensive
Costly
Training
Coders
![Page 9: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/9.jpg)
Engagement Measurement
9
Physiology Wedel & Pieters, 2000
LaBarbera, Tucciarone, 1995
Silveira et al, 2013
Hernandez et al 2013
Lang, 1990
![Page 10: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/10.jpg)
Social Engagement
10
Interactional Synchrony “Coordination of behaviors between individuals during social interactions.”
Physiological linkage
Levenson & Gottman , 1983
30 married couples
Marci et al, 2007
20 patient-therapist dyads
![Page 11: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/11.jpg)
Data
Collection
![Page 12: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/12.jpg)
12 Project: http://cbs.gatech.edu
Computational Behavioral Science Modeling, Analysis, and Visualization of Social and Communicative Behavior
Dataset: http://cbi.gatech.edu/mmdb
![Page 13: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/13.jpg)
Good Indicator
• Arousal
• Cognitive Load
• Wireless
• Comfortable
• 32 Hz
• 4 sensors
Limitations
• Specificity
• Artifacts
Electrodermal Activity
Affectiva QTM
![Page 14: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/14.jpg)
Social Interaction Greet Ball Play Book Hat Tickle
5 scripted activities with an adult
External coder
“Amount of effort required to get child’s attention” Ratings for each Activity
• 0 – little effort
• 1 – some effort
• 2 – extensive effort 14
![Page 15: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/15.jpg)
Social Interaction
23 sessions exclude due to artifacts
51 clean sessions 24 males, 27 females
Average age: 21 months (STD: 5.23)
Average duration: 2.72 minutes (STD: 1.02)
Engagement groups
• Easier to engage (N = 29)
• Harder to engage (N = 22)
Greet Ball Play Book Hat Tickle
15
![Page 16: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/16.jpg)
EDA Response
Characterization
![Page 17: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/17.jpg)
Preprocessing 1. Normalize EDA values (Lykken, 1971)
2. Reduce noise (Hanning filter, 1 sec. window)
3. Extract tonic and phasic EDA (Benedek and Kaembach, 2010)
17
EDA
Phasic Tonic
![Page 18: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/18.jpg)
Feature Extraction
18
Ton
ic/P
ha
sic
C
om
po
ne
nts
Time
Child Adult
Synchrony Features
Individual Features
Mean
# Peaks
Area under the curve
Position Maximum
Position Minimum
Standard Deviation
Slope
Average Peak Amplitude
Pearson Product-moment Correlation (standard)
Canonical Correlation (explores different representations)
Dynamic Time Warping (allows for temporal flexibility)
L2-norm between means
L2-norm between #peaks
L2 norm between averages of peak amplitudes
Ton
ic/P
ha
sic
C
om
po
ne
nts
![Page 19: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/19.jpg)
Feature Extraction
19
Ton
ic/P
ha
sic
C
om
po
ne
nts
Time
Child Adult
Synchrony Features
Individual Features
Mean
# Peaks
Area under the curve
Position Maximum
Position Minimum
Standard Deviation
Slope
Average Peak Amplitude
Pearson Product-moment Correlation (standard)
Canonical Correlation (explores different representations)
Dynamic Time Warping (allows for temporal flexibility)
L2-norm between means
L2-norm between #peaks
L2 norm between averages of peak amplitudes
Ton
ic/P
ha
sic
C
om
po
ne
nts
![Page 20: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/20.jpg)
Feature Extraction
20
Ton
ic/P
ha
sic
C
om
po
ne
nts
Time
Child Adult
Synchrony Features
Individual Features
Mean
# Peaks
Area under the curve
Position Maximum
Position Minimum
Standard Deviation
Slope
Average Peak Amplitude
Pearson Product-moment Correlation (standard)
Canonical Correlation (explores different representations)
Dynamic Time Warping (allows for temporal flexibility)
L2-norm between means
L2-norm between #peaks
L2 norm between averages of peak amplitudes
Ton
ic/P
ha
sic
C
om
po
ne
nts
![Page 21: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/21.jpg)
Engagement
Recognition
![Page 22: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/22.jpg)
Experiment Details
22
10 fold cross validation for training and testing
http://www.csie.ntu.edu.tw/~cjlin/libsvm/
Easier to Engage VS Harder to Engage
Misclassification Costs: 2^[ 0:1:18]
Support Vector Machines
Linear Kernel
Easier: 29 samples
Harder: 22 samples
Boser et al., 1992
Cortes and Vapnik, 1995
Validation
Sequential Feature Selection
![Page 23: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/23.jpg)
0 20 40 60 80 100 120 1400.25
0.3
0.35
0.4
0.45
0.5
0.55
0.6
0.65
0.7
0.75
Avg. Time (seconds)
Avg
. N
orm
aliz
ed
ED
A
0 20 40 60 80 100 120 140 1600.25
0.3
0.35
0.4
0.45
0.5
0.55
0.6
0.65
0.7
0.75
Avg. Time (seconds)
Avg
. N
orm
aliz
ed
ED
A
Avg. Normalized EDA
(±Standard Error)
Easier to Engage Children Harder to Engage Children
Average EDA Response
23
![Page 24: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/24.jpg)
Results
50
55
60
65
70
75
80
85
90
IF+SF IF SF IF SF
IF: Individual Features (only from the child’s EDA) SF: Synchronized Features (from the dyad’s EDA)
Best performance 69% Similar performance for both IF and SF IF better from tonic SF better from phasic Best IF: Position Max. from Tonic Best SF: #Peaks from Phasic (>13% than correlation) Tonic and Phasic decomposition improves >6% and interpretability
One feature at a time
Av
erag
e A
UC
s (R
OC
s an
d P
reci
sio
n/R
ecal
l)
24
![Page 25: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/25.jpg)
Results
50
55
60
65
70
75
80
85
90
IF+SF IF SF IF SF
IF: Individual Features (only from the child’s EDA) SF: Synchronized Features (from the dyad’s EDA)
Feature selection improves >11% SF slightly better than IF (>2%) Tonic and phasic equally represented SF and IF are complementary (81%) Some of most selected features: Difference between peaks from phasic STD and DTW from tonic
Combinations of features
Av
erag
e A
UC
s (R
OC
s an
d P
reci
sio
n/R
ecal
l)
25
![Page 26: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/26.jpg)
Conclusions
![Page 27: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/27.jpg)
• 51 child-adult dyads in a structured social interaction
• Evaluation of individual and synchrony features from
tonic and phasic components
• Promising results using SVMs and feature selection (81%)
• Main benefits: automated and scalable • Biggest challenge: 31% of sessions had to be excluded
• Comparison with other modalities
• Goals: better quantify and understand behavior, and
detect developmental delays
Conclusions
27
![Page 28: Using Electrodermal Activity to Recognize Ease of ...web.media.mit.edu/~javierhr/files/14.Hernandez_et... · Using Electrodermal Activity to Recognize Ease of Engagement in Children](https://reader035.vdocuments.mx/reader035/viewer/2022070111/605062154472807c397dff94/html5/thumbnails/28.jpg)
[email protected] [email protected]
Javier Hernandez
Ivan Riobo
Agata Rozga
Gregory D. Abowd
Rosalind W. Picard
Using Electrodermal Activity to
Recognize Ease of Engagement in
Children during Social Interactions
Expeditions in Computing
Project: http://cbs.gatech.edu Dataset: http://cbi.gatech.edu/mmdb