![Page 1: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/1.jpg)
Multi‐Modal Affective Data Analytics
Mykola Pechenizkiy
SDAD 2012 @ ECMLPKDD2012 2 September 2012
Bristol, UK
http://www.win.tue.nl/stressatwork
![Page 2: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/2.jpg)
Affective data Social media
• Social media leads to masses of affective data related to peoples’ emotions, sentiments and opinions
• In the recent past was used mainly for marketing needs– Web analytics Social Media
• Whatever the incentive was to study this, sentiment classification has become much more accurate
3
![Page 3: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/3.jpg)
Multilingual Sentiment Classification
4
![Page 4: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/4.jpg)
Rule‐based polarity detectionRule‐based emission model: 8 kinds of rules:
Emission 5
![Page 5: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/5.jpg)
SentiCorrHow much positive and negative content do we read or write?
6
![Page 6: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/6.jpg)
Mobile SentiCorr AppWhat a fantastic idea, now ifThis app is esigned to make someone else (or a computer) read our e‐mails for us and “protect” us from WHAT??? How lazy can we get? Like someone commented on CNN reactions, if we are getting upset by tones and/or scoldings in e‐mails, we certainly have bigger issues that need to be dealt with. C’mon, guys, go invent something useful. Not to mention, does it detect irony? Will it weed out the liars?Pleeeeezzzeeee, what a WASTE OF SOMEONE’S COLLEGIATE TIME AND ENERGY. Don’t we have houses to clean and poor people to feed and old folks to help with their shopping? Go do something useful with your time, inventors of his app!!!
Great idea! Get it on iOSsoon (anonymous)“Stress is often made worse by the anticipation of an unpleasant event and actually dissipated once you tackle the problem directly”
Pamela BriggsBritish Psychological Society
7
![Page 7: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/7.jpg)
OLAP Style Exploration of Data Summaries
8
![Page 8: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/8.jpg)
Exploration of Individual Cases, e.g. e‐Mails
9
![Page 9: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/9.jpg)
Sentiment vs. Fact Classification
News in media or business are considered to be sentiment neutral,– but they often contain positive or negative information, e.g.
“You will be fired in 3 months because of the serious budget cuts.”
– no sentiment, but negative informationSimilarly, in work‐related correspondence there could be stressing information:– How can we identify it?
10
![Page 10: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/10.jpg)
Sentiment discovery: State‐of‐the‐art• Sentiment analysis/classification is mature!
– Commercial products, free services, open‐source, variety of apps, evolves in many directions
• Several great overviews:– Sentiment Analysis in Practice ‐ ICDM2011 tutorial by Tiger Zhang (eBay Research Labs)
• http://web.cs.dal.ca/~yongzhen/publication/paper/ICDM2011_SentimentAnalysisInPracticeTutorial.pdf
– Modeling Opinions and Beyond in Social Media by Bing Liu (UIC)
• http://kdd2012.sigkdd.org/sites/images/summerschool/Bing‐Liu.pptx
![Page 11: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/11.jpg)
Outline
• Framework for Stress Analytics:–Data management, OLAP support– Shape‐based Query‐by‐Example
• Stress detection from speech and GSR–Predictive features and classification– From controlled experiments to real life
![Page 12: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/12.jpg)
What is stress? Is it a bad thing?
![Page 13: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/13.jpg)
Stress in NL according to Coosto.nl
Not really job related
![Page 14: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/14.jpg)
Impact of Stress at WorkWHO: by 2020 Top 5 diseases will be stress related. USA: health care expenditures are ~50% greater for workers who report high levels of stress at work (J. Occup. Env. Med, 40:843‐854).
the Netherlands: (TNO, 2006; TU/e Cursor 2012):• The direct costs of stress are 4 billion Euro per year. • Every year 150.000 300.000 employees become ill because of stress at work.
• 1/7 disabled because of stress at work.• In TU Delft, 53% of surveyed students indicated that they experienced huge stress during their studies.
15
![Page 15: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/15.jpg)
What do organizations try (not) to do?
Discuss psychological load (28%)
Change work processes (17%)
Source: (TNO, dossier Werkdruk)
Extend regulations (9%)
Reduce workload (33%)
Improve managers’ skills (13%)
Improve work/life balance (14%)
![Page 16: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/16.jpg)
What can go wrong?• They are not always aware of the problem ordon’t know the exact cause
• People do not always want to share what they experience with others
• Not always timely enough• Expensive to organize meeting with psychologists, interventions
• The individual causes are different and notalways well understood
• Giving practical advises is not trivial
![Page 17: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/17.jpg)
Types of Stress and StressorsDifferent types of stress:• Survival stress – a response to a physical danger• Environmental stress ‐ noise, crowding, pressure from work or family
• Internal stress ‐ worrying about things we can't control; putting ourselves in situations we know will cause us stress (addicted to stress – expanding todolist with more and more conference deadlines)
• Fatigue and overwork ‐ in a long term perspective
Stress affects both body and mind
18
![Page 18: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/18.jpg)
Types of Stress and StressorsThree kinds of stress:• Acute: caused by an acute short‐term stress factor.• Episodic acute: occurs more frequently & periodically.• Chronic: caused by long‐term stress factors ‐ harmful.
Factors causing stress@work:• long work hours, work overload, time pressure, difficult, demanding or complex tasks, high responsibility, lack of breaks, lack of training
• conflicts, underpromotion, job insecurity, lack of variety, and poor physical work conditions (limited space, temperature and lighting conditions)
19
![Page 19: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/19.jpg)
Concept
20
Be‐eep!Be‐eep!
![Page 20: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/20.jpg)
StressAnalyticsMake people aware of their stress and stressors
Overview of stressors Exploration of relations
Access to evidence, i.e. annotated, measured stressEmpowerment by awareness (+ implicit/explicit advice)
![Page 21: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/21.jpg)
Our approach to StressAnalytics
What, When, Where, with Whom
Physiological signs
OLAP cube
Pattern Mining
22
![Page 22: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/22.jpg)
Our approach to Stress Analytics• Make a person aware of what is happening
– how they spend their time and when and from where the stress comes in
• Provide valuable input for pattern mining/knowledge discovery– Much richer data sources
• Visual analytics– Interactive exploration of stress‐related data– Collecting subjective data/labels from a person through the interaction
23
![Page 23: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/23.jpg)
GSR, temp., voice, heart rate, facial expressions
Physiological signsrelated data
External user‐related data
KPI, E‐mail, calendar, social media, news
environmentExternal
environment
Zoom in&out, slice&diceOLAP
Zoom in&out, slice&dice
Pattern mining, prediction, query‐by‐exampleData Mining
Feature extraction, peak/change detection, classification
Raw data, objective evidence
GUIExploration, Interaction, Visual Analytics
temperature, lighting, noise, airconditioning
![Page 24: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/24.jpg)
Evidence: physiol. signals & external sourcesGSR, Temperature, Speech, Facial expressions, Sentiment in text
![Page 25: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/25.jpg)
Alignment of Information Sources
• What person reads and writes: SentiCorr• What person does in general according to agenda• Environment context (lighting, noise, temp etc.)• Annotate data from video, sound, text processing, and vital signs
• …• What person does with the computer
– http://wakoopa.com/
Different aspect with pre‐processing, storing, managing
26
![Page 26: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/26.jpg)
Stress Data Cube/OLAPQuick data summaries wrt predefined dimensions
27
![Page 27: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/27.jpg)
Stress Analytics Visualization• OLAP‐style exploration: selecting multi‐dimension, zoom‐in, zoom‐out.
• Navigating to the evidences: i.e. raw data:– GSR, skin temperature, speech, and email
• Shape‐based time‐series similarity search–State‐of‐the‐art UCR‐Suite (Keogh et al.)
• Demo:http://www.win.tue.nl:8080/saw_analytics/stress_v
isualization.jsp
![Page 28: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/28.jpg)
OLAP system, a Star Schema
![Page 29: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/29.jpg)
Shape‐Based Query‐by‐Example
• Find a similar shape time‐series with s
• Given a subsequence of GSR time series s
Query
Result
![Page 30: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/30.jpg)
Shape‐based QBE• Euclidean Distance:
• Dynamic Time Warping (DTW)
State‐of‐the‐art UCR‐Suite (Keogh et al.)
![Page 31: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/31.jpg)
How to measure stress
Determine stress levelbased on observed sweat production
32
![Page 32: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/32.jpg)
Detection and Categorization of Stress
Based on GSR data alone ‐ not as easy as the following figure may suggest:
33
![Page 33: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/33.jpg)
Challenges in Stress Detection• All kinds of noise, e.g. loosing contact with the skin
• Activity (exercising) , environment (cold/hot) context and personal differences may impact GSR we observe
34
![Page 34: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/34.jpg)
Interpretation isn’t straightforward
35
![Page 35: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/35.jpg)
Detection as Classification
• Total number of GSR response.• The sum of GSR amplitude.• The sum of rising time response.• The sum of energy response.
• Mean, SD, min and max of GSR.• Mean, min and max of peak height.
GSR features
![Page 36: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/36.jpg)
Adding more data to disambiguate
• Skin and room temperature, noise, accelerometer, voice, face, …
37
![Page 37: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/37.jpg)
e.g. activity recognition can helpWriting vs. typing vs. walking vs. teaching vs …
Analyzing accelerometer data only (wrist band) 38
![Page 38: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/38.jpg)
Uncontrolled and semi‐controlled
• Philips Research employees wearing the device during their working hours
• Students passing the written and multiple choice exams
• Students presenting demos/posters with course project results
• More to come via HumanCapitalCare
39
![Page 39: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/39.jpg)
Experiment demo
![Page 40: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/40.jpg)
41
![Page 41: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/41.jpg)
Measuring GSR in (un)controlled settings
• Philips prototype
• Self‐made, the LEGO Mindstorms NXT
42
![Page 42: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/42.jpg)
Multi‐Source Affective Data Classification
Stress/Emotion classification from text, GSR & speech
GSR & other sensors
Facial expression analysis
43
![Page 43: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/43.jpg)
Automatic Stress Detectionfeature enrichment ensemble learning
speech GSR
speechfeatures
GSRfeatures
combinefeatures
classification
speech GSR
speechfeatures
GSRfeatures
ensemble
classificationclassification
speech
speechfeatures
classification
GSR
GSRfeatures
classification
speech model GSR model
![Page 44: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/44.jpg)
Stress and Skin Conductance
StressChanges in Autonomic Nervous System (ANS)
activation of sweat glands
Changes of the amount of the produced sweatChanges of skin conductance
• Relax skin is drier skin conductance is lower
• Stress sweat increasesskin conductance is higher
![Page 45: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/45.jpg)
GSR features
• Total number of GSR response.• The sum of GSR amplitude.• The sum of rising time response.• The sum of energy response.
• Mean, SD, min and max of GSR.• Mean, min and max of peak height.
![Page 46: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/46.jpg)
Change detection approachOnline settings
![Page 47: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/47.jpg)
Preprocessing steps
50
![Page 48: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/48.jpg)
Stress and Speech
Stress Respiration Rateincreases
Increased subglottal pressure
Increased Pitch
Voice is a good indicator of stress [scherer, 1986]
![Page 49: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/49.jpg)
Speech Features• Voiced and unvoiced speech
![Page 50: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/50.jpg)
Speech Features• Pitch / Fundamental frequency
![Page 51: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/51.jpg)
Speech FeaturesMel Frequency Cepstral Coefficients (MFCCs)are coefficients that approximate human perception auditory response.
Audio(temporal)
frequency filtered frequency
log frequency
FFT Mel scale filter
logs power
DCT representation
DCTStore the first coefficientsMFCCs
![Page 52: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/52.jpg)
Classification Methods
• Support Vector Machine (SVM) – State of the arts.
• Decision Tree classifier.• K‐means using Vector Quantization (VQ). This method is chosen as a baseline.
• Gaussian Mixture Model (GMM). This method works well for speaker recognition task.
• Change detectors: ADWIN, thresholding
![Page 53: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/53.jpg)
Stress Dataset
• Three types of GSR patterns.
• First type:• Second type:• Third type:
![Page 54: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/54.jpg)
Aligning of data sources
Instance 1 Instance 2 Instance 3
Instance 1 Instance 2 Instance 3
60 seconds
GSR
speech
![Page 55: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/55.jpg)
Stress Dataset: Speech Features
![Page 56: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/56.jpg)
Stress Model using GSR features
• SVM outperformed other methods.• Recognizing light vs heavy workload is harder than between recovery vs heavy workload.
46.12
55.54 53.21
70.5174.9
66.82
79.66 80.72
70.673.45
77.81
62.52
0
10
20
30
40
50
60
70
80
90
Recovery vs workloads Recovery vs heavy workload Light vs heavy workload
Accuracy (p
ercent)
k‐means
GMM
SVM
Decision Tree
10-times 10-fold CV (not subject independent)
![Page 57: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/57.jpg)
Stress Model using speech features
• SVM outperforms the other classifier.• K‐means and GMM do not perform well for speech.• MFCC is a good indicator for stress detection.
49.6555.39
49.17 50.6
58.82 56.78 59.08
52.3
62.08
92.39 92.56 91.69
55.6
68.86 70.69 71.47
0
10
20
30
40
50
60
70
80
90
100
Pitch MFCC MFCC‐Pitch RASTA
Accuracy (p
ercent)
k‐means
GMM
SVM
Decision Tree
![Page 58: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/58.jpg)
1‐subject‐leave‐out cross‐validation(subject independent model)
79.66 80.72
70.674.84 75
63.04
0
10
20
30
40
50
60
70
80
90
Recovery vs workloads Recovery vs heavy workload Light vs heavy workload
Accuracy (p
ercent)
GSR Tasks
10‐times 10‐fold CV 1‐Subject‐Leave‐Out CV
62.08
92.39 92.56 91.69
53.04
67.82 70 72.17
0
10
20
30
40
50
60
70
80
90
100
Pitch MFCC MFCC‐Pitch RASTA PLP
Accuracy (p
ercent)
Speech Features
10‐times 10‐fold CV 1‐subject‐leave‐out CV
It is better to address the problem of stress detection using a subject dependent model
![Page 59: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/59.jpg)
Fusion ApproachesFeature
enrichmentEnsemble learning
![Page 60: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/60.jpg)
Fusion of GSR and Speech90.73 91.34
69.04
92.43 92.47
70.17
0
10
20
30
40
50
60
70
80
90
100
MFCC and GSR MFCC‐Pitch and GSR Pitch and GSR
Accuracy (p
ercent)
Enriching Feature Space Logistic Regression as MetaLearner
Light vs. heavy workload, balanced data
![Page 61: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/61.jpg)
Kappa Agreement for Classifiers
• Measure agreement between two model using Cohen’s Kappa test.
• Kappa = 1 complete agreement.• Kappa = 0 complete disagreement.
![Page 62: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/62.jpg)
Stress detection summary• Speech is more reliable (in lab settings) than GSR, but more subject dependent.
• SVM is performing better on both GSR and Speech signal.
• ADWIN & thresholding detectors do well on GSR
• Combining GSR and Speech is not trivial:– Speech and GSR predictions are highly independent (low kappa value)
– This diversity may be exploited with dynamic integrations methods
![Page 63: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/63.jpg)
Further directions• Extend the notion of stress (positive and negative) in the stress analytics framework.
• Stress analytics affective data analytics• Collect more data to enable OLAP KDD part of the framework.
• Combine with other signals, such as facial expression, heart rate, nutrition.
• Long path from lab setting to real‐life situation; but both are needed.
![Page 64: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/64.jpg)
Is Acute Stress Good or Bad?
69
![Page 65: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/65.jpg)
What is the Relaxation Then?
70
![Page 66: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/66.jpg)
Is “Normal” Condition Good or Bad?
What if someone’s patterns looks like NNNNNNNNNNNNNNNN ……
71
![Page 67: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/67.jpg)
Summary• The fun parts come from
– The fact that not much is known about stress–Playing with heterogeneous/multi‐modal data–Multi‐disciplinary (data collection, data management, data mining, visual analytics)
– Engineering approach to data mining
• How to show the utility – i.e. what we do–helps to understand better stress as a phenomenon, and the stressors, and how to
–helps people at the end 72
![Page 68: Multi Modal Affective Data Analytics - win.tue.nlmpechen/talks/sdad2012_invited.pdf · Stress/Emotion classification from ... • K‐means using Vector Quantization (VQ). This method](https://reader033.vdocuments.mx/reader033/viewer/2022042319/5f082f207e708231d420c1cb/html5/thumbnails/68.jpg)
Take home messages• Lab settings vs. real world• Availability and quality of the signal
– Voice recorded– Someone’s else voice recorded– Noise and missing data, uncertainty– A person cannot speak (during the meeting while someone else is speaking)
• Ground truth, labels, subjective vs. objective• A large problem space
– If you know how to help us with any part on StressAnalytics – talk to me 73