1. Affective Computing Quantifying and Augmenting Peoples ExperiencesHyungil Ahn, Shani B Daily, Rana el Kaliouby, Micah EckhardtMIT Media Laboratory

2. Outline of Workshop

Roundtable introductions

The challenge of measuring customer experiences

Affective sensors and technologies + Demo

Consumer decision-making

Affect as index

3. Measuring Beverage Taste Preference (with Pepsi)

Evaluation method used today:

• Cleanse palate (crackers, water)

• Consume at least half of the beverage (food)

• Fill out questionnaire about liking/buying

• Other method: Focus Groups

Problem: These do not predict marketplace buying

Challenge: Construct an evaluation method and a computational model to accurately predict consumer preferences and marketplace buying behavior.

VS.

4. Self-Report

Most attempts to date that try to predict marketplace decisions are based on studies where people are asked what they would do

Self-report captures cognitive elements of liking or wanting (what you think you should say you like or want) but may or may not capture the actual feelings of liking or wanting.

Self-reported liking can be rated instantly (after any sip), but obtaining an accurate value for wanting or motivation may require a longer experience.

5. Why Technology to Tag Experiences?

Measuringin situExperiences: MIT Media Lab has over 100 industry sponsors -> unique opportunity to develop and test with real-world applications

Advertising Customer delight Customer preferences Learning Medical training - empathy Cognitive load measurement 6. Why Technology to Tag Experiences?

Measuring Experiences:

• Advertising, marketing, usability, Learning, Training, Customer relationship management

Augment Communication

• Advance social-emotional intelligence in machines to improve peoples experience with technology

• Enhance peoples ability to connect with others (autism spectrum disorders)

7. Physiology Sensing 8. Traditional SC sensing off the fingers, wired to a box Current version:wirelessly communicating SC off the wrist,Media Lab Galvactivator LED on the hand reflects SC (1999) Skin Conductance (SC) Sensors 9. Mindreader Platform 10. Automated Facial Analysis Affective + Cognitive States Concentrating Disagreeing Interested Thinking Unsure Absorbed Concentrating Vigilant Disapproving Discouraging Disinclined Asking Curious Impressed Interested Brooding Choosing Thinking Thoughtful Baffled Confused Undecided Unsure Agreeing Assertive Committed Persuaded Sure Happy Happy Delighted Enjoying Content 11. A Smile for Every Emotion!! 12. Mindreader Platform Feature point tracking (Nevenvision) Head pose estimation Facial feature extraction Head & facial action unit recognition Head & facial display recognition Mental state inference Hmm Let me think about this 13. Demo 14. Generalization Level 1 Train and test on Mind Reading DVD 15. Generalization Level 2 Train>Mind Reading DVD; Test>posed corpus

A challenge with machine learning + with interventions for autism

96 videos from CVPR 2004 tested on our system and on 18 people

Accuracy: 80% for best 11% of videos

Agreeing Disagreeing Confused Concentrating Thinking Interested 16. Generalization Level 2 Train>Mind Reading DVD; Test>posed corpus

96 videos from CVPR 2004 tested on our system and on 18 people

Accuracy: 80% for best 11% of videos

Accuracy of panel of 18 people Average = 54.5% Accuracy of computerAverage = 63.5% (better than 17 of 18 panelists) 17. Mindreader Platform Components MindReaderAPI Wrappers SDK (for developers) Application(for non-developers) Sample apps Tracker OpenCV nPlot External Libs/APIs MindreaderPlatform Downloadables 18. Features of the platform

Facial information accessible at multiple levels (action units, expressions and head gesture, affective and/or cognitive states)

Interface for training of new and existing mental states

Sensor support (skin conductance, motion, temperature)

Socket communication

Wrappers in python, java

LINUX (ubunto) version coming soon

Porting to camera phone, xo

19. To use the platform

Accessible to sponsors for download (the stand-alone application or SDK for developers)

Email: Rana el Kaliouby,[email_address] . mit . edu

20. ConsumerDecision -making 21. Consumer Decision-Making BehavioralMeasure (Number of Choices, Amount Consumed, etc.) CognitiveMeasure (Self-Reports,Focus Groups, etc.) AffectiveMeasure (Facial Valence, Skin Conductance,etc.) Decision -Making ComputationalModel 22.

Experienced utility: affective or hedonic experience, which can occur in a moment-based measure or memory-based measure.

Decision utility is inferred from observed behavioral choices.

Predicted utility is a belief about future experienced utility.

Moment utility is a measure of current affective or hedonic experience.

Total utility is derived from statistically aggregating a series of moment utilities.

Remembered utility is a single memory-based measure of affective or hedonic experience, which is based on retrospective assessments of episodes or periods of life.

23. Wanting vs. Liking

Rather, multiple valuation systems such as cognitive and affective

processing systematically influence human decision-making. Also, the neural substrates of liking (pleasure) are separate from those of wanting (motivation) in the human brain, so there is evidence from neuroscience that supports treating these concepts differently when modeling how people make decisions. Our model will separate these.

24. Experienced Utility

The memory-based approach accepts a persons retrospective evaluations of past episodes and situations as valid data. Theremembered utility of an episode of experienceis defined by retrospective global assessment of it the self-reported recollection ofliking, e.g., how much did you like the drink?

(ii) The moment-based approach derives the experienced utility of an episode, e.g., a

series of sips, from real-time measures of the pleasure and pain that the participant

experienced during that episode. Moment utility refers to the valence (good or bad) and

to the intensity (mild to extreme) of current affective or hedonic experience, e.g. the

experience of the current sip. Kahnemanstotal utilityof an episode is derived

exclusively and statistically from the record of moment utilities during that episode. We

measure moment utility by measuring emotional expressions elicited by the episode in

real-time.

People sometimes also attempt to forecast the affective or hedonic experience the

experienced utility that is associated with various life circumstances. These are called

predicted utilityor affective forecasting.

25. Experiment Setup 26. Experiment Setup Machine SelectionSip on Resulted BeverageAnswer Questions (each trial) 27. 28. AffectiveMeasure (Facial Valence, Skin Conductance,etc.) AnticipationDisappointment - Satisfaction Liking / Disliking 25 consumers, 30 trials, 30 min. videos! 29. BehavioralMeasure (Number of Choices, Amount Consumed, etc.) Choosing a Vending Machine on the Computer Screen 30. BehavioralMeasure (Number of Choices, Amount Consumed, etc.) Beverage Outcome and Sipping 31. CognitiveMeasure (Self-Reports,Focus Groups, etc.)? Asking Self-Reported Beverage Liking (asked every trial) 32. CognitiveMeasure (Self-Reports,Focus Groups, etc.)? Asking Machine Liking (asked every 5 trials) 33. Asking Expectation Comparison and Purchase Intent (asked every 5 trials) 34. Participants Men WomenTotal participants 17 22 Video analysis 15 19 Facially expressive 7 10 35. Analysis overview

Cognitive - Self-Report on Questionnaire:

• Cognitive wanting

• Cognitive liking

Behavioral

• Consumption

• Machine selection

Affective

• valence (positive or negative) from the facial expression

• • Affective outcome valence (satisfaction/disappointment, associated with affective wanting value)

• • Affective evaluation valence (liking/disliking, associated with affective liking value)

• arousal (calm or excited) from the skin conductance data

• • Affective anticipatory arousal (associated with risk and uncertainty, possibly valenced feelings of hope or dread)

• • Affective outcome arousal (associated with affective wanting value)

• • Affective evaluation arousal (associated with affective liking value)

36. Cognitive Analysis

Vanilla vs. SummerMix

Three different ways of looking questionnaire data:

• Ultimate after the 30 trials: 20:18

• 20:17

• 18:16.

People were fairly evenly split in preferring Vanilla or SummerMix, with only a slight preference for Vanilla

37. Behavioral Analysis

Consumption:

• an average of 15 sips of each beverage

• slightly more Vanilla (6.7 oz) than SummerMix (6.1 oz)

• Vanilla favorers reported lower purchase intent than those who preferred SummerMix

• suggests that the SummerMix favorers, while a slightly smaller group, were even more likely to buy the soda.

Machine Selection:

• a subtle bias (54.6%) toward the SummerMix machine

38. Cognitive + Behavioral Anslysis

SummerMix should have had nearly as good of a chance to succeed in the marketplace as Vanilla had.

Going to market looks like a reasonable decision.

But this turns out to be only part of the story.

39. Physiology Sensing 40. Traditional SC sensing off the fingers, wired to a box Current version:wirelessly communicating SC off the wrist,Media Lab Galvactivator LED on the hand reflects SC (1999) Skin Conductance (SC) Sensors 41. Patriots Touchdown Doritos Mouse Patriots 1st Down End Zone Overthrown Leading up to PatriotsTouchdown Super Bowl XLII 42. Sponsor Week Event:Randi vs. Raphael with live audience feedback Magician correctly identifies search term

• http://web.media.mit.edu/~sbdaily/AaI/AaIGroupVisualizationSkin.html#

43. 44. FacialAnalysis 45. Satisfaction DisappointmentLiking Disliking 46. (Obtained Outcome What She Wanted) Satisfaction DisappointmentLiking Disliking 47. Satisfaction DisappointmentLiking Disliking 48. (Obtained Outcome = What She Wanted) Satisfaction DisappointmentLiking Disliking 49. Satisfaction DisappointmentLiking Disliking 50. (Sipped Soda = What He Disliked) Satisfaction DisappointmentLiking Disliking 51. Satisfaction DisappointmentLiking Disliking 52. (Sipped Soda = What He Liked) Satisfaction Disappointment Liking Disliking 53. Facial Valence Analysis 54. Asymmetry

Vanilla favorers showed absolutely no positive expressions while tasting SummerMix, while nearly half of the SummerMix favorers showed something positive while tasting Vanilla.

The complete lack of any positive expressions in the Vanilla group may be a red flag.

If positive FVs were mapped into purchasing behavior, then one might expect slightly less than half the SummerMix favorers to buy both products, while no Vanilla favorers would buy SummerMix.

55. 56. Automated Facial Analysis Tracker Accuracy Average = 77.4%; highest = 96.87%; lowest = 23.98% 57. Automated Facial Analysis Sip Detection 58. Automated Facial Analysis Sip Detection 700 Sips, 82% of the sips detected 59. Automated Facial Analysis Real-time Analysis 60. Automated Facial Analysis Speedup over manual coding

3 minutes on average for each minute of video.

At least 2 or 3 coders are needed to establish validity of the coding, resulting in 6 to 9 minutes of coding per minute of video

A very labor-intensive and time-consuming approach

Each participant has about 30 minutes of video, so we can expect two coders to take over 180 minutes (3 hours) to code the 30 minutes of data.

61. Automated Facial Analysis Speedup over manual coding

With our automated sip detection algorithm, the algorithm can fast forward to sip events and can be made to look at outcome segments, the only two places where we coded for FVs in the analysis above.

If positioned to the right spots in the video, the coder only has to look at about 20 seconds of video per trial. Over 30 trials per participant, this is 600 seconds, or 10 minutes of video to code, instead of 30.

At 3 minutes of coding time per minute of video, plus occasional breaks, this results in ~30-40 minutes for each coder to code each participants experience, or ~60-80 minutes if we use two coders.

Combining our sip detection algorithm with human coding would cut the total manual coding time required (man hours) for sip events down by at least a factor of three.

62. Conclusion Next steps 63. Affect asIndex 64. Multi-person aggregate

X People are watching same video, or doing same task (+physiology) can we aggregate this data in real-time?

Advertising and TV/Cable programming

Web applications

65. Patriots Touchdown Doritos Mouse Patriots 1st Down End Zone Overthrown Leading up to PatriotsTouchdown Super Bowl XLII 66. Sponsor Week Event:Randi vs. Raphael with live audience feedback Magician correctly identifies search term

• http://web.media.mit.edu/~sbdaily/AaI/AaIGroupVisualizationSkin.html#

67. Shani to add more slides here? 68. ResearchRoadmap forQuantifyingExperiences 69. Two-person interaction

Measuring:

• Mirroring

• Synchrony

• Initiation

For service interactions (e.g. Bank of America)

For autism (e.g., monitoring real-time social interactions, parent-child interactions Baby Siblings project (UK), Playlamp

70. Mobile Phone Application

Port API to Googles Android platform (Java)

Develop some applications to work on mobiles

HTC Android phones coming out Dec 08

71. Mobile Phone Application

Port API to Googles Android platform (Java)

Develop some applications to work on mobiles

HTC Android phones coming out Dec 08

72. Sociable Robots 73. Acknowledgements http://affect.media.mit.edu [email_address] MIT Media Lab Affective Computing GroupComputer Science, American University in Cairo Abdel Rahman Nasser, Youssef Abdallah. Mina Mikhail, Tarek HefniSponsors:National Science Foundation Nancy Lurie Marks Family Foundation MIT Media Lab TTT Consortium Seagate, Google, Robeez