Happiness Recognitionfrom Mobile Phone Data

Andrey Bogomolov1 Bruno Lepri2 Fabio Pianesi2

1University of Trento,Via Sommarive, 5

I-38123 Povo - Trento, Italy

2Fondazione Bruno KesslerVia Sommarive, 18

I-38050 Povo - Trento, Italy

EmoPAR group meeting 2013-JUN-19, Trento, Italy.

1 / 49

Outline

IntroductionProblem StatementSource Data

Recognition ModelBasic FeaturesFinal Feature Space

Results

Limitations

Summary

2 / 49

Happiness as an emotional state – why is it important?

Your ideas?. . .

3 / 49

General Problem Statement

Inputs

I Pervasive technology data.I Multimodal data.

Outputs

I Emotion recognition.I Mood recognition.I Personality recognition.

4 / 49

Our Problem Statement

Inputs

I Smartphone call log.I Smartphone sms log.I Smartphone Bluetooth proximity hits.

Outputs

I Daily happiness recognition.I 3-classes: {not happy, neutral, happy}.

5 / 49

Data Collection

I 117 subjectsI dates: 21 February, 2010 – 16 July, 2011

Source Space Dataset: Living Laboratoryphone calls 33497

sms 22587Bluetooth hits 1460939

6 / 49

Happiness Data

Recorded Happiness Scores Density

0.0

0.2

0.4

0.6

0.8

0 2 4 6Score

Den

sity

7 / 49

Happiness Data

Descriptive Statisticsnumber of records 12991

mean 4.84standard deviation 1.26

median 5.00mean average deviation 1.48

min 1.00max 7.00

range 6.00skew -0.39

kurtosis -0.07

8 / 49

Happiness Data

Within-person and between-subject variance

0 1 2 3 4 5 6

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

density.default(x = r1[, 1])

Variance

Den

sity

Within−Person VarianceBetween−Person Variance

9 / 49

Feature Space

Basic Features

I General Phone UsageI DiversityI Active BehaviorsI Regularity

10 / 49

Feature Space

Basic Features: General Phone Usage1. Total Number of Calls (Outgoing+Incoming)2. Total Number of Incoming Calls3. Total Number of Outgoing Calls4. Total Number of Missed Calls5. Number of SMS received6. Number of SMS sent

11 / 49

Feature Space

Basic Features: Diversity7. Number of Unique Contacts Called8. Number of Unique Contacts who Called9. Number of Unique Contacts Communicated with (Incoming+Outgoing)10. Number of Unique Contacts Associated with Missed Calls11. Entropy of Call Contacts12. Call Contacts to Interactions Ratio13. Number of Unique Contacts SMS received from14. Number of Unique Contacts SMS sent to15. Entropy of SMS Contacts16. Sms Contacts to Interactions Ratio

12 / 49

Feature Space

Basic Features: Active Behaviors17. Percent Call During the Night18. Percent Call Initiated19. Sms response rate20. Sms response latency21. Percent SMS Initiated

13 / 49

Feature Space

Basic Features: Regularity22. Average Inter-event Time for Calls (time elapsed between two events)23. Average Inter-event Time for SMS (time elapsed between two events)24. Variance Inter-event Time for Calls (time elapsed between two events)25. Variance Inter-event Time for SMS (time elapsed between two events)

14 / 49

Feature Space

Proximity FeaturesGeneral Bluetooth Proximity1. Number of Bluetooth IDs2. Times most common Bluetooth ID is seen3. Bluetooth IDs accounting for n% of IDs seen4. Bluetooth IDs seen for more than k time slots5. Time interval for which a Bluetooth ID is seen6. Entropy of Bluetooth contactsDiversity7. Contacts to interactions ratioRegularity8. Average Bluetooth interactions inter-event time(time elapsed between two events)9. Variance of the Bluetooth interactions inter-event time(time elapsed between two events)

15 / 49

Feature Space Innovation

“Background Noise” Features

I a) peoples activity, as detected through their smartphonesI b) the weather conditions {humidity, wind speed, pressure,

total precipitation and visibility}I c) personality traits {“Big Five"}

Functional Innovation

I a) time domain – sliding window functionsI b) Miller-Madow correction for entropy calculation

HMM(θ) ≡ −p∑

i=1

θML,i log θML,i +m − 1

2N

16 / 49

Top-30 Features

-1 0 1 MeanDecreaseAccuracy MeanDecreaseGinimeanTemperature 0.0099 0.0033 0.0094 0.0082 154.8379

humidity 0.0047 -0.0033 0.0115 0.0074 149.7668pressure -0.0002 0.0008 0.0015 0.0011 149.0302

windSpeed 0.0028 0.0017 0.0051 0.0040 142.7727visibility 0.0024 -0.0009 0.0051 0.0034 120.8683

neuroticism 0.0690 0.0288 0.0399 0.0419 90.5721conscientiousness 0.0659 0.0480 0.0668 0.0627 90.2708

extraversion 0.0511 0.0357 0.0472 0.0454 76.9467openness 0.0656 0.0340 0.0406 0.0429 73.9181

totalPrecipitation 0.0007 -0.0000 0.0012 0.0009 73.5273agreeableness 0.0536 0.0282 0.0235 0.0289 70.7261

bluetoothQ95TimeForWhichIdSeen 0.0233 0.0120 0.0149 0.0155 22.4018bluetoothQ90TimeForWhichIdSeen 0.0161 0.0082 0.0141 0.0131 19.8364

smsRepliedEventsLatencyMedian 0.0093 0.0085 0.0096 0.0093 18.7719bluetoothIdsMoreThan04TimeSlotsSeen 0.0114 0.0066 0.0124 0.0110 15.6738

bluetoothMaxTimeForWhichIdSeen 0.0141 0.0072 0.0095 0.0097 15.2546bluetoothTotalEntropyMillerMadow 0.0058 0.0017 0.0035 0.0035 13.9000

bluetoothTotalEntropyShannon 0.0065 0.0009 0.0060 0.0050 13.1826callMeanInterEventTimePerDay -0.0003 -0.0000 0.0014 0.0009 13.1180

incomingAndOutgoingCallsPerDay -0.0000 -0.0002 0.0024 0.0015 12.3962bluetoothQ50TimeForWhichIdSeen 0.0104 0.0104 0.0091 0.0095 12.2270

callStandardDeviationInterEventTimePerDay 0.0004 -0.0005 0.0007 0.0004 10.1723bluetoothIdsMoreThan19TimeSlotsSeen 0.0087 0.0033 0.0089 0.0077 9.7388

incomingCallsPerDay 0.0003 -0.0005 0.0009 0.0005 9.6572outgoingContactsToInteractionsRatioPerDay 0.0005 -0.0004 0.0013 0.0009 9.2016

callsInitiatedRatioPerDay -0.0001 -0.0001 0.0014 0.0008 9.0245entropyMillerMadowCallsOutgoingWindow3Days 0.0001 -0.0008 0.0014 0.0008 8.7199

bluetoothIdsMoreThan09TimeSlotsSeen 0.0074 0.0046 0.0040 0.0046 8.6006bluetoothQ75TimeForWhichIdSeen 0.0027 0.0018 0.0032 0.0028 8.4454

outgoingCallsPerDay -0.0001 -0.0001 0.0012 0.0007 8.3368

17 / 49

Top-20 Features Correlation

agreeableness

bluetoothIdsMoreThan04TimeSlotsSeen

bluetoothMaxTimeForWhichIdSeen

bluetoothQ90TimeForWhichIdSeen

bluetoothQ95TimeForWhichIdSeen

bluetoothTotalEntropyMillerMadow

bluetoothTotalEntropyShannon

callMeanInterEventTimePerDay

conscientiousness

extraversion

humidity

incomingAndOutgoingCallsPerDay

meanTemperature

neuroticism

openness

pressure

smsRepliedEventsLatencyMedian

totalPrecipitation

visibility

windSpeed

agreeablenessbluetoothIdsMoreThan04TimeSlotsSeenbluetoothMaxTimeForWhichIdSeenbluetoothQ90TimeForWhichIdSeenbluetoothQ95TimeForWhichIdSeenbluetoothTotalEntropyMillerMadowbluetoothTotalEntropyShannoncallMeanInterEventTimePerDayconscientiousnessextraversionhumidityincomingAndOutgoingCallsPerDaymeanTemperatureneuroticismopennesspressuresmsRepliedEventsLatencyMediantotalPrecipitationvisibilitywindSpeedVar1

Var2

0.0

0.5

1.0value

18 / 49

Results

Final Classifier Performance Metrics ComparisonTraining set Test set

Accuracy 0.8081 0.8036Kappa 0.5879 0.5743

AccuracyLower 0.8004 0.7878AccuracyUpper 0.8156 0.8187

AccuracyNull 0.6415 0.6419AccuracyPValue 2.139e-303 8.826e-73McnemarPValue 5.647e-208 1.738e-57

19 / 49

Results

Final Classifier Confusion Matrix for Training Set-1 0 1

-1 782 119 750 153 1170 1451 600 903 6448

Final Classifier Confusion Matrix for Test Set-1 0 1

-1 197 30 140 34 274 371 152 243 1616

20 / 49

Results

What We Learnt: Final Model ROC curve

Specificity

Sen

sitiv

ity

0.0

0.2

0.4

0.6

0.8

1.0

1.0 0.8 0.6 0.4 0.2 0.0

AUC: 0.844

21 / 49

Limitations

Data

I Data loss not registered as NA’sI BatteryI Temporal resolution

Model

I Requires personality dataI Requires 1 week data collection periodI Not tested on diverse cultural groups

22 / 49

Summary

I Automatic recognition of people’s daily happiness frommobile phone data is feasible.

I Accuracy is approaching the results of multi-modalobtrusive methods.

I Future work should be focused on multi-step recognitionmodel development.

23 / 49

Thank you!{andrey.bogomolov@unitn.it}

24 / 49

References I

Aharony, Nadav, Wei Pan, Cory Ip, Inas Khayal, andAlex Pentland (2011). “Social fMRI: Investigating and shapingsocial mechanisms in the real world”. In: Pervasive and MobileComputing 7.6, pp. 643–659.Anguita, Davide, Alessandro Ghio, Luca Oneto, Xavier Parra,and Jorge L. Reyes-Ortiz (2012). “Human Activity Recognitionon Smartphones Using a Multiclass Hardware-Friendly SupportVector Machine”. In: IWAAL, pp. 216–223.Bickmore, Timothy W., Rosalind, and W. Picard (2005).“Establishing and Maintaining Long-Term Human-ComputerRelationships”. In: ACM Transactions on Computer HumanInteraction 12, pp. 293–327.Bishop, C. M. (1995). “Training with Noise is Equivalent toTikhonov Regularization”. In: Neural Computation 7.1,pp. 108–116.

25 / 49

References II

Both, Fiemke, Mark Hoogendoorn, Michel C. A. Klein, andJan Treur (2010). “Computational modeling and analysis of therole of physical activity in mood regulation and depression”. In:Proceedings of the 17th international conference on Neuralinformation processing: theory and algorithms - Volume Part I.ICONIP’10. Sydney, Australia: Springer-Verlag, pp. 270–281.Brave, Scott and Clifford Nass (2003). “The human-computerinteraction handbook”. In: ed. by Julie A. Jacko andAndrew Sears. Hillsdale, NJ, USA: L. Erlbaum Associates Inc.Chap. Emotion in human-computer interaction, pp. 81–96.Breiman, Leo (Oct. 2001). “Random Forests”. In: Mach. Learn.45.1, pp. 5–32.Broek, Egon L. (Jan. 2013). “Ubiquitous emotion-awarecomputing”. In: Personal Ubiquitous Comput. 17.1, pp. 53–67.Buhmann, Martin D. and M. D. Buhmann (2003). Radial BasisFunctions. New York, NY, USA: Cambridge University Press.

26 / 49

References III

Busso, Carlos, Zhigang Deng, Serdar Yildirim, Murtaza Bulut,Chul Min Lee, Abe Kazemzadeh, Sungbok Lee,Ulrich Neumann, and Shrikanth Narayanan (2004). “Analysis ofemotion recognition using facial expressions, speech andmultimodal information”. In: in Sixth International Conferenceon Multimodal Interfaces ICMI 2004. ACM Press, pp. 205–211.Campbell, Angus, Philip E. Converse, and Willard L Rodgers(1976). The quality of American life : perceptions, evaluations,and satisfactions / Angus Campbell, Philip E. Converse, WillardL. Rodgers. English. Russell Sage Foundation New York, xi,583 p. :Carlson, Charles R., Frank P. Gantz, and John C. Masters(1983). “Adults’ emotional states and recognition of emotion inyoung children”. English. In: Motivation and Emotion 7.1,pp. 81–101.

27 / 49

References IV

Carver, Charles S. and Michael F. Scheier (Jan. 1990). “Originsand Functions of Positive and Negative Affect: AControl-Process View”. In: Psychological Review 97.1,pp. 19–35.Chittaranjan, Gokul, Jan Blom, and Daniel Gatica-Perez (Mar.2013). “Mining large-scale smartphone data for personalitystudies”. In: Personal Ubiquitous Comput. 17.3, pp. 433–450.Cohen, Jacob (Apr. 1960). “A Coefficient of Agreement forNominal Scales”. In: Educational and PsychologicalMeasurement 20.1, pp. 37–46.Cohn, Jeffrey F. (2007). “Foundations of human computing:facial expression and emotion”. In: Proceedings of the ICMI2006 and IJCAI 2007 international conference on Artificalintelligence for human computing. ICMI’06/IJCAI’07. Banff,Canada: Springer-Verlag, pp. 1–16.

28 / 49

References VCosta, P T and R R McCrae (1992). Revised NEO personalityinventory (NEO PI-R) and neo five-factor inventory (NEO-FFI).Vol. 0. October. Psychological Assessment Resources,pp. 17–20.— (1995). “Domains and facets: hierarchical personalityassessment using the revised NEO personality inventory.” In:Journal of Personality Assessment 64.1, pp. 21–50.Costa, Paul T. and Robert R. McCrae (1980). “Influence ofextraversion and neuroticism on subjective well-being: Happyand unhappy people”. In: Journal of Personality and SocialPsychology 38 (4), pp. 668–678.Cowie, R., E. Douglas-Cowie, K. Karpouzis, G. Caridakis,M. Wallace, and S. Kollias (2008). “Recognition of EmotionalStates in Natural Human-Computer Interaction”. English. In:Multimodal User Interfaces. Ed. by Dimitrios Tzovaras. Signalsand Commmunication Technologies. Springer BerlinHeidelberg, pp. 119–153.

29 / 49

References VICowie, Roddy, Ellen Douglas-Cowie, Nicolas Tsapatsoulis,George Votsis, Stefanos Kollias, Winfried Fellenz, andJohn G Taylor (2001). “Emotion recognition in human-computerinteraction”. In: Signal Processing Magazine, IEEE 18.1,pp. 32–80.DeNeve, K.M. and H. Cooper (1998). “The happy personality: ameta-analysis of 137 personality traits and subjectivewell-being.” In: Psychol Bull 124.2, pp. 197–229.Denissen, Jaap J. A., Ligaya Butalid, Lars Penke, and MarcelA. G. Van Aken (2008). “The effects of weather on daily mood:A multilevel approach.” In: Emotion Researcher 8.5,pp. 662–667.Diener, Ed (1994). “Assessing subjective well-being: Progressand opportunities”. English. In: Social Indicators Research31.2, pp. 103–157.Diener, Ed and Martin E.P. Seligman (2002). “Very HappyPeople”. In: Psychological Science 13.1, pp. 81–84.

30 / 49

References VII

Dolan, R. J. (Nov. 2002). “Emotion, cognition, and behavior.”In: Science (New York, N.Y.) 298.5596.Dong, Wen, Bruno Lepri, and Alex Pentland (2011). “Modelingthe co-evolution of behaviors and social relationships usingmobile phone data”. In: MUM, pp. 134–143.Eagle, Nathan and Alex Pentland (2006). “Reality mining:sensing complex social systems”. In: Personal and UbiquitousComputing 10.4, pp. 255–268.Eagle, Nathan, Alex (Sandy) Pentland, and David Lazer (2009).“Inferring friendship network structure by using mobile phonedata”. In: Proceedings of the National Academy of Sciences106.36, pp. 15274–15278.Feldman, Lisa A (1995). “Valence focus and arousal focus:Individual differences in the structure of affective experience”.In: Journal of personality and social psychology 69,pp. 153–153.

31 / 49

References VIIIFUNF Open Sensing Framework. (Access date: 12-JUN-2013).http://www.funf.org.Garcia, Salvador and Francisco Herrera (Dec. 2008). “AnExtension on "Statistical Comparisons of Classifiers overMultiple Data Sets" for all Pairwise Comparisons”. In: Journalof Machine Learning Research 9, pp. 2677–2694.Gray, J. A. (1991). “Neural systems, emotion, and personality.”English. In: Neurobiology of learning, emotion, and affect /editor, John Madden IV. Raven Press New York, pp. 273–306.Greenberg, Ariel M., William G. Kennedy, and Nathan Bos,eds. (2013). Social Computing, Behavioral-Cultural Modelingand Prediction - 6th International Conference, SBP 2013,Washington, DC, USA, April 2-5, 2013. Proceedings.Vol. 7812. Lecture Notes in Computer Science. Springer.Gunes, H. and M. Pantic (2010). “Automatic, Dimensional andContinuous Emotion Recognition”. In: Int’l Journal of SyntheticEmotion 1.1, pp. 68–99.

32 / 49

References IX

Gunes, H., B. Schuller, M. Pantic, and R. Cowie (2011).“Emotion Representation, Analysis and Synthesis inContinuous Space: A Survey”. In: Proceedings of IEEEInternational Conference on Automatic Face and GestureRecognition (FG’11), EmoSPACE 2011 - 1st InternationalWorkshop on Emotion Synthesis, rePresentation, and Analysisin Continuous spacE. Santa Barbara, CA, USA, pp. 827–834.Hand, DavidJ. and RobertJ. Till (2001). “A SimpleGeneralisation of the Area Under the ROC Curve for MultipleClass Classification Problems”. English. In: Machine Learning45 (2), pp. 171–186.Hernandez, Javier, Mohammed E. Hoque, andRosalind W. Picard (2012). “Mood Meter: large-scale andlong-term smile monitoring system”. In: ACM SIGGRAPH 2012Emerging Technologies. SIGGRAPH ’12. Los Angeles,California: ACM, 15:1–15:1.

33 / 49

References XJohn, Oliver P. and Sanjay Srivastava (1999). “The Big Five traittaxonomy: History, measurement, and theoreticalperspectives”. In: Handbook of Personality: Theory andResearch. Ed. by Lawrence A. Pervin and Oliver P. John.Second. New York: Guilford Press, pp. 102–138.Kapoor, Ashish, Winslow Burleson, and Rosalind W. Picard(Aug. 2007). “Automatic prediction of frustration”. In: Int. J.Hum.-Comput. Stud. 65.8, pp. 724–736.Kapoor, Ashish and Rosalind W. Picard (2005). “Multimodalaffect recognition in learning environments”. In: Proceedings ofthe 13th annual ACM international conference on Multimedia.MULTIMEDIA ’05. Hilton, Singapore: ACM, pp. 677–682.Keller, Matthew C., Barbara L. Fredrickson, Oscar Ybarra,Stephane Cote, Kareem Johnson, Joe Mikels, Anne Conway,and Tor Wager (Sept. 2005). “A Warm Heart and a Clear Head:The Contingent Effects of Weather on Mood and Cognition”.In: Psychological Science 16.9, pp. 724–731.

34 / 49

References XI

Krzanowski, Wojtek J. (2005). “Misclassification Rates”. In:Encyclopedia of Statistics in Behavioral Science.Lane, Nicholas, Mashfiqui Mohammod, Mu Lin, Xiaochao Yang,Hong Lu, Shahid Ali, Afsaneh Doryab, Ethan Berke,Tanzeem Choudhury, and Andrew Campbell (Apr. 2012).“BeWell: A Smartphone Application to Monitor, Model andPromote Wellbeing”. In: IEEE.Lane, Nicholas D., Emiliano Miluzzo, Hong Lu, Daniel Peebles,Tanzeem Choudhury, and Andrew T. Campbell (Sept. 2010). “Asurvey of mobile phone sensing”. In: Comm. Mag. 48.9,pp. 140–150.Lee, Gary R. and Masako Ishii-Kuntz (Dec. 1987). “SocialInteraction, Loneliness, and Emotional Well-Being among theElderly”. In: Research on Aging 9.4, pp. 459–482.

35 / 49

References XII

Lester, Jonathan, Tanzeem Choudhury, and Gaetano Borriello(2006). “A Practical Approach to Recognizing PhysicalActivities”. In: Pervasive Computing. Ed. by KennethP. Fishkin,Bernt Schiele, Paddy Nixon, and Aaron Quigley. Vol. 3968.Lecture Notes in Computer Science. Springer BerlinHeidelberg, pp. 1–16.LEWIS, M.E. and J.M.E. HAVILAND-JONES (2004). Handbookof Emotions. Guilford Publications, Incorporated.Lin, Wei-Jiun and James J. Chen (Jan. 2013).“Class-imbalanced classifiers for high-dimensional data”. In:Briefings in Bioinformatics 14.1, pp. 13–26.Lyubomirsky, Sonja, Laura King, and Ed Diener (Nov. 2005).“The Benefits of Frequent Positive Affect: Does Happiness Leadto Success?” In: Psychological Bulletin 131.6, pp. 803–855.

36 / 49

References XIIILyubomirsky, Sonja, Chris Tkach, and Robin M. Dimatteo(2006). “WHAT ARE THE DIFFERENCES BETWEENHAPPINESS AND SELF-ESTEEM?” In: Social IndicatorsResearch 78, pp. 363–404.Ma, Yuanchao, Bin Xu, Yin Bai, Guodong Sun, and Run Zhu(2012). “Daily Mood Assessment Based on Mobile PhoneSensing”. In: Wearable and Implantable Body Sensor Networks(BSN), 2012 Ninth International Conference on, pp. 142–147.Madan, Anmol, Manuel Cebrián, David Lazer, andAlex Pentland (2010). “Social sensing for epidemiologicalbehavior change”. In: UbiComp, pp. 291–300.McCrae, Robert R. and Paul T. Costa (1991). “Adding Liebeund Arbeit: The Full Five-Factor Model and Well-Being”. In:Personality and Social Psychology Bulletin 17.2, pp. 227–232.Miller, G. (1955). “Note on the bias of information estimates”.In: Information theory in psychology: Problems and methods2.95, p. 100.

37 / 49

References XIV

Miller, Geoffrey (2012). “The Smartphone PsychologyManifesto”. In: Perspectives on Psychological Science 7.3,pp. 221–237.Montjoye, Yves-Alexandre, Jordi Quoidbach, Florent Robic, andAlex(Sandy) Pentland (2013a). “Predicting Personality UsingNovel Mobile Phone-Based Metrics”. In: Social Computing,Behavioral-Cultural Modeling and Prediction. Ed. byArielM. Greenberg, WilliamG. Kennedy, and NathanD. Bos.Vol. 7812. Lecture Notes in Computer Science. Springer BerlinHeidelberg, pp. 48–55.Montjoye, Yves-Alexandre de, Jordi Quoidbach, Florent Robic,and Alex Pentland (2013b). “Predicting Personality Using NovelMobile Phone-Based Metrics”. In: SBP. Ed. byAriel M. Greenberg, William G. Kennedy, and Nathan Bos.Vol. 7812. Lecture Notes in Computer Science. Springer,pp. 48–55.

38 / 49

References XVMoturu, Sai T., Inas Khayal, Nadav Aharony, Wei Pan, andAlex Pentland (2011). “Using Social Sensing to Understand theLinks between Sleep, Mood, and Sociability”. In:SocialCom/PASSAT, pp. 208–214.Muaremi, Amir, Bert Arnrich, and Gerhard Tröster (2012). “ASurvey on Measuring Happiness with Smart Phones”. In:Myers, David G. and Ed Diener (1995). “Who Is Happy?” In:Psychological Science 6.1, pp. 10–19.Myers, I.B. and P.B. Myers (1980). Gifts differing: understandingpersonality type. Davies-Black Pub.Nicolaou, M. A., H. Gunes, and M. Pantic (2011).“Output-Associative RVM Regression for Dimensional andContinuous Emotion Prediction”. In: Proceedings of IEEEInternational Conference on Automatic Face and GestureRecognition (FG’11). Santa Barbara, CA, USA, pp. 16–23.Nicu Sebe Erwin Bakker, Ira Cohen Theo Gevers andThomas Huang. Bimodal Emotion Recognition.

39 / 49

References XVI

Oikonomopoulos, A., I. Patras, M. Pantic, and N. Paragios(2007). “Trajectory-based Representation of Human Actions”.In: Lecture Notes in Artificial Intelligence, Special Volume onArtificial Intelligence for Human Computing. Ed. by T. Huang,A. Nijholt, A. Pentland, and M. Pantic. Vol. 4451, pp. 133–154.Paninski, Liam (June 2003). “Estimation of entropy and mutualinformation”. In: Neural Comput. 15.6, pp. 1191–1253.Pantic, M. (2005). “Affective Computing”. In: in TheEncyclopedia of Multimedia Technology and Networking, M.Pagani, Ed. Hershy, USA: Idea Group Reference. Vol. 1,pp. 8–14.— (2009a). “Affective Computing (revisited)”. In: in TheEncyclopedia of Multimedia Technology and Networking - 2ndEdition, M. Pagani, Ed., Hershy, USA: Idea Group Reference.Vol. 1, pp. 15–21.

40 / 49

References XVII

Pantic, M. (2009b). “Machine analysis of facial behaviour:Naturalistic and dynamic behaviour”. In: PhilosophicalTransactions of Royal Society B 364, pp. 3505–3513.Pantic, M., A. Pentland, A. Nijholt, and T. Huang (2007).“Human Computing and Machine Understanding of HumanBehavior: A Survey”. In: Lecture Notes in Artificial Intelligence,Special Volume on Artificial Intelligence for Human Computing.Ed. by T. Huang, A. Nijholt, A. Pentland, and M. Pantic.Vol. 4451, pp. 47–71.Pantic, M., R. Cowie, F. D’ericco, D. Heylen, M. Mehu,C. Pelachaud, I. Poggi, M. Schroder, and A. Vinciarelli (2011).“Social Signal Processing: The Research Agenda”. In: VisualAnalysis of Humans, pp. 511–538.Pentland, Alex and Andrew Liu (Jan. 1999). “Modeling andprediction of human behavior”. In: Neural Comput. 11.1,pp. 229–242.

41 / 49

References XVIII

Picard, Rosalind W. (1997). Affective computing. Cambridge,MA, USA: MIT Press.Quercia, Daniele (2013). Donât Worry, Be Happy: TheGeography of Happiness on Facebook. http://profzero.org/publications/websci13_dontworry.pdf.Quercia, Daniele, Jonathan Ellis, Licia Capra, andJon Crowcroft (2012). “Tracking "gross community happiness"from tweets”. In: CSCW, pp. 965–968.Rabbi, Mashfiqui, Shahid Ali, Tanzeem Choudhury, andEthan Berke (2011). “Passive and In-Situ assessment of mentaland physical well-being using mobile sensors”. In: Proceedingsof the 13th international conference on Ubiquitous computing.UbiComp ’11. Beijing, China: ACM, pp. 385–394.

42 / 49

References XIX

Rachuri, Kiran K., Mirco Musolesi, Cecilia Mascolo,Peter J. Rentfrow, Chris Longworth, and Andrius Aucinas(2010). “EmotionSense: a mobile phones based adaptiveplatform for experimental social psychology research”. In:UbiComp, pp. 281–290.Robert LiKamWa Yunxin Liu, Nicholas D. Lane Lin Zhong(2013). MoodScope: Building a Mood Sensor from SmartphoneUsage Patterns.http://niclane.org/pubs/mobisys_moodscope.pdf.Salah, Albert Ali and Bruno Lepri (2011). “Challenges ofHuman Behavior Understanding for Inducing BehavioralChange”. In: AmI Workshops, pp. 249–251.

43 / 49

References XXSalah, AlbertAli, Javier Ruiz-del Solar, Ãetin Meriçli, andPierre-Yves Oudeyer (2012). “Human Behavior Understandingfor Robotics”. In: Human Behavior Understanding. Ed. byAlbertAli Salah, Javier Ruiz-del Solar, Ãetin Meriçli, andPierre-Yves Oudeyer. Vol. 7559. Lecture Notes in ComputerScience. Springer Berlin Heidelberg, pp. 1–16.Sanderson C.A., Cantor N. (1999). “A life task perspective onpersonality coherence: Stability versus change in tasks, goals,strategies, and outcomes.” In: D. Cervone & Y. Shoda (Eds.),The Coherence of Personality: Social-cognitive Bases ofConsistency, Variability, and Organization. New York, NY, USA:Guilford Press, pp. 372–392.Schuller, B., R.J. Villar, G. Rigoll, and M. Lang (2005).“Meta-Classifiers in Acoustic and Linguistic FeatureFusion-Based Affect Recognition”. In: Acoustics, Speech, andSignal Processing, 2005. Proceedings. (ICASSP ’05). IEEEInternational Conference on. Vol. 1, pp. 325–328.

44 / 49

References XXI

Sebe, Nicu, Michael S. Lew, Ira Cohen, Ashutosh Garg, andThomas S. Huang (2002). “Emotion Recognition Using aCauchy Naive Bayes Classifier”. In: in Proc. ICPR, pp. 17–20.Soleymani, M. and M. Pantic (2013). “Emotionally Aware TV”.In: Proceedings of TVUX-2013: Workshop on Exploring andEnhancing the User Experience for TV at ACM CHI 2013.Paris, France.Soleymani, M., M. Pantic, and T. Pun (2012). “MultimodalEmotion Recognition in Response to Videos”. In: IEEE Trans.on Affective Computing 3.2. in press, pp. 211–223.Staiano, Jacopo, Bruno Lepri, Ramanathan Subramanian,Nicu Sebe, and Fabio Pianesi (2011). “Automatic modeling ofpersonality states in small group interactions”. In: Proceedingsof the 19th ACM international conference on Multimedia. MM’11. Scottsdale, Arizona, USA: ACM, pp. 989–992.

45 / 49

References XXII

Staiano, Jacopo, Bruno Lepri, Nadav Aharony, Fabio Pianesi,Nicu Sebe, and Alex Pentland (2012). “Friends don’t lie:inferring personality traits from social network structure”. In:Proceedings of the 2012 ACM Conference on UbiquitousComputing. ACM, pp. 321–330.Stehman, Stephen V. (1997). “Selecting and interpretingmeasures of thematic classification accuracy”. In: RemoteSensing of Environment 62.1, pp. 77–89.Track your Happiness Web Application. (Access date:12-JUN-2013). http://www.trackyourhappiness.org.Tsutsui, Yoshiro (2012). “Weather and Individual Happiness”.In: Weather, Climate, and Society 5.1, pp. 70–82.Unser, Michael (2000). “Sampling–50 years after Shannon”. In:Proceedings of the IEEE, pp. 569–587.Vapnik, Vladimir N. (1995). The nature of statistical learningtheory. New York, NY, USA: Springer-Verlag New York, Inc.

46 / 49

References XXIII

Vitters, Joar (2001). “Personality traits and subjectivewell-being: emotional stability, not extraversion, is probably theimportant predictor”. In: Personality and Individual Differences31.6, pp. 903 –914.Wagner, Johannes, Florian Lingenfelser, Elisabeth Andre,Jonghwa Kim, and Thurid Vogt (2011a). “Exploring FusionMethods for Multimodal Emotion Recognition with MissingData”. In: IEEE Transactions on Affective Computing 2.4,pp. 206–218.Wagner, Johannes, Elisabeth Andre, Florian Lingenfelser, andJonghwa Kim (Oct. 2011b). “Exploring Fusion Methods forMultimodal Emotion Recognition with Missing Data”. In: IEEETrans. Affect. Comput. 2.4, pp. 206–218.Wandishin, Matthew S. and Steven J. Mullen (2009). MulticlassROC Analysis. American Meteorological Society.

47 / 49

References XXIV

Wang, Yongjin and Ling Guan (2008). “Recognizing HumanEmotional State From Audiovisual Signals”. In: Multimedia,IEEE Transactions on 10.4, pp. 659–668.Zajonc, Robert B and Daniel N McIntosh (1992). “Emotionsresearch: Some promising questions and some questionablepromises”. In: Psychological Science, pp. 70–74.Zeng, Z., M. Pantic, and T. Huang (2009). “Emotion Recognitionbased on Multimodal Information”. In: Affective InformationProcessing. Ed. by T. Tan and J. Tao. Springer, pp. 241–266.Zeng, Z., M. Pantic, G. Roisman, and T. Huang (2009). “ASurvey of Affect Recognition Methods: Audio, Visual, andSpontaneous Expressions”. In: IEEE Transactions on PatternAnalysis and Machine Intelligence 31.1, pp. 39–58.

48 / 49

References XXV

Zeng, Zhihong, Jilin Tu, Ming Liu, Tong Zhang,Nicholas Rizzolo, Zhenqiu Zhang, Thomas S. Huang,Dan Roth, and Stephen Levinson (2004). “Bimodal HCI-relatedaffect recognition”. In: Proceedings of the 6th internationalconference on Multimodal interfaces, pp. 137–143.

49 / 49