analyzing user emotions via physiology signals user emotions via physiology signals 13 of user’s...

Data Science and Pattern Recognition c©2017 ISSN 2520-4165

Ubiquitous International Volume 1, Number 2, December 2017

Analyzing User Emotions via Physiology Signals

Bohdan Myroniv∗, �Cheng-Wei Wu∗ , Yi Ren† , Albert Christian∗ , Ensa Bajo∗ and Yu-Chee Tseng∗

∗Department of Computer ScienceNational Chiao Tung University, Hsinchu, Taiwan, ROC

[email protected]

†School of Computing ScienceUniversity of East Anglia, Norwich, UK

Abstract. Recently some studies have shown that the major influential factor of ourhealth is not only physical activities, but the states of our emotion that we experiencethrough our daily life, which continuously build our behavior and affect our physical healthsignificantly. Therefore, emotion recognition draws more and more attention for manyresearchers in recent years. In this paper, we propose a system that uses off-the-shelfwearable sensors, including heart rate, galvanic skin response, and body temperature sen-sors, to read physiological signals from the user and apply machine learning techniquesto recognize emotional states of the user. These states are key steps, toward improvingnot only the physical health but also emotional intelligence in advanced human-machineinteraction. Moreover, we consider three types of emotional states and conduct exper-iments on real-life scenarios. Experimental results shows that the proposed system hasgood performance for emotion recognition.Keywords: Emotion recognition, Machine learning, Physiological signals, Wearable de-vices.

1. Introduction. The Internet of Things (IoT) [30] have rapidly become one of themost popular research topics due to its wide range of application scenarios. In general,these applications can be divided into three categories: (1) smart home [32], (2) smarttransportation [33], and (3) e-health [34]. Wearable technology [31] is often touted as oneof the important applications of the IoT. It consists of low-cost and low-power sensorswhich widely used in the market. The existing wearable applications are in their earlyphase and currently dominated by physical activities tracking devices. So-called smart-band can monitor some of our daily activities, such as walking, running, cycling, andswimming. It also can track when and how long we sleep. Some of these act as personaltrainers and count how many calories was burned. Although the application of physicalhealth monitoring applications are booming, only few studies have been conducted onthat of emotional health.

Emotion is an integral part of our health and has a huge impact on our bodies, mentalhealth, and behavior. Weak emotional state can affect our immune system, making usmore likely to get cold and other minor infections. According to American PsychologicalAssociation (APA) [10], more than 53% of Americans report personal health problems asa source of stress. This issue of stress which has left unchecked can contribute many healthproblems, such as high blood pressure, heart disease, obesity, and diabetes. According to

11

12 Bohdan Myroniv, Cheng-Wei Wu, Yi Ren, Albert Budi Christian, Ensa Bajo, and Yu-Chee Tseng

a study conducted by the American College Health Association (ACHA) [11], a consid-erable proportion of students said that mental health problems affected their academicperformance, causing them to dropping courses, or receiving low grades in the classes.

Most of the existing studies of emotion recognition are based on visual camera-basedapproaches [2, 3, 6], audio speech-based approaches [1, 4, 5, 7] and physiology-based emo-tion recognition [27, 28]. The visual camera-based approaches that used image processingtechnologies to detect users’ emotions. However, this approach requires the users to bewell seen by the cameras, thus made this approaches may fall significantly. With regardsto the audio speech-based approaches [1, 4, 5, 7], the main idea of these approaches isto use the speakers’ loudness, tone, and intonation patterns in a speech to detect theiremotions. However, conversational properties may vary in different cultures and nation-alities, which may lead to worse recognition results from the system. Recently manystudies have shown that less attention was paid to physiological signals analysis for emo-tion recognition using wearable devices [27, 28]. However, to detect users’ emotions usingphysiological signals required a constant and reliable data. It means the autonomous ner-vous system cannot be controlled consciously by users themselves. Such data will not beaffected by the factors, such as light requirements in video-based or cultural peculiaritiesin audio-based approaches.

Figure 1. a)Heart rate sensor; b)Galvanic skin response sensor; c) Tem-perature sensor

In this paper, we use three non-invasive sensors as shown in Fig. 1: Heart rate sensorto sense user’s pulse (Beats Per Minute; BPM), Galvanic Skin Response (GSR) sensorto sense the skin conductivity to modulates the amount of sweat secretion from sweatglands, and body Temperature sensor (T) to sense temperature value of the user’s body.Then, we apply sliding window-based segmentation method to segment collected dataand extract candidate features from the segments. After that, we feed extracted featuresto classifiers to identify the type of users’ emotional states. We implement the prototypeof the proposed methods on Arduino platform and evaluate the performance of six clas-sification algorithms to train emotion classification models: Random Tree, J48 DecisionTree, Naive Bayes, SVM, KNN, and Multilayer Perceptron Neural Network.

We collect data from 10 participants using Geneva Affective Picture Database (GAPED)and International Affective Picture System (IAPS) as a triggering mechanism for experi-mental investigations of emotion. These databases consist of a set of labeled pictures whichare designed to stimulate particular emotions. The results show high accuracy above 97%

Analyzing User Emotions via Physiology Signals 13

of user’s emotional conditions recognition using low-cost sensors that are available in cur-rent wearable devices. In recognition stage, we consider three general emotional states:Negative, Neutral, and Positive.

The rest of this paper is organized as follows: Section 2 reviews the related works.Section 3 introduces the details of the proposed methods and experimental methodol-ogy. Performance evaluation and prototype implementation of the proposed system aredescribed in Section 4. Finally, Section 5 gives conclusion and future works.

2. Related Works. This section discusses about existing studies that fall into threecategories: emotion recognition based on video, audio and physiology approaches.

2.1. Video-based emotion recognition. The approaches in this category can be di-vided into two types: emotion recognition (ER) based on facial expression [18, 19, 20]and body movements [17, 21]. Both of them rely on cameras and image processing tech-nologies to detect emotional states. However, in order to detect emotional states, userhas to be in a range of camera vision and even if so, user has to be well seen. In otherwords, such system will output an unreliable results or even will not work in a poorly litenvironment. Besides, the user can consciously control facial expression to false classifi-cation results. In addition, camera-based emotion recognition systems are stationary andcannot be properly used as a day-long wearable solution.

2.2. Audio-based emotion recognition. Besides information that we intend to say,our speech contain information about our emotions [22]. Different methods and algorithmswere designed for audio-based emotion recognition, such as [23, 24]. These methods relyon microphones and based on speakers’ loudness, tone, and intonation. However, con-versational properties vary in different cultures, nationalities, or even differ from humanbeing to human being [7], so the universality of such systems could be low and may betight to a specific area with a specific conversation property. Moreover, in order to detectusers’ emotional states, users have to constantly or continuously talking, which is veryinconvenient in many applications.

2.3. Physiology-based emotion recognition. Using biosensors to detect emotionalstates becomes more popular among researchers in recent years. These biosensors aremonitoring physiological signals generated by an autonomic nervous system of human,which produces visceral sensations that shape emotional experience. Autonomous nervoussystem cannot be controlled consciously and it allows to collect reliable data, which isnot affected by factors, such as cultural and national diversities in audio-based approachor range of vision and light requirements in video-based approach. Besides, both of theseapproaches require users to do some actions to know their emotional states. Studies[27, 28] take advantage of using biosensors for emotion recognition. These studies usehigh cost and precision sensors and achieve high recognition accuracy of 96.6% [26]. Thesystem developed in [28] achieves even higher accuracy. It uses electrocardiogram (ECG)sensor to read heart’s electrical activity, electromyograph (EMG) sensor to read electricalactivity produced by skeletal muscles, and electroencephalogram (EEG) sensor to detectelectrical activity in brain and respiration rate sensor to detect how fast and deep weare breathing. However, these devices are costly and cannot be properly integrated insmart-watches and fitness bands.

3. The Proposed Method. In this section, we introduce the proposed methods andexplain the procedure of physiological signals processing for obtaining user’s emotionalstates.


3.1. System Architecture Overview. In this subsection, we explain the procedure ofphysiological signals processing for obtaining user’s emotional states. In our research, weuse three sensors which are connected to Arduino Yun microcontroller to collect users’data. The users’ data are collected with 10 Hz sampling rate frequency, that furtherprocess three physiological signals collected from non-invasive sensors below:

1. Heart rate (Grove Ear-clip Heart rate Sensor) is used to measure the amount ofheart beats per minute. Heart rate changes are triggered while different emotionsare triggered [13] or even listening to different music styles [14].

2. Galvanic Skin Response (Grove GSR sensor) is used to measure the electrical con-ductance of the skin. GSR modulates the amount of sweat secreted from sweatglands. In this case, emotion results to stimulate our sympathetic nervous system.The stimulus itself has impact on our body which becomes sweaty due to secretionsfrom sweat glands [15].

3. Temperature sensor (MLX90614) is used to measure the body temperature. Asindicated in [16] body temperature changes while different emotions are triggered.

Figure 2. The Input-Process-Output diagram of the proposed system

The collected sensor data are sent to a server via Bluetooth for further processing. Fig.2 shows the Input-Process-Output diagram of the proposed system. In this work, weconsider the recognition of three states of emotion; positive, neutral, and negative. Thedata generated from the sensors go through four phases before finally recognizing an indi-vidual user’s emotional state, which are pre-processing, segmentation, feature extraction,and emotion classification.

Figure 3. Labeling process for sensor data


Table 1. The extracted candidate features

ID Feature Formula

1 Mean µ(A) = 1n

∑ni=1 ai

2 Variance V ar(A) = 1n

∑ni=1(ai − µ(A))2

3 Energy Eng(A) = 1n

∑ni=1(ai)

2

4 Average Absolute Difference Aav(A) 1n

∑ni=1 | ai |

6 Skewness Skew(A) =1n

∑ni=1(ai−µ(A))3

[ 1n

∑ni=1(ai−µ(A))2]3/2

7 Kurtosis Kurt(A)1n

∑ni=1(ai−µ(A))4

[ 1n

∑ni=1(ai−µ(A))2]2

8 Zero Crossing Rate ZCR(A) = 12

∑ni=2 | sign(ai)− sign(ai−1) |

9 Mean Crossing Rate MCR(A) =∑n

i=2|sign(ai−µ(A))−sign(ai−1−µ(A))|2

10 Root Mean Square Rms(A) =√

1n

∑ni=1 a

2i

Pre-processing Phase: In this phase, we observe that the collected data tend tohave missing values during a short period (i.e., one to two seconds) just after theconnection stability between the devices are established. In order to address suchproblem, we therefore cut off the first two seconds’ data from the collected data toavoid the interference of the missing data. Such issue may happen due to the ongoingestablishment of the Bluetooth connection.

Feature Extraction Phase: In this phase, we extract candidate features from thesensor signals in each segment. For instance, let A =< a1, a2, ..., an > be a timeseries data of a sensor in a segment, where ai (1 ≤ i ≤ n) is the ith sample in A. Theextracted candidate features are shown in Table 1.

• Mean - It measures the central tendency of the data.

• Variance - It represents the expected squared deviation from the mean.

• Energy - It measures the magnitude of A.

• Average Absolute Difference - It is a measurement of statistical dispersionequal to the average absolute difference of two independent values drawn from aprobability distribution.

• Average Absolute Value - It is the average of the absolute values in A.

• Skewness - It is used to describe asymmetry of A from the normal distributionin a set of statistical data.

• Kurtosis - It measures whether A are heavy-tailed or light-tailed relative to anormal distribution.

• Zero Crossing Rate (ZCR) - It is a measurement that reflects the times thatsigns of two adjacent values in A, change from positive to negative or vice versa,


where sign(.) returns 1 if one of the two inputs is positive and another one isnegative; otherwise it returns 0.

• Mean Crossing Rate (MCR) - It is a measurement that reflects how manytimes the sign of two adjacent values in A cross mean.

• Root Mean Square - It measures the square root of the data’s mean square.

Figure 4. Data segmentation and overlapping sliding window method

Segmentation Phase: Fig. 4 describes how we apply the overlapping sliding window-based segmentation to segment the collected sensor data. The segmented windowsize is set to θ(θ > 0) seconds and the size of overlapping is set to δ(0 ≤ δ ≤ θ)seconds. In the experiments, we evaluate the effect of varied window sizes and over-lapping sizes on the recognition results of the proposed system.

Emotion Classification Phase: In this phase, the extracted candidate features areused to build classifiers using classification algorithms in Weka [12]. We consider thefollowing six different types of classification algorithms: K -Nearest Neighbor (abbr.KNN), J48 Decision Tree (abbr. J48), Naive Bayes (abbr. NB), Random Tree (abbr.RT), Support Vector Machine (abbr. SVM), and Multilayer Perceptron Neural Net-works (abbr. MP).

3.2. Experiment Methodology. In the experiment, we recruit ten participants, whichare divided into two groups: seven males and three females. Their ages are betweentwenty-two and thirty. Fig. 3 shows how we label sensor data using emotional trigger-ing databases. All the experiments are conducted in separate rooms. We request theparticipants to leave their phones out during the experiments. Emotion triggering is avery important part of the experiment. In the emotional triggering, we use two affectiveemotion photo databases in our experiments. The first one is Geneva Affective PictureDatabase (GAPED) [8] provided by Swiss Center for Affective Sciences, and the secondone is International Affective Picture System (IAPS) [9] provided by Center for The Studyof Emotion and Attention. We choose twenty photos for each type of emotional states(i.e., positive, neutral, and negative) from the above mentioned databases. Each photowas shown to the participant for five seconds. We have tried different orders of datacollection, i.e., different emotion stimuli were shown in different days, and different con-sistency. Such approach was applied to minimize dependencies of one emotional conditionfrom another one.

We held the experiments three times. The data collected from the first experiment wasnever used in the processing. It happened because we realize the participants felt nervousand strange due to the bunch of sensors and cables connected to their arms, which wouldinfluence participants’ emotion and lead to noise data. However, in the next experiment,our participants were more familiar with the sensors setup. Therefore, the data collectionfrom the second and the third experiments are used in the processing phase.


Table 2. The accuracy of the group models under varied segmentation sizes

ClassifiersAccuracy of each segment size

1 sec. 2 sec. 3 sec. 4 sec. 5 sec.

RT 83.69% 82.90% 82.58% 73.29% 80.52%

J48 90.35% 84.49% 86.48% 87.25% 80.55%

NB 39.46% 39.16% 45.34% 41.62% 34%

KNN 72.76% 74.75% 76.57% 80.45% 79.56%

SVM 47.11% 42.94% 42.64% 44.84% 49%

MP 86.28% 82.30% 79.87% 77.28% 72%

4. Performance Evaluation.

4.1. Experimental Evaluation. In this subsection, we evaluate the recognition perfor-mance of the proposed sensing system using different types of classifiers, including RT,J48, NB, KNN, SVM, and MP. We apply 10-fold cross validation to conduct the perfor-mance evaluation. We consider two kinds of classification models, which are named groupmodel and personalized model, respectively. For the group model, the emotion classi-fier is constructed by considering all the participants’ physiology signals in the trainingphase. While for the personalized model, it is constructed by considering only individualparticipant’s physiology signals.

4.2. Experimental Results. Table 2 shows the accuracy of the group models underdifferent sizes of data segmentation. In this experiment, the overlapping size δ is setto 0. In Table 2, we can see that the J48 classifier performs better than the other fiveclassifiers. Besides, when the segmentation size θ is set to 1 seconds, the J48 classifierachieves up to 90.35% recognition accuracy. Table 3 shows the average, minimum, andmaximum accuracies of the group models in this experiment. As shown in Table 3, theSVM and NB classifiers are not very effective, their average accuracies are 45.59% and39.91%, respectively.

Then, we test the effect of δ on the overall accuracy of the proposed system. In thisexperiment, the segmentation size θ is set to 5 seconds. Table 4 shows the accuracies ofthe group models when the overlapping size is varied from 0.5 seconds to 4.5 seconds.As shown in Table 4, RT, J48, NB, and KNN generally have better recognition accuracythan SVM and MP.

Fig. 5 shows the accuracy of the MP, RT, KNN, and J48 classifiers when θ = 5 andδ= 4.5. As shown in Fig. 5, the accuracy increases with increasing overlapping size ingeneral. Table 5 shows the performance improvement when the proposed system appliesthe overlapping sliding window-based segmentation. The left column of the Table 5 showsthe accuracy of MP, KNN, RT, and J48 when θ = 1 and δ = 0, while the right one showsthat of the four classifiers when θ = 5 and δ = 4.5. As shown in Table 5, the overallperformance of the system is generally enhanced.

From the above experiments, we can observe that the proposed system using KNNhas the best accuracy when θ and δ are set to 5 seconds and 4.5 seconds, respectively.Therefore, we then evaluate its performance in terms of precision, recall, and F-Measure


Table 3. Average, minimum, and maximum accuracies of the group models

Classifier Min Average Max

NB 34% 39.92% 45.34%

SVM 42.64% 45.31% 49%

KNN 72.76% 76.82% 80.45%

RT 73.29% 80.59% 83.69%

MP 72% 79.55% 86.28%

J48 80.55% 85.82% 90.35%

Figure 5. The recognition accuracy of different classifiers when θ = 5 andθ = 4.5

for each class. Table 6 shows the experimental results. As shown in Table 6, the systemusing KNN classifier also has good recognition result for each class.

After evaluating the recognition performance of the group models, we then evaluate thatof the personalized models. In this experiments, we consider to build ten personalizedmodels from data collected from ten participants. Because KNN has the best performancein the previous experiments, in this experiment, we only consider the KNN classifier andset the parameters θ and δ to 5 and 4.5 seconds, respectively. Fig. 6 shows the classifi-cation results for each personalized model built by the training data collected from eachuser. As shown in Fig. 6, the average accuracy of all the personalized models is 97.78%.Moreover, the maximum and minimum accuracies are 98.2% and 96.8%, respectively.

4.3. Prototype Implementation. In this subsection, we introduce the prototype resultof the proposed system. Fig. 7 shows the detail of system prototype components. Wedeveloped an Android application to recognize users’ emotional state online. It consistsof Arduino side and Android side. On the Arduino side, heart rate, body temperature,and GSR sensors will continuously collect users’ physiology signals, which will be sent


Table 4. The accuracy of the group models under different sizes of overlaps

δ RT J48 NB KNN SVM MP

0.5 sec. 84.68 81.92 39.63 78.37 49.54 77.92

1.0 sec. 79.51 84.33 38.95 79.11 42.57 80.72

1.5 sec. 85.31 86.01 38.11 77.62 44.40 76.92

2.0 sec. 85.58 85.88 44.44 84.08 48.34 83.18

2.5 sec. 86.53 90.77 44.38 85.28 46.38 83.54

3.0 sec. 87.41 89.27 41.38 84.58 45.28 85.15

3.5 sec. 89.82 88.62 49.11 87.27 49.85 88.47

4.0 sec. 93.11 92.31 46.30 92.11 48.41 88.92

4.5 sec. 94.22 95.46 52.14 97.31 48.41 91.78

Table 5. The accuracy of different classifiers with/without applying over-lapping sliding window-based segmentation

Classifier Without Overlapping Method With Overlapping method Improvement

MP 79.55% 84.07% 4.52%

KNN 76.82% 85.08% 8.26%

RT 80.59% 87.35% 6.76%

J48 85.82% 88.29% 2.47%

Table 6. The precision, recall, and F-measure of the proposed systemusing KNN

Classifier:

KNN

Group model / 5sec.segm / 4.5 sec. overlap

Training Testing

Class Precision Recall F-Measure Precision Recall F-Measure

Positive 0.984 0.980 0.982 0.973 0.974 0.974

Neutral 0.974 0.979 0.976 0.961 0.969 0.965

Negative 0.985 0.983 0.984 0.987 0.976 0.981

to the users’ smart-phone via Bluetooth. On the Android side, the collected data willgo through pre-processing, segmentation, feature extraction, and emotion classificationphases. Fig. 8 shows the prototype of the implemented App. On the first screen of theApp, user can click “Connect” button to connect the App with the Arduino device.

When Bluetooth connection is established, the App will go to the second screen, whereuser can see three boxes on which shown his/her detected heart beats per minute (show.as Heart rate. in the interface), GSR (abbr. as Skin resp. in the interface), and bodytemperature (abbr. as Tempr. in the interface) data in a real-time fashion. Detected


Figure 6. The accuracy of each personalized model

Figure 7. System prototype components

user’s emotional state is shown in the upper right side of the App. Online recognition isdone by using the pre-trained classifier inside the smart-phone.

4.4. System Performance Testing. We perform the testing of our system in real-life scenario. We consider different actions, such as: reading paper, scrolling Facebook,watching funny video, reading article, and watching negative pictures to see what emotionswill be triggered and see if they match our expected results. In Fig. 9, we can see a userreading article about newly released laptop. While on the left side of the screen, it showsthe application’ prototype result. The output of the App shows “Positive”, which meansthe user’s emotional state is identified as “positive”.


Figure 8. The User Interface of the implemented system prototyping

Based on our experiment, the transition between different emotional states usually takesapproximately 45 - 60 seconds to be perfectly steady and accurate on a given emotionalstate. This situation appears because heart rate, temperature, and other physiologicalsignals cannot change quickly.

Figure 9. A user is reading an article about newly released laptop

5. Conclusions and Future Work. In this work, we consider heart rate, body temper-ature and galvanic skin response sensors to design wearable sensing system for effectiverecognition of users’ emotional states. The proposed system allows recognizing three typesof emotions, including positive, neutral, and negative, in an online fashion. We apply themachine learning technology to process the physiology signals collected from the user.The process consists of four main phases: data pre-processing, data segmentation, featureextraction, and emotion classification. We implement the prototype of the system on Ar-duino platform with Android smart-phone. Extensive experiments on real-life scenarios


show that the proposed system achieves up to 97% recognition accuracy when it adoptsthe k -nearest neighbor classifier. In the future work, we will consider more types of user’semotional states and consider to find a correlation between these emotions and physicalactivities for more diverse and novel applications. Below is the demo video, source code,and dataset of our system (https : //goo.gl/KhhcyA).

Acknowledgment. This work was support in part by the Ministry of Science and Tech-nology of Taiwan, ROC., under grant MOST 104-2221-E-009-113-MY3, 105-2221-E-009-101-MY3, 105-2218-E-009-004.

REFERENCES

[1] Amin S., Andriluka M., Bulling A., Mller M. P., Verma P.,“Emotion recognition from embedded bod-ily expressions and speech during dyadic interactions,” IEEE International Conference on AffectiveComputing and Intelligent Interaction, pp. 663–669, 2015.

[2] Bulut M., Busso C., Deng Z., Kazemzadeh A., Lee C.M., Lee S., Narayanan S., Neumann U., YildirimS.,“Analysis of emotion recognition using facial expres-sions, speech and multimodal information,”International Conference on Multimodal Interfaces, pp. 205–211, 2004.

[3] Bilakhia S., Cowie R., Eyben F., Jiang B., Pantic M., Schnieder S., Schuller B., Smith K., Val-star M.,“The continuous audio/visual emotion and depression recognition challenge,” InternationalWorkshop on Audio/Visual Emotion Challenge, pp. 3–10, 2013.

[4] Aswathi E., Deepa T. M., Rajan S., Shameema C. P., Sinith M. S.,“Emotion recognition fromaudio signals using Support Vector Machine,” IEEE Recent Advances in Intelligent ComputationalSystems, pp. 139–144, 2015.

[5] Dai K., Fell J. H., MacAuslan J.,“Recognizing emotion in speech using neural net-works,” Interna-tional Conference on Telehealth/Assistive Technologies, pp. 31–36, 2008.

[6] Chakraborty A., Chakraborty U. K., Chatterjee A., Konar A.,“Emotion Recognition From FacialExpressions and Its Control Using Fuzzy Logic,” IEEE Trans-actions on Systems, Man, and Cyber-netics - Part A: Systems and Humans, Vol.39, Issue. 4, pp. 726–743, 2009.

[7] Pao T. L., Tsai Y. W., Yeh J. H.,“Recognition and analysis of emotion transition in Mandarin speechsignal,” IEEE International Conference on Systems Man and Cybernetics, pp. 3326–3332, 2010.

[8] Dan-Glauser E. S., Scherer K. R.,“The Geneva affective picture database (GAPED): a new 730-picture database focusing on valence and normative significance,” Behavior Research Methods, Vol.43, pp. 468-477, 2011.

[9] Bradley, M.M., Cuthbert, B.N., Lang, P.J.,“International Affective Picture System (IAPS): TechnicalManual and Affective Ratings. ,” NIMH Center for the Study of Emotion and Attention, 1997 ,http : //csea.phhp.ufl.edu/media.html.

[10] American Psychological Association, http : //www.apa.org/index.aspx[11] Consequences of Poor Mental Health, www.campushealthandsafety.org/mentalhealth/consequences/[12] Weka 3: Data Mining Software in Java, http : //www.cs.waikato.ac.nz/ml/weka/[13] Ralf Kbele, Mandy Koschke, Steffen Schulz, Gerd Wagner, ShravyaYeragani, Chaitra T. Ra-

machandraiah, Andreas Voss, Vikram K. Yeragani,Karl-Jrgen Br,“The influence of negative moodon heart rate complexity measures and baroreflex sensitivity in healthy subjects,” Indian Journal ofPsychiatry, pp.4247, 52(1):. 2010.

[14] Gregory D. Webster, Catherine G. Weir. ,“Emotional Responses to Music: Interactive Effects ofMode, Texture, and Tempo,” Motivation and Emotion,Volume 29, Issue 1, pp 1939, 2005.

[15] Wolfram Boucse . ,“Electrodermal Activity,” University of Wuppertal , 2012.[16] Lauri Nummenma, Enrico Glereana, Riitta Harib, Jari K. Hietanend,“Bodily maps of emotions,”

Developmental science, pp 1-8, 2016.[17] Ginevra Castellano, Santiago D. Villalba, Antonio Camurri,“Recognizing Human Emotions from

Body Movement and Gesture Dynamics,” ACII, pp. 71-82, 2007.[18] Carlos Busso, Zhigang Deng, Serdar Yildirim, Murtaza Bulut, Chul MinLee, Abe Kazemzadeh,

Sungbok Lee, Ulrich Neumann, Shrikanth Narayanan,“Analysis of emotion recognition using facialexpressions, speech and multi modal information,” ICMI, pp 205-211, 2004.

[19] G.U. Kharat, S.V. Dudul,“Neural network classifier for human emotion recognition from facial ex-pressions using discrete cosine transform,” First International Conference on Emerging Trends inEngineering and Technology , pp.653-658, 2008.


[20] R.Cowie, E.D. Cowie, J.G. Taylor, S.Ioannou, M.Wallace, S. Kollias,“An Intelligent system for facialrecognition,” IEEE International Conference on Multimedia and Expo, 2005.

[21] A. Cammuri, I. Lagerlof, G. Volpe,“Recognizing emotion from dance movement: comparison ofspectator recognition and automated techniques,” International journal of human-computer studies,vol. 59, pp. 213-255,2003.

[22] J.H.Jeon, R.Xia, Y.Liu .,“Sentence level emotion recognition based on decisions from subsentencesegments,” IEEE, ICASSP, pp.4940-4943, 2011.

[23] A.A. Razak, M.I.Zainal Abidin, R. Komiya,“Comparison between fuzzy and NN method for speechemotion recognition,” International conference on Information Technology and Applications, 2015.

[24] S.G.Koolagudi, R.Reddy, K.S. Rao,“Emotion recognition from speech signal using epoch parame-ters,” SPCOM, 2010.

[25] T.L.Pao, J.H.Yeh, Y.W.Tsai,“Recognition and analisis of emotion transition in Mandarin speechsignal,” IEEE, SMC, pp.3326-3332, 2010.

[26] Andreas Haag, Silke Goronzy, Peter Schaich, Jason Williams,“Emotion Recognition Using Biosen-sors: First Steps towards an Automatic System,” Affective Dialogue Systems, Tutorial and ResearchWorkshop, ADS, Kloster Isree, 2004.

[27] Eun-Hye Jang, Byoung-Jun Park, Sang-Hyeob Kim, Myoung-Ae Chung,Mi-Sook Park, Jin-HunSohn,“ Classification of Human Emotions from Physiological signals using Machine Learning Algo-rithms,” IARIA, pp.395-400, 2013.

[28] Eun-Hye Jang, Byoung-Jun Park, Sang-Hyeob Kim,“Emotion Classification based on PhysiologicalSignals inducted by Negative emotions,” IEEE, ICNSC, 2012.

[29] Bohdan Myroniv, Cheng-Wei Wu, Yi Ren, and Yu Chee Tseng,“Analysis of Users’ Emotions ThroughPhysiology,” The 11th International Conference on Genetic and Evolutionary Computing, 2017.

[30] A. Al-Fuqaha, M. Guizani, M. Mohammadi, M. Aledhari, and M. Ayyash,“Internet of Things: ASurvey on Enabling Technologies, Protocols, and Applications,” IEEE Communications Surveys andTutorials, Vol. 17, No.4, pp.2347-2376, 2015.

[31] Oscar D. Lara and Miguel A. Labrador,“A Survey an Human Activity Recognition Using WearableSensors,” IEEE Communications Surveys and Tutorials, Vol. 15, No. 3, 2013.

[32] Smart Home, https : //en.wikipedia.org/wiki/Homeautomation[33] Smart Transportation, https : //en.wikipedia.org/wiki/V ehicularautomation[34] Wearables / e-Health, https : //en.wikipedia.org/wiki/EHealth


Bohdan Myroniv received his BSc. degree in Information Securityfrom Yuriy Fedkovych Chernivtsi National University, Ukraine in 2015and M.S. degree in National Chiao Tung University, Taiwan, in 2017.His research interests include Machine Learning, Networking, and DataMining.

Cheng-Wei Wu received the Ph.D. degree in Department of Com-puter Science and Information Engineering from National Cheng KungUniversity, Taiwan, in 2015. Currently, he is hired as a assistant re-searcher in College of Computer Science, National Chiao Tung Univer-sity, Taiwan. His research interests include data mining, utility mining,pattern discovery, machine learning and big data analytics.

Yi Ren received the Ph.D. degree in information communication andtechnology from the University of Agder, Norway, in 2012. He waswith the Department of Computer Science, National Chiao Tung Uni-versity (NCTU), Hsinchu, Taiwan, as a Postdoctoral Fellow, an Assis-tant Research Fellow, and an Adjunct Assistant Professor from 2012 to2017. He is currently a Lecturer in the School of Computing Science atUniversity of East Anglia (UEA), Norwich, U.K. His current researchinterests include security and performance analysis in wireless sensornetworks, ad hoc, and mesh networks, LTE, and e-health security. Hereceived the Best Paper Award in IEEE MDM 2012.

Albert Budi Christian received his B.C.S degree from Soegi-japranata Catholic University, Indonesia, in April 2014 and M.S. degreefrom Chang Gung University, Taiwan, in June 2017. Currently, he ispursuing the Ph.D. degree in the Department of Electrical Engineeringand Computer Science, National Chiao Tung University, Taiwan. Hisresearch interests include Data Mining, Internet of Things(IoT), andHuman-Computer Interaction (HCI).


Ensa Bajo received his BSc. degree in Electrical Engineering from theRepublic of China Military Academy, Taiwan. Currently, he is pursuinga M.Sc. in the Department of Computer Science at National ChiaoTung University. His research interests include smart-phone sensingand Data mining.

Yu-Chee Tseng received the PhD degree in Computer and infor-mation science from the Ohio State University in January of 1994.He was/is chairman (2005-2009) and dean (2011-present), College ofComputer Science, National Chiao-Tung University, Taiwan. He hasbeen awarded as NCTU chair professor (2011-present) and Y.Z. HsuScientific chair professor (2012-2013). He received the Outstanding Re-search Award (National Science Council, 2001, 2003, and 2009), BestPaper Award (International Conference on Parallel Processing, 2003),Elite I. T. Award (2004), Distinguished Alumnus Award (Ohio StateUniversity, 2005), and Y. Z. Hsu Scientific Paper Award (2009). Hisresearch interests include mobile computing, wireless communication,and sensor networks. He served/ serves on the editorial boards of IEEE

Transactions on Vehicular Technology, IEEE Transactions on Mobile Computing, IEEE Trans-actions on Parallel and Distributed Systems, and IEEE Internet of Things Journal. His h-indexis more than 50. He is a fellow of the IEEE.

analyzing user emotions via physiology signals user emotions via physiology signals 13 of user’s...

Documents