Emotion detection from QRS complex of ECG signals using Hurst Exponent for different age groups

Jerritta S#1, M Murugappan#2, Khairunizam Wan #3, Sazali Yaacob #4

#School of Mechatronic Engineering, Universiti Malaysia Perlis (UniMAP) Campus Pauh Putra, 02600, Arau, Perlis, Malaysia

1sn.jerritta@gmail.com 2murugappan@unimap.edu.my

3 khairunizam@unimap.edu.my 4s.yaacob@unimap.edu.my

Abstract— Emotion recognition using physiological signals is one of the key research areas in Human Computer Interaction (HCI). In this work, we identify the six basic emotional states (Happiness, sadness, fear, surprise, disgust and neutral) from the QRS complex of electrocardiogram (ECG) signals. We focus specifically on the nonlinear feature ‘Hurst exponent’ computed using two methods namely rescaled range statistics (RRS) and finite variance scaling (FVS). The study is done on emotional ECG data obtained using audio visual stimuli from sixty subjects belonging to three different age groups – children (9 to 16 years), young adults (18 to 25 years) and adults (39 to 68 years). The performance of the Hurst exponent computed using RRS and FVS for individual age groups resulted in a maximum average accuracy of 78.21%. The combined analysis of the all the age groups had a maximum average accuracy of 70.23%. In general, the results of all the six emotional states indicate better performance compared to previous research works. However, the performance needs to be further improved in order to develop a reliable and robust emotion recognition system.

Keywords— Emotion, Inducement Stimuli, Physiological signals, Signal Processing Techniques.

I. INTRODUCTION

Researchers in Human Computer Interaction (HCI) are working on different ways to equip machines and computers with some amount of emotional intelligence. Methods depending on the expression of emotions such as facial actions, gestures and speech are prone to social masking. There are many social circumstances and instances where the expression of emotion is completely different from the real emotion experienced [1]. This paved way to emotion recognition using physiological signals. Researchers are working on a number of physiological signals such as electroencephalogram (EEG), electrocardiogram (ECG), electromyogram (EMG), galvanic skin response (GSR) and blood volume pulse (BVP) to understand the underlying and true emotional state of the person [2, 3].

Electrocardiogram (ECG) has been widely used by researchers in Human Computer Interaction (HCI) to understand the emotional state of a person. It is used either individually or along with a combination of physiological signals such as EMG, GSR and BVP [2, 4-6]. However, the

different research works vary vastly in performance due to factors such as the number of emotional states, number of subjects, methodology, dependency of classifiers on subjects and number of physiological signals used [7]. Of late, the QRS complex of ECG signals are also used in different applications such as the recognition of imperative cardiac arrhythmias [8] and acute myocardial ischemia [9]. The QRS-complex provides information about the rhythm and conduction path of the activation pulse in the heart and hence can be used in identifying the emotional states.

Despite a number of methods present, recently, nonlinear methods and features have been used widely in processing biological signals especially for the prediction of sleep apnea, mental state, arrhythmias and emotions [10-13]. Nonlinear analysis is based on chaos theory and helps in identifying the irregular behaviors that are present in the signal [14]. Non-linear features such as approximate entropy (APEN), largest Lyapunov exponent (LLE), correlation dimension (CD), Hurst exponent (H) and non-linear prediction error have been learned widely [10, 15]. These features convey information related to properties such as similarity, predictability, reliability and sensitivity of the signal.

Hurst exponent, based on self similarity and correlation properties measure the presence or absence of long-range dependence and its degree on a time series [11, 15]. The momentary and short range correlations can be measured using Hurst exponent. The variations in emotional state being transient and occurring for minute instances of time can be captured easily using Hurst. In this work, we concentrate on identifying the emotional information from QRS complex of ECG signals using the Hurst parameter computed using two methods namely rescaled range statistics (RRS) and finite variance scaling (FVS). The six basic emotional states (happiness, sadness, fear, surprise, disgust and neutral) are taken into consideration.

Researchers have studied the age dependence of emotions by inducing the different valance and arousal states in people belonging to different age groups. Mostly they have worked on two extreme age groups with mean ages around 20 for younger adults and 65 to 70 for older adults. The age

2013 Humaine Association Conference on Affective Computing and Intelligent Interaction

978-0-7695-5048-0/13 $26.00 © 2013 IEEE

DOI 10.1109/ACII.2013.159

849

difference was more evident in the pleasant region with young adults finding the stimuli to be more pleasant compared to older adults. The age dependence of the different emotional states is also studied in this work, for three different age groups. Understanding the emotional experience of different age groups helps in developing a common emotion recognition system which would benefit all people in the society.

The organization of this paper is as follows: Section I introduces the work and section II talks about the acquisition of emotional data. Section III details the data processing including preprocessing, classification and the two methods used for finding the value of Hurst exponent. Section IV illustrates and discusses the different observations from the results and section V concludes with the findings and remarks on future work.

II. EMOTIONAL DATA

One of the challenges in emotion recognition using physiological signals is the acquisition of proper emotional data. Many methods using visual images, sounds, audio visual clips, recall paradigm and dyadic interaction has been used by researchers to elicit the target emotions in the subject [16-20]. However, researchers have found audio visual elicitation to be more dynamic and natural in eliciting the target emotions [18]. Hence audio visual clips were used to induce the target emotions in the subject.

Short video clips were collected from different sources on the internet and a pilot study was conducted to validate the video clips that could elicit the target emotions in a better way. Sixty video clips (ten for each emotional state) were used to collect the emotional data. The emotion ‘anger’ was omitted as a result of the pilot study. The different clips were displayed to the subjects by means of a self-guided protocol. More details about the protocol can be found in our work here [21]. The subjects also filled in a self assessment questionnaire which was used to understand the intensity and validate the emotions experienced by the subjects.

Sixty subjects belonging to three different age groups (15 children aged 9 to 16 years, 30 undergraduate students aged 18 to 25 years and 15 adults aged 39 to 68 years) participated in the data collection experiment. ECG data was obtained by placing two electrodes on the wrists (one each on the right and left) and the reference electrode on the leg. Power Lab data acquisition system with chart software, developed by AD instruments was used to collect the emotional data. The signals were sampled at 1000 Hz.

III. EMOTIONAL DATA PROCESSING The raw ECG data is prone to noise that occurs mainly due

to power line interference, high frequency and movement artifacts. Baseline wander that occur due to movement and other artifacts were removed by means of a wavelet based

algorithm[22]. Butterworth low pass digital filter at 45Hz was used to remove both power line interference and high frequency noises. QRS complex was obtained from ECG signal by means of a derivative based algorithm used by WEN Wan-Hui et al., [5]. Hurst exponent was derived from the QRS complex using two methods.

A. Rescaled Range Statistics (RRS) This method analyzes the smoothness of a fractal time

series based on the asymptotic pattern of the rescaled range. First, the accumulated deviation of mean of time series over time is computed. The rescaled range R/S follows a power law relationship with time T as,

HTSR ~/ (1) where R is the difference between the maximum and minimum deviation from the mean and S represents the standard deviation. Hurst, H is then derived as,

)log()/log( TSRH = (2) where T is the length of sample data and R/S represents the corresponding value of rescaled range. [10].

B. Finite Variance Scaling (FVS)

Finite variance scaling method is also known as standard deviation analysis and is based on the standard deviation D(t) of the variable x(t).

Considering the time series x(t) to be of length n, the standard deviation is computed as,

( ) ( )2/12

112

)(���

�

�

���

�

�

�

��

��

� �

−�

��

��

� �

= �� ==

jtx

jtx

tDj

i ij

i ij

(3) for j=1,2,….,n.

Eventually, ( ) HttD ∝ (4) where H is the Hurst exponent. It is evaluated by finding

the gradient of the best fitted log-log plot of D(t) and t [23].

C. Classification of emotional states

The performance of the different features were analysed by three classifiers namely regression tree, K- Nearest Neighbour (KNN) and fuzzy KNN (FKNN) classifiers. Regression tree classifier creates a decision tree for predicting the classes based on Gini’s diversity index. KNN assigns a class based on the predominant class among the k nearest neighbours using Euclidean distance. FKNN works similar to KNN but the classes are assigned using the membership values. The value of k was chosen to vary from six to fifteen as the number of emotional states considered for classification is six.

Sixty subjects with six emotions and ten trials per emotion resulted in a total of 3600 samples. However, four trials of one subject had loose electrode contact because of which the data was ignored. Data from four kids and three young adults were also ignored because of unreliability captured using the

850

NN/RR ratio. This resulted in a total of 3300 samples, which were processed

The performance of the classifiers was evaluated using random-cross validation. Hurst derived using RRS and FVS from all the subjects were permutated and 70% of it was used for training the classifiers and 30% for testing. The percentage accuracy was derived from the predicted classes as,

100% ×=Emotion

EmotionEmotion samplestestedofnumberTotal

samplesclassifiedcorrectlyofNumberAccuracy

(5) where Emotion refers to the six emotional states namely happiness, sadness, fear, surprise, disgust and neutral.

IV. RESULTS AND DISCUSSION

Hurst exponent was analyzed in three frequency ranges commonly used in emotion recognition algorithms high (0.15 to 0.4 Hz), low (0.04-0.15 Hz) and very low frequency ranges (<0.04 Hz) [2]. The values of Hurst exponent, obtained in both RRS and FVS methods were statistically significant at the very low frequency range. The values are tabulated in Table I. Hurst exponent ranges from 0.000104 to 0.000456, similar to the work done by Peng et al., where the scaling exponent � is close to 0 for healthy heart beat data [24].

Statistical validation, done using Post hoc tests on the Hurst exponent derived using both the methods showed statistical significance (p<0.01) among the different emotional states. Table II and Table III shows the accuracy obtained for classifying the different emotional states for the three age groups using three classifiers namely regression tree, k –nearest neighbor (KNN) and Fuzzy k-nearest neighbor (FKNN). The value of k in the tables indicates the k value with the best classification accuracy for KNN and FKNN.

TABLE I VALUES OF HURST EXPONENT

RRS FVS Neutral 0.000395 0.000190

Happiness 0.000384 0.000186 Sadness 0.000350 0.000133

Fear 0.000398 0.000128 Surprise 0.000256 0.000108 Disgust 0.000456 0.000104 P value < 0.001 < 0.001

The maximum average accuracy of the RRS based method is 68.44% using KNN classifier for the adult category. The overall performance varied from 60.48% to 68.44% for the

different age groups and the different classifiers. The results are higher than the previous research works used for classifying the six emotional states [21, 25]. Table II also indicates that accuracy of the six emotional states did not vary much among the classifiers and among the age groups. The accuracy of emotions happiness and sadness is higher for children compared to young adults and adults.

FIGURE 1 AVERAGE ACCURACY OF DIFFERENT AGE GROUPS USING RRS

FIGURE 2 AVERAGE ACCURACY OF DIFFERENT AGE GROUPS USING FVS

The FVS based analysis in Table III shows an overall higher performance compared to the RRS based method. The performance of FVS varies from 66.78% to 78.21%. A difference in the average performance can be observed among the age groups where accuracy of children is highest, followed by adults and then young adults. The performance of the six emotional states also shows a demarcation among the three age groups. It should also be noted that the emotion disgust was captured effectively in the FVS based Hurst with maximum accuracy of 100%, 90.91% and 83.58% for the children, adults and young adults respectively. The negative valance emotions (sadness, fear and disgust) performed similar across all age groups. The positive and higher arousal emotion, happiness decreased in accuracy with increasing age groups. Its accuracy was highest for children and least for adults. In general, the emotions neutral and happiness performed least in this analysis. This may be because of more similarity and lesser deviation between the two emotional states. FVS, being an analysis of the standard deviation couldn’t capture the difference between closely related emotions effectively.

851

TABLE III ACCURACY OF RRS METHOD FOR THE DIFFERENT AGE GROUPS

Age Group Classifier Neutral (%)

Happiness (%)

Sadness (%)

Fear (%)

Surprise (%)

Disgust (%)

Average (%)

Children

Regression Tree

51.43 60.00 77.14 65.71 57.14 51.43 60.48

KNN (K=7) 57.14 77.14 82.86 74.29 65.71 45.71 67.14

FKNN (K=9) 57.14 62.86 77.14 57.14 68.57 54.29 62.86

Young adults

Regression Tree

69.01 62.86 62.86 70.00 71.83 55.71 65.38

KNN (K=8) 59.15 68.57 65.71 65.71 69.01 54.29 63.74

FKNN (K=7) 64.79 65.71 60.00 62.86 70.42 60.00 63.96

Adults

Regression Tree

68.18 68.18 68.18 54.55 60.47 65.91 64.24

KNN (K=12) 77.27 65.91 75.00 61.36 67.44 63.64 68.44

FKNN (K=6) 70.45 68.18 61.36 65.91 69.77 61.36 66.17

TABLE IIIII ACCURACY OF FVS METHOD FOR THE DIFFERENT AGE GROUPS

Age Group Classifier Neutral (%)

Happiness (%)

Sadness (%)

Fear (%)

Surprise (%)

Disgust (%)

Average (%)

Children

Regression Tree

58.33 77.14 77.14 82.86 85.71 82.86 77.34

KNN (K=12) 69.44 57.14 62.86 80.00 77.14 100.00 74.43 FKNN (K=7) 75.00 57.14 80.00 77.14 88.57 91.43 78.21

Young adults

Regression Tree

52.17 61.19 80.60 76.12 75.00 74.63 69.95

KNN (K=8) 47.83 68.66 67.16 79.10 57.35 80.60 66.78 FKNN (K=10) 52.17 65.67 76.12 77.61 67.65 83.58 70.47

Adults

Regression Tree

61.36 52.27 79.55 75.00 86.05 79.55 72.30

KNN (K=6) 63.64 45.45 79.55 86.36 83.72 90.91 74.94 FKNN (K=8) 61.36 52.27 84.09 79.55 76.74 90.91 74.15

TABLE IV

ACCURACY OF RRS AND FVS METHODS FOR ALL THE AGE GROUPS

Method Classifier Neutral (%)

Happiness (%)

Sadness (%)

Fear (%)

Surprise (%)

Disgust (%)

Average (%)

RRS Regression Tree 62.91 76 67.78 71.81 73.83 59.06 68.56

KNN (K=7) 71.52 74 73.83 73.15 71.81 57.04 70.23

FKNN (K=8) 61.59 72 65.1 73.83 73.15 64.43 68.35

FVS Regression Tree 54.72 61.9 67.12 65.07 60.96 76.03 64.3

KNN (K=6) 58.78 62.58 64.38 65.06 64.38 82.19 66.23

852

FKNN (K=11) 55.4 59.18 69.17 63.01 66.43 78.76 65.33

The average accuracy of the different age group using the three classifiers for the RRS analysis and FVS analysis is shown in Figures 1 and 2 respectively. The accuracy of the different age groups in the RRS analysis did not vary much whereas in the FVS analysis, the accuracy was highest in children. This indicates that the physiological response of the emotions felt by children is better compared to adults and young adults. Hence the algorithms were able to capture the emotional states easily. The emotional states are induced effectively in children compared to other age groups. The young adults indicated least performance in the FVS based analysis.

Table IV shows the accuracy of the different emotional states for all the sixty subjects obtained using the two methods. We can see that the RRS methods worked better for all the age groups compared to the FVS based method with the maximum accuracy of 70.23%. The maximum accuracy of the FVS based method is 66.23%. The individual emotional states also performed better in the RRS based analysis with almost similar accuracy for all the emotional states except neutral.

A contrasting performance can be observed in the accuracy of the emotions captured using the FVS and RRS based methods. The emotion ‘disgust’ in the negative valance was captured best by the FVS method and least by the RRS method. Contrastingly, the positive emotion ‘happy’ and the ‘neutral’ state of the subject were effectively captured by the RRS method compared to FVS. Analyzing both these methods and trying to identify new features or methods that can combine both RRS and FVS may help to identify all the basic emotions in an effective way. The confusion matrix for the RRS and FVS based analysis for all the age groups are specified in tables V and VI respectively.

TABLE V CONFUSION MATRIX FOR RRS BASED ANALYSIS FOR ALL AGE GROUPS

Neutral Happy Sad Fear Surprise Disgust

Neutral 98 15 4 8 6 20 Happy 18 94 7 15 8 8

Sad 4 5 97 4 22 17 Fear 6 16 7 106 6 8

Surprise 6 4 21 4 111 3 Disgust 29 13 18 10 1 78

The confusion matrix of the RRS based method (Table V) shows that the wrongly classified samples are wide spread across all the other emotional states. In the case of FVS based analysis, wrong classifications are seen among adjacent emotional states. Table VI shows that some of the neutral emotions are wrongly classified as happiness and

vice versa. They are not classified into any of the other emotional states. Similar is the case between sadness and fear, surprise and disgust, though two fear samples are classified as surprise. It can be observed that the valance and arousal components are rightly captured by the FVS based analysis. However, distinguishing the emotions that lie in the same quadrant or emotions that are knitted together still remains a challenge.

TABLE VI CONFUSION MATRIX FOR FVS BASED ANALYSIS FOR ALL AGE GROUPS

Neutral Happy Sad Fear Surprise Disgust

Neutral 97 51 0 0 0 0 Happy 65 82 0 0 0 0

Sad 2 0 108 36 0 0 Fear 0 0 56 88 2 0

Surprise 0 0 0 0 93 53 Disgust 0 0 0 0 29 117

Though the Hurst exponent is able to identify the emotional states, the classification accuracy of the emotion recognition system needs further improvement. This would require more investigation into other nonlinear methods and features that may provide more information about the hidden emotions. Further different physiological signals such as EEG, EMG, BVP and GSR can be combined to develop a robust and reliable emotion recognition system.

V. CONCLUSION Emotions being subjective and transient require efficient

methodologies to identify the underlying and true emotional state of the person. In this work, we explore the usage of Hurst exponent in extracting the emotional information pertaining to six emotional states (happiness, sadness, surprise, fear, disgust and neutral) from the QRS complex of electrocardiogram (ECG) signals. Two methods namely rescaled range statistics (RRS) and finite variance scaling (FVS) was used to capture the Hurst exponent from sixty subjects belonging to three different age groups. The results indicate that Hurst computed FVS based method performs well on categorizing six emotional states for the individual age groups. The children felt their emotions better than other age groups. The results obtained by combining all the age groups shows that the RRS method captures emotional information effectively for all the emotional states. Another observation shows the RRS and FVS to extract positive and negative emotions respectively in an enhanced way. Hence, further analysis can be done by combining RRS and FVS to extract both positive and negative emotional features effectively thereby improving the performance of the emotion recognition system.

853

ACKNOWLEDGMENT This research work is supported by the Fundamental

Research Grant Scheme (FRGS ), Malaysia. Grant Number: 9003-00341

REFERENCES [1] K. Kim, et al., "Emotion recognition system using

short-term monitoring of physiological signals," Medical and Biological Engineering and Computing, vol. 42, pp. 419-427, 2004.

[2] K. Jonghwa and E. Ande, "Emotion Recognition Based on Physiological Changes in Music Listening," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, pp. 2067-2083, 2008.

[3] R.-N. Duan, et al., "EEG-Based Emotion Recognition in Listening Music by Using Support Vector Machine and Linear Dynamic System," in Neural Information Processing. vol. 7666, T. Huang, et al., Eds., ed: Springer Berlin Heidelberg, 2012, pp. 468-475.

[4] F. Agrafioti, et al., "ECG Pattern Analysis for Emotion Detection," Affective Computing, IEEE Transactions on, vol. 3, pp. 102-115, 2012.

[5] W. Wan-Hui, et al., "Electrocardiography Recording, Feature Extraction and Classification for Emotion Recognition," presented at the 2009 WRI World Congress on Computer Science and Information Engineering, Los Angeles, CA, 2009.

[6] R. W. Picard, et al., "Toward machine emotional intelligence: analysis of affective physiological state," IEEE Transactions on Pattern Analysis and Machine Intelligence., vol. 23, pp. 1175 - 1191, 2001.

[7] J. Arroyo-Palacious and D. M. Romano, "Towards a Standardization in the Use of Physiological Signals for Affective Recognition Systems," in Measuring Behaviour, Maastricht,The Netherlands, 2008, pp. 121-14.

[8] C.-H. Lin and Y.-C. Du, "Fractal QRS-complexes pattern recognition for imperative cardiac arrhythmias," Digit. Signal Process., vol. 20, pp. 1274-1285, 2010.

[9] D. Romero, et al., "Depolarization Changes During Acute Myocardial Ischemia by Evaluation of QRS Slopes: Standard Lead and Vectorial Approach," IEEE Transactions on Biomedical Engineering, , vol. 58, pp. 110-120, 2011.

[10] U. R. Acharya, et al., "Automated detection of sleep apnea from electrocardiogram signals using nonlinear parameters," Physiological Measurement, vol. 32, pp. 287-303, 2011.

[11] T. Balli and R. Palaniappan, "Classification of biological signals using linear and nonlinear features," Physiological Measurement, vol. 31, pp. 903-920, 2010.

[12] A. Jovic and N. Bogunovic, "Electrocardiogram analysis using a combination of statistical, geometric, and nonlinear heart rate variability features," Artificial Intelligence in Medicine, vol. 51, pp. 175-186.

[13] G. Valenza, et al., "The Role of Nonlinear Dynamics in Affective Valence and Arousal Recognition," Affective Computing, IEEE Transactions on, vol. 3, pp. 237-249, 2012.

[14] J. Gao, et al., "Facilitating Joint Chaos and Fractal Analysis of Biosignals through Nonlinear Adaptive Filtering," Plos one, vol. 6, 2011.

[15] K. Natarajan, et al., "Nonlinear analysis of EEG signals at different mental states," BioMedical Engineering Online, 2004.

[16] P. J. Lang, et al., "International Affective Picture System(IAPS): INstruction Manual and Affective Ratings," Technical Report A-5, 2001.

[17] R. A. Stevenson and T. W. James, "Affective auditory stimuli: Characterization of the International Affective Digitized Sounds (IADS) by discrete emotional categories," Behavior Research Methods, vol. 40, pp. 315-321, 2008.

[18] J. J. Gross and R. W.Levenson, "Emotion Elicitation using Films," Cognition and Emotion, vol. 9, pp. 87-108, 1995.

[19] G. Chanel, et al., "Short-term emotion assessment in a recall paradigm," International Journal of Human-Computer Studies, vol. 67, pp. 607-627, 2009.

[20] N. Roberts and J. T. J. Coan, "Emotion Elicitation Using Dyadic Interaction Tasks," in The Handbook of Emotion Elicitation and Assessment J. A. Coan and J. J. B. Allen, Eds., ed: Oxford University Press, 2005.

[21] S. Jerritta, et al., "Electrocardiogram-based emotion recognition system using empirical mode decomposition and discrete Fourier transform," Expert Systems, pp. n/a-n/a, 2013.

[22] C. Bunluechokchai and T. Leeudomwong, "Discrete Wavelet Transform -based Baseline Wandering Removal for High Resolution Electrocardiogram," International Journal of Applied Biomedical Engineering, vol. 3, pp. 26-31, 2010.

[23] C. Das and M. H. Khondekar, "Statistical Investigation of ECG Signal of Sleep Apnea Patient," Global Journal of Researches in Engineering, 2011.

[24] C. K. Peng, et al., "Long-Range Anticorrelations and Non-Gaussian Behavior of the Heartbeat," Physical Review Letters, vol. 70, pp. 1343-1346, 1993.

[25] C. Maaoui and A. Pruski, "Emotion Recognition through Physiological Signals for Human-Machine Communication," presented at the Cutting Edge Robotics 2010, Vedran Kordic (Ed.), 2010.

854