Affective Mood Mining Through Deep Recurrent Neural Network

Md. Golam Rabiul Alam, Sarder Fakhrul Abedin, Seung Il Moon, Ashis Talukder, Anupam Kumar

Bairagi, Md. Shirajum Munir, Saeed Ullah , Choong Seon Hong

Department of Computer Science and Engineering, Kyung Hee University

{robi, saab0015, moons85,ashis, anupam, munir, saeed}@khu.ac.kr, cshong@khu.ac.kr

Abstract

Affective computing is becoming a pioneer research domain to understand human’s emotion through scientific

methods. From genome sequence to face recognition and from neuroimaging to social-post mining each of this

domain tries to use their scientific methodology to recognize, realize and predict human’s affective state. The

pen and paper-based affective state determination methods are not so accurate and impressive therefore due to

the advancement of intelligent technology researchers are trying to apply some intelligent learning methods to

realize individuals affective state. This research uses biosensors data to realize humans’ affective state.

Humans’ psychophysiological data is collected through Electroencephalogram (ECG), Electro-Dermal

Activity (EDA), Electromyography (EMG) and Photoplethysmogram (PPG) and analyzed those data using the

deep recurrent neural network to determine affective mood. Here, based on Russell’s circumplex four primary

affective mood i.e. Joy, Sad, Surprise, and Disgust is considered for realization. The benchmark DEAP dataset

is used to analyze the performance of the proposed method. The higher accuracy in classification of the primary

affective mood justifies the performance of the proposed method.

1. Introduction

The human mind is so complex and still remains unexplored

black-box to the scientific community. According to

neuroimaging research, the brain of a human consists 100 billion

neurons and glial cells and they form a well-structured

communication network among themselves, and therefore

pinpointing the neurons, which are responsible for which mental

state still remains haze [1]. However, one of the pioneering

research by J. D. Haynes et al. [2], disclosed the fact and

feasibility of decoding the mental state of humans such as

conscious experience and covert attitude by evaluating non-

invasive fMRI signals. Apart from that, H. L. Niculescu et al. [3]

successfully classified the blood biomarkers, which is a state-of-

the-art method to recognize suicidality in the patients with mood

disorder. There is some research on affective computing for

determining affective state as an emotional state from face

recognition [4]. The authors of [5] derive the emotional state from

social network post. However, face recognition based, social

network post based or tradition questionnaire based emotion

recognition has many flaws. Because sometimes facial

expression cannot express the internal affective state, individuals’

are not posting on the social network in every time, and people

may hide information in case of questionnaire-based emotion

recognition. Therefore, biosensor’s physiological information is

relatively more appropriate to analyze humans’ affective state.

The DEAP [6] is a pioneer research in analyzing affective state

from physiological data. In that research, the authors’ used 45

channels biosensors data including EEG, EMG, ECG, PPG, GSR

[7] etc. and used simple Naïve Bayes classification to analyze

affective mood. In contrast, we used only four channels of

biosensors i.e. Electroencephalogram (ECG), Electro-Dermal

Activity (EDA), Electromyography (EMG) and

Photoplethysmogram (PPG) to classify affective state. In this

research, we determine the Valence and Arousal level of a human

subject while watching the video stimuli of different affective

state. Based on the Valence and Arousal level, the affective state

is determined according to Russell’s circumplex [] as shown in

Fig. 1.

2. Deep Recurrent Neural Network (RNN) based Affective

Mood Mining

Biosensors signals, which are collected from the subject are

Fig. 1. Simplified Russell’s mood circumplex [3]

2017년 한국소프트웨어종합학술대회 논문집

501

continuous and sequential data signals. There are several machine

learning methods for processing sequential data like HMM,

MEMM, MRF, and CRF. However, the deep recurrent neural

network has the capability to work with the massive dataset.

Therefore, the deep recurrent neural network (RNN) is used to

classify the affective moods of biosensor’s signal.

The human mood mostly depends on the current environment,

facts, stimulus and past emotional states. Therefore, for

predicting affective mood we need to consider the current input

and past states. In RNN model, the long short-time memory

(LSTM) has the capability to memorize the near past and derive

the dependency from the previous states and current biosensors

observations. The used RNN model with LSTM architecture is

presented in Fig. 2.

Fig. 2. LSTM based RNN model for mood mining. The inputs are the biosensor observations and outputs are the

valence and arousal level induced by the video stimuli to determine the mood of the user.

The Long Short-Term Memory (LSTM) helps the model to hold

long-term dependencies. The forget gate layer 𝐹𝑡 determines

what information we’re going to throw away from the cell state

as determine using (1)

The input gate layer it helps to decide which values we’ll update

is derived from (2).

Also, the tanh layer creates a vector of new candidate values

Ĉ𝑡to store in the cell state as derive in (3).

Then update the old cell state 𝐶𝑡−1 to new cell state 𝐶𝑡

according to (4).

Later, we run a sigmoid layer which decides what parts of the cell

state we’re going to output as derived from (5).

Then, we put the cell state through tanh to push the values to be

between -1 to +1, as shown in (6).

Finally, the softmax function as in (7) or normalized exponential

function highlights the largest values and suppress values which

are significantly below the maximum value to determine the

accurate valence and arousal level with highest values.

3. Experimental Results

The deep RNN based affective mood classification method, we

applied in DEAP [6] dataset.

3.1. Dataset Description

The dataset DEAP is an open access (required registration and

authorization) standard dataset used in many emotion analysis

research. To prepare the dataset authors’ were placed 45 recorded

channels at 512Hz, among them 32 EEG channels, 12 peripheral

channels, and 1 status channel. Out of the 45 channel, we used

the data of four channels i.e. Electroencephalogram (ECG),

Electro-Dermal Activity (EDA), Electromyography (EMG) and

𝐶𝑡 = 𝐹𝑡 ∗ 𝐶𝑡−1 + 𝑖𝑡 ∗ Ĉ𝑡 (4)

𝐹𝑡 = 𝜎(𝑊𝑓 . [𝐻𝑡−1, 𝑋𝑡) + 𝑏𝑓 (1)

𝑖𝑡 = 𝜎(𝑊𝑖. [𝐻𝑡−1, 𝑋𝑡) + 𝑏𝑖 (2)

Ĉ𝑡 = 𝑡𝑎𝑛ℎ(𝑊𝐶 . [𝐻𝑡−1, 𝑋𝑡) + 𝑏𝐶 (3)

𝑂𝑡 = 𝜎(𝑊𝑜. [𝐻𝑡−1, 𝑋𝑡) + 𝑏𝑜 (5)

𝐻𝑡 = 𝑂𝑡 ∗ 𝑡𝑎𝑛ℎ(𝐶𝑡) (6)

𝑌𝑇 = 𝑠𝑜𝑓𝑡𝑚𝑎𝑥(𝑊𝑓 . 𝐻𝑇) (7)

2017년 한국소프트웨어종합학술대회 논문집

502

Photoplethysmogram (PPG). Signals are taken from the subjects’

body surface of the above mentioned 45 channels while watching

the emotional video clips. There were 32 volunteers watched 40

of the music video clips. Using the Likert scale the participants

rated all the video clips during the experiment. Valence, arousal,

dominance, and liking were rated directly after each trial on a

continuous 9-point scale.

The length of each video clip is 2 minutes. Total 8064 samples

from each channel are taken from the attached body sensors while

watching each of the 2 minutes video clip. Therefore, the total

data size for our experiment is 32x40x4x8064 of 32 participants,

40 video clips, 4 channels, and 8064 samples data points.

3.2. Performance study

The classification accuracy of four primary affective moods is

determined in the form of confusion matrix as presented in Table

1. According to the confusion matrix, the overall accuracy is 89%.

Table 1: Confusion Matrix of affective Mood

Classification (Unit: %)

Joy Surprise Sad Distressed

Joy 22 3 0 0

Surprise 2 20 0 0

Sad 0 0 24 2

Distressed 1 2 1 23

Average

Accuracy 88% 80% 96%

92%

Overall

Accuracy 89%

4. Conclusion

Wearable internet of things market is rapidly growing especially

for health status monitoring and athletics training. Affective

mood recognition from wearable biosensors can complement

context aware recommendation, mood stabilization, stress and

depression management, especially for mental well-being. The

accuracy of the proposed deep RNN based affective mood mining

method is better but not impressive. The discriminative feature

extraction and more deep architecture may enhance the accuracy

of the proposed method.

Acknowledgement

This research was supported by the MSIT(Ministry of Science

and ICT), Korea, under the Grand Information Technology

Research Center support program (IITP-2017-2015-0-00742)

supervised by the IITP(Institute for Information &

communications Technology Promotion)" *Dr. CS Hong is the

corresponding author.

References

[1] Alam, Md Golam Rabiul and Cho, Eung Jun and Huh, Eui-

Nam and Hong, Choong Seon, “Cloud-based mental state

monitoring system for suicide risk reconnaissance using wearable

bio-sensors,” Proceedings of the 8th International Conference on

Ubiquitous Information Management and Communication, ACM,

pp. 56, 2014.

[2] J.D. Haynes and G. Rees, “Decoding mental states from brain

activity in humans”, Nature Reviews Neuroscience, vol. 7,

pp. 523–534, July 2006.

[3] H. L. Niculescu, et al., "Discovery and validation of blood

biomarkers for suicidality", Molecular psychiatry, vol. 18,

no. 12, pp. 1249-1264, Aug. 2013.

[4] Siddiqi, Muhammad Hameed and Alam, Md Golam Rabiul

and Hong, Choong Seon and Khan, Adil Mehmood and

Choo, Hyunseung, “A Novel Maximum Entropy Markov

Model for Human Facial Expression Recognition,” PloS one,

vol. 11, no. 9, 2016.

[5] Ahmed A. A. Esmin; Roberto L. De Oliveira Jr., Stan Matwin,

“Hierarchical Classification Approach

to Emotion Recognition in Twitter,” 11th International

Conference on Machine Learning and Applications, vol. 2,

pp. 381-385, 2012.

[6] Sander Koelstra, Christian M¨uhl, Mohammad Soleymani,

Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry

Pun, Anton Nijholt and Ioannis Patras, “DEAP: A Database

for Emotion Analysis Using Physiological Signals,” IEEE

Transaction on Affective Computing, vol. 3, no. 1, pp. 18-31,

2012.

[7] Md Golam Rabiul Alam,, Rim Haw, Sung Soo Kim, Md Abul

Kalam Azad, Sarder Fakhrul Abedin, and Choong Seon

Hong. "EM-psychiatry: An Ambient Intelligent System for

Psychiatric Emergency." IEEE Transactions on Industrial

Informatics, vol. PP, No. 99, 2016.

[8] Russell JA. “A circumplex model of affect”, Journal of

Personality and Social Psychology, vol. 39, No. 6, 1980.

[9] Md Golam Rabiul Alam, Sarder Fakhrul Abedin ,

Moshaddique Al Ameen and Choong Seon Hong, " Web of

Objects Based Ambient Assisted Living Framework for

Emergency Psychiatric State Prediction," Sensors, Vol. 16,

No. 9, September 2016.

[10] J. Schmidhuber, Deep learning in neural networks: An

overview, Neural networks, Elsevier, vol. 61, pp. 85-117,

2015.

2017년 한국소프트웨어종합학술대회 논문집

503