Abstract—The implementation of the emotion reasoning

systems to these days has been on software level; the software

evaluates emotional states based on the amount of changes in

biological signals. Therefore, the memory consumption and

computational overload would increase with the number of

parameters used, and the overall effect would be an increased

the response time needed to acquire the needed emotion

reasoning results. In this paper, we go over the design of a

One-Chip hardware reasoning block dedicated to emotion

based applications that are capable of processing multiple,

real-time information, and the design an emotion reasoning

system utilizing this chip. Our system provides an emotion

reasoning API for the interaction between sensing and

reasoning blocks so that the hardware emotion reasoning device

can be easily utilized by emotion based service applications. In

addition, it is possible to manufacture accessory devices for

smartphones based on downsized reasoning block hardware

with USB and Bluetooth interface for use with emotion based

mobile services.

Index Terms—Emotion reasoning, emotion inference.

I. INTRODUCTION

In order to acquire numerical values that can be used to

infer about individuals’ emotional states, sensors that

measure variables such as PPG (Photoplethysmogram),

GSR(Galvanic Skin Response), and SKT(Skin Temperature)

are used [1], [2]. However, individuals’ actual emotional

states are likely to differ even under identical conditions [3],

[4], and environmental factors such as temperature and

humidity affect individuals’ emotions differently. Therefore,

not only the biological data gathered from the mentioned

sensors but also additional factors that could influence

individuals’ emotions should be considered. Diverse types of

data along with the feedback from users’ unique emotional

states can help obtain a more truthful picture of human

emotional states.

An emotion reasoning system can be divided largely two

parts, a sensing block and a reasoning block. The sensing

block utilizes a few sensors to send biological signals

acquired from sensors for PPG, GSR, SKT, HR (heart rate),

etc. from the human body to the reasoning block. Depending

on whether the sensor output values have increased,

decreased, or remained the same, the reasoning block then

makes an evaluation of the current emotional state based on a

Manuscript received January 25, 2013; revised March 25, 2013. This

work was supported by the IT R&D program of MKE/KEIT.

[2009-S-014-01, On the development of Sensing based Emotive Service

Mobile Handheld Devices]

The authors are with the Wireless Convergence Platform Research Center,

Korea Electronics Technology Institute, Seoul, Korea (e-mail: busytom@

keti.re.kr, solim@keti.re.kr, ycpark@keti.re.kr, bhpark34@keti.re.kr,

tipsiness@gmail.com).

rule-base table. Hence, the number of entries in the rule-base

table in the reasoning block increases with the number of

parameters needed for making decision, and this degrades the

computational speed.

In this paper, we discuss the design and implementation for

an emotion reasoning system based on One-Chip integral

emotion reasoning hardware. By constructing the reasoning

block on a single hardware device, one can achieve lower

memory consumption and faster computational speed. Since

the one-chip emotion reasoning hardware can be refined for

use in smartphones and smartpads as an accessory device,

this design can form the basis for diversification of emotion

based contents.

II. EMOTION REASONING

In emotion reasoning systems, the sensing block collects

biological data and transmits to the reasoning block. In

software-based systems, the reasoning block receives the

data collected in the sensing block, processes it as necessary,

and defines the current emotional state as one of the

following: pleasant, unpleasant, aroused, relaxed, pleasantly

aroused, pleasantly-relaxed, unpleasantly aroused,

unpleasantly-relaxed, or neutral. Additionally, parameters

such as environmental temperature that could affect

emotional states can be used to further define emotional

Fig. 1. Emotion reasoning results.

Fig. 2. Basic rule-base table.

For emotion reasoning, a rule-based table [6], [7]

pertaining to sensed data value decrease or increase is used.

Say there are four parameters being used, namely PPG, GSR,

A Design and Implementation of an Emotion Reasoning

Chip Based Emotion Reasoning System

Yong-Seok Lim, Seung-Ok Lim, Young-Choong Park, Byoung-Ha Park, and Joongjin Kook

131DOI: 10.7763/LNSE.2013.V1.29

states [5]. The final resulting evaluation is shown in Fig. 1.

Lecture Notes on Software Engineering, Vol. 1, No. 2, May 2013

Fig. 3. Compound rule-base table.

sin _ 3sen g paramsThe Number of Entries Pos Time THI

(1)

When each field uses 1 byte of memory, one entry takes up

8 bytes and the size of the entire table is 11644bytes

(11.4KB). The price of including the rule-base table into the

emotion based application program is an increase in the size

of the program and a drop in the computational speed.

III. ONE-CHIP HARDWARE BASED EMOTION REASONING

SYSTEM

Fig. 4. Emotion reasoning system architecture.

The computational overload of an emotion reasoning

system is lower when the reasoning block, the essential

component of the system, is made into hardware instead; the

emotion-based application can use an API to send the

sensor-measured data to the reasoning block and then receive

the result from the reasoning block. Moreover, since the

Fig. 5. USB-type one-chip emotion reasoning device.

However, since Android smartphones/pads released in the

market so far do not officially support devices other than

popular USB-based products such as keyboards, mouses,

memory sticks, it is necessary to modify the kernel to run

hardware based emotion reasoning system. Galaxy S2 and S3,

the smartphones used in this paper, use a customized Linux

kernel source from Samsung Electronics. One can use the

One-Chip emotion reasoning device on these models by

modifying the kernel setting so that it provides USB function

Fig. 6. Kernel setting for USB-to-Serial communication.

IV. EMOTION REASONING DEVICE AND APIS

An emotion based application should have the capability to

collect and process data from an external sensing device. The

application should also periodically send a query containing

the processed data and additional information to the emotion

132

SKT, and HR. Each parameter can have three possible stored

values: increase (+1), decrease (-1), no change (0). Such

rule-base table can be laid out as Fig. 2. Depending on how

much information is being considered for emotion reasoning,

the number of entries in the table can increase dramatically as

shown in Fig. 3.

Newly introduced parameters in Fig. 3 are position, time,

and temperature. States of position is categorized as either

indoor or outdoor, and of time as AM, PM, and late night.

States of temperature is categorized into level 1, 2, and 3. The

number of entries in the rule-base table is calculated as

follows:

rule-base table is included in the reasoning block, the size of

the emotion-based application minimized as well. Our

One-Chip hardware based reasoning device has a microUSB

interface so that it can be easily utilized by smartphones or

smartpads. The overall structure is shown in Fig. 4. In Fig. 4,

emotion related applications use Bluetooth to receive data

from the sensing device and communicates with the

reasoning block using USB or Bluetooth.

An USB-type emotion reasoning device can be connected

to the smartphones/pads in the form of an USB accessory,

and a Bluetooth emotion reasoning device can be connected

to the smartphones/pads wirelessly. Fig. 5 shows the

USB-type emotion reasoning device.

needed for communication. Fig. 6 shows the result of

modifying the kernel setting for USB-to-Serial

communication.

Lecture Notes on Software Engineering, Vol. 1, No. 2, May 2013

reasoning engine. The application should receive the result

from the engine when the computation is complete. These

functions realized as an emotion API make it possible for

emotion based applications to utilize the reasoning engine.

TABLE I: EMOTION REASONING APIS

Function Description

BioDevConfig() Configure Bluetooth for use with sensing

device

BioDevConnect() Connect using Bluetooth with sensing

device

BioDevDisconnect() Disconnect Bluetooth connection to

sensing device

BioDevRecv() Receive biological signal

EREngInit() Initialize emotion reasoning engine

EREngConnect() Connect emotion reasoning engine

EREngDisconnect() Disconnect emotion reasoning engine

EREngSendQuery() Send query for emotion reasoning

EREngRecvResult() Receive emotion reasoning result

EREngSendERTable() Send rule-base table

EREngUpdateERTable() Update rule-base table

In order to test our emotion reasoning system’s operations,

one needs a sensing device to measure and transfer biological

signals to the application. The application needs to convert

the sensing data into the format required by the emotion

reasoning engine before sending it to the engine, and the

engine in return sends the emotion results to the application.

In order to process such procedures, we can use the emotion

API laid out in Table I. The general procedure for emotion

reasoning is shown in “Fig. 7”.

Fig. 7. Emotion reasoning procedure.

Fig. 8. Emotion reasoning system make up and Android applications.

“Fig. 8” and “Fig. 9” show the realized the emotion

reasoning system using the One-Chip emotion reasoning

hardware device and emotion API, and the demonstration of

emotion reasoning.

In case of an emotion reasoning system realized purely on

software level on Galaxy S2 and S3, the CPU occupancy

level is around 24.7%. One-Chip hardware reasoning block,

on the other hand, shows an occupancy level of around 5.3%.

Using software-based emotion reasoning applications on

Smartphones/pads that support multitasking not only causes a

significant burden on the system but also shortens battery life,

defeating the purpose of commercialization.

Fig. 9. Emotion reasoning results using one-chip emotion reasoning device

and emotion API.

V. CONCLUSION

Emotion reasoning systems can be divided into two parts, a

sensing block and a reasoning block, and the sensing block

collects and transfers data to the reasoning block. In this

scheme, the size of the rule-base table increases with the

number of parameters and this decreases the computational

speed. In this paper, we showed how the reasoning block can

be separated from the application by utilizing a One-Chip

hardware emotion reasoning device so that the memory

consumption and computation are kept low. The hardware

reasoning device has USB and Bluetooth interface,

facilitating its use as an accessory device in

smartphones/pads. This type of device can be widely applied

to mobile applications targeted at emotion related services.

ACKNOWLEDGMENT

This work was supported by the IT R&D program of

MKE/KEIT. [2009-S-014-01, On the development of

Sensing based Emotive Service Mobile Handheld Devices]

REFERENCES

[1] R. W. Picard, Affective Computing, Cambridge, MA: The MIT Press,

1997.

[2] N. Sebe, I. Cohen, T. Gevers, and T. S. Huang, “Multimodal

approaches for emotion recognition: a survey,” in Proc. the SPIE:

Internet Imaging, 2004, pp. 56-67.

[3] M. Whang, “The Emotional Computer Adaptive to Human Emotion,”

Probing Experience, Philips Research Book Series, vol. 8, pp. 209-219,

2008.

[4] M. Whang and J. Lim, “A Physiological Approach to Affective

Computing,” Affective Computing: Focus on Emotion Expression,

Synthesis and Recognition, In-Tech Education and Publishing, pp.

309-318, 2008.

[5] Y. Park, B. Park, K. Choi, K. Jung, and S. Kim, “A Development of

Human Emotion Interaction Platform in Smart Environments,” in Proc.

the 2nd International Conference onComputers, Networks,

Systems,and Industrial Applications, 2012, pp. 571-575.

[6] M. Black and Y. Yacoob, “Tracking and recognizing rigid and

non-rigid facial motions using local parametric models of image

motion,” in Proc. International Conference on Computer Vision, 1995,

pp. 374-381.

[7] Y. Yacoob and L. Davis, “Recognizing human facial expressions from

long image sequences using optical flow,” IEEE Transactions on

Pattern Analysis and Machine Intelligence, vol. 18, no. 6, pp. 636-642,

1996.

133

Lecture Notes on Software Engineering, Vol. 1, No. 2, May 2013

134

Yong-Seok Lim received his B.S. degree in Electronics

Engineering from Korea University, South Korea, in

2001 and his M.S. degree in Ultra Large Scale

Integration Digital System from Korea University in

2003. He is currently a senior researcher of Wireless

Convergence Platform Research Center of the Korea

Electronics Technology Institute. His areas of expertise

include wireless/network SOC system architecture and

wireless power transfer.

Young-Choong Park received his B.S., M.S., and

Ph.D. degrees in Computer Engineering from SeJong

University, South Korea, in 1999, 2001, and 2010. He is

currently a senior researcher of Realistic Media

Platform Research Center of the Korea Electronics

Technology Institute. His areas of expertise include

embedded software, wireless sensor network,

smart-learning, and mobile platforms.

Byoung-Ha Park received his B.S. and M.S. degrees

in Computer Engineering from SeJong University,

South Korea, in 1999, and 2001. He is currently a

senior researcher of Realistic Media Platform Research

Center of the Korea Electronics Technology Institute.

His areas of expertise include embedded software,

wireless sensor network, smart-learning, and mobile

platforms.

Joongjin Kook received his B.S. and M.S. degrees in

Computer Engineering from KwangWoon University,

South Korea, in 2005, and 2007, and his Ph.D. degree

in School of Computer from Soongsil University in

2012. He is currently a Post-Doc. of Realistic Media

Platform Research Center of the Korea Electronics

Technology Institute. His areas of expertise include

embedded software, wireless sensor network,

smart-learning, and mobile platforms.

Seung-Ok Lim respectively received B.S., M.S., and

Ph.D. degrees in Electrical Engineering from KonKuK

University, Seoul, Korea, in 1997, 1999, and 2005. He

is now a director at the Wireless Convergence Platform

Research Center in KETI (Korea Electronics

Technology Institute), Seoul, Korea. His research

interests include wireless communication systems for

harsh environments such as underground and

underwater environments, and wireless power transfer systems based on

magnetic coupling mechanism.

Lecture Notes on Software Engineering, Vol. 1, No. 2, May 2013