i

Master’s Thesis

Image-based Tactile Emojis:

Higher Emotional Clarity in Computer Mediated

Communication(CMC) for the Visually Imparied

Yuri Choi (최 유 리)

Department of Culture Technology

KAIST

2016

ii

Image-based Tactile Emojis:

Higher Emotional Clarity in Computer Mediated

Communication(CMC) for the Visually Imparied

1 iii

Image-based Tactile Emojis:

Higher Emotional Clarity in Computer Mediated

Communication(CMC) for the Visually Imparied

Advisor : Professor Ji-Hyun Lee

by

Yuri Choi Department of Culture Technology

KAIST

A thesis submitted to the faculty of KAIST in partial fulfillment of the re-

quirements for the degree of Master of Science in Engineering in the Depart-

ment of Culture Technology. The study was conducted in accordance with

Code of Research Ethics1

1, 10, 2016 Approved by Professor Ji-Hyun Lee

1Declaration of Ethical Conduct in Research: I, as a graduate student of KAIST, hereby declare that I have not committed any acts that may damage the credibility of my research. These include, but are not limited to: falsi-fication, thesis written by someone else, distortion of research findings or plagiarism. I affirm that my thesis contains honest conclusions based on my own careful research under the guidance of my thesis advisor.

iv

Image-based Tactile Emojis:

Higher Emotional Clarity in Computer Mediated

Communication(CMC) for the Visually Imparied

Yuri Choi

The present dissertation has been approved by the dissertation committee

as a master’s thesis at KAIST

December 9, 2015

Committee head Ji-Hyun Lee

Committee member Woontack Woo

Committee member Juyong Park

- 2 -

DICE

Student ID

최유리. Choi, Yuri.Image-based Tactile Emojis Provide Higher Emotional Clarity in Computer-Mediated-Communication(CMC) for the Visually impaired. 이미지 기반 촉각에모지: 시각장애인의 컴퓨터 매개 커뮤니케이션(CMC)에서의 더 명확한 감정전달에 관한 연구 Department of Culture Technology. 2016. 93 p. Advisor Prof. Lee, Jihyun.

ABSTRACT In order to enhance the missing non-verbal cues in a computer-mediated communication (CMC) using text, the

sighted use emojis or emoticons. Although emojis for the sighted have improved immensely over the years to

contain audio effects and animations, the emojis for the visually impaired still remains underdeveloped. This

study tested how tactile emojis based on visual imagery can enhance the clarity in the CMC environment for the

visually impaired. Based on previous studies that support the fact that the visually impaired (even the

congenitally blind) have their own understanding of visual imagery and can recognize simple objects and facial

expression through previous experience, the tactile emojis are expected to be easier to learn compared to other

tactile services which do not use imagery as a reference for its design. In this study, three things were confirmed:

first, the visually impaired subjects’ ability to connect the emotional emojis to its emotion without any prior

guidance; second, comparing the learnability of imagery-based tactile emoji and non imagery-based tactile

emoji; and lastly, the improvement of clarity when using an emoji with text (braille). Thirty subjects were asked

to complete the user test and as a result, the visually impaired subjects were able to match an average of 4.03 out

of six emotions without prior guidance, and three out of the four subjects who matched perfectly before and after

the guidance were congenitally blind. After the guidance, the imagery-based tactile design had an average of

correct answers that were 4.87 out of six, while the non-imagery based tactile design had an average of 2.19 out

of six, which shows that the imagery-based tactile design where more affective when learning. The clarity of the

sentence also improved. Out of the thirty subjects, only two subjects had different interpretation of the sentence

after the tactile emoji were added to the sentence. This study shows that the imagery-based tactile icons for the

visually impaired can upgrade their texting experience to a whole new level where they can communicate the

subtle emotional cues through tactile imagery. The advancement could minimize the service gap between the

sighted and the visually impaired and offer a much more abundant CMC environment for the visually impaired.

Keywords: Visually Impaired, CMC, Emoji, Instant Messaging (IM), Tactile Feedback

- 3 -

Table of Contents

ABSTRACT ............................................................................................................................ 2

TABLE OF CONTENTS ....................................................................................................... 3

LIST OF FIGURES ............................................................................................................... 5

LIST OF TABLES .................................................................................................................. 6

CHAPTER 1. INTRODUCTION ........................................................................................... 8 1.1 Research Background ................................................................................................... 8 1.2 Research Purpose ......................................................................................................... 9 1.3 Dissertation Structure ................................................................................................. 9

CHAPTER 2. RELATED WORK ........................................................................................ 13 2.1 Understanding and Storing of Images for the Visually Impaired. .......................... 13 2.2 CMC for the Visually Impaired ................................................................................. 14 2.3 Emojis in CMC ............................................................................................................ 16

CHAPTER 3. PRELIMINARY STUDY: EMOJIS TO EMOTIONS SURVEY ................. 23 3.1 Preliminary Study to Identify the Connection Between Emotions to Emojis. ........ 23 3.2 Organize the Facial Features Separately and Convert the Images into Textual

Descriptions. ..................................................................................................................... 27 3.3 Rank the Facial Features by Scoring the Most Chosen Features. .......................... 29

CHAPTER 4. MAKING OF APPARATUS ......................................................................... 33 4.1 Create Image-Based Tactile Emoji Design based on the Previously Defined

Descriptions of the Six Emotions. .................................................................................... 33 4.2 Creating the Tactile Apparatus (Letterpress) .......................................................... 36 4.3 Creating the Tactile Apparatus (3D Printer) ........................................................... 37

CHAPTER 5. INTERVIEW ON IMAGE-BASED TACTILE EMOJIS ............................. 40 5.1 Contents of the Interview .......................................................................................... 40 5.2. Subjects, Time, and Place ......................................................................................... 41

CHAPTER 6. CONCLUSION AND FUTURE WORK ...................................................... 45 6.1 Summary .................................................................................................................... 45 6.2 Contributions .............................................................................................................. 47

- 4 -

6.3 Limitation and Future Work ..................................................................................... 48

APPENDIX .......................................................................................................................... 49

REFERENCES .................................................................................................................... 51

SUMMARY .......................................................................................................................... 58

ACKNOWLEDGEMENT .................................................................................................... 59

- 5 -

List of Figures

Figure 1. Flowchart of the Study Process .......................................................................... 10 Figure 2. Tactile Studies on Remote Communication (A-

Instant Messaging and Haptic Presence Prototype, B- CheekTouch Prototype, C-

InTouch Prototype) ............................................................................................ 15 Figure 3. Patent on Refreshable Braille Display and How Emoticons can be Incorporated16 Figure 4. Advantages of Emojis ......................................................................................... 18 Figure 5. FACS Action Units Chart ................................................................................... 20 Figure 6. Emoji Design Factors ......................................................................................... 23 Figure 7. Apple Emojis and KakaoTalk Emojis ................................................................ 24 Figure 8. The set of emojis used for user survey ............................................................... 25 Figure 9. Demographic of the User Survey ....................................................................... 26 Figure 10. Result Overview of User Survey ...................................................................... 27 Figure 11. Top Ten Selected Emojis for Each Emotions ................................................... 28 Figure 12. Standard Braille Size (U.S. and South Korea) ................................................. 34 Figure 13. Different Resolutions of Emoji Design ............................................................ 34 Figure 14. Contoured and Filled Version of Emoji Design ............................................... 35 Figure 15. Final Tactile Emoji Design ............................................................................... 35 Figure 16. Detailed Tactile Design Description ................................................................ 36 Figure 17. Letterpress Apparatus ....................................................................................... 37 Figure 18. Final Apparatus. 3D Printed Image-based Tactile Design ............................... 37 Figure 19. Final Apparatus 3D Printed Image-based Tactile Design ................................ 38 Figure 20. Visually Impaired Subjects Using Tactile Emoji for the User Test ................. 42

- 6 -

List of Tables

Table 1. Ten Most Selected Emoji Features Categorized and Put into Textual Descriptions

............................................................................................................................. 29 Table 2. Emoji Facial Feature Description in Words ........................................................ 30 Table 3. Description of Participants ................................................................................... 42

- 7 -

1 ____________________________________________________________________________________

Introduction

1.1 Research Background

1.2 Research Purpose

1.3 Dissertation Structure

- 8 -

Chapter 1. Introduction

1.1 Research Background

As the Internet and mobile phones evolved over the years, technology to improve remote communication has

developed immensely. According to the Pew Institute survey, it is clearly shown that mobile phone users

preferred text messaging over phone calls and the number of monthly text messages in the U.S. have

increased from 14 billion in 2000 to 188 billion in 2010 and is still growing (Pew Internet & American Life

Project, 2010).

In a computer-mediated communication (CMC) such as text messaging, a cues-filtered-out approach and

social presence theory assume that it lacks nonverbal communication cues and fails to communicate emotions

and attitudes to receivers (Sauer, 2000). Also, prior studies showed that viewing online text without emoticons

(emotion + icons) caused misinterpretation of the nature of the message and the sender’s attitude (Lo, 2008).

Many researchers have suggested that written communication can be enhanced through visual cues in the

same way visual or body language supports verbal communication (Derks et al., 2008; Rezabek & Cochenour,

1998). During CMC, which has grown to be so common nowadays, to fill in the missing socio-emotional

context, users attach emoticons in their text messages (Tossell, 2012). It is stated in studies that emoticons can

provide the missing information and enhance CMC (Derks et al., 2008). These visual cues have been noted as

the primary way to express emotion in CMC (Riva, 2002) and a way to replace non-verbal communications

when face-to-face (F2F) is not possible (Walther & D’Addario, 2001). Emoticons are defined as ‘graphic

representations of facial expressions that are embedded in electronic messages’ by Walther and D’Addario

(2001). In this paper, the term ‘emoji’ is used instead of ‘emoticons’ for the commonly known distinction of

the two terms is that emoticons are punctuations combined on the keyboard such as the smiley face :), or a

wink ;D, while emojis are depicted as actual pictures such as these: ☺☹ and the final tactile design used in this

paper represents imagery more than combination of punctuation marks. Emojis nowadays are used

worldwide in countries that use electronic devices to communicate. As Apple created its Apple Color Emoji

providing support for Unicode emojis, it has been used in all iOS and OS X devices and many more. Japan

and Korea, countries with the largest mobile market in the world, use Line and Kakaotalk for their messaging

platforms. It is now said that there are over twenty-two million emojis in KakaoTalk and Line, and

KakaoTalk counted that out of all the messages with emojis in it, emojis took up 15% of the conversation. In

2012, the daily sales of paid emojis were approximately one hundred million won (MBC News, 2014).

- 9 -

Throughout this vast development and change in online communication, the visually impaired has often been

left out. While companies such as Apple, KakaoTalk, and Google are continuously developing technologies

for the visually impaired, most of this is based on audio guidance that is still incomplete and uncomfortable in

the sense that it does not provide any privacy and cannot be easily heard in a noisy environment. To prove

the inconvenience of the lack of non-verbal cues or the currently provided service, there have been a one-

person protest infront of KakaoTalk in 2014 where a visually impaired person proclaimed that they should

provide more variety of audio emoticons for equality. Also through personal interviews, the visually impaired

shared experiences in which they experienced misunderstanding with the other party for there were no

emotional implication in the text message.

Studies on emotional cues using tactile delivery have also been progressing such as Hapticons (Rovers & Van

Essen, 2004), InTouch (Brave & Dahley, 1997), CheekTouch (Park et al., 2010), or vSmileys (Mathew, 2005),

where each of which uses vibration patterns, person-to-person haptic responses, and vibrotactile displays.

However, as most of these were not developed for the visually impaired, it still cannot not provide the ultimate

solution for the visually disabled people’s underdeveloped texting experience.

1.2 Research Purpose

The goal of this study, is to prove that image-based tactile emojis can become the solution for future texting

experience for the visually impaired. Also based on prior studies that indicate the congenially blind people’s

ability to comprehend facial expressions, the fact that even the congenitally blind, who supposedly has no

prior visual knowledge as the sighted, are capable of understanding the emojis that the sighted people use with

a short introduction will be tested.

1.3 Dissertation Structure

In order to achieve this goal, several steps were followed:

1. Conduct preliminary study (survey) to identify the connection between emotions to emojis.

2. For the ten most selected emojis for each emotions, organize the facial features separately into eyes,

- 10 -

eyebrows, and mouth (like descriptions in EMFACS) and convert the images into textual descriptions.

3. Rank the separately organized facial features by scoring the most chosen features.

4. Create image-based tactile emoji design based on the previously defined descriptions of the six emotions.

5. Conduct interview using the apparatus and analyze the results.

The more specific chart are as below:

Figure 1. Flowchart of the Study Process

For the first part, for there were no studies that defined the connection between emotions to emojis, a

preliminary test was conducted in order to look into how the users connected the two concepts. For the test,

the emotions to be defined and the emoji sets to be used in the preliminary test were selected. Next, the result

from the preliminary test was analyzed, using EMFACS(Emotional Facial Action Coding System) as a model.

Like the EMFACS, the facial features from the ten most selected emojis from each emotions were seperatly

- 11 -

rated and the images were converted into descriptive words. Thirdly, the descriptive words were used to

create the actual image-based emoji tactile apparatus using the 3D printer. Lastly, the apparatus was used to

conduct interviews with the visually impaired to test whether the subjects can connect the emoji to emotion

without any guidance, to test the learnability compared to the non-visual tactile emoji, and to test its

effectiveness of improving the emotional understanding of the sentence with and without the emoji.

- 12 -

2 ____________________________________________________________________________________

Related Work

2.1 Understanding and Storing of Images for the Visually Impaired

2.2 CMC for the Visually Impaired

2.3 Emojis in CMC

- 13 -

Chapter 2. Related Work

2.1 Understanding and Storing of Images for the Visually Impaired.

2.1.1 The Visually Impaired and Raised Line Drawings

Many studies have proven that the visually impaired have different approach to understanding images

through touch when compared to the sighted. Révész assumed that the haptic and vision function differently

because in haptic, there is active and passive touch. And when touch is generally active and intentional, it

leads to an intension of form that is more cognitive than perceptual (Révész, 1934, 1950). When a sighted

individual touches an object, most of the time, it is to manipulate the object, whereas when the individual is

visually impaired, they often touch for pattern perception (Heller & Gentaz, 2014). Therefore, the blind

individuals are more suited to use their sense of touch for developing a keener sense in recognizing imagery

compared to the sighted who rely more on their vision to understand the object geometry, spatial relations,

and mobility. More supporting facts were found when methods from neuroscience were used to understand

the sense of touch. The methods involved brain imaging and transcranial magnetic stimulation (TMS) –

which lets a researcher turn off a part of a brain to see which functions (cognitive or perceptual) are affected–

and it was found that the occipital lobe of congenitally blind individuals were involved when processing

Braille, which did not happen for the sighted subjects who were blind-folded (Yuasa et al., 2006). This meant

that for the congenitally blind, the visual cortex was used to understand the sense of touch, even when it

comes to understanding text in braille. Also, if any shortcomings of the visually impaired not being as skilled

as the sighted in recognizing 2D drawings is found, Heller states that the cause to be more from the flaw of

education rather than the absence of visual imagery or visual experience (Heller & Gentaz, 2014). In most

studies which looks into the visually impaired’s understanding of 2-dimensional pictures, raised line drawings

are used. A line drawing is not like normal visual stimuli, for it does not depict planes, and are made-up

(Gibson, 1966, 1979), but people’s perception of the world generally cannot be veridical in any case and the

visual illusions cannot be taken out(Heller & Schiff, 2013). It is also argued that the untrained blind subjects

can recognize raised line drawings of ordinary objects and have the same pictorial abilities and uses the same

principles as the sighted (D’Angiulli et al., 1998).

- 14 -

2.1.2 The Visually Impaired and Facial Expression

It has been proven in studies that the congenitally blind have the understanding of facial expressions even

when they have not seen an actual face before. In Picard’s study (2011), fifteen sighted and fifteen congenitally

blind subjects were asked to classify raised line drawings of facial expressions, and whereas the accuracy did

not vary significantly, the blind adults were faster with the task (Lebaz et al., 2011). The study stated the

reason for this result may be because the shape and location of the features fit with the tactile and the

proprioceptive knowledge that the blind have of their own facial expression of basic emotions. To support this

fact, it is said that the understanding and recognition of the facial expression is innate rather than culturally

encrypted and that it is possibly known from birth (Darwin et al., 1998) for young children uses the emotion

read from their caregiver to decide how to handle different situations (Russell, & Fernández-Dols, 1997).

2.2 CMC for the Visually Impaired

2.2.1 Audio Services in CMC for the Visually Impaired

One of the major reasons for the increase of text messaging is related to its benefit of privacy regardless of the

time, place, and occation. The fact that it can only be seen by the receiver is an appealing factor for people

nowadays. For the visually impaired, most of the smartphone technologies are related to sound such as

Apple’s advanced screen reader and KakaoTalk’s audio emoticons, but the problem with messaging with an

audio output is that it does not yet provide the privacy that naturally comes with visual text messaging, and

when there is a lot of surrounding noise, it is difficult to hear. Furthermore it is dangerous for the visually

impaired who cannot see to use earphones.

Another aspect of audio output of emojis is that it cannot provide the visual quality that the tactile and visual

share. There are two types of audio output where one uses sounds that goes well with the image, for instance,

a smiling face will be translated into the sound “haha~”, and for the other type, the sound will describe the

image, for instance, “a girl is holding a heart and smiling.” The first one communicates the emotion directly

but does not get the image across, and the second one has its limitation that it cannot deliver the instant

emotion that an image can give. Also they are both difficult for a congenitally blind person to imagine the

image. Although the audio has its own benefits when filling in the missing emotional cues, it still has many

aspects where it can improve.

- 15 -

Although there have been many attempts to evolve the text messaging experience into a multisensory

experience using audio, tactile, and tactile-vibro, the texting experience for the visually impaired still has a

long way to go. As attempts to improve the nonverbal aspect of texting experience for the sighted are

progressing at a fast pace, this study has hopes to improve the texting experience for the visually impaired by

creating a new way to get across their non-verbal signals in CMC.

2.2.2 Tactile Studies in CMC which Can be Used for the Visually Impaired

Studies on haptic have been a rising trend for recent years and many studies related to remote

communication has also been done in various ways. The vSmiley (Mathew, 2005) is a method developed to

communicate emotion through different vibration patterns, inTouch (Brave & Dahley, 1997) uses force

feedback for haptic interpersonal communication. CheekTouch (Park et al., 2010) gives a vibro-tactile

feedback to the cheeks while speaking on the mobile phone. The term, Hapticon (haptic + icons), is usually

used to refer to the vibro-tactile feedback used in instant messaging. These methods give the users a

multimodal texting experience and the feeling of intimacy. However, these methods were not exclusively

developed for the visually impaired and there still lies controversy on the issue of how the emotions and the

vibration patterns can be connected objectively.

Figure 2. Tactile Studies on Remote Communication (A- Instant Messaging and Haptic Presence Prototype,

B- CheekTouch Prototype, C- InTouch Prototype)

- 16 -

There are not many studies on in text messaging using tactile feedback, but there is a patent on refreshable

haptic braille display and how the emoticons can be incorporated in the system (U.S. Patent No. 717,756,

2012). The method does not have any logical rule to it, and the rules must be learned from the beginning but

the intention was to be able to incorporate a system for emoticons inside the current braille system.

Figure 3. Patent on Refreshable Braille Display and How Emoticons can be Incorporated

2.3 Emojis in CMC

2.3.1 Emotional Communication in CMC

As mentioned before, the main purpose of an emoji is to supplement the exchange of social information by

providing additional social cues beyond what is found in the text (Thompson & Foulger, 1996) and serve the

same role as non-verbal signals in an F2F conversation (Derk et al., 2008). The three roles of nonverbal signals

in a F2F conversation is said to be: (a)providing information; (b)regulating interaction; and (c)expressing

intimacy (Ekman & Friesen, 1969; Harrison, 1973), and as a compensation to overcome the visual

shortcomings of CMC due to its lack of visibility, and thus reduced social presence, (Derks et al., 2008) the use

of emojis grew emensely over the years. Line Inc., one of the biggest text messaging platform in Korea, stated

that the portion that emojis takes up in the conversations is known to have well gone over twenty percent.

Also, as more users used emojis, the related business has grown to make 500 billion won in the past three

- 17 -

years (JoongAng Ilbo, 2015; CNBC, 2013). It has been proven in studies that text messages which used

emoticons worked better when interpreting the precise meaning of the sender (Lo, 2008; Gajadhar & Green,

2005). When Trend Monitor (2014) asked 1,000 subjects between 19 to 59 years-old on their thought on

emojis, 52.4% of the subjects stated that online communication that uses emojis are the norm these days and

the main reason for using emojis were related to enjoyment, variety and exaggeration in expression, softening

negative expressions, concealing embarrassment and awkwardness, and saving time. The growth in the emoji

industry proves the need and potential for advancement and the spread of emojis.

There have been several studies on how the faces are depicted in emoticons and differences were found in the

preference of horizontal or vertical designs or its depiction of eyes and mouth according to the user's culture

or the language they speak (Park et al., 2013; Yuki et al., 2007). While these factors were not considered when

emojis were designed by various designers, common components still exists to serve as the key to making the

emoji become used in its particular way and mean what it means. The fact that emojis have common

meaning to its users were proven in several studies that used emoticons as classifiers for sentiment analysis

(Pak & Paroubek, 2010; Go et al., 2009; Read, 2005) and how the emojis are actively used as an

communication tool in CMC.

2.3.2 Use of Emojis and Emoticons in CMC

As Trend Monitor(trendmonitor.co.kr), a surveying organization in South Korea, stated after asking 1,000

people between 19 to 59 years-old on their thought on mobile emojis, 64.9% users expressed that it was more

enjoyable to use emojis in a conversation and more than 52.4% thought that online communication that uses

emojis are the norm these days. The reason for using emojis were as in Figure 4. Most of the reasons were

related to the enjoyment and adding more fun and others equally important reasons were for the variety and

exaggeration in expression, softening the negative expressions, concealing embarrassment and awkwardness,

and saving time. (Trend Monitor, 2014)

- 18 -

Figure 4. Advantages of Emojis

Also the situations where people mostly used emojis were when they wanted to express affection (81.3%),

when they wanted to lighten the mood (79.8%), and was also used when they wanted to look good to the

other person by appearing trendy (75.2%) or appearing cute (71.5%). It was also used when they wanted to

graphically show their facial expression (70.4%), and express regret for what they did (69.1%). There was also

a high number for when there is not much to say (66.8%)(Trend Monitor, 2014). KakaoTalk reported that

out of all the messages with emojis in it, emojis took up 15% of the conversation. (Joongang Daily News, 2014)

When the reasons for success in KakaoTalk and Line - the two largest texting platforms in Korea and Japan –

is discussed, the effect of emojis (i.e. ‘Emoticons’ in KakaoTalk and ‘Mobile Stickers’ in Line) cannot be

factored out. In 2012, the daily sales of paid emojis were approximately one hundred million won. (MBC

News, 2014) The portion that emojis business takes up in the Naver’s Line business is known to have well

gone over twenty percent and as more users used emojis, the related business has grown up to five hundred

billion won in past three years (MBC News, 2014). It is now said that there are over twenty-two million emojis

in KakaoTalk and Line. Line has eight times more character than KakaoTalk which goes up to 5600

characters with each character having 40 expressions.

- 19 -

Seven out of ten people replied that there is potential in the mobile sticker industry(69.1%) and that there will

be increasing amount of users for mobile stickers (70.7%). Also majority (64.1%) of the people thought that

there will be increase in the users globally as well and that advertisement using mobile stickers will increase

(60.6%) (Trend Monitor, 2014).

2.3.4 FACS and EMFACS

In this study, emojis which depict the human facial expression (smiley emojis) were focused on for they show

the most emotional clarity. For the reference, studies on the emotional interpretation of human facial

expressions were looked into. The systems of FACS (Facial Action Coding System) (Ekman & Friesen, 1977)

provides an objective means of measurement for the facial muscle contractions when a facial expression is

made. FACS combines three different approaches which involves holistic spacial analysis based on principle

components of graylevel images; feature measurement using local image features such as wrinkles; and optic

flow, but since emojis are consisted of simplified components, and because this study will only use static emojis

without movement, the sequential and time aspect of the FACS were not considered and only the description

of the facial muscles that consist each emotions were looked into.

FACS later branches out to the EMFACS (Emotional Facial Action Coding System) (Friesen & Ekman, 1983)

which shows how the movement of facial muscles connect to emotions. FACS and EMFACS both uses the six

basic emotions (Batty & Taylor, 2003) – happiness, sadness, surprise, fear, anger, and disgust – by Ekman, for

it was both developed by him. And as seen in Figure 5., The Action Units are divided into upper and lower

face and includes visual descriptions on the eye lids, eyebrows, nose, cheek, mouth, chin and more.

- 20 -

Figure 5. FACS Action Units Chart

For the EMFACS, All emotions are connected with numbers of Action Units (AUs) and the description of

EMFACS are as follows:

EMFACS (Emotional Facial Action Coding System)

• Happiness : Cheek Raiser + Lip Corner Puller

• Sadness: Inner Brow Raiser + Brow Lowerer + Lip Corner Depressor

• Surprise: Inner Brow Raiser + Outer Brow Raiser + (Slight) Upper Lid Raiser + Jaw Drop

• Fear: Inner Brow Raiser + Outer Brow Raiser + Brow Lowerer + Upper Lid Raiser + Lid Tightener

+ Lip Stretcher + Jaw Drop

- 21 -

Anger: Brow Lowerer + Upper Lid Raiser + Lid Tightener + Lip Tightener

Disgust: Nose Wrinkler + Lip Corner Depressor + Lower Lip Depressor

- 22 -

3 ____________________________________________________________________________________

Preliminary Study: Emojis to Emotions Survey

3.1 Understanding and Storing of Images for the Visually Impaired

3.2 Organize Facial Features Separately

- 23 -

Chapter 3. Preliminary Study: Emojis to Emotions Survey

3.1 Preliminary Study to Identify the Connection Between Emotions to Emo-jis.

3.1.1. Visual Elements in Emoji-Smileys

As it has been implied previously, there have been no studies that disassembled or looked into what visual

factors of an emoji caused it to hold the meaning in which the users use them as. For there are no former

studies, The basic elements in an emoji were form, color, size, and placement. Most of the time, size and color

were used to intensify the emotion the shape represented and the core factor that determined the emotion was

the shape of facial features and actions on the face. But in this study, only the facial features such as the

mouth, eyes, and eyebrows and the iconic shape of the heart was used to communicate the emotion. First the

color was difficult to deliver using tactile, and actions on the face such as an exaggerated sweat, nose air, or

veins on the temple that represented anger, were often taken from the expression from Japanese manga which

may be difficult for the visually impaired to understand instantly.

Figure 6. Emoji Design Factors

- 24 -

3.1.2 Select smiley emojis to use for the preliminary study.

In order to conduct the preliminary study, an emoji set had to be selected. Between the Apple Colored

Emoji and Kakao Talk Emojis, the Apple Colored Emoji was selected. The two emoji sets are both familiar to

the mobile users of South Korea, for Apple Colored Emoji is universally used and is the basic emoji for all

Apple smartphones, and as Kakao Talk is the biggest texting platform in South Korea, Kakao Talk Emojis

are also widely used. Out of these two options, the Apple Colored Emoji was chosen for the Kakao Talk

Emojis had a more decorative quality which could effect the outcomes of the survey, and the Apple Colored

Emoji set had more variety of smiley designs of similar shapes of mouth or eyes which were easier to analyze

when the results were out.

Next, out of the various sets of emojis – such as nature, places, symbols and more – the smileys were

selected for they focused on depicting facial expressions which has the most direct emotional delivery. Because

the focus of this process was to examine the shape of the facial features, emojis which concealed or did not

depict the eyes or the mouth were eliminated. The symbol mark of the heart was added for the heart was

always located at the top of ‘Most Frequently Used Emoji’ list and seemed essential to the set.

Figure 7. Apple Emojis and KakaoTalk Emojis

- 25 -

Figure 8. The set of emojis used for user survey

3.1.3 Select 6 (+ 1) emotions to use for the preliminary study.

For the emotions to use in the preliminary test, six emotional adjectives were taken from Ekman’s six basic

emotions for it has been widely used in the studies of FACS and many other studies related to emotional

delivery. In an addition, the emotion of ‘affection’, was included for the most frequently used emoji often

contained the shape of the heart which widely symbolizes the emotion of ‘affection’. As a result, seven

emotional adjectives – happy, angry, disgust, sad, surprised, fear, and love – were used in the survey.

3.1.4 Conduct preliminary study in order to categorize the selected emojis to the selected seven emotions.

After the selection of emotions and an emoji set, a preliminary test to categorize the formerly chosen emojis

into the seven emotions was conducted. The survey was conducted in order to see how the emoji users

connected each emotion to emojis. For the preliminary study, the seven emotional adjectives selected from the

former process were asked to be connected with the chosen section of Apple Color Emojis.

The subjects were asked to select as many emojis they thought that applied to the emotion they were trying to

- 26 -

express. For an example, for the emotion ‘happy’, the question asked “Which emoji will you use if you want

to express ‘happy’?”, and this applied to all seven emotions.

The user test was conducted online on ‘surveymonkey.com’ and 205 subjects who are familiar with emojis

from different gender and age groups picked as many emojis which seemed to apply to the emotion. Out of

the 205 replies, only the completed 190 were used in the result. As seen in Figure 9., there were a total of 115

female users’ replies and 75 male replies out of 190, and for the different age groups, 118 replies were from

ages 20s to 30s users, 70 replies from ages 40s~50s users, and 1 replies each from the teenage group and 60s

and more.

Figure 9. Demographic of the User Survey

For the overall result, as seen in Figure 10., because the question allowed multiple answers, each emotions had

different number of responses where ‘Happy’ had the most response of 1075 and ‘Disgust’ had the least of

425. ‘Angry’ and ‘Love’ showed a definite contrast in number for the ones that were chosen more and the

ones that were least chosen. The most chosen three emojis for ‘Angry’ had over 70% users who agreed upon

the answers where the percentage dropped to 19% on the fourth emoji, and the most chosen three emojis for

‘Love’ each had 90%, 74%, 66%, but dropped to 24% on the fourth emoji, which shows that the first three

emojis were agreed upon strongly. ‘Happy’, ‘Sad’, ‘Surprised’, and ‘Fear’ had a more distributed amount of

choice for the first few emojis, but had emojis which had more than half of the subjects agreeing upon.

‘Disgust’, which had relatively little agreement on which emojis represented the emotion and 13% of the

- 27 -

subjects claiming that there were no emojis that represented the emotion, was eliminated from the set.

Figure 10. Result Overview of User Survey

3.2 Organize the Facial Features Separately and Convert the Images into Textual Descriptions.

The result was first put into a chart where the most selected top ten emojis could be seen together with the

total percentage of the number of subjects which agreed upon the specific emoji. As seen in Figure 11.,

although they are all top ten emojis, the percentage of agreement differs depending on the emotion. Through

this process, the emotion ‘disgust’ had to be eliminated, for the total number of selection was very weak and

14 percent of the subjects felt that there were no emojis that applied to the emotion. For the rest of the emojis,

although they differ in the percentage of selection, the answers showed noticeable differences in the overall

selection and thus were altogether used as a data.

- 28 -

Figure 11. Top Ten Selected Emojis for Each Emotions

Next, all the facial features which appeared in the ten most selected emoji features were categorized and

converted into textual descriptions. This is the same as FACS and EMFACS where the descriptions are

written textually. There have been several alterations in doing so, for FACS and EMFACS uses the real

human face, whereas this study uses an emoji which is different from the real face in terms of omission,

exaggeration, and simplification.

EMFACS uses descriptions in the lid, upper lid, inner brow, outer brow, cheek, nose, lip, lower lip, and jaw,

emojis have far simpler feature which are basically the eyes, eyebrows, and mouth. Because of this difference,

some descriptions had to be converted. For example, the ‘cheek raiser’ in ‘happiness’ was reassigned as

‘crescent shaped eyes with the curves facing up’, because when the cheek raises upward, the eyes

automatically closes into forming a curve. Also, with its simplified graphic lines, some descriptions such as

‘inner brow raiser and outer brow raiser’ can be combined as ‘brow raiser’ and ‘jaw drop and lip stretcher’

could be combined as ‘upper lip raiser’. The descriptions focused more on the final appearance of the face

rather than the action it went through. However some of the descriptions in EMFACS such as ‘lip corner

raiser’ or ‘lip corner depressor’ stayed the same, for they were already directly stating the appearance.

- 29 -

Table 1. Ten Most Selected Emoji Features Categorized and Put into Textual

Descriptions

In Table 1., the emoji features which were categorized together, and how they were described in text, can be

seen.

3.3 Rank the Facial Features by Scoring the Most Chosen Features.

3.3.1 Score the features to see the importance of each feature.

In order to see the relevance of the importance of each feature, the numbers of the percentage of people who

agreed upon the feature when expressing each emotion were added up. For example, if the ‘brow lowerer’ for

the emotion ‘angry’ appeared three times and for each time it was selected as the percentage of ‘75%’, ‘74%’,

‘70%’ out of the total survey respondent, the numbers were added up and became 219 to differentiate the

significance of the feature.

As seen in Table 2., the added up weight of each features of eyes and eyebrows, and mouth were put in with

the highest number on top, and were compared with the EMFACS to see how it matched with the results.

Although some descriptions from EMFACS showed weakly in the preliminary study results such as the ‘inner

brow raiser and outer brow raiser’ for ‘fear’, almost all of the descriptions from EMFACS applied to the

results of the preliminary study results. One part that EMFACS could not cover was ‘tears’ for sadness, for

EMFACS only covered the muscle movements on the face, and ‘tears’ are a relatively extreme reaction to the

emotion.

- 30 -

Table 2. Emoji Facial Feature Description in Words

3.3.2 Write the final description for the eyes, eyebrows, and mouth.

Taken from the final description, the EMFACS for emojis came out as follows.

EMFACS for Emojis

• Happy: Crescent shaped eyes + Mouth Corner Raiser

• Sad: Brows Corners Depressor + Tears + Mouth Corner Depressor

• Surprised: Round Eyes + Brow Raiser + Round Mouth

• Fear: Round Eyes + Brows Corners Depressor + Upper Lip Raiser

- 31 -

• Angry: Eyelid Tightner + Brow Corners Raiser + Mouth Corner Depressor

• Love: Heart Shaped Eyes + Mouth Corner Raiser

- 32 -

4 ____________________________________________________________________________________

Making of Apparatus

4.1 Creating Image-Based Tactile Design based on Previously Defined Descriptions of the

Six Emotions.

4.2 Creating Tactile Apparatus (Letterpress)

4.3 Creating Tactile Apparatus (3D Printed)

- 33 -

Chapter 4. Making of Apparatus

4.1 Create Image-Based Tactile Emoji Design based on the Previously De-fined Descriptions of the Six Emotions.

For the image-based tactile emojis which are finger-sized and uses the sense of touch instead of the vision,

several rules had to be considered when converting the written descriptions of the visual emoji.

l Dot ratio that can also imprint braille.

One of the biggest characteristic of emoji is that it is commonly used together with text (in this case, braille),

the resolution must be considered so that it can depict braille as well. The standard size of braille is as in

Figure 12. The matrix of tactile dots is based on the standard braille cell dimension composed of six dots

arranged in 2 x 3 in order to display letters on it along with emoticons. Fig. 12 shows the dot numbering in a

cell and Korean standard braille cell dimension. Dot base diameter is 1.5mm and distance between two dots

in the same cell is 2.5mm. Distance between corresponding dots in adjacent cells is 4.5mm. Distance between

corresponding dots from one cell directly below is 10 to 10.2 mm. And dot height is around 0.6 to 0.7 mm.

Since the maximum number of braille cells for a single syllable in Korean braille is four, four braille cells in a

line were arranged so that at least one syllable can be displayed at a time.

- 34 -

Figure 12. Standard Braille Size (U.S. and South Korea)

l Resolution which can depict a continuous curved line.

In the pilot test, three resolutions that could cover the braille system were tested out. The lowest resolution

had the closest resemblance of the braille size but could not depict the curvy lines of the heart and the smiling

mouth which affected the comprehension of the emoji. Therefore, the minimum resolution was determined

by its capability to depict curved lines in the finger-sized area.

Figure 13. Different Resolutions of Emoji Design

- 35 -

l Contoured Depiction

The emojis were expressed in line drawing, instead of filled in design for it is easier to decipher contoured

drawings with the tip of the finger. The visually impaired are also accustomed to raised line drawing which

uses the contoured depiction.

Figure 14. Contoured and Filled Version of Emoji Design

l Final Design

The final design was made incorporating all the considerations from the above; a resolution of 17 by 26 dots

that can also apply to the dimensions of a braille and contoured design. Each of the emotional expressions

were designed using the facial feature description from Table 2. The details were kept simple as possible and

distinctive so that it would not confuse the subjects.

Figure 15. Final Tactile Emoji Design

- 36 -

The apparatus was created in 21 by 12 millimeter blocks with 17 by 26 dots tactile emojis imprinted on it.

The dot resolution of 17 by 26 came from the ratio that can also express the braille. The detailed design size

description are as Figure 16. The black part of the designs were protruded in the final apparatus.

Figure 16. Detailed Tactile Design Description

4.2 Creating the Tactile Apparatus (Letterpress)

After several trials, Object Eden350 3D printer was used for the final apparatus. In the beginning of creating

the apparatus, it was created with 3D System’s z450 3D printer with gypsum plaster. It had a high resolution,

but the plaster did not feel good on the skin which could affect the result.

For the second apparatus, letterpress was used for softer feeling to the skin and its closer texture to the braille,

but when it was tried hands-on, the small dots did not protrude as much as it should (as seen in Figure.17) for

the dot distance was too short, and when I looked further into companies that specialized in braille printing, it

came clear that a specialized technology which applies heat from both sides of the paper had to be used, and

the companies who had this kind of technology did not want to take a chance in the experiment which was

only for a small amount of customized production.

- 37 -

Figure 17. Letterpress Apparatus

4.3 Creating the Tactile Apparatus (3D Printer)

Finally, the apparatus was created using the 3D printer, but this time plastic was used instead of gypsum

plaster for better texture. The final apparatus came out as Figure 18 and 19.

Figure 18. Final Apparatus. 3D Printed Image-based Tactile Design

- 38 -

Figure 19. Final Apparatus 3D Printed Image-based Tactile Design

- 39 -

5 ____________________________________________________________________________________

Interview on Image-based Tactile Emojis

5.1 Contents of the Interview

5.2 Subjects, Time and Place

- 40 -

Chapter 5. Interview on Image-based Tactile Emojis

The interview was conducted in three parts and took around 20 to 30 minutes for each subjects. All the

instructions had to be read out and blocks of apparatus was asked to be matched throughout the interview. In

order to make them relaxed during the interview, the word selection was carefully picked so they did not feel

pressured. Word such as ‘test’ ‘time limit’ ‘experiment’ were avoided and for young students, overly formal

language (in Korean) could seem intimidating, so the tone was adjusted to make them comfortable. The

detailed scenario can be found in the appendix.

5.1 Contents of the Interview

5.1.1. Part A. Understanding of Emojis without Former Guidance.

In order to see how naturally understandable the design is, the subjects were asked to match the

emotion of the emoji by touching the finger-sized block apparatus. After telling the subjects which way is up,

and that the eyes and mouth are depicted on the frame (for ‘affection’, I explained that it was a symbol.), the

six emotions were slowly read out to them and were asked to match which seemed closest to the pattern they

were holding. Whether they were right or wrong were not told.

5.1.2 Part B. Learnability of Image-Based Emoji Set and Non-Image-Based Emoji Set.

The image-based tactile emoji which was created for this study was compared against a non-image-

based emoji which comes from a patent. The image-based emoji and the non-image based emoji were both

given to them one by one and after being told in a very concise sentence “This is a wink,” and they had the

chance to feel the shape for fifteen seconds each. After the six shapes were felt, I gave them the blocks one by

one again and asked them to match the emotion they think in which applied to the emoji they were holding

and slowly read out the emotions for reminder.

- 41 -

5.1.3 Part C. Effect Of Emojis When Interpreting the Tone of a Sentence.

For the third part, a strip of paper which had a neutrally understood sentence written on in braille was

handed to the subject, and they were told that this came in a text message. The sentences were emotionally

and intentionally ambiguous. For example, “Today’s menu is curry again.”, “He visited again.”, or “You’re

being silly.” These sentences can have different meanings and intensions depending on the emoji which

followed behind. When the subject finished reading the sentence, three or four options of interpretation were

given to them including “I don’t know.” After the subject answered to the question, an emoji was handed to

them and were told that the emoji came along with the former text. The subjects were asked again on the

interpretation of the sentence and the first and second answers were recorded. Two out of three sentences

were randomly chosen and given out to the subjects, and how their interpretation of the sentence became to

an agreement after the use of emoji was observed.

5.2. Subjects, Time, and Place

The interview was conducted in a quiet room where there were no disturbance in order to conduct the

interview vocally for the visually impaired. All interviews were conducted through cooperation with Daejeon

City Sansung Welfare Center (Taejeon), Braille Library (Taejeon), and a Taejeon Visually Impaired School

(Taejeon), Siloam Welfare Center (Seoul), and Song-pa Visually Impaired Welfare Center (Seoul) and was

also conducted at these places. I asked the centers if there was a quiet room with a table and a chair to sit side

by side with the subjects so it was easier to hand out the apparatus.

The interview was done from September to November, 2015, although the very first pilot test was conducted

in November, 2014. It was difficult to match the institutions’ schedule and it was difficult for them to predict

when many of the subjects had time available so many visits had to be made.

For the visually impaired subjects, thirty subjects from welfare centers, teachers and students from visually

impaired school, and a braille library were gathered. There were 16 male and 14 female subjects, 20 blind

subjects and 10 low-visioned subjects (the ones who can see the tactile braille were eliminated), 20 who lost

their sight after birth and 10 who had low vision or could not see from birth. Five more subjects that were

included in the beginning, had mental health issues that affected the results of the test, or were disturbed by

the interphone announcement which affected the result of the test, and thus, had to be eliminated.

- 42 -

Figure 20. Visually Impaired Subjects Using Tactile Emoji for the User Test

Table 3. Description of Participants

Participant Occupation Education Level Sex Age Degree of Blindness Age at Onset of Blindness

(Years)

I. EARLY BLIND

1 Unemployeed F 43 Very Low Vision 0 2 High School Student F 18 Very Low Vision 0 3 Masseuse College Graduate M 27 Blind 0 4 Unemployeed College Graduate M 26 Blind 0 5 Unemployeed College Graduate F 24 Blind 0 6 Library Employee College Graduate F 35 Very Low Vision 0 7 High School Student F 17 Blind 0 8 Welfare Center Teacher College Graduate M 44 Blind 0 9 University Student College Enrolled F 25 Blind 0

10 Health Keeper; Student on a leave of absence College Enrolled F 22 Very Low Vision 0

II. LATE BLIND

11 Masseuse College Graduate M 44 Blind 12 Welfare Center Employee F 30s Blind

13 Teacher Graduate School Graduate M 42 Blind Late 20s

14 Ex-Principle College Graduate F 56 Blind 3

15 Teacher Graduate School Graduate F 38 Blind High School

16 Elementary Teacher College Graduate M 35 Blind

17 Welfare Center General Manager College Graduate M 58 Blind 48

- 43 -

18 Unemployeed Middle School Dropout M 60

Retinitis pigmentosa ; Very Low Vision; Nyctalopia

19 Unemployeed College Enrolled M 43 Blind 23

20 Unemployeed College Enrolled F 43 Very Low Vision; Retinitis pigmentosa; Can only see contours

21 Unemployeed High School Graduate F 35 Blind; Became Completely Blind last year

22 Housewife College Enrolled F 71 Very Low Vision; Can see the light slightly

23 Church Activity Graduate School Graduate M 42 Very Low Vision 37

24 Private Business College Graduate M 50 Very Low Vision 40 25 Unemployeed High School Graduate M 56 Blind

26 Unemployeed High School Graduate F ff F

53 Very Low Vision 28

27 College Graduate M 55 Blind 53 28 High School Graduate M 40 Blind 30 29 Unemployeed No Education M 43 Blind 31 30 High School Graduate M 70 Blind

- 44 -

6 ____________________________________________________________________________________

Conclusion and Future Work

5.1 Summary

5.2 Contributions

- 45 -

Chapter 6. Conclusion and Future Work

6.1 Summary

6.1.1. The Overall Performance of the Subjects

For Part A., subjects were able to match an average of 4.03 out of 6 emotions without prior guidance, and 3

out of the 4 subjects who matched perfectly before and after the guidance were congenitally blind. The 3

congenitally blind subjects who matched all 6 emotions were the fastest with the task and non-hesitant when

answering the question.

When asked, on what bases they were matching the imagery with the emotions, one subject, who was

completely blind from birth, answered “I used my own face as a reference because the corner of her mouth

pulls up when I’m happy, and my eyes and mouth opens up when I am surprised.” This was congruent with

Picard’s assumption and also Darwin’s study that says that the understanding and recognition of the facial

expression is innate rather than culturally encrypted and that it is possibly known from birth (Darwin, Ekman,

and Prodger, 1998) for young children uses the emotion read from their caregiver to decide how to handle

different situations (Russell, & Fernández-Dols, 1997). Another congenitally blind subject stated “The

straight angular lines seemed angry and felt far from happy, and the curvy lines seemed positive.” In the user

test, there were several remarks about the angle and the characteristics of the line which included the words

‘round’ or ‘pointy.’ In Lundholm’s study in 1921 (Lundholm, 1921), participants were asked to draw lines to

express the affective tone of different adjectives and more angles were drawn for adjectives such as "hard",

"harsh", and "cruel", and more curves were drawn for adjectives such as "weak", " gentle", and "mild." There

seemed to be a connection between the character of the lines that helped interpret the emotion of the emoji.

Lastly, two subjects mentioned that because they know that when depicting a smiley face, caret marks(“^^”)

are used, they assumed that the eyes for “happiness” were similar to that form. It is unknown where they were

able to learn the form of the caret marks, but it was proven that the education and former experience on the

connection of form to its connotation has helped them understand the facial features better. Lastly, three

subjects simply stated that the angry depiction of tactile emoji “just seemed scary.” Many subjects had

difficulty explaining how they matched the emoji to emotion and even the comparably articulate subjects

sometimes did not know when they have learned that the form connected to a specific emotion.

There were no apparent gap between of the subjects ability according to gender, time of sight loss, or level of

- 46 -

blindness, but some subjects became discouraged when they could not tell how to connect the emotion to the

emoji seemed to give up on their effort in exploring the rest of the emojis. Although I tried hard not to give

the subjects any impression of a “test” since I did not want the pressure in anyway affect their performances

when conducting the user test, 4 subjects matched less emotions after my guidance. It is assumed that this

comes from the pressure to match the “correct answer” and trying to remember the clues instead of

instinctively feeling it. As for the first test, the subjects were asked to match the emotional adjective which

seemed right to them with what they were feeling, but for the second test, they were asked to remember the

emotions that were told to them and state the matching emotion when asked which caused some of them to

perform lower than their first try.

After the guidance, the image-based tactile design had an average of correct answers that were 4.87 out of 6,

while the non-image-based tactile design had an average of 2.19 out of 6, which showed the contrast that the

image-based tactile design where far more effective for learning. Many subjects commented that they did not

understand the connection between the non image-based tactile designs and the emotions.

The clarity of the sentence also improved where for the first try for the first sentence, 6 subjects out of the 30

subjects matched the intended answer, when the emoji was attached 26 out of 30 subjects matched the

intended answer and out of the 4 who had the wrong answer was not completely wrong, for the answer

‘affectionate’ and happy’ was a bit similar. are a only 2 subjects had different interpretation of the sentence

after the tactile emoji were added to the sentence.

6.1.2. Design of the Image-based Tactile Emojis

Out of the six image-based tactile emojis, the smile connected to ‘happiness’ and the heart connected to

‘affection’ had the highest scores, where 28 subjects out of 30 subjects got the correct answers. The next easier

emotion was ‘anger’ which had 22 subjects getting correct answers followed by ‘surprise’ and ‘sadness’ which

had 18 subjects with right answers. The lowest correct answer was ‘fear’ which was often confused with

‘sadness’ or ‘surprise’ for it shared the same traits of sloping down eyebrows or big eyes. When thirty sighted

subjects were asked to connect the emoji design with the emotion, they also had difficult time connecting ‘fear’

and even the ones who got the correct answer stated that it was tricky. Also, when conducting the user test,

many subjects tried to matched the tactile design of ‘fear’ by eliminating the other options first and asked

‘which one did I not say yet?’, and when considering that the ‘fear’ was a tricky design from the start, the

subjects who were able to match the emotion may have been better with their ability to remember the visual

tactile images than the others.

- 47 -

The emojis that many subjects mismatched were ‘sadness’ and ‘anger’ which shared the same mouth. This

can mean two things: first, that the subjects mostly uses the mouth to discriminate the emotions; and second,

that the subjects had difficult time processing the difference between the downward shaped eyebrow and

upward shaped eyebrow. They both seem true, for there were subjects who said that they were both the same,

or “This is ‘angry’ because the mouth goes down.” while touching ‘sadness’ or “This definitely seems happy

because of the mouth.”. For the second theory, one of the subject asked ‘Is this going downward?’ when

feeling the brows and another subject described the eyebrows that they are “gathered in the middle and

spreads in the end” and gets the emotion right the first time, but had ‘sadness’ mistaken with ‘anger’ for the

second try.

6.2 Contributions

This study provides a kickoff point for a new CMC experience for the visually impaired and the direction of

design when it can later be produced. The study proves that the visually impaired, even those who are

completely blind from birth, can connect the emoji to emotion and has a brief understanding of the form.

With the previous knowledge of facial expressions, they can easily learn the emojis with a short introduction

and thus makes this method a more accessible means of communication compared to the ones which has to

be learned from the beginning.

The image-based tactile design also provides visual cues to the visually impaired in a context that has never

been done before. For Part C. of the user test where the subjects were to interpret sentences with and without

the emojis, many were amused when the emojis changed the entire interpretation of the sentence. The

experience of image inside a text using tactile and how this changes the entire context of the image usage in

the sense that the image affects the text was a new experience for the subjects. Also, because CMC using

smartphone devices are for daily conversation regardless of time and place, when it becomes available in the

market for wide use, it will provide the visually impaired with a highly accessible image from a brand new

channel that will make their conversations more interesting.

The image-based tactile design, unlike the audio feedbacks that are often used by the visually impaired, can

also provide the privacy that all texting CMCs should support, and it will also be more easier to use in a loud

environment. Assuming that this will be manufactured as a wearable device, using the fingertip to feel and

receive the message will create a more intimate environment to communicate with the other party.

- 48 -

6.3 Limitation and Future Work

For the study, it was difficult to find visually impaired subjects for the user test, and due to the small sample

size, it was difficult to make meaning out of the numbers. In the interview, it seemed that the congenitally

blind were better at matching the emoji to emotion, but because there were only four subjects that were

completely blind from birth, and one subject did not have high scores as the others, the number was too weak

to differentiate with the other subjects.

Also when it comes to actual production of the image-based tactile emojis, the small scale and the resolution it

must depict can become a hindrance. When briefly planning out the final design as a wearable device worn as

a wristband, the device must be braille-size and light-weight. The pins must be refreshable with pins pushing

up and down, and in order to communicate the images as understandable image, it is essential to maintain the

level of resolution as planned in this paper. More studies from the mechanical engineering can be done to

look into the production.

The biggest limitation of for the image-based tactile emoji design is the small scale of the apparatus and its

difficulty to depict detailed images. This can mean that it will face some limitation when producing an

abundant variety of designs. But as the emojis for the sighted started with a simple smiley :-) and a frown :-( ,

the tactile emojis also have the chance of starting as a small set and expanding. Also, using a multisensory

approach, using audio and vibro-tactile along with the image-based tactile emojis may lead to more exciting

results. When gathering the subjects for the user test, it seemed that there were less braille-users than initially

expected, and one subject went as far as to say that they did not need braille since there was audio feedback. If

this is the actual case, a new usage of emojis can be looked into for the visually impaired. However the

limitations, an iconified pictorial display in a CMC is still an interesting direction to look into for the visually

impaired and the sighted as well.

- 49 -

Appendix

Appendix 1. User Test Scenario

Basic Information

Hello, I am Yuri Choi who came for my Master’s dissertation. Nice to meet you. Is it okay for me to record our interview as video or voice recording? Before starting I would like to gather some information.(name, age, occupation, degree of visual impairment, age of loss of sight, familiarity of emojis, if they are using the emoji)

Interview

PART A. Understanding of emojis without former guidance.

“First, there will be eyes and mouth imprinted on this small block I hand you. I will give you six emotional words, so please pick one that you fit best with the block you are holding.” “This is the first block. Which emotion do you think best fits this block. Happy, love, surprised, fear, sadness, anger.” (repeat for all six blocks) “I will not be telling you whether you got it right or wrong”

PART B. Learnability of image-based emoji set and non-image-based emoji set.

“This time, you will feel a different block from the first one and I will tell you which emotion it goes with. While you feel this block for some time, please remember which emotion goes with which block.” “Now I will hand you the block. (hand the block) This is ‘happiness’.” (after giving some time, take away the block and hand another block) “This is ‘wink’.” (repeat for all six blocks) “Now I will give you the emotions so let’s see if you can remember the blocks. This is the first block. Please choose the emotion you think went with this block. Happy, angry, surprised, tongue sticking out, sadness, wink.” (repeat for all six blocks) “Next, we will try the same thing for the first block you used. I will tell you which emotion goes with each blocks as I just did, so please remember which emotion goes with which block. Now I will hand you the block. (hand the block) This is ‘happiness’.” (after giving some time, take away the block and hand another block) “This is ‘sadness’.” (repeat for all six blocks) “Now I will give you the emotions so let’s see if you can remember the blocks. This is the first block. Please choose the emotion you think went with this block. Happy, love, surprised, fear, sadness, anger.” (repeat for all six blocks)

PART C. Effect of emojis when interpreting the tone of a sentence.

“For the last part, I will hand you a strip of paper with braille imprinted on it. When you finish reading, please tell me.” (For the ones who could not read braille, read it out in a monotone voice.) “Now, pretend that this sentence came in a text message from a friend and choose the answer that seems closest to the the friend’s intension or emotion.”

1) For “He visited again.”

“This friend’s emotion is… 1. Annoyed 2. Happy. 3. Affectionatly Excited. 4. (I don’t know.)” “Now, pretend that this emotion came together with the text, (hand out a emoji), now please answer the question again.” (LJ♥) (read out the examples again.)

- 50 -

2) For “Today’s menu is curry again.”

“This friend… 1. likes curry. 2. is tired of curry. 3. (I don’t know.)” “Now, pretend that this emotion came together with the text, (hand out a emoji), now please answer the question again.” (LJ♥) (read out the examples again.)

3) For “You’re being silly.”

“This friend… 1. is being playful 2. is disapproving. 3. (I don’t know.)” “Now, pretend that this emotion came together with the text, (hand out a emoji), now please answer the question again.” (LJ) (read out the examples again.)

- 51 -

References

[1] Abramowicz, A., Klatzky, R. L., & Lederman, S. J. (2010). “Learning and generali-

zation in haptic classification of 2-D raised-line drawings of facial expressions of

emotion by sighted and adventitiously blind observers.” Perception, 39(9), 1261.

[2] Adolphs, R. (2002). “Recognizing emotion from facial expressions: psychological

and neurological mechanisms.” Behavioral and Cognitive Neuroscience Reviews, 1(1), pp.

21-62.

[3] Bartlett, M. S., Viola, P. A., Sejnowski, T. J., Golomb, B. A., Larsen, J., Hager, J.

C., & Ekman, P. (1996). “Classifying facial action.” Advances in neural information pro-

cessing systems, pp. 823-829.

[4] Bassili, J. N. (1979). “Emotion recognition: the role of facial movement and the rela-

tive importance of upper and lower areas of the face.” Journal of personality and social

psychology, 37(11), 2049.

[5] Batty, M., & Taylor, M. J. (2003). “Early processing of the six basic facial emotional

expressions.” Cognitive Brain Research, 17(3), pp. 613-620.

[6] Bosker, B. (2014, June 27). How Emoji Get Lost In Translation. from

http://www.huffingtonpost.com/2014/06/27/emoji-meaning_n_5530638.html

[7] Bourdon, P., Le Clerc, F., & Jouet, P. (2012). U.S. Patent Application 13/717,756.

[8] Brave, S., & Dahley, A. (1997, March). “inTouch: a medium for haptic interperson-

al communication.” In CHI'97 Extended Abstracts on Human Factors in Computing Systems

(pp. 363-364). ACM.

[9] Bryan, T. H. (1977). “Learning disabled children's comprehension of nonverbal

- 52 -

communication.” Journal of Learning Disabilities, 10(8), pp. 501-506

[10] Churches, O., Nicholls, M., Thiessen, M., Kohler, M., & Keage, H. (2014). Emoti-

cons in mind: An event-related potential study. Social neuroscience, 9(2), 196-202.

[11] Darwin, C., Ekman, P., & Prodger, P. (1998). The expression of the emotions in man and

animals. Oxford University Press, USA.

[12] D’Angiulli, A., Kennedy, J. M., & Helle, M. A. (1998). Blind children recognizing

tactile pictures respond like sighted children given guidance in exploration. Scandina-

vian Journal of Psychology, 39(3), 187x-190.

[13] Derks, D., Bos, A. E., & Grumbkow, J. V. (2007). Emoticons and social interaction

on the Internet: the importance of social context. Computers in Human Behavior, 23(1),

842-849.

[14] Derks, D., Fischer, A. H., & Bos, A. E. (2008). The role of emotion in computer-

mediated communication: A review. Computers in Human Behavior, 24(3), 766-785.

[15] Eimer, M. (2000). Event-related brain potentials distinguish processing stages in-

volved in face perception and recognition. Clinical neurophysiology, 111(4), 694-705.

[16] Ekman, P. (2006). Darwin and facial expression: A century of research in review. Ishk.

[17] Ekman, P. (1993). Facial expression and emotion. American psychologist, 48(4),

384.

[18] Ekman, P., & Friesen, W. V. (1977). Facial action coding system.

[19] Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969). Pan-cultural elements in facial

displays of emotion. Science, 164(3875), 86-88.

[20] Emojitracker: Real time emoji use on twitter. (n.d.). Retrieved March 18, 2015,

from http://emojitracker.com/

[21] Friesen, W. V., & Ekman, P. (1983). EMFACS-7: Emotional facial action coding

system. Unpublished manuscript, University of California at San Francisco, 2, 36.

- 53 -

[22] Gajadhar, J., & Green, J. (2005). The importance of nonverbal elements in online

chat. Educause Quarterly, 28(4), 63.

[23] Gibson, J. J. (1966). The senses considered as perceptual systems.

[24] Go, A., Bhayani, R., & Huang, L. (2009). Twitter sentiment classification using dis-

tant supervision. CS224N Project Report, Stanford, 1, 12.

[25] Goldmine Emoticon Markets…Creating Profit through Humorous Pictures (In Ko-

rean)｜ MBC News. (2014, December 8). Retrieved April 5, 2015, from

http://imnews.imbc.com/replay/2014/nwdesk/article/3569182_13490.html

[26] Heller, M., & Gentaz, E. (2014). Psychology of touch and blindness. New York: Psy-

chology Press.

[27] Heller, M. A., Calcaterra, J. A., Burson, L. L., & Tyler, L. A. (1996). Tactual pic-

ture identification by blind and sighted people: Effects of providing categorical in-

formation. Perception & Psychophysics, 58(2), 310-323.

[28] Hu, M., & Liu, B. (2004, August). Mining and summarizing customer reviews. In

Proceedings of the tenth ACM SIGKDD international conference on Knowledge

discovery and data mining (pp. 168-177). ACM.

[29] Itier, R. J., & Taylor, M. J. (2002). Inversion and contrast polarity reversal affect

both encoding and recognition processes of unfamiliar faces: a repetition study using

ERPs. Neuroimage, 15(2), 353-372.

[30] Kelly, C. (2015). Do you know what I mean >:(: A linguistic study of the under-

standing of emoticons and emojis in text messages.

[31] Kim, Minsang. (n.d.). [On the Site] Emoticons which are more various than human

emotions. (In Korean) Retrieved March 17, 2015, from

http://life.joins.com/news/article/article.asp?Total_ID=15426823&ctg=12&sid=6

692.

- 54 -

[32] Lebaz, S., Jouffrais, C., & Picard, D. (2011). Identifying basic emotions in raised-line

pictures of the human face: Do we need vision?. Perception ECVP abstract, 40, 156-

156.

[33] Lenhart, A., Ling, R., Campbell, S., & Purcell, K. (2010). Teens and mobile phones:

Text messaging explodes as teens embrace it as the centerpiece of their communica-

tion strategies with friends. Pew Internet & American Life Project.

[34] Lundholm, H. (1921). The Affective Tone of Lines: Experimental Researches. Psy-

chological Review, 28(1), 43.

[35] Lo, S. K. (2008). The nonverbal communication functions of emoticons in comput-

er-mediated communication. CyberPsychology & Behavior, 11(5), 595-597.

[36] Mathew, D. (2005). vSmileys: Imaging emotions through vibration patterns. Alterna-

tive Access: Feeling and Games.

[37] Maurer, D., Le Grand, R., & Mondloch, C. J. (2002). The many faces of configural

processing. Trends in cognitive sciences, 6(6), 255-260.

[38] McDougald, B. R., Carpenter, E. D., & Mayhorn, C. B. (2011, September). Emoti-

cons What does this one mean?. In Proceedings of the Human Factors and Ergonomics Socie-

ty Annual Meeting (Vol. 55, No. 1, pp. 1948-1951). SAGE Publications.

[39] ‘Mobile Stickers are easier to express negative emotions in difficult conversations.

“Conversations get interesting with Mobile Stickers. (In Korean) (2014, August

22).Retrieved March 29, 2015, from

https://www.trendmonitor.co.kr/tmweb/trend/allTrend/detail.do?bIdx=1200&co

de=0102&trendType=CKOREA

[40] Nelson, C. A. (2001). The development and neural bases of face recognition. Infant

and child development, 10(1-2), 3-18.

- 55 -

[41] Opinion Mining, Sentiment Analysis, Opinion Extraction. (2004, May 15). Re-

trieved April 4, 2015, from http://www.cs.uic.edu/~liub/FBS/sentiment-

analysis.html#lexicon

[42] Ortony, A., Clore, G. L., & Foss, M. A. (1987). The referential structure of the af-

fective lexicon. Cognitive science, 11(3), 341-364.

[43] Pascual-Leone, A., Theoret, H., Merabet, L., Kauffman, T., & Schlaug, G. (2006).

The role of visual cortex in tactile processing: A metamodal brain. Touch and

blindness: Psychology and neuroscience, 171-195.

[44] Pak, A., & Paroubek, P. (2010, May). Twitter as a Corpus for Sentiment Analysis

and Opinion Mining. In LREC (Vol. 10, pp. 1320-1326).

[45] Park, J., Barash, V., Fink, C., & Cha, M. (2013, June). Emoticon Style: Interpreting

Differences in Emoticons Across Cultures. In ICWSM.

[46] Park,Y.W., Lim,C.Y., & Nam,T.J. (2010, April). CheekTouch: an affective interac-

tion technique while speaking on the mobile phone. In CHI'10 Extended Abstracts

on Human Factors in Computing Systems (pp. 3241-3246). ACM.

[47] Picard, D., Jouffrais, C., & Lebaz, S. (2011). Haptic recognition of emotions in

raised-line drawings by congenitally blind and sighted adults. IEEE Transactions on

Haptics, 4(1), 67-71.

[48] Pew Internet & American Life Project (2010). <http://www.pewinternet.org>.

[49] Read, J. (2005, June). Using emoticons to reduce dependency in machine learning

techniques for sentiment classification. In Proceedings of the ACL student research workshop

(pp. 43-48). Association for Computational Linguistics.

[50] Révész, G. (1934). System der optischen und haptischen Raumtäuschungen.

Zeitschrift für Psychologie und Physiologie der Sinnesorgane. Abt. 1. Zeitschrift für Psychologie.

[51] Révész, G. (1950). Psychology and art of the blind.

- 56 -

[52] Rivera, K., Cooke, N. J., & Bauhs, J. A. (1996, April). The effects of emotional icons

on remote communication. In Conference Companion on Human Factors in Computing Sys-

tems (pp. 99-100). ACM.

[53] Rovers, A. F., & van ESSEN, H. A. (2004, April). HIM: a framework for haptic in-

stant messaging. In CHI'04 Extended Abstracts on Human Factors in Computing

Systems (pp. 1313-1316). ACM.

[54] Rovers, L., & Van Essen, H. A. (2004, June). Design and evaluation of hapticons for

enriched instant messaging. In Proceedings of EuroHaptics (Vol. 4, No. 4).

[55] Russell, J. A., & Mehrabian, A. (1977). Evidence for a three-factor theory of emo-

tions. Journal of research in Personality, 11(3), 273-294.

[56] Russell, J. A., & Fernández-Dols, J. M. (1997). The psychology of facial expression. Cam-

bridge university press.

[57] Sauer, J., Schramme, S., & Rüttinger, B. (2000). Knowledge acquisition in ecologi-

cal product design: The effects of computer-mediated communication and elicita-

tion method. Behaviour & Information Technology, 19(5), 315-327.

[58] Tossell, C. C., Kortum, P., Shepard, C., Barg-Walkow, L. H., Rahmati, A., &

Zhong, L. (2012). A longitudinal study of emoticon use in text messaging from

smartphones. Computers in Human Behavior, 28(2), 659-663.

[59] Van Boven, R. W., Hamilton, R. H., Kauffman, T., Keenan, J. P., & Pascual–

Leone, A. (2000). Tactile spatial resolution in blind Braille readers. Neurology, 54(12),

2230-2236.

[60] Walther, J. B., & D’Addario, K. P. (2001). The impacts of emoticons on message in-

terpretation in computer-mediated communication. Social science computer review,

19(3), 324-347.

[61] Yuki, M., Maddux, W. W., & Masuda, T. (2007). Are the windows to the soul the

- 57 -

same in the East and West? Cultural differences in using the eyes and mouth as cues

to recognize emotions in Japan and the United States. Journal of Experimental Social

Psychology, 43(2), 303-311.

[62] Yuasa, M., Saito, K., & Mukawa, N. (2006, April). Emoticons convey emotions

without cognition of faces: an fMRI study. In CHI'06 Extended Abstracts on Hu-

man Factors in Computing Systems (pp. 1565-1570). ACM.

- 58 -

Summary

이미지 기반 촉각에모지:

시각장애인의 컴퓨터 매개 커뮤니케이션(CMC)에서의

더 명확한 감정전달에 관한 연구

일반적으로 CMC(컴퓨터 매개 커뮤니케이션)에서 문자로는 전달이 불가한 비언어적 메시지전달을 위해

에모지, 또는 이모티콘을 사용한다. 에모지는 문자가 더 활성화가 되는 시대의 흐름에 따라 음성 또는

움직이는 애니메이션들이 추가되는 등 많은 발전을 이루어왔으나, 시각장애인들을 위한 에모지는 그

흐름에 따라 크게 발전되지 못하고 기술의 발전에서 소외되었다. 본 연구는 이미지기반의

촉각에모지라는 새로운 대안이 시각장애인을 위한 CMC 환경에서의 의사소통의 명확도를 얼마나

높여줄 수 있고, 그 학습도가 이미지 기반이 아닌 에모지와 비교하였을 때 얼만큼 빨라질 수 있는지를

보기위한 연구이다. 본 연구에서는 3 가지를 확인하였다: 첫째, 학습 없이 시각장애인들이 감정과

에모지를 연결할 수 있는지, 둘째, 이미지기반 촉각 에모지와 이미지기반이 아닌 촉각에모지의 학습도

차이, 세번째, 문장 속에 에모지를 포함하였을 때와 그렇지 않았을 때의 문장의 명확도 차이.

사용자조사에는 30 명의 피험자들이 사용자조사에 참여하였으며, 학습 없이도 평균적으로 6 가지

감정들 중 4.03 개를 맞출 수 있었다. 학습 후에는 이미지기반 촉각에모지의 맞춘 숫자는 4.87 개로

올랐으며, 이미지기반이 아닌 에모지는 학습 후에 2.19 개를 맞출 수 있어, 이미지기반의 에모지가

학습도가 더 높았음을 확인 할 수 있었다. 문장의 명확도 역시 에모지가 있을 때에 더 높아졌음을 볼

수 있었다. 본 연구는 이미지기반 촉각에모지가 시각장애인들에게 활용되기에 학습도가 높으며,

이미지를 통한 더 섬세한 감정적 표현의 장을 열어줌으로써 그들의 CMC 환경을 더 높은 수준으로

향상시켜줄 수 있음을 확인할 수 있었다. 또한, 현재 크게 벌어진 비시각장애인과 시각장애인들의

CMC 환경수준을 조금이라도 좁힐 수 있는 대안 중 하나로 활용될 수 있을 것이다.

Keywords: 시각장애인, CMC, 에모지, 이모티콘, 촉각피드백

- 59 -

Acknowledgement

I would like to thank all the people who have helped me throughout the process of this

study.

First, I give all my thanks to Dr. Taesaeng Choi and Dr. Eunjoo Baik for advising me in

clarifying the objective of this study and giving me helpful advices in the process of this

research. I also express my sincere gratitude to Dr. Ji-Hyun Lee, my advisor for the

encouragement and excellent support. I would like to also express thanks to Hyeryeong Jun

for being a great friend, constantly talking me through the process of this study and making

it clear for me, taking her personal time to help me with the user test and creating the 3D

apparatus.

I would like to also give thanks to the Jong-geun Lee from the school administration office

who helped me use the 3D printer. And my thanks also goes to all the centers and schools

(Daejeon City Sansung Welfare Center, Taejeon Braille Library, Taejeon Visually

Impaired School, Siloam Welfare Center, and Song-pa Visually Impaired Welfare Center

who helped me with the study without cause.

I would not have been able to complete my study without all the help and I am truly

blessed.

- 60 -

Curriculum Vitae

Name: Yuri Choi Date of Birth: 1986/08/15

Education

‘06 ~‘09 Rhode Island School of Design (RISD) Bachelor of Fine Arts, Graphic Design (GPA: 3.6 / 4.0)

’08 Fall Semester, Brown University

Advanced Japanese Class

’07 Summer Studies, Sophia University

Japanese Society, Japanese Art History

Career

’09 Kkotsbom

Editorial Design, Internship

’10 ~’13 Daiso Asung Industries

Package Design, Store R&D Team, Section Manager

‘15~ Team Interface

Service Design Team, Senior Consultant

- 61 -

Exhibitions & Works

‘15 "Robot Essay.", National Museum of Modern and Contemporary Art (MMCA)

Seoul, Korea. Installation (Design Planning)

‘14 "Togetherness, which is Peace.", Ungno Lee Museum of Art

Taejeon, Korea. Interaction Installation (Design Planning)

‘14 “SCAN and GO” Explanation Video, Electronics and Telecommunications Research Institute (ETRI)

Taejeon, Korea. Explanation Video (Design and Production)

-----------------------------------------------------------------------------