i
Master’s Thesis
Image-based Tactile Emojis:
Higher Emotional Clarity in Computer Mediated
Communication(CMC) for the Visually Imparied
Yuri Choi (최 유 리)
Department of Culture Technology
KAIST
2016
ii
Image-based Tactile Emojis:
Higher Emotional Clarity in Computer Mediated
Communication(CMC) for the Visually Imparied
1 iii
Image-based Tactile Emojis:
Higher Emotional Clarity in Computer Mediated
Communication(CMC) for the Visually Imparied
Advisor : Professor Ji-Hyun Lee
by
Yuri Choi Department of Culture Technology
KAIST
A thesis submitted to the faculty of KAIST in partial fulfillment of the re-
quirements for the degree of Master of Science in Engineering in the Depart-
ment of Culture Technology. The study was conducted in accordance with
Code of Research Ethics1
1, 10, 2016 Approved by Professor Ji-Hyun Lee
1Declaration of Ethical Conduct in Research: I, as a graduate student of KAIST, hereby declare that I have not committed any acts that may damage the credibility of my research. These include, but are not limited to: falsi-fication, thesis written by someone else, distortion of research findings or plagiarism. I affirm that my thesis contains honest conclusions based on my own careful research under the guidance of my thesis advisor.
iv
Image-based Tactile Emojis:
Higher Emotional Clarity in Computer Mediated
Communication(CMC) for the Visually Imparied
Yuri Choi
The present dissertation has been approved by the dissertation committee
as a master’s thesis at KAIST
December 9, 2015
Committee head Ji-Hyun Lee
Committee member Woontack Woo
Committee member Juyong Park
- 2 -
DICE
Student ID
최유리. Choi, Yuri.Image-based Tactile Emojis Provide Higher Emotional Clarity in Computer-Mediated-Communication(CMC) for the Visually impaired. 이미지 기반 촉각에모지: 시각장애인의 컴퓨터 매개 커뮤니케이션(CMC)에서의 더 명확한 감정전달에 관한 연구 Department of Culture Technology. 2016. 93 p. Advisor Prof. Lee, Jihyun.
ABSTRACT In order to enhance the missing non-verbal cues in a computer-mediated communication (CMC) using text, the
sighted use emojis or emoticons. Although emojis for the sighted have improved immensely over the years to
contain audio effects and animations, the emojis for the visually impaired still remains underdeveloped. This
study tested how tactile emojis based on visual imagery can enhance the clarity in the CMC environment for the
visually impaired. Based on previous studies that support the fact that the visually impaired (even the
congenitally blind) have their own understanding of visual imagery and can recognize simple objects and facial
expression through previous experience, the tactile emojis are expected to be easier to learn compared to other
tactile services which do not use imagery as a reference for its design. In this study, three things were confirmed:
first, the visually impaired subjects’ ability to connect the emotional emojis to its emotion without any prior
guidance; second, comparing the learnability of imagery-based tactile emoji and non imagery-based tactile
emoji; and lastly, the improvement of clarity when using an emoji with text (braille). Thirty subjects were asked
to complete the user test and as a result, the visually impaired subjects were able to match an average of 4.03 out
of six emotions without prior guidance, and three out of the four subjects who matched perfectly before and after
the guidance were congenitally blind. After the guidance, the imagery-based tactile design had an average of
correct answers that were 4.87 out of six, while the non-imagery based tactile design had an average of 2.19 out
of six, which shows that the imagery-based tactile design where more affective when learning. The clarity of the
sentence also improved. Out of the thirty subjects, only two subjects had different interpretation of the sentence
after the tactile emoji were added to the sentence. This study shows that the imagery-based tactile icons for the
visually impaired can upgrade their texting experience to a whole new level where they can communicate the
subtle emotional cues through tactile imagery. The advancement could minimize the service gap between the
sighted and the visually impaired and offer a much more abundant CMC environment for the visually impaired.
Keywords: Visually Impaired, CMC, Emoji, Instant Messaging (IM), Tactile Feedback
- 3 -
Table of Contents
ABSTRACT ............................................................................................................................ 2
TABLE OF CONTENTS ....................................................................................................... 3
LIST OF FIGURES ............................................................................................................... 5
LIST OF TABLES .................................................................................................................. 6
CHAPTER 1. INTRODUCTION ........................................................................................... 8 1.1 Research Background ................................................................................................... 8 1.2 Research Purpose ......................................................................................................... 9 1.3 Dissertation Structure ................................................................................................. 9
CHAPTER 2. RELATED WORK ........................................................................................ 13 2.1 Understanding and Storing of Images for the Visually Impaired. .......................... 13 2.2 CMC for the Visually Impaired ................................................................................. 14 2.3 Emojis in CMC ............................................................................................................ 16
CHAPTER 3. PRELIMINARY STUDY: EMOJIS TO EMOTIONS SURVEY ................. 23 3.1 Preliminary Study to Identify the Connection Between Emotions to Emojis. ........ 23 3.2 Organize the Facial Features Separately and Convert the Images into Textual
Descriptions. ..................................................................................................................... 27 3.3 Rank the Facial Features by Scoring the Most Chosen Features. .......................... 29
CHAPTER 4. MAKING OF APPARATUS ......................................................................... 33 4.1 Create Image-Based Tactile Emoji Design based on the Previously Defined
Descriptions of the Six Emotions. .................................................................................... 33 4.2 Creating the Tactile Apparatus (Letterpress) .......................................................... 36 4.3 Creating the Tactile Apparatus (3D Printer) ........................................................... 37
CHAPTER 5. INTERVIEW ON IMAGE-BASED TACTILE EMOJIS ............................. 40 5.1 Contents of the Interview .......................................................................................... 40 5.2. Subjects, Time, and Place ......................................................................................... 41
CHAPTER 6. CONCLUSION AND FUTURE WORK ...................................................... 45 6.1 Summary .................................................................................................................... 45 6.2 Contributions .............................................................................................................. 47
- 4 -
6.3 Limitation and Future Work ..................................................................................... 48
APPENDIX .......................................................................................................................... 49
REFERENCES .................................................................................................................... 51
SUMMARY .......................................................................................................................... 58
ACKNOWLEDGEMENT .................................................................................................... 59
- 5 -
List of Figures
Figure 1. Flowchart of the Study Process .......................................................................... 10 Figure 2. Tactile Studies on Remote Communication (A-
Instant Messaging and Haptic Presence Prototype, B- CheekTouch Prototype, C-
InTouch Prototype) ............................................................................................ 15 Figure 3. Patent on Refreshable Braille Display and How Emoticons can be Incorporated16 Figure 4. Advantages of Emojis ......................................................................................... 18 Figure 5. FACS Action Units Chart ................................................................................... 20 Figure 6. Emoji Design Factors ......................................................................................... 23 Figure 7. Apple Emojis and KakaoTalk Emojis ................................................................ 24 Figure 8. The set of emojis used for user survey ............................................................... 25 Figure 9. Demographic of the User Survey ....................................................................... 26 Figure 10. Result Overview of User Survey ...................................................................... 27 Figure 11. Top Ten Selected Emojis for Each Emotions ................................................... 28 Figure 12. Standard Braille Size (U.S. and South Korea) ................................................. 34 Figure 13. Different Resolutions of Emoji Design ............................................................ 34 Figure 14. Contoured and Filled Version of Emoji Design ............................................... 35 Figure 15. Final Tactile Emoji Design ............................................................................... 35 Figure 16. Detailed Tactile Design Description ................................................................ 36 Figure 17. Letterpress Apparatus ....................................................................................... 37 Figure 18. Final Apparatus. 3D Printed Image-based Tactile Design ............................... 37 Figure 19. Final Apparatus 3D Printed Image-based Tactile Design ................................ 38 Figure 20. Visually Impaired Subjects Using Tactile Emoji for the User Test ................. 42
- 6 -
List of Tables
Table 1. Ten Most Selected Emoji Features Categorized and Put into Textual Descriptions
............................................................................................................................. 29 Table 2. Emoji Facial Feature Description in Words ........................................................ 30 Table 3. Description of Participants ................................................................................... 42
- 7 -
1 ____________________________________________________________________________________
Introduction
1.1 Research Background
1.2 Research Purpose
1.3 Dissertation Structure
- 8 -
Chapter 1. Introduction
1.1 Research Background
As the Internet and mobile phones evolved over the years, technology to improve remote communication has
developed immensely. According to the Pew Institute survey, it is clearly shown that mobile phone users
preferred text messaging over phone calls and the number of monthly text messages in the U.S. have
increased from 14 billion in 2000 to 188 billion in 2010 and is still growing (Pew Internet & American Life
Project, 2010).
In a computer-mediated communication (CMC) such as text messaging, a cues-filtered-out approach and
social presence theory assume that it lacks nonverbal communication cues and fails to communicate emotions
and attitudes to receivers (Sauer, 2000). Also, prior studies showed that viewing online text without emoticons
(emotion + icons) caused misinterpretation of the nature of the message and the sender’s attitude (Lo, 2008).
Many researchers have suggested that written communication can be enhanced through visual cues in the
same way visual or body language supports verbal communication (Derks et al., 2008; Rezabek & Cochenour,
1998). During CMC, which has grown to be so common nowadays, to fill in the missing socio-emotional
context, users attach emoticons in their text messages (Tossell, 2012). It is stated in studies that emoticons can
provide the missing information and enhance CMC (Derks et al., 2008). These visual cues have been noted as
the primary way to express emotion in CMC (Riva, 2002) and a way to replace non-verbal communications
when face-to-face (F2F) is not possible (Walther & D’Addario, 2001). Emoticons are defined as ‘graphic
representations of facial expressions that are embedded in electronic messages’ by Walther and D’Addario
(2001). In this paper, the term ‘emoji’ is used instead of ‘emoticons’ for the commonly known distinction of
the two terms is that emoticons are punctuations combined on the keyboard such as the smiley face :), or a
wink ;D, while emojis are depicted as actual pictures such as these: ☺☹ and the final tactile design used in this
paper represents imagery more than combination of punctuation marks. Emojis nowadays are used
worldwide in countries that use electronic devices to communicate. As Apple created its Apple Color Emoji
providing support for Unicode emojis, it has been used in all iOS and OS X devices and many more. Japan
and Korea, countries with the largest mobile market in the world, use Line and Kakaotalk for their messaging
platforms. It is now said that there are over twenty-two million emojis in KakaoTalk and Line, and
KakaoTalk counted that out of all the messages with emojis in it, emojis took up 15% of the conversation. In
2012, the daily sales of paid emojis were approximately one hundred million won (MBC News, 2014).
- 9 -
Throughout this vast development and change in online communication, the visually impaired has often been
left out. While companies such as Apple, KakaoTalk, and Google are continuously developing technologies
for the visually impaired, most of this is based on audio guidance that is still incomplete and uncomfortable in
the sense that it does not provide any privacy and cannot be easily heard in a noisy environment. To prove
the inconvenience of the lack of non-verbal cues or the currently provided service, there have been a one-
person protest infront of KakaoTalk in 2014 where a visually impaired person proclaimed that they should
provide more variety of audio emoticons for equality. Also through personal interviews, the visually impaired
shared experiences in which they experienced misunderstanding with the other party for there were no
emotional implication in the text message.
Studies on emotional cues using tactile delivery have also been progressing such as Hapticons (Rovers & Van
Essen, 2004), InTouch (Brave & Dahley, 1997), CheekTouch (Park et al., 2010), or vSmileys (Mathew, 2005),
where each of which uses vibration patterns, person-to-person haptic responses, and vibrotactile displays.
However, as most of these were not developed for the visually impaired, it still cannot not provide the ultimate
solution for the visually disabled people’s underdeveloped texting experience.
1.2 Research Purpose
The goal of this study, is to prove that image-based tactile emojis can become the solution for future texting
experience for the visually impaired. Also based on prior studies that indicate the congenially blind people’s
ability to comprehend facial expressions, the fact that even the congenitally blind, who supposedly has no
prior visual knowledge as the sighted, are capable of understanding the emojis that the sighted people use with
a short introduction will be tested.
1.3 Dissertation Structure
In order to achieve this goal, several steps were followed:
1. Conduct preliminary study (survey) to identify the connection between emotions to emojis.
2. For the ten most selected emojis for each emotions, organize the facial features separately into eyes,
- 10 -
eyebrows, and mouth (like descriptions in EMFACS) and convert the images into textual descriptions.
3. Rank the separately organized facial features by scoring the most chosen features.
4. Create image-based tactile emoji design based on the previously defined descriptions of the six emotions.
5. Conduct interview using the apparatus and analyze the results.
The more specific chart are as below:
Figure 1. Flowchart of the Study Process
For the first part, for there were no studies that defined the connection between emotions to emojis, a
preliminary test was conducted in order to look into how the users connected the two concepts. For the test,
the emotions to be defined and the emoji sets to be used in the preliminary test were selected. Next, the result
from the preliminary test was analyzed, using EMFACS(Emotional Facial Action Coding System) as a model.
Like the EMFACS, the facial features from the ten most selected emojis from each emotions were seperatly
- 11 -
rated and the images were converted into descriptive words. Thirdly, the descriptive words were used to
create the actual image-based emoji tactile apparatus using the 3D printer. Lastly, the apparatus was used to
conduct interviews with the visually impaired to test whether the subjects can connect the emoji to emotion
without any guidance, to test the learnability compared to the non-visual tactile emoji, and to test its
effectiveness of improving the emotional understanding of the sentence with and without the emoji.
- 12 -
2 ____________________________________________________________________________________
Related Work
2.1 Understanding and Storing of Images for the Visually Impaired
2.2 CMC for the Visually Impaired
2.3 Emojis in CMC
- 13 -
Chapter 2. Related Work
2.1 Understanding and Storing of Images for the Visually Impaired.
2.1.1 The Visually Impaired and Raised Line Drawings
Many studies have proven that the visually impaired have different approach to understanding images
through touch when compared to the sighted. Révész assumed that the haptic and vision function differently
because in haptic, there is active and passive touch. And when touch is generally active and intentional, it
leads to an intension of form that is more cognitive than perceptual (Révész, 1934, 1950). When a sighted
individual touches an object, most of the time, it is to manipulate the object, whereas when the individual is
visually impaired, they often touch for pattern perception (Heller & Gentaz, 2014). Therefore, the blind
individuals are more suited to use their sense of touch for developing a keener sense in recognizing imagery
compared to the sighted who rely more on their vision to understand the object geometry, spatial relations,
and mobility. More supporting facts were found when methods from neuroscience were used to understand
the sense of touch. The methods involved brain imaging and transcranial magnetic stimulation (TMS) –
which lets a researcher turn off a part of a brain to see which functions (cognitive or perceptual) are affected–
and it was found that the occipital lobe of congenitally blind individuals were involved when processing
Braille, which did not happen for the sighted subjects who were blind-folded (Yuasa et al., 2006). This meant
that for the congenitally blind, the visual cortex was used to understand the sense of touch, even when it
comes to understanding text in braille. Also, if any shortcomings of the visually impaired not being as skilled
as the sighted in recognizing 2D drawings is found, Heller states that the cause to be more from the flaw of
education rather than the absence of visual imagery or visual experience (Heller & Gentaz, 2014). In most
studies which looks into the visually impaired’s understanding of 2-dimensional pictures, raised line drawings
are used. A line drawing is not like normal visual stimuli, for it does not depict planes, and are made-up
(Gibson, 1966, 1979), but people’s perception of the world generally cannot be veridical in any case and the
visual illusions cannot be taken out(Heller & Schiff, 2013). It is also argued that the untrained blind subjects
can recognize raised line drawings of ordinary objects and have the same pictorial abilities and uses the same
principles as the sighted (D’Angiulli et al., 1998).
- 14 -
2.1.2 The Visually Impaired and Facial Expression
It has been proven in studies that the congenitally blind have the understanding of facial expressions even
when they have not seen an actual face before. In Picard’s study (2011), fifteen sighted and fifteen congenitally
blind subjects were asked to classify raised line drawings of facial expressions, and whereas the accuracy did
not vary significantly, the blind adults were faster with the task (Lebaz et al., 2011). The study stated the
reason for this result may be because the shape and location of the features fit with the tactile and the
proprioceptive knowledge that the blind have of their own facial expression of basic emotions. To support this
fact, it is said that the understanding and recognition of the facial expression is innate rather than culturally
encrypted and that it is possibly known from birth (Darwin et al., 1998) for young children uses the emotion
read from their caregiver to decide how to handle different situations (Russell, & Fernández-Dols, 1997).
2.2 CMC for the Visually Impaired
2.2.1 Audio Services in CMC for the Visually Impaired
One of the major reasons for the increase of text messaging is related to its benefit of privacy regardless of the
time, place, and occation. The fact that it can only be seen by the receiver is an appealing factor for people
nowadays. For the visually impaired, most of the smartphone technologies are related to sound such as
Apple’s advanced screen reader and KakaoTalk’s audio emoticons, but the problem with messaging with an
audio output is that it does not yet provide the privacy that naturally comes with visual text messaging, and
when there is a lot of surrounding noise, it is difficult to hear. Furthermore it is dangerous for the visually
impaired who cannot see to use earphones.
Another aspect of audio output of emojis is that it cannot provide the visual quality that the tactile and visual
share. There are two types of audio output where one uses sounds that goes well with the image, for instance,
a smiling face will be translated into the sound “haha~”, and for the other type, the sound will describe the
image, for instance, “a girl is holding a heart and smiling.” The first one communicates the emotion directly
but does not get the image across, and the second one has its limitation that it cannot deliver the instant
emotion that an image can give. Also they are both difficult for a congenitally blind person to imagine the
image. Although the audio has its own benefits when filling in the missing emotional cues, it still has many
aspects where it can improve.
- 15 -
Although there have been many attempts to evolve the text messaging experience into a multisensory
experience using audio, tactile, and tactile-vibro, the texting experience for the visually impaired still has a
long way to go. As attempts to improve the nonverbal aspect of texting experience for the sighted are
progressing at a fast pace, this study has hopes to improve the texting experience for the visually impaired by
creating a new way to get across their non-verbal signals in CMC.
2.2.2 Tactile Studies in CMC which Can be Used for the Visually Impaired
Studies on haptic have been a rising trend for recent years and many studies related to remote
communication has also been done in various ways. The vSmiley (Mathew, 2005) is a method developed to
communicate emotion through different vibration patterns, inTouch (Brave & Dahley, 1997) uses force
feedback for haptic interpersonal communication. CheekTouch (Park et al., 2010) gives a vibro-tactile
feedback to the cheeks while speaking on the mobile phone. The term, Hapticon (haptic + icons), is usually
used to refer to the vibro-tactile feedback used in instant messaging. These methods give the users a
multimodal texting experience and the feeling of intimacy. However, these methods were not exclusively
developed for the visually impaired and there still lies controversy on the issue of how the emotions and the
vibration patterns can be connected objectively.
Figure 2. Tactile Studies on Remote Communication (A- Instant Messaging and Haptic Presence Prototype,
B- CheekTouch Prototype, C- InTouch Prototype)
- 16 -
There are not many studies on in text messaging using tactile feedback, but there is a patent on refreshable
haptic braille display and how the emoticons can be incorporated in the system (U.S. Patent No. 717,756,
2012). The method does not have any logical rule to it, and the rules must be learned from the beginning but
the intention was to be able to incorporate a system for emoticons inside the current braille system.
Figure 3. Patent on Refreshable Braille Display and How Emoticons can be Incorporated
2.3 Emojis in CMC
2.3.1 Emotional Communication in CMC
As mentioned before, the main purpose of an emoji is to supplement the exchange of social information by
providing additional social cues beyond what is found in the text (Thompson & Foulger, 1996) and serve the
same role as non-verbal signals in an F2F conversation (Derk et al., 2008). The three roles of nonverbal signals
in a F2F conversation is said to be: (a)providing information; (b)regulating interaction; and (c)expressing
intimacy (Ekman & Friesen, 1969; Harrison, 1973), and as a compensation to overcome the visual
shortcomings of CMC due to its lack of visibility, and thus reduced social presence, (Derks et al., 2008) the use
of emojis grew emensely over the years. Line Inc., one of the biggest text messaging platform in Korea, stated
that the portion that emojis takes up in the conversations is known to have well gone over twenty percent.
Also, as more users used emojis, the related business has grown to make 500 billion won in the past three
- 17 -
years (JoongAng Ilbo, 2015; CNBC, 2013). It has been proven in studies that text messages which used
emoticons worked better when interpreting the precise meaning of the sender (Lo, 2008; Gajadhar & Green,
2005). When Trend Monitor (2014) asked 1,000 subjects between 19 to 59 years-old on their thought on
emojis, 52.4% of the subjects stated that online communication that uses emojis are the norm these days and
the main reason for using emojis were related to enjoyment, variety and exaggeration in expression, softening
negative expressions, concealing embarrassment and awkwardness, and saving time. The growth in the emoji
industry proves the need and potential for advancement and the spread of emojis.
There have been several studies on how the faces are depicted in emoticons and differences were found in the
preference of horizontal or vertical designs or its depiction of eyes and mouth according to the user's culture
or the language they speak (Park et al., 2013; Yuki et al., 2007). While these factors were not considered when
emojis were designed by various designers, common components still exists to serve as the key to making the
emoji become used in its particular way and mean what it means. The fact that emojis have common
meaning to its users were proven in several studies that used emoticons as classifiers for sentiment analysis
(Pak & Paroubek, 2010; Go et al., 2009; Read, 2005) and how the emojis are actively used as an
communication tool in CMC.
2.3.2 Use of Emojis and Emoticons in CMC
As Trend Monitor(trendmonitor.co.kr), a surveying organization in South Korea, stated after asking 1,000
people between 19 to 59 years-old on their thought on mobile emojis, 64.9% users expressed that it was more
enjoyable to use emojis in a conversation and more than 52.4% thought that online communication that uses
emojis are the norm these days. The reason for using emojis were as in Figure 4. Most of the reasons were
related to the enjoyment and adding more fun and others equally important reasons were for the variety and
exaggeration in expression, softening the negative expressions, concealing embarrassment and awkwardness,
and saving time. (Trend Monitor, 2014)
- 18 -
Figure 4. Advantages of Emojis
Also the situations where people mostly used emojis were when they wanted to express affection (81.3%),
when they wanted to lighten the mood (79.8%), and was also used when they wanted to look good to the
other person by appearing trendy (75.2%) or appearing cute (71.5%). It was also used when they wanted to
graphically show their facial expression (70.4%), and express regret for what they did (69.1%). There was also
a high number for when there is not much to say (66.8%)(Trend Monitor, 2014). KakaoTalk reported that
out of all the messages with emojis in it, emojis took up 15% of the conversation. (Joongang Daily News, 2014)
When the reasons for success in KakaoTalk and Line - the two largest texting platforms in Korea and Japan –
is discussed, the effect of emojis (i.e. ‘Emoticons’ in KakaoTalk and ‘Mobile Stickers’ in Line) cannot be
factored out. In 2012, the daily sales of paid emojis were approximately one hundred million won. (MBC
News, 2014) The portion that emojis business takes up in the Naver’s Line business is known to have well
gone over twenty percent and as more users used emojis, the related business has grown up to five hundred
billion won in past three years (MBC News, 2014). It is now said that there are over twenty-two million emojis
in KakaoTalk and Line. Line has eight times more character than KakaoTalk which goes up to 5600
characters with each character having 40 expressions.
- 19 -
Seven out of ten people replied that there is potential in the mobile sticker industry(69.1%) and that there will
be increasing amount of users for mobile stickers (70.7%). Also majority (64.1%) of the people thought that
there will be increase in the users globally as well and that advertisement using mobile stickers will increase
(60.6%) (Trend Monitor, 2014).
2.3.4 FACS and EMFACS
In this study, emojis which depict the human facial expression (smiley emojis) were focused on for they show
the most emotional clarity. For the reference, studies on the emotional interpretation of human facial
expressions were looked into. The systems of FACS (Facial Action Coding System) (Ekman & Friesen, 1977)
provides an objective means of measurement for the facial muscle contractions when a facial expression is
made. FACS combines three different approaches which involves holistic spacial analysis based on principle
components of graylevel images; feature measurement using local image features such as wrinkles; and optic
flow, but since emojis are consisted of simplified components, and because this study will only use static emojis
without movement, the sequential and time aspect of the FACS were not considered and only the description
of the facial muscles that consist each emotions were looked into.
FACS later branches out to the EMFACS (Emotional Facial Action Coding System) (Friesen & Ekman, 1983)
which shows how the movement of facial muscles connect to emotions. FACS and EMFACS both uses the six
basic emotions (Batty & Taylor, 2003) – happiness, sadness, surprise, fear, anger, and disgust – by Ekman, for
it was both developed by him. And as seen in Figure 5., The Action Units are divided into upper and lower
face and includes visual descriptions on the eye lids, eyebrows, nose, cheek, mouth, chin and more.
- 20 -
Figure 5. FACS Action Units Chart
For the EMFACS, All emotions are connected with numbers of Action Units (AUs) and the description of
EMFACS are as follows:
EMFACS (Emotional Facial Action Coding System)
• Happiness : Cheek Raiser + Lip Corner Puller
• Sadness: Inner Brow Raiser + Brow Lowerer + Lip Corner Depressor
• Surprise: Inner Brow Raiser + Outer Brow Raiser + (Slight) Upper Lid Raiser + Jaw Drop
• Fear: Inner Brow Raiser + Outer Brow Raiser + Brow Lowerer + Upper Lid Raiser + Lid Tightener
+ Lip Stretcher + Jaw Drop
- 21 -
Anger: Brow Lowerer + Upper Lid Raiser + Lid Tightener + Lip Tightener
Disgust: Nose Wrinkler + Lip Corner Depressor + Lower Lip Depressor
- 22 -
3 ____________________________________________________________________________________
Preliminary Study: Emojis to Emotions Survey
3.1 Understanding and Storing of Images for the Visually Impaired
3.2 Organize Facial Features Separately
- 23 -
Chapter 3. Preliminary Study: Emojis to Emotions Survey
3.1 Preliminary Study to Identify the Connection Between Emotions to Emo-jis.
3.1.1. Visual Elements in Emoji-Smileys
As it has been implied previously, there have been no studies that disassembled or looked into what visual
factors of an emoji caused it to hold the meaning in which the users use them as. For there are no former
studies, The basic elements in an emoji were form, color, size, and placement. Most of the time, size and color
were used to intensify the emotion the shape represented and the core factor that determined the emotion was
the shape of facial features and actions on the face. But in this study, only the facial features such as the
mouth, eyes, and eyebrows and the iconic shape of the heart was used to communicate the emotion. First the
color was difficult to deliver using tactile, and actions on the face such as an exaggerated sweat, nose air, or
veins on the temple that represented anger, were often taken from the expression from Japanese manga which
may be difficult for the visually impaired to understand instantly.
Figure 6. Emoji Design Factors
- 24 -
3.1.2 Select smiley emojis to use for the preliminary study.
In order to conduct the preliminary study, an emoji set had to be selected. Between the Apple Colored
Emoji and Kakao Talk Emojis, the Apple Colored Emoji was selected. The two emoji sets are both familiar to
the mobile users of South Korea, for Apple Colored Emoji is universally used and is the basic emoji for all
Apple smartphones, and as Kakao Talk is the biggest texting platform in South Korea, Kakao Talk Emojis
are also widely used. Out of these two options, the Apple Colored Emoji was chosen for the Kakao Talk
Emojis had a more decorative quality which could effect the outcomes of the survey, and the Apple Colored
Emoji set had more variety of smiley designs of similar shapes of mouth or eyes which were easier to analyze
when the results were out.
Next, out of the various sets of emojis – such as nature, places, symbols and more – the smileys were
selected for they focused on depicting facial expressions which has the most direct emotional delivery. Because
the focus of this process was to examine the shape of the facial features, emojis which concealed or did not
depict the eyes or the mouth were eliminated. The symbol mark of the heart was added for the heart was
always located at the top of ‘Most Frequently Used Emoji’ list and seemed essential to the set.
Figure 7. Apple Emojis and KakaoTalk Emojis
- 25 -
Figure 8. The set of emojis used for user survey
3.1.3 Select 6 (+ 1) emotions to use for the preliminary study.
For the emotions to use in the preliminary test, six emotional adjectives were taken from Ekman’s six basic
emotions for it has been widely used in the studies of FACS and many other studies related to emotional
delivery. In an addition, the emotion of ‘affection’, was included for the most frequently used emoji often
contained the shape of the heart which widely symbolizes the emotion of ‘affection’. As a result, seven
emotional adjectives – happy, angry, disgust, sad, surprised, fear, and love – were used in the survey.
3.1.4 Conduct preliminary study in order to categorize the selected emojis to the selected seven emotions.
After the selection of emotions and an emoji set, a preliminary test to categorize the formerly chosen emojis
into the seven emotions was conducted. The survey was conducted in order to see how the emoji users
connected each emotion to emojis. For the preliminary study, the seven emotional adjectives selected from the
former process were asked to be connected with the chosen section of Apple Color Emojis.
The subjects were asked to select as many emojis they thought that applied to the emotion they were trying to
- 26 -
express. For an example, for the emotion ‘happy’, the question asked “Which emoji will you use if you want
to express ‘happy’?”, and this applied to all seven emotions.
The user test was conducted online on ‘surveymonkey.com’ and 205 subjects who are familiar with emojis
from different gender and age groups picked as many emojis which seemed to apply to the emotion. Out of
the 205 replies, only the completed 190 were used in the result. As seen in Figure 9., there were a total of 115
female users’ replies and 75 male replies out of 190, and for the different age groups, 118 replies were from
ages 20s to 30s users, 70 replies from ages 40s~50s users, and 1 replies each from the teenage group and 60s
and more.
Figure 9. Demographic of the User Survey
For the overall result, as seen in Figure 10., because the question allowed multiple answers, each emotions had
different number of responses where ‘Happy’ had the most response of 1075 and ‘Disgust’ had the least of
425. ‘Angry’ and ‘Love’ showed a definite contrast in number for the ones that were chosen more and the
ones that were least chosen. The most chosen three emojis for ‘Angry’ had over 70% users who agreed upon
the answers where the percentage dropped to 19% on the fourth emoji, and the most chosen three emojis for
‘Love’ each had 90%, 74%, 66%, but dropped to 24% on the fourth emoji, which shows that the first three
emojis were agreed upon strongly. ‘Happy’, ‘Sad’, ‘Surprised’, and ‘Fear’ had a more distributed amount of
choice for the first few emojis, but had emojis which had more than half of the subjects agreeing upon.
‘Disgust’, which had relatively little agreement on which emojis represented the emotion and 13% of the
- 27 -
subjects claiming that there were no emojis that represented the emotion, was eliminated from the set.
Figure 10. Result Overview of User Survey
3.2 Organize the Facial Features Separately and Convert the Images into Textual Descriptions.
The result was first put into a chart where the most selected top ten emojis could be seen together with the
total percentage of the number of subjects which agreed upon the specific emoji. As seen in Figure 11.,
although they are all top ten emojis, the percentage of agreement differs depending on the emotion. Through
this process, the emotion ‘disgust’ had to be eliminated, for the total number of selection was very weak and
14 percent of the subjects felt that there were no emojis that applied to the emotion. For the rest of the emojis,
although they differ in the percentage of selection, the answers showed noticeable differences in the overall
selection and thus were altogether used as a data.
- 28 -
Figure 11. Top Ten Selected Emojis for Each Emotions
Next, all the facial features which appeared in the ten most selected emoji features were categorized and
converted into textual descriptions. This is the same as FACS and EMFACS where the descriptions are
written textually. There have been several alterations in doing so, for FACS and EMFACS uses the real
human face, whereas this study uses an emoji which is different from the real face in terms of omission,
exaggeration, and simplification.
EMFACS uses descriptions in the lid, upper lid, inner brow, outer brow, cheek, nose, lip, lower lip, and jaw,
emojis have far simpler feature which are basically the eyes, eyebrows, and mouth. Because of this difference,
some descriptions had to be converted. For example, the ‘cheek raiser’ in ‘happiness’ was reassigned as
‘crescent shaped eyes with the curves facing up’, because when the cheek raises upward, the eyes
automatically closes into forming a curve. Also, with its simplified graphic lines, some descriptions such as
‘inner brow raiser and outer brow raiser’ can be combined as ‘brow raiser’ and ‘jaw drop and lip stretcher’
could be combined as ‘upper lip raiser’. The descriptions focused more on the final appearance of the face
rather than the action it went through. However some of the descriptions in EMFACS such as ‘lip corner
raiser’ or ‘lip corner depressor’ stayed the same, for they were already directly stating the appearance.
- 29 -
Table 1. Ten Most Selected Emoji Features Categorized and Put into Textual
Descriptions
In Table 1., the emoji features which were categorized together, and how they were described in text, can be
seen.
3.3 Rank the Facial Features by Scoring the Most Chosen Features.
3.3.1 Score the features to see the importance of each feature.
In order to see the relevance of the importance of each feature, the numbers of the percentage of people who
agreed upon the feature when expressing each emotion were added up. For example, if the ‘brow lowerer’ for
the emotion ‘angry’ appeared three times and for each time it was selected as the percentage of ‘75%’, ‘74%’,
‘70%’ out of the total survey respondent, the numbers were added up and became 219 to differentiate the
significance of the feature.
As seen in Table 2., the added up weight of each features of eyes and eyebrows, and mouth were put in with
the highest number on top, and were compared with the EMFACS to see how it matched with the results.
Although some descriptions from EMFACS showed weakly in the preliminary study results such as the ‘inner
brow raiser and outer brow raiser’ for ‘fear’, almost all of the descriptions from EMFACS applied to the
results of the preliminary study results. One part that EMFACS could not cover was ‘tears’ for sadness, for
EMFACS only covered the muscle movements on the face, and ‘tears’ are a relatively extreme reaction to the
emotion.
- 30 -
Table 2. Emoji Facial Feature Description in Words
3.3.2 Write the final description for the eyes, eyebrows, and mouth.
Taken from the final description, the EMFACS for emojis came out as follows.
EMFACS for Emojis
• Happy: Crescent shaped eyes + Mouth Corner Raiser
• Sad: Brows Corners Depressor + Tears + Mouth Corner Depressor
• Surprised: Round Eyes + Brow Raiser + Round Mouth
• Fear: Round Eyes + Brows Corners Depressor + Upper Lip Raiser
- 31 -
• Angry: Eyelid Tightner + Brow Corners Raiser + Mouth Corner Depressor
• Love: Heart Shaped Eyes + Mouth Corner Raiser
- 32 -
4 ____________________________________________________________________________________
Making of Apparatus
4.1 Creating Image-Based Tactile Design based on Previously Defined Descriptions of the
Six Emotions.
4.2 Creating Tactile Apparatus (Letterpress)
4.3 Creating Tactile Apparatus (3D Printed)
- 33 -
Chapter 4. Making of Apparatus
4.1 Create Image-Based Tactile Emoji Design based on the Previously De-fined Descriptions of the Six Emotions.
For the image-based tactile emojis which are finger-sized and uses the sense of touch instead of the vision,
several rules had to be considered when converting the written descriptions of the visual emoji.
l Dot ratio that can also imprint braille.
One of the biggest characteristic of emoji is that it is commonly used together with text (in this case, braille),
the resolution must be considered so that it can depict braille as well. The standard size of braille is as in
Figure 12. The matrix of tactile dots is based on the standard braille cell dimension composed of six dots
arranged in 2 x 3 in order to display letters on it along with emoticons. Fig. 12 shows the dot numbering in a
cell and Korean standard braille cell dimension. Dot base diameter is 1.5mm and distance between two dots
in the same cell is 2.5mm. Distance between corresponding dots in adjacent cells is 4.5mm. Distance between
corresponding dots from one cell directly below is 10 to 10.2 mm. And dot height is around 0.6 to 0.7 mm.
Since the maximum number of braille cells for a single syllable in Korean braille is four, four braille cells in a
line were arranged so that at least one syllable can be displayed at a time.
- 34 -
Figure 12. Standard Braille Size (U.S. and South Korea)
l Resolution which can depict a continuous curved line.
In the pilot test, three resolutions that could cover the braille system were tested out. The lowest resolution
had the closest resemblance of the braille size but could not depict the curvy lines of the heart and the smiling
mouth which affected the comprehension of the emoji. Therefore, the minimum resolution was determined
by its capability to depict curved lines in the finger-sized area.
Figure 13. Different Resolutions of Emoji Design
- 35 -
l Contoured Depiction
The emojis were expressed in line drawing, instead of filled in design for it is easier to decipher contoured
drawings with the tip of the finger. The visually impaired are also accustomed to raised line drawing which
uses the contoured depiction.
Figure 14. Contoured and Filled Version of Emoji Design
l Final Design
The final design was made incorporating all the considerations from the above; a resolution of 17 by 26 dots
that can also apply to the dimensions of a braille and contoured design. Each of the emotional expressions
were designed using the facial feature description from Table 2. The details were kept simple as possible and
distinctive so that it would not confuse the subjects.
Figure 15. Final Tactile Emoji Design
- 36 -
The apparatus was created in 21 by 12 millimeter blocks with 17 by 26 dots tactile emojis imprinted on it.
The dot resolution of 17 by 26 came from the ratio that can also express the braille. The detailed design size
description are as Figure 16. The black part of the designs were protruded in the final apparatus.
Figure 16. Detailed Tactile Design Description
4.2 Creating the Tactile Apparatus (Letterpress)
After several trials, Object Eden350 3D printer was used for the final apparatus. In the beginning of creating
the apparatus, it was created with 3D System’s z450 3D printer with gypsum plaster. It had a high resolution,
but the plaster did not feel good on the skin which could affect the result.
For the second apparatus, letterpress was used for softer feeling to the skin and its closer texture to the braille,
but when it was tried hands-on, the small dots did not protrude as much as it should (as seen in Figure.17) for
the dot distance was too short, and when I looked further into companies that specialized in braille printing, it
came clear that a specialized technology which applies heat from both sides of the paper had to be used, and
the companies who had this kind of technology did not want to take a chance in the experiment which was
only for a small amount of customized production.
- 37 -
Figure 17. Letterpress Apparatus
4.3 Creating the Tactile Apparatus (3D Printer)
Finally, the apparatus was created using the 3D printer, but this time plastic was used instead of gypsum
plaster for better texture. The final apparatus came out as Figure 18 and 19.
Figure 18. Final Apparatus. 3D Printed Image-based Tactile Design
- 38 -
Figure 19. Final Apparatus 3D Printed Image-based Tactile Design
- 39 -
5 ____________________________________________________________________________________
Interview on Image-based Tactile Emojis
5.1 Contents of the Interview
5.2 Subjects, Time and Place
- 40 -
Chapter 5. Interview on Image-based Tactile Emojis
The interview was conducted in three parts and took around 20 to 30 minutes for each subjects. All the
instructions had to be read out and blocks of apparatus was asked to be matched throughout the interview. In
order to make them relaxed during the interview, the word selection was carefully picked so they did not feel
pressured. Word such as ‘test’ ‘time limit’ ‘experiment’ were avoided and for young students, overly formal
language (in Korean) could seem intimidating, so the tone was adjusted to make them comfortable. The
detailed scenario can be found in the appendix.
5.1 Contents of the Interview
5.1.1. Part A. Understanding of Emojis without Former Guidance.
In order to see how naturally understandable the design is, the subjects were asked to match the
emotion of the emoji by touching the finger-sized block apparatus. After telling the subjects which way is up,
and that the eyes and mouth are depicted on the frame (for ‘affection’, I explained that it was a symbol.), the
six emotions were slowly read out to them and were asked to match which seemed closest to the pattern they
were holding. Whether they were right or wrong were not told.
5.1.2 Part B. Learnability of Image-Based Emoji Set and Non-Image-Based Emoji Set.
The image-based tactile emoji which was created for this study was compared against a non-image-
based emoji which comes from a patent. The image-based emoji and the non-image based emoji were both
given to them one by one and after being told in a very concise sentence “This is a wink,” and they had the
chance to feel the shape for fifteen seconds each. After the six shapes were felt, I gave them the blocks one by
one again and asked them to match the emotion they think in which applied to the emoji they were holding
and slowly read out the emotions for reminder.
- 41 -
5.1.3 Part C. Effect Of Emojis When Interpreting the Tone of a Sentence.
For the third part, a strip of paper which had a neutrally understood sentence written on in braille was
handed to the subject, and they were told that this came in a text message. The sentences were emotionally
and intentionally ambiguous. For example, “Today’s menu is curry again.”, “He visited again.”, or “You’re
being silly.” These sentences can have different meanings and intensions depending on the emoji which
followed behind. When the subject finished reading the sentence, three or four options of interpretation were
given to them including “I don’t know.” After the subject answered to the question, an emoji was handed to
them and were told that the emoji came along with the former text. The subjects were asked again on the
interpretation of the sentence and the first and second answers were recorded. Two out of three sentences
were randomly chosen and given out to the subjects, and how their interpretation of the sentence became to
an agreement after the use of emoji was observed.
5.2. Subjects, Time, and Place
The interview was conducted in a quiet room where there were no disturbance in order to conduct the
interview vocally for the visually impaired. All interviews were conducted through cooperation with Daejeon
City Sansung Welfare Center (Taejeon), Braille Library (Taejeon), and a Taejeon Visually Impaired School
(Taejeon), Siloam Welfare Center (Seoul), and Song-pa Visually Impaired Welfare Center (Seoul) and was
also conducted at these places. I asked the centers if there was a quiet room with a table and a chair to sit side
by side with the subjects so it was easier to hand out the apparatus.
The interview was done from September to November, 2015, although the very first pilot test was conducted
in November, 2014. It was difficult to match the institutions’ schedule and it was difficult for them to predict
when many of the subjects had time available so many visits had to be made.
For the visually impaired subjects, thirty subjects from welfare centers, teachers and students from visually
impaired school, and a braille library were gathered. There were 16 male and 14 female subjects, 20 blind
subjects and 10 low-visioned subjects (the ones who can see the tactile braille were eliminated), 20 who lost
their sight after birth and 10 who had low vision or could not see from birth. Five more subjects that were
included in the beginning, had mental health issues that affected the results of the test, or were disturbed by
the interphone announcement which affected the result of the test, and thus, had to be eliminated.
- 42 -
Figure 20. Visually Impaired Subjects Using Tactile Emoji for the User Test
Table 3. Description of Participants
Participant Occupation Education Level Sex Age Degree of Blindness Age at Onset of Blindness
(Years)
I. EARLY BLIND
1 Unemployeed F 43 Very Low Vision 0 2 High School Student F 18 Very Low Vision 0 3 Masseuse College Graduate M 27 Blind 0 4 Unemployeed College Graduate M 26 Blind 0 5 Unemployeed College Graduate F 24 Blind 0 6 Library Employee College Graduate F 35 Very Low Vision 0 7 High School Student F 17 Blind 0 8 Welfare Center Teacher College Graduate M 44 Blind 0 9 University Student College Enrolled F 25 Blind 0
10 Health Keeper; Student on a leave of absence College Enrolled F 22 Very Low Vision 0
II. LATE BLIND
11 Masseuse College Graduate M 44 Blind 12 Welfare Center Employee F 30s Blind
13 Teacher Graduate School Graduate M 42 Blind Late 20s
14 Ex-Principle College Graduate F 56 Blind 3
15 Teacher Graduate School Graduate F 38 Blind High School
16 Elementary Teacher College Graduate M 35 Blind
17 Welfare Center General Manager College Graduate M 58 Blind 48
- 43 -
18 Unemployeed Middle School Dropout M 60
Retinitis pigmentosa ; Very Low Vision; Nyctalopia
19 Unemployeed College Enrolled M 43 Blind 23
20 Unemployeed College Enrolled F 43 Very Low Vision; Retinitis pigmentosa; Can only see contours
21 Unemployeed High School Graduate F 35 Blind; Became Completely Blind last year
22 Housewife College Enrolled F 71 Very Low Vision; Can see the light slightly
23 Church Activity Graduate School Graduate M 42 Very Low Vision 37
24 Private Business College Graduate M 50 Very Low Vision 40 25 Unemployeed High School Graduate M 56 Blind
26 Unemployeed High School Graduate F ff F
53 Very Low Vision 28
27 College Graduate M 55 Blind 53 28 High School Graduate M 40 Blind 30 29 Unemployeed No Education M 43 Blind 31 30 High School Graduate M 70 Blind
- 44 -
6 ____________________________________________________________________________________
Conclusion and Future Work
5.1 Summary
5.2 Contributions
- 45 -
Chapter 6. Conclusion and Future Work
6.1 Summary
6.1.1. The Overall Performance of the Subjects
For Part A., subjects were able to match an average of 4.03 out of 6 emotions without prior guidance, and 3
out of the 4 subjects who matched perfectly before and after the guidance were congenitally blind. The 3
congenitally blind subjects who matched all 6 emotions were the fastest with the task and non-hesitant when
answering the question.
When asked, on what bases they were matching the imagery with the emotions, one subject, who was
completely blind from birth, answered “I used my own face as a reference because the corner of her mouth
pulls up when I’m happy, and my eyes and mouth opens up when I am surprised.” This was congruent with
Picard’s assumption and also Darwin’s study that says that the understanding and recognition of the facial
expression is innate rather than culturally encrypted and that it is possibly known from birth (Darwin, Ekman,
and Prodger, 1998) for young children uses the emotion read from their caregiver to decide how to handle
different situations (Russell, & Fernández-Dols, 1997). Another congenitally blind subject stated “The
straight angular lines seemed angry and felt far from happy, and the curvy lines seemed positive.” In the user
test, there were several remarks about the angle and the characteristics of the line which included the words
‘round’ or ‘pointy.’ In Lundholm’s study in 1921 (Lundholm, 1921), participants were asked to draw lines to
express the affective tone of different adjectives and more angles were drawn for adjectives such as "hard",
"harsh", and "cruel", and more curves were drawn for adjectives such as "weak", " gentle", and "mild." There
seemed to be a connection between the character of the lines that helped interpret the emotion of the emoji.
Lastly, two subjects mentioned that because they know that when depicting a smiley face, caret marks(“^^”)
are used, they assumed that the eyes for “happiness” were similar to that form. It is unknown where they were
able to learn the form of the caret marks, but it was proven that the education and former experience on the
connection of form to its connotation has helped them understand the facial features better. Lastly, three
subjects simply stated that the angry depiction of tactile emoji “just seemed scary.” Many subjects had
difficulty explaining how they matched the emoji to emotion and even the comparably articulate subjects
sometimes did not know when they have learned that the form connected to a specific emotion.
There were no apparent gap between of the subjects ability according to gender, time of sight loss, or level of
- 46 -
blindness, but some subjects became discouraged when they could not tell how to connect the emotion to the
emoji seemed to give up on their effort in exploring the rest of the emojis. Although I tried hard not to give
the subjects any impression of a “test” since I did not want the pressure in anyway affect their performances
when conducting the user test, 4 subjects matched less emotions after my guidance. It is assumed that this
comes from the pressure to match the “correct answer” and trying to remember the clues instead of
instinctively feeling it. As for the first test, the subjects were asked to match the emotional adjective which
seemed right to them with what they were feeling, but for the second test, they were asked to remember the
emotions that were told to them and state the matching emotion when asked which caused some of them to
perform lower than their first try.
After the guidance, the image-based tactile design had an average of correct answers that were 4.87 out of 6,
while the non-image-based tactile design had an average of 2.19 out of 6, which showed the contrast that the
image-based tactile design where far more effective for learning. Many subjects commented that they did not
understand the connection between the non image-based tactile designs and the emotions.
The clarity of the sentence also improved where for the first try for the first sentence, 6 subjects out of the 30
subjects matched the intended answer, when the emoji was attached 26 out of 30 subjects matched the
intended answer and out of the 4 who had the wrong answer was not completely wrong, for the answer
‘affectionate’ and happy’ was a bit similar. are a only 2 subjects had different interpretation of the sentence
after the tactile emoji were added to the sentence.
6.1.2. Design of the Image-based Tactile Emojis
Out of the six image-based tactile emojis, the smile connected to ‘happiness’ and the heart connected to
‘affection’ had the highest scores, where 28 subjects out of 30 subjects got the correct answers. The next easier
emotion was ‘anger’ which had 22 subjects getting correct answers followed by ‘surprise’ and ‘sadness’ which
had 18 subjects with right answers. The lowest correct answer was ‘fear’ which was often confused with
‘sadness’ or ‘surprise’ for it shared the same traits of sloping down eyebrows or big eyes. When thirty sighted
subjects were asked to connect the emoji design with the emotion, they also had difficult time connecting ‘fear’
and even the ones who got the correct answer stated that it was tricky. Also, when conducting the user test,
many subjects tried to matched the tactile design of ‘fear’ by eliminating the other options first and asked
‘which one did I not say yet?’, and when considering that the ‘fear’ was a tricky design from the start, the
subjects who were able to match the emotion may have been better with their ability to remember the visual
tactile images than the others.
- 47 -
The emojis that many subjects mismatched were ‘sadness’ and ‘anger’ which shared the same mouth. This
can mean two things: first, that the subjects mostly uses the mouth to discriminate the emotions; and second,
that the subjects had difficult time processing the difference between the downward shaped eyebrow and
upward shaped eyebrow. They both seem true, for there were subjects who said that they were both the same,
or “This is ‘angry’ because the mouth goes down.” while touching ‘sadness’ or “This definitely seems happy
because of the mouth.”. For the second theory, one of the subject asked ‘Is this going downward?’ when
feeling the brows and another subject described the eyebrows that they are “gathered in the middle and
spreads in the end” and gets the emotion right the first time, but had ‘sadness’ mistaken with ‘anger’ for the
second try.
6.2 Contributions
This study provides a kickoff point for a new CMC experience for the visually impaired and the direction of
design when it can later be produced. The study proves that the visually impaired, even those who are
completely blind from birth, can connect the emoji to emotion and has a brief understanding of the form.
With the previous knowledge of facial expressions, they can easily learn the emojis with a short introduction
and thus makes this method a more accessible means of communication compared to the ones which has to
be learned from the beginning.
The image-based tactile design also provides visual cues to the visually impaired in a context that has never
been done before. For Part C. of the user test where the subjects were to interpret sentences with and without
the emojis, many were amused when the emojis changed the entire interpretation of the sentence. The
experience of image inside a text using tactile and how this changes the entire context of the image usage in
the sense that the image affects the text was a new experience for the subjects. Also, because CMC using
smartphone devices are for daily conversation regardless of time and place, when it becomes available in the
market for wide use, it will provide the visually impaired with a highly accessible image from a brand new
channel that will make their conversations more interesting.
The image-based tactile design, unlike the audio feedbacks that are often used by the visually impaired, can
also provide the privacy that all texting CMCs should support, and it will also be more easier to use in a loud
environment. Assuming that this will be manufactured as a wearable device, using the fingertip to feel and
receive the message will create a more intimate environment to communicate with the other party.
- 48 -
6.3 Limitation and Future Work
For the study, it was difficult to find visually impaired subjects for the user test, and due to the small sample
size, it was difficult to make meaning out of the numbers. In the interview, it seemed that the congenitally
blind were better at matching the emoji to emotion, but because there were only four subjects that were
completely blind from birth, and one subject did not have high scores as the others, the number was too weak
to differentiate with the other subjects.
Also when it comes to actual production of the image-based tactile emojis, the small scale and the resolution it
must depict can become a hindrance. When briefly planning out the final design as a wearable device worn as
a wristband, the device must be braille-size and light-weight. The pins must be refreshable with pins pushing
up and down, and in order to communicate the images as understandable image, it is essential to maintain the
level of resolution as planned in this paper. More studies from the mechanical engineering can be done to
look into the production.
The biggest limitation of for the image-based tactile emoji design is the small scale of the apparatus and its
difficulty to depict detailed images. This can mean that it will face some limitation when producing an
abundant variety of designs. But as the emojis for the sighted started with a simple smiley :-) and a frown :-( ,
the tactile emojis also have the chance of starting as a small set and expanding. Also, using a multisensory
approach, using audio and vibro-tactile along with the image-based tactile emojis may lead to more exciting
results. When gathering the subjects for the user test, it seemed that there were less braille-users than initially
expected, and one subject went as far as to say that they did not need braille since there was audio feedback. If
this is the actual case, a new usage of emojis can be looked into for the visually impaired. However the
limitations, an iconified pictorial display in a CMC is still an interesting direction to look into for the visually
impaired and the sighted as well.
- 49 -
Appendix
Appendix 1. User Test Scenario
Basic Information
Hello, I am Yuri Choi who came for my Master’s dissertation. Nice to meet you. Is it okay for me to record our interview as video or voice recording? Before starting I would like to gather some information.(name, age, occupation, degree of visual impairment, age of loss of sight, familiarity of emojis, if they are using the emoji)
Interview
PART A. Understanding of emojis without former guidance.
“First, there will be eyes and mouth imprinted on this small block I hand you. I will give you six emotional words, so please pick one that you fit best with the block you are holding.” “This is the first block. Which emotion do you think best fits this block. Happy, love, surprised, fear, sadness, anger.” (repeat for all six blocks) “I will not be telling you whether you got it right or wrong”
PART B. Learnability of image-based emoji set and non-image-based emoji set.
“This time, you will feel a different block from the first one and I will tell you which emotion it goes with. While you feel this block for some time, please remember which emotion goes with which block.” “Now I will hand you the block. (hand the block) This is ‘happiness’.” (after giving some time, take away the block and hand another block) “This is ‘wink’.” (repeat for all six blocks) “Now I will give you the emotions so let’s see if you can remember the blocks. This is the first block. Please choose the emotion you think went with this block. Happy, angry, surprised, tongue sticking out, sadness, wink.” (repeat for all six blocks) “Next, we will try the same thing for the first block you used. I will tell you which emotion goes with each blocks as I just did, so please remember which emotion goes with which block. Now I will hand you the block. (hand the block) This is ‘happiness’.” (after giving some time, take away the block and hand another block) “This is ‘sadness’.” (repeat for all six blocks) “Now I will give you the emotions so let’s see if you can remember the blocks. This is the first block. Please choose the emotion you think went with this block. Happy, love, surprised, fear, sadness, anger.” (repeat for all six blocks)
PART C. Effect of emojis when interpreting the tone of a sentence.
“For the last part, I will hand you a strip of paper with braille imprinted on it. When you finish reading, please tell me.” (For the ones who could not read braille, read it out in a monotone voice.) “Now, pretend that this sentence came in a text message from a friend and choose the answer that seems closest to the the friend’s intension or emotion.”
1) For “He visited again.”
“This friend’s emotion is… 1. Annoyed 2. Happy. 3. Affectionatly Excited. 4. (I don’t know.)” “Now, pretend that this emotion came together with the text, (hand out a emoji), now please answer the question again.” (LJ♥) (read out the examples again.)
- 50 -
2) For “Today’s menu is curry again.”
“This friend… 1. likes curry. 2. is tired of curry. 3. (I don’t know.)” “Now, pretend that this emotion came together with the text, (hand out a emoji), now please answer the question again.” (LJ♥) (read out the examples again.)
3) For “You’re being silly.”
“This friend… 1. is being playful 2. is disapproving. 3. (I don’t know.)” “Now, pretend that this emotion came together with the text, (hand out a emoji), now please answer the question again.” (LJ) (read out the examples again.)
- 51 -
References
[1] Abramowicz, A., Klatzky, R. L., & Lederman, S. J. (2010). “Learning and generali-
zation in haptic classification of 2-D raised-line drawings of facial expressions of
emotion by sighted and adventitiously blind observers.” Perception, 39(9), 1261.
[2] Adolphs, R. (2002). “Recognizing emotion from facial expressions: psychological
and neurological mechanisms.” Behavioral and Cognitive Neuroscience Reviews, 1(1), pp.
21-62.
[3] Bartlett, M. S., Viola, P. A., Sejnowski, T. J., Golomb, B. A., Larsen, J., Hager, J.
C., & Ekman, P. (1996). “Classifying facial action.” Advances in neural information pro-
cessing systems, pp. 823-829.
[4] Bassili, J. N. (1979). “Emotion recognition: the role of facial movement and the rela-
tive importance of upper and lower areas of the face.” Journal of personality and social
psychology, 37(11), 2049.
[5] Batty, M., & Taylor, M. J. (2003). “Early processing of the six basic facial emotional
expressions.” Cognitive Brain Research, 17(3), pp. 613-620.
[6] Bosker, B. (2014, June 27). How Emoji Get Lost In Translation. from
http://www.huffingtonpost.com/2014/06/27/emoji-meaning_n_5530638.html
[7] Bourdon, P., Le Clerc, F., & Jouet, P. (2012). U.S. Patent Application 13/717,756.
[8] Brave, S., & Dahley, A. (1997, March). “inTouch: a medium for haptic interperson-
al communication.” In CHI'97 Extended Abstracts on Human Factors in Computing Systems
(pp. 363-364). ACM.
[9] Bryan, T. H. (1977). “Learning disabled children's comprehension of nonverbal
- 52 -
communication.” Journal of Learning Disabilities, 10(8), pp. 501-506
[10] Churches, O., Nicholls, M., Thiessen, M., Kohler, M., & Keage, H. (2014). Emoti-
cons in mind: An event-related potential study. Social neuroscience, 9(2), 196-202.
[11] Darwin, C., Ekman, P., & Prodger, P. (1998). The expression of the emotions in man and
animals. Oxford University Press, USA.
[12] D’Angiulli, A., Kennedy, J. M., & Helle, M. A. (1998). Blind children recognizing
tactile pictures respond like sighted children given guidance in exploration. Scandina-
vian Journal of Psychology, 39(3), 187x-190.
[13] Derks, D., Bos, A. E., & Grumbkow, J. V. (2007). Emoticons and social interaction
on the Internet: the importance of social context. Computers in Human Behavior, 23(1),
842-849.
[14] Derks, D., Fischer, A. H., & Bos, A. E. (2008). The role of emotion in computer-
mediated communication: A review. Computers in Human Behavior, 24(3), 766-785.
[15] Eimer, M. (2000). Event-related brain potentials distinguish processing stages in-
volved in face perception and recognition. Clinical neurophysiology, 111(4), 694-705.
[16] Ekman, P. (2006). Darwin and facial expression: A century of research in review. Ishk.
[17] Ekman, P. (1993). Facial expression and emotion. American psychologist, 48(4),
384.
[18] Ekman, P., & Friesen, W. V. (1977). Facial action coding system.
[19] Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969). Pan-cultural elements in facial
displays of emotion. Science, 164(3875), 86-88.
[20] Emojitracker: Real time emoji use on twitter. (n.d.). Retrieved March 18, 2015,
from http://emojitracker.com/
[21] Friesen, W. V., & Ekman, P. (1983). EMFACS-7: Emotional facial action coding
system. Unpublished manuscript, University of California at San Francisco, 2, 36.
- 53 -
[22] Gajadhar, J., & Green, J. (2005). The importance of nonverbal elements in online
chat. Educause Quarterly, 28(4), 63.
[23] Gibson, J. J. (1966). The senses considered as perceptual systems.
[24] Go, A., Bhayani, R., & Huang, L. (2009). Twitter sentiment classification using dis-
tant supervision. CS224N Project Report, Stanford, 1, 12.
[25] Goldmine Emoticon Markets…Creating Profit through Humorous Pictures (In Ko-
rean)| MBC News. (2014, December 8). Retrieved April 5, 2015, from
http://imnews.imbc.com/replay/2014/nwdesk/article/3569182_13490.html
[26] Heller, M., & Gentaz, E. (2014). Psychology of touch and blindness. New York: Psy-
chology Press.
[27] Heller, M. A., Calcaterra, J. A., Burson, L. L., & Tyler, L. A. (1996). Tactual pic-
ture identification by blind and sighted people: Effects of providing categorical in-
formation. Perception & Psychophysics, 58(2), 310-323.
[28] Hu, M., & Liu, B. (2004, August). Mining and summarizing customer reviews. In
Proceedings of the tenth ACM SIGKDD international conference on Knowledge
discovery and data mining (pp. 168-177). ACM.
[29] Itier, R. J., & Taylor, M. J. (2002). Inversion and contrast polarity reversal affect
both encoding and recognition processes of unfamiliar faces: a repetition study using
ERPs. Neuroimage, 15(2), 353-372.
[30] Kelly, C. (2015). Do you know what I mean >:(: A linguistic study of the under-
standing of emoticons and emojis in text messages.
[31] Kim, Minsang. (n.d.). [On the Site] Emoticons which are more various than human
emotions. (In Korean) Retrieved March 17, 2015, from
http://life.joins.com/news/article/article.asp?Total_ID=15426823&ctg=12&sid=6
692.
- 54 -
[32] Lebaz, S., Jouffrais, C., & Picard, D. (2011). Identifying basic emotions in raised-line
pictures of the human face: Do we need vision?. Perception ECVP abstract, 40, 156-
156.
[33] Lenhart, A., Ling, R., Campbell, S., & Purcell, K. (2010). Teens and mobile phones:
Text messaging explodes as teens embrace it as the centerpiece of their communica-
tion strategies with friends. Pew Internet & American Life Project.
[34] Lundholm, H. (1921). The Affective Tone of Lines: Experimental Researches. Psy-
chological Review, 28(1), 43.
[35] Lo, S. K. (2008). The nonverbal communication functions of emoticons in comput-
er-mediated communication. CyberPsychology & Behavior, 11(5), 595-597.
[36] Mathew, D. (2005). vSmileys: Imaging emotions through vibration patterns. Alterna-
tive Access: Feeling and Games.
[37] Maurer, D., Le Grand, R., & Mondloch, C. J. (2002). The many faces of configural
processing. Trends in cognitive sciences, 6(6), 255-260.
[38] McDougald, B. R., Carpenter, E. D., & Mayhorn, C. B. (2011, September). Emoti-
cons What does this one mean?. In Proceedings of the Human Factors and Ergonomics Socie-
ty Annual Meeting (Vol. 55, No. 1, pp. 1948-1951). SAGE Publications.
[39] ‘Mobile Stickers are easier to express negative emotions in difficult conversations.
“Conversations get interesting with Mobile Stickers. (In Korean) (2014, August
22).Retrieved March 29, 2015, from
https://www.trendmonitor.co.kr/tmweb/trend/allTrend/detail.do?bIdx=1200&co
de=0102&trendType=CKOREA
[40] Nelson, C. A. (2001). The development and neural bases of face recognition. Infant
and child development, 10(1-2), 3-18.
- 55 -
[41] Opinion Mining, Sentiment Analysis, Opinion Extraction. (2004, May 15). Re-
trieved April 4, 2015, from http://www.cs.uic.edu/~liub/FBS/sentiment-
analysis.html#lexicon
[42] Ortony, A., Clore, G. L., & Foss, M. A. (1987). The referential structure of the af-
fective lexicon. Cognitive science, 11(3), 341-364.
[43] Pascual-Leone, A., Theoret, H., Merabet, L., Kauffman, T., & Schlaug, G. (2006).
The role of visual cortex in tactile processing: A metamodal brain. Touch and
blindness: Psychology and neuroscience, 171-195.
[44] Pak, A., & Paroubek, P. (2010, May). Twitter as a Corpus for Sentiment Analysis
and Opinion Mining. In LREC (Vol. 10, pp. 1320-1326).
[45] Park, J., Barash, V., Fink, C., & Cha, M. (2013, June). Emoticon Style: Interpreting
Differences in Emoticons Across Cultures. In ICWSM.
[46] Park,Y.W., Lim,C.Y., & Nam,T.J. (2010, April). CheekTouch: an affective interac-
tion technique while speaking on the mobile phone. In CHI'10 Extended Abstracts
on Human Factors in Computing Systems (pp. 3241-3246). ACM.
[47] Picard, D., Jouffrais, C., & Lebaz, S. (2011). Haptic recognition of emotions in
raised-line drawings by congenitally blind and sighted adults. IEEE Transactions on
Haptics, 4(1), 67-71.
[48] Pew Internet & American Life Project (2010). <http://www.pewinternet.org>.
[49] Read, J. (2005, June). Using emoticons to reduce dependency in machine learning
techniques for sentiment classification. In Proceedings of the ACL student research workshop
(pp. 43-48). Association for Computational Linguistics.
[50] Révész, G. (1934). System der optischen und haptischen Raumtäuschungen.
Zeitschrift für Psychologie und Physiologie der Sinnesorgane. Abt. 1. Zeitschrift für Psychologie.
[51] Révész, G. (1950). Psychology and art of the blind.
- 56 -
[52] Rivera, K., Cooke, N. J., & Bauhs, J. A. (1996, April). The effects of emotional icons
on remote communication. In Conference Companion on Human Factors in Computing Sys-
tems (pp. 99-100). ACM.
[53] Rovers, A. F., & van ESSEN, H. A. (2004, April). HIM: a framework for haptic in-
stant messaging. In CHI'04 Extended Abstracts on Human Factors in Computing
Systems (pp. 1313-1316). ACM.
[54] Rovers, L., & Van Essen, H. A. (2004, June). Design and evaluation of hapticons for
enriched instant messaging. In Proceedings of EuroHaptics (Vol. 4, No. 4).
[55] Russell, J. A., & Mehrabian, A. (1977). Evidence for a three-factor theory of emo-
tions. Journal of research in Personality, 11(3), 273-294.
[56] Russell, J. A., & Fernández-Dols, J. M. (1997). The psychology of facial expression. Cam-
bridge university press.
[57] Sauer, J., Schramme, S., & Rüttinger, B. (2000). Knowledge acquisition in ecologi-
cal product design: The effects of computer-mediated communication and elicita-
tion method. Behaviour & Information Technology, 19(5), 315-327.
[58] Tossell, C. C., Kortum, P., Shepard, C., Barg-Walkow, L. H., Rahmati, A., &
Zhong, L. (2012). A longitudinal study of emoticon use in text messaging from
smartphones. Computers in Human Behavior, 28(2), 659-663.
[59] Van Boven, R. W., Hamilton, R. H., Kauffman, T., Keenan, J. P., & Pascual–
Leone, A. (2000). Tactile spatial resolution in blind Braille readers. Neurology, 54(12),
2230-2236.
[60] Walther, J. B., & D’Addario, K. P. (2001). The impacts of emoticons on message in-
terpretation in computer-mediated communication. Social science computer review,
19(3), 324-347.
[61] Yuki, M., Maddux, W. W., & Masuda, T. (2007). Are the windows to the soul the
- 57 -
same in the East and West? Cultural differences in using the eyes and mouth as cues
to recognize emotions in Japan and the United States. Journal of Experimental Social
Psychology, 43(2), 303-311.
[62] Yuasa, M., Saito, K., & Mukawa, N. (2006, April). Emoticons convey emotions
without cognition of faces: an fMRI study. In CHI'06 Extended Abstracts on Hu-
man Factors in Computing Systems (pp. 1565-1570). ACM.
- 58 -
Summary
이미지 기반 촉각에모지:
시각장애인의 컴퓨터 매개 커뮤니케이션(CMC)에서의
더 명확한 감정전달에 관한 연구
일반적으로 CMC(컴퓨터 매개 커뮤니케이션)에서 문자로는 전달이 불가한 비언어적 메시지전달을 위해
에모지, 또는 이모티콘을 사용한다. 에모지는 문자가 더 활성화가 되는 시대의 흐름에 따라 음성 또는
움직이는 애니메이션들이 추가되는 등 많은 발전을 이루어왔으나, 시각장애인들을 위한 에모지는 그
흐름에 따라 크게 발전되지 못하고 기술의 발전에서 소외되었다. 본 연구는 이미지기반의
촉각에모지라는 새로운 대안이 시각장애인을 위한 CMC 환경에서의 의사소통의 명확도를 얼마나
높여줄 수 있고, 그 학습도가 이미지 기반이 아닌 에모지와 비교하였을 때 얼만큼 빨라질 수 있는지를
보기위한 연구이다. 본 연구에서는 3 가지를 확인하였다: 첫째, 학습 없이 시각장애인들이 감정과
에모지를 연결할 수 있는지, 둘째, 이미지기반 촉각 에모지와 이미지기반이 아닌 촉각에모지의 학습도
차이, 세번째, 문장 속에 에모지를 포함하였을 때와 그렇지 않았을 때의 문장의 명확도 차이.
사용자조사에는 30 명의 피험자들이 사용자조사에 참여하였으며, 학습 없이도 평균적으로 6 가지
감정들 중 4.03 개를 맞출 수 있었다. 학습 후에는 이미지기반 촉각에모지의 맞춘 숫자는 4.87 개로
올랐으며, 이미지기반이 아닌 에모지는 학습 후에 2.19 개를 맞출 수 있어, 이미지기반의 에모지가
학습도가 더 높았음을 확인 할 수 있었다. 문장의 명확도 역시 에모지가 있을 때에 더 높아졌음을 볼
수 있었다. 본 연구는 이미지기반 촉각에모지가 시각장애인들에게 활용되기에 학습도가 높으며,
이미지를 통한 더 섬세한 감정적 표현의 장을 열어줌으로써 그들의 CMC 환경을 더 높은 수준으로
향상시켜줄 수 있음을 확인할 수 있었다. 또한, 현재 크게 벌어진 비시각장애인과 시각장애인들의
CMC 환경수준을 조금이라도 좁힐 수 있는 대안 중 하나로 활용될 수 있을 것이다.
Keywords: 시각장애인, CMC, 에모지, 이모티콘, 촉각피드백
- 59 -
Acknowledgement
I would like to thank all the people who have helped me throughout the process of this
study.
First, I give all my thanks to Dr. Taesaeng Choi and Dr. Eunjoo Baik for advising me in
clarifying the objective of this study and giving me helpful advices in the process of this
research. I also express my sincere gratitude to Dr. Ji-Hyun Lee, my advisor for the
encouragement and excellent support. I would like to also express thanks to Hyeryeong Jun
for being a great friend, constantly talking me through the process of this study and making
it clear for me, taking her personal time to help me with the user test and creating the 3D
apparatus.
I would like to also give thanks to the Jong-geun Lee from the school administration office
who helped me use the 3D printer. And my thanks also goes to all the centers and schools
(Daejeon City Sansung Welfare Center, Taejeon Braille Library, Taejeon Visually
Impaired School, Siloam Welfare Center, and Song-pa Visually Impaired Welfare Center
who helped me with the study without cause.
I would not have been able to complete my study without all the help and I am truly
blessed.
- 60 -
Curriculum Vitae
Name: Yuri Choi Date of Birth: 1986/08/15
Education
‘06 ~‘09 Rhode Island School of Design (RISD) Bachelor of Fine Arts, Graphic Design (GPA: 3.6 / 4.0)
’08 Fall Semester, Brown University
Advanced Japanese Class
’07 Summer Studies, Sophia University
Japanese Society, Japanese Art History
Career
’09 Kkotsbom
Editorial Design, Internship
’10 ~’13 Daiso Asung Industries
Package Design, Store R&D Team, Section Manager
‘15~ Team Interface
Service Design Team, Senior Consultant
- 61 -
Exhibitions & Works
‘15 "Robot Essay.", National Museum of Modern and Contemporary Art (MMCA)
Seoul, Korea. Installation (Design Planning)
‘14 "Togetherness, which is Peace.", Ungno Lee Museum of Art
Taejeon, Korea. Interaction Installation (Design Planning)
‘14 “SCAN and GO” Explanation Video, Electronics and Telecommunications Research Institute (ETRI)
Taejeon, Korea. Explanation Video (Design and Production)
-----------------------------------------------------------------------------