Int. J. Human-Computer Studies 109 (2018) 79–88

Contents lists available at ScienceDirect

International Journal of Human-Computer Studies

journal homepage: www.elsevier.com/locate/ijhcs

Designing motion marking menus for people with visual impairments

Nem Khan Dim

a , Kibum Kim

a , b , Xiangshi Ren

a , ∗

a The Center for Human-Engaged Computing, Kochi University of Technology, 185 Miyanokuchi, Tosayamada-Cho, Kami-Shi, Kochi, 782–8502, Japan b Department of Game and Mobile Engineering, Keimyung University, Daegu, 42601, South Korea

a r t i c l e i n f o

Keywords:

Marking menus

Motion gestures

Accessibility

People with visual impairments

a b s t r a c t

Current smartphone accessibility for people with visual impairments relies largely on screen readers and voice

commands. However, voice commands and screen readers are often not ideal because users with visual impair-

ments rely mostly on hearing ambient sound from the environment for their safety in mobile situations. Recent

research has shown that marking menus in mobile devices provide fast and eyes-free access for sighted users

Francone et al., 2010; Oakley and Park, 2007, 2009. However, the literature is lacking design implications and

adaptations that meet the needs of users with visual impairments. This paper investigates the capabilities of vi-

sually impaired people to invoke smartphone functions using marking menus via 3D motions. We explore and

present the optimal numbers of menu items (breadth) and menu levels (depth) for marking menus that people

with visual impairments can successfully adopt. We also compared a marking menu prototype to TalkBack TM

which is an accessibility menu system in Android smartphones. The experimental results show that our partici-

pants could perform menu selections using marking menus faster than when using TalkBack. Based on the study

results, we provide implications and guidelines for designing marking menus and motion gesture interfaces for

people with visual impairments.

© 2017 Elsevier Ltd. All rights reserved.

1

r

i

a

t

a

m

q

m

T

s

r

i

w

i

t

b

2

u

w

p

q

(

b

w

m

s

t

h

v

p

a

w

t

o

p

H

m

h

R

A

1

. Introduction

Even though some accessibility issues remain with smartphones, the

eliance of visually impaired people on smartphones has been increas-

ng ( Ye et al., 2014 ). Although smartphones support screen readers, such

s VoiceOver TM and TalkBack TM ( WebAIM, 2015 ), and voice commands,

hese features can be inefficient in noisy environments and inappropri-

te in quiet public environments. These systems enable users to browse

enu items on touchscreens using speech feedback. However, they re-

uire users to perform long sequences of touch gestures to browse the

enus. This might result in increased user fatigue and dissatisfaction.

here is an increasing need for more efficient interaction techniques as

upplements or alternatives to the accessibility features that are cur-

ently available for users with visual impairments.

Marking menus allow fast and eyes-free menu selections. In mark-

ng menus, menu items are arranged in certain directions (north, south,

est, east, and so on). Users perform menu selections by drawing marks

n the direction of the desired menu item without the need of visual at-

ention. Recently, marking menus have been adapted to mobile devices

ecause they offer fast and eyes-free interaction ( Oakley and Park, 2007;

009 ). This means marking menus may offer significant benefits to the

sers with visual impairments via eyes-free mobile interactions. Thus,

∗ Corresponding author.

E-mail address: xsren@acm.org (X. Ren).

u

ttp://dx.doi.org/10.1016/j.ijhcs.2017.09.002

eceived 13 August 2016; Received in revised form 25 August 2017; Accepted 7 September 201

vailable online 8 September 2017

071-5819/© 2017 Elsevier Ltd. All rights reserved.

e propose marking menus working together with motion gestures to

rovide users with fast access to smartphones using only one hand. Ade-

uate motion sensors are now available on most common mobile devices

Negulescu et al., 2012 ), and we developed 3-D space motion gesture-

ased marking menu selection called Motion Marking Menus (MMM)

hich offer users with visual impairments ready access to smartphone

enus ( Fig. 1 ).

In spite of the need and potential, the capability of people with vi-

ual impairments to perform motion marking menus has not been inves-

igated until now. A thorough understanding of such capabilities will

elp in the design of more efficient and accessible interfaces. We pro-

ide valuable design implications regarding the spatial ability of smart

hone users to navigate motion marking menus, and more especially to

pply the benefits of this work to people with visual impairments.

Research questions include: Q1. How many directions can people

ith visual impairments distinguish? In other words, we wanted to de-

ermine how many user-discernable directions there are at each level

f efficient marking menus. Q2. How many hierarchic levels can peo-

le with visual impairments navigate in marking menu selections? Q3.

ow receptive are users with visual impairments to motion marking

enu systems?

To answer the aforementioned research questions, we performed two

ser studies. In Study 1, we investigated Q1 and Q2. In Study 2, we in-

7

N.K. Dim et al. Int. J. Human-Computer Studies 109 (2018) 79–88

Fig. 1. Motion Marking Menu (MMM) interfaces in smartphones. Menus are assigned

according to the movement of the device in certain directions (e.g., the Phone-call menu

is assigned to appear at the right side) and users perform menu selections by moving the

phone in the direction of the desired menu. Submenus are assigned in the same manner.

Users select a submenu by continuously moving the hand in the direction of the main

menu, then in the direction of a submenu (e.g. move hand to the right, then to down

directions).

v

m

t

o

2

i

2

p

B

m

t

i

n

w

2

t

m

O

m

s

o

i

m

i

o

r

p

n

2

f

s

S

e

i

t

f

w

E

a

o

i

w

b

I

h

t

h

i

m

a

O

t

o

e

n

b

p

t

w

o

l

t

r

a

s

a

t

2

i

e

t

c

m

p

l

h

i

(

d

2

a

t

f

O

f

a

u

m

d

f

p

b

i

estigated Q3. To understand the relative efficiency of motion marking

enu systems, we compared the efficiency of our marking menu pro-

otype to TalkBack TM , an accessibility menu system currently available

n Android smartphones.

. Related work

Related work includes marking menus for mobile devices, non-visual

nteractions with menus and mobile spatial interactions.

.1. Marking menus on mobile devices

Kurtenbach (1993) introduced marking menus that allow users to

erform fast, eyes-free menu selections. According to Kurthenbach and

uxton ’s case study of marking menus in a real-world situation, marking

akes interaction more efficient, easier to learn, and faster than selec-

ion using the menu ( Kurtenbach and Buxton, 1994 ). Recently, mark-

ng menus have been adapted to mobile devices. Jain and Balakrish-

an (2012) developed a marking gesture based mobile text entry system

hich requires less visual attention from users. pieTouch ( Ecker et al.,

009 ) is a marking gesture based vehicle information system designed

o reduce visual demand. Francone et al. (2010) presented touch-based

arking menus for navigating data hierarchies on mobile phones. Also,

akley et al. ( Oakley and Park, 2007; 2009 ) demonstrated a marking

enu based eyes-free menu system with 3D rotational strokes. Their

ystem deviated from the traditional marking menu system which was

ne dimensional and involved dividing a 90 ° portion of rotational space

nto three targets of 30 degrees each. Bauer et al. (2013) presented a

arking menu system for eyes-free interactions with a large display us-

ng smartphones and tablets. In their study, a marking menu was placed

n the touch screen of a smartphone or tablet so that the user could

emotely interact with a large display.

Despite the fact that marking menus are promising for eyes-free in-

ut, questions about the capability of people with visual impairments to

avigate marking menus has not been investigated.

.2. Non-visual interactions with menus

Research studies have attempted to provide more accessible inter-

aces for users with visual impairments. Kane et al. (2008) presented a

pecialized touch-based interface for menu selections called Slide Rule.

lide Rule is a set of audio-based multi-touch interaction techniques that

nable people with visual impairments to access smartphone functions

ncluding making phone calls, mailing, and music performance func-

ions. Zhao et al. (2007) developed EarPod using touch input and sound

80

eedback for eyes-free menu selections. Audio-based text entry systems

ere also developed by Sánchez and Aguayo (2007) and Yfantidis and

vreinov (2006) . These systems used multi-tap, directional gestures and

udio feedback, to enable users to enter text on touchscreens. On the

ther hand, Oliveira et al. (2011) showed that spatial ability plays an

mportant role in the blind user ’s ability to use and perform accurately

ith touch-based text-entry methods.

The literature has demonstrated several accessibility features on mo-

ile devices that utilized touch-based gestures and speech feedback.

n mobile situations, people with visual impairments usually have one

and occupied with a cane or a guide dog leash ( Ye et al., 2014 ), while

ouch-based interfaces may require users to use both hands, one hand to

old the phone and the other to perform gestures. Although touch-based

nterfaces enable one hand operation using the thumb, user perfor-

ance in thumb-based interactions greatly relies on several factors such

s surface size, hand size and hand posture ( Bergstrom-Lehtovirta and

ulasvirta, 2014 ). For the purpose of use “on the go ”, 3D motion ges-

ures are suitable because they provide fast access and enable users to

perate the system with only one hand ( Negulescu et al., 2012; Ruiz

t al., 2011 ).

Some researchers studied the design space of motion gestures for

on-visual interfaces ( Wolf et al., 2011 ) and other researchers validated

ody-space gestures with a view to improving on-the-move interaction

erformance ( Guerreiro et al., 2008 ). Also, a previous study has reported

hat motion gesture interfaces were efficient and well received by users

ith visual impairments ( Dim and Ren, 2014 ). Some previously devel-

ped motion gesture interfaces in ( Dim and Ren, 2014 ) were based on

inear gestures performed to the left or to the right, e.g., to select a con-

act, users repeated the gesture, to the left for the previous contact and

ight for the next contact, until they found the desired contact name in

list.

To increase efficiency of motion gesture interfaces for users with vi-

ual impairments, we propose motion marking menus (MMM). MMM

llow users to perform menu selections by drawing marks in the direc-

ion of the desired menu with six degrees of freedom in 3D space.

.3. Mobile spatial interactions

Recently, there has been a growing interest in research regard-

ng input techniques in mobile devices that allow spatial input. For

xample, SideSight ( Butler et al., 2008 ) allows multitouch interac-

ions around the device. HoverFlow ( Kratz and Rohs, 2009 ) and Abra-

adabra ( Harrison and Hudson, 2009 ) provided spatial interactions in

obile devices using motion input in space around the device. Min-

ut ( Harrison and Hudson, 2010 ) is also a spatial interaction that al-

ows users to manipulate the whole device for input. On the other

and, Skinput ( Harrison et al., 2010 ) provides spatial input by allow-

ng users to appropriate the body for finger input. Peephole displays

Yee, 2003 ) offered spatial input that maps physical movements of a

evice to movements in a virtual world. VirtualShelves ( Li et al., 2009;

010 ) extends these techniques by treating the space around the user as

discrete set of regions (shelves), so that the user can access contents on

hese virtual shelves. Gustafson et al. (2010) presented Imaginary Inter-

aces i.e., spatial interactions that occur only in the user ’s imagination.

h et al. (2013) proposed and evaluated a gesture sonification inter-

ace which generates various sounds based on finger touches, creating

n audio representation of gesture.

Romano et al. (2015) conducted a preliminary elicitation study to

nderstand the preferences of blind people with respect to touch and

otion gestures on mobile devices. In addition, several studies have

emonstrated spatial interaction in mobile devices as a promising eyes-

ree input modality.

Our study investigates the spatial ability of people with visual im-

airments to perform motion marking menus for eyes-free input in mo-

ile phones. MMM differ from other solutions such as VirtualShelves

n that MMM inherit the advantages of traditional marking menus:

N.K. Dim et al. Int. J. Human-Computer Studies 109 (2018) 79–88

(

m

q

b

3

u

a

t

o

a

b

0

r

E

p

m

s

T

r

p

q

a

3

u

a

i

m

G

o

s

t

t

f

p

3

t

t

m

t

h

t

t

t

t

s

m

u

b

3

p

o

E

o

r

w

h

w

b

q

p

c

m

t

s

t

c

t

c

4

i

r

p

p

W

f

4

t

m

o

8

c

t

g

2

o

T

O

w

n

m

e

s

p

t

E

t

E

(

o

t

d

l

o

T

p

4

f

1) MMM efficiently facilitate the transition from novice to expert

ode (our participants learned the motion marking menus reasonably

uickly), (2) MMM support hierarchical menus which can be performed

y compound markings.

. Preliminary interviews

In previous work ( Shinohara and Wobbrock, 2011 ), an interview was

sed to find out how assistive technologies are used by participants. We

lso conducted a preliminary interview with 10 visually impaired par-

icipants (8 males and 2 females). Ages ranged from 27 to 78 years. Five

f them were totally blind, two of them could distinguish between light

nd dark, and three could see objects, but none were able to distinguish

etween objects. Eight of the participants were early-blind (ages from

to 3 years) and two of them were late-blind (at 6 years and 18 years

espectively). The interviews took around one hour for each participant.

ach participant was paid $10 for their participation.

The interview questions were designed to investigate (i) current

roblems with the mobile phones they were using, (ii) the potential of

arking menus and spatial interactions for eyes-free interaction, (iii) the

patial awareness in the daily lives of people with visual impairments.

he final set of the interview questions achieved a high inter-rater

eliability among authors and one independent rater (Kappa = 0.919,

< 0.001). After the interview, data were analyzed by clustering the

uotes and identifying common themes using a bottom-up inductive

nalysis approach.

.1. Current problems with mobile phones

Seven of the participants used feature phones and the other three

sed smartphones. When asked about their current mobile phone us-

ge, the feature phone users commented mostly on the limited features

n their mobile phones and the need for faster operation. All of them

entioned that they would like to have access to more utilities such as

PS, calendar and weather. The smartphone users mostly commented

n the fatigue they experienced using their smartphones. Current acces-

ibility features in smartphones support flick gestures and speech output

o browse menus on the screen. One of our participants stated, “Many

imes (when using the smartphone), I want to jump the cursor to the

unction I want to select. Sometimes the guiding voice is frustrating in

ublic places. ”

.2. Potential of marking menus and spatial interactions

Through the interview study, we learned that many interactions in

he daily lives of people with visual impairments are facilitated by spa-

ial awareness and kinesthetic memory. All participants consistently

entioned that performing their daily tasks was mostly facilitated by

ouch, relocation of objects that they use each day and by memorizing

abits they learned through repetition. When asked about daily tasks

hey could successfully perform by repetition and habit, one of the par-

icipants replied, “I can put an appropriate amount of water into a jug

o make coffee even though I cannot see it. ” The same participant men-

ioned, “It is not that difficult to do daily tasks. But I am in trouble if

omeone has moved things that I usually use. ”

The important point to note with marking menus is that the physical

ovements performed when selecting a menu item are embedded in the

ser ’s muscle memory. This in turn suggests that marking menus could

e used as eyes-free interactions for people with visual impairments.

.3. Spatial awareness

When asked about their awareness of directions, all participants ex-

ressed difficulty with cardinal directions (i.e. North, West, etc.). Most

f the participants were not familiar with directions of a compass (N, S,

, W) or the twelve divisions on a clockface (e.g. “at 2 o ’clock ”). One

81

f the participants mentioned, “I used a tactile watch before, so I can

oughly say the directions of hours on a clock. For example, 8 o ’clock

ould be at the lower-left of my body. ” Another participant stated, “I

ave never used a compass though a compass with sound or texture

ould be usable. ” All participants agreed that directions relative to their

ody (i.e. left, right, etc.) were easy to understand and that they fre-

uently used those directions to arrange items they used each day. One

articipant mentioned, “I put my daily items around my chair where I

an easily access them, for example, my mobile phone at my right side,

y radio or charger at my left side. ” All participants mentioned that

hey preferred saying directions using left/right because they are con-

tant relationships with regard to the body. Answers regarding the spa-

ial awareness of our participants suggested that their familiarity with

ompass or clock layouts depends on the individuals ’ training in visual

hinking (e.g., whether they were exposed to and taught to read a braille

lock or a raised compass).

. Study 1: Suitable menu items and levels for people with visual

mpairments

Study 1 was conducted to answer two questions, Q1. How many di-

ections can people with visual impairments distinguish to successfully

erform marking menu selections? Q2. How many hierarchic levels can

eople with visual impairments navigate in marking menu selections?

hat is the optimum number of hierarchical levels on a marking menu

or efficient navigation by people with visual impairments?

.1. Experiment design

Experimental trials to establish the number of items (breadth) and

he number of levels (depth) were designed as within-subject repeated

easures. The participants performed menu selections in 16 menu lay-

uts of 4 breadths (angular width 90°), 6 breadths (angular width 60°),

breadths (angular width 45°) and 12 breadths (angular width 30°)

rossed with depths from 1 to 4 levels. The angular threshold was

hree degrees. For example, in the 4-breadth menu, the acceptable an-

ular ranges for the ’down ’ gesture was between 138 ( = 135+3) and

22 ( = 225-3) degrees, considering clock-wise increase starting from 12

’clock origin. Each menu breadth had four depths from 1 to 4 levels.

he schematic diagram of menu breadth and levels is shown in Fig. 2 .

ur interview study informed us that people with visual impairments

ere more familiar with body-centric directions (left, right) than cardi-

al directions (east, west). According to these findings, we labeled all

enus in relation to the human body (left, right).

The rationale for selecting the number of menu items and menu lev-

ls was based on the experimental design from a previous marking menu

tudy ( Kurtenbach, 1993 ). We also added 6-item menus because we hy-

othesized that this menu layout could be a good option if menu selec-

ions in 8-item menus were found too error prone for our participants.

In each menu layout, three different menu selections were presented.

ach selection was presented three times. Three different menu selec-

ions were configured to include both easy and difficult target menus.

asy menus were those that existed along vertical and horizontal axes

i.e. left, right, up, down). Difficult menus were those that existed in

ff-axis positions (i.e. upper-left, upper-right, etc.). We paid attention

o ensure that menu selections included easy, moderately difficult and

ifficult menus. Each participant performed 432 trials in total (16 menu

ayouts × 3 menu selections × 3 repetitions × 3 blocks = 432). The

rder of the menu selections was counterbalanced using a Latin Square.

he occurrences of menu layouts were randomized among the partici-

ants.

.2. Participants

Twelve participants (2 females and 10 males), with ages ranging

rom 27 to 78 years, participated in the experiment. Seven of them par-

N.K. Dim et al. Int. J. Human-Computer Studies 109 (2018) 79–88

Fig. 2. Schematic diagram of menus used in the experiment. (a) menu items (breadth), (b) menu levels (depth). To select menus at more than one level, users perform continuous

marking gestures in the direction of the target menus (e.g., right, then downward). c) Tactile patterns for twelve directions. The participants were trained with regard to the directions

using tactile patterns at the parctice session.

t

v

a

o

s

t

i

4

n

l

t

o

2

×

T

t

C

v

c

s

d

g

t

V

n

s

4

U

c

e

f

Fig. 3. Participant performing menu selection.

w

f

u

w

t

t

a

icipated in the preliminary interviews. Two of the participants had light

isual perception and three of them could see objects, but none were

ble to distinguish between objects. The rest were totally blind. Two of

ur participants were late blind (onset at age 22 years and 48 years re-

pectively), the rest were early blind (onset from ages 0 to 3 years). All

he participants were right-handed. Each was paid $15 for their partic-

pation.

.3. Apparatus

Our study required a sound motion capturing system to avoid sensor

oise confusion. Thus, for capturing user hand movement in menu se-

ections, we used twelve Bonita 10 cameras (frame rate: 250 fps, resolu-

ion: 1 megapixel (1024 × 1024), lens operating range: up to 13m, angle

f view wide (4mm): 70.29° × 70.29°, angle of view narrow (12mm):

6.41° × 26.41°).

The participants held an Alcatel OneTouch Dimension - 5.37 × 2.74

0.30 in, Weight - 1.6g) smartphone for performing motion gestures.

en markers (14mm wide) were attached on the smartphone, the par-

icipant ’s wrist, forearm, elbow, arm and shoulder ( Fig. 3 ).

For the Vicon system, Nexus 2.2.1 motion capture software was used.

ustom software was also developed to track user hand movement data

ia the Vicon system. When a user started and ended a gesture, the

ustom software logged frame numbers from the Vicon cameras and po-

ition data from the markers which were attached to the body. Those

ata were later used for calculating errors and the response times of

estures. The software was also used to trigger real-time instructions to

he participant. Gesture recognition was stroke-by-stroke recognition.

elocity threshold of the movements (i.e., 4mm/s) was used to recog-

ize the final stroke. The motion tracking system ’s accuracy was tested

everal times to confirm that the desired movements were achievable.

.4. Building models

Each participant used their dominant hand to perform the gestures.

ser models were built for both left and right hands. Each model in-

luded four segments: shoulder, arm, forearm and a phone. Three mark-

rs were used for the shoulder segment, two for the arm, two for the

orearm and three for the phone.

82

One marker on the shoulder, one on the elbow and one on the phone

ere used to connect two segments, that is, connecting the hand and the

orearm, connecting the forearm and the upper arm, and connecting the

pper arm and the shoulder.

To enable the system to differentiate between the left and right hands

ith no errors, markers on the left arm were placed in different positions

o those on the right arm. Markers on the left arm were placed higher

han those on the right hand so that the two ratios would be different

nd the system would not confuse the left and the right arms.

N.K. Dim et al. Int. J. Human-Computer Studies 109 (2018) 79–88

4

t

s

t

e

a

t

t

a

d

a

t

t

c

e

l

i

s

f

s

p

W

(

t

t

t

w

f

s

p

r

O

p

t

t

fi

t

c

o

a

4

a

m

(

m

a

e

b

W

b

W

d

f

b

1

s

R

Table 1

Mean error rates. Standard errors are shown in paren-

theses.

Breadth Level

1 2 3 4

4 0.0 0.0 0.93 0.0

(0.0) (0.0) (0.93) (0.0)

6 0.93 2.78 10.04 20.34

(0.93) (2.46) (3.03) (4.01)

8 0.0 2.78 7.41 15.74

(0.0) (1.45) (3.16) (2.54)

12 35.18 24.07 36.33 37.96

(2.68) (4.90) (2.38) (3.73)

w

(

f

b

a

a

d

(

m

s

m

t

W

t

t

b

m

h

g

p

(

I

r

w

o

0

4

a

d

f

b

t

f

b

m

b

e

8

s

s

b

m

a

F

.5. Procedure

Participants ’ consent forms were gathered before the trials. The par-

icipants were then introduced to the purpose of the study and to the

tudy procedure. Four of our participants used their left hands to hold

he phone and to perform the gestures. Markers were put in place, and

ach participant was handed a smartphone on which three markers were

ttached.

The participants were then taught the menu layouts. To help the par-

icipants learn the menu layouts, we put tactile patterns of directions on

he wall ( Fig. 2 c). The participants were exposed to the tactile directions

nd allowed to practice until they felt that they were familiar with all

irections and direction labels. The participants were also trained with

ll menu layouts via verbal instructions. Then they were trained with

he experimental custom software to help them become familiar with

he experimental setup. To minimize the learning effect, we used 4 × 1

ondition which was the easiest layout for menu selections.

After training and practice sessions, the experiment started. Each

xperimental trial occurred as follows. The participant stood in a ‘re-

axed ’ state with the arms beside the body. Then the system ’s speaker

nstructed the participants to adopt a ‘ready ’ state. The participant re-

ponded by moving the dominant hand to any position in which they

elt comfortable to start the menu selection. After 500 milliseconds, the

ystem read out the menu name that the participant had to select. The

articipant responded by moving the dominant hand to select the menu.

hen the speed of the user ’s hand movements exceeded the threshold

i.e. 4mm/s) the system logged that point as the starting point of a ges-

ure. When the user slowed the speed down or stopped arm movement,

hat is, when the movement speed was less than the pre-established

hreshold and if the distance between the start point and the end point

as more than 20 mm, the system logged that point as the ending point

or the gesture. For menu selections with more than one menu level, the

ystem read out the next menu to be selected while the participant was

erforming the gesture for the current menu level. This procedure was

epeated for each gesture stroke until all menu levels were completed.

nce the participants finished menu selection, the system instructed the

articipant to return to the ‘relaxed ’ state. After three seconds, the next

rial started with the same procedure. No backward or reselecting of

argets was allowed. All the experimental instructions were made in the

rst language of each participant. We measured errors and menu selec-

ion times.

After the experiment, the participants were asked for their subjective

omments on each menu layout and direction that were particularly easy

r difficult for them to understand and to perform. Participants were also

sked for comments on the gestures.

.6. Results

We measured errors and menu selection times. Errors were defined

s errors in the angle of the participants ’ hand movements in respective

enu layouts. Angular errors were calculated for each gesture stroke

each menu level). Thus, if one of the gesture strokes with a particular

enu level had an angle error, that menu selection was regarded as

n erroneous menu selection. Response time was defined as the time

lapsed until the completion of the menu from the start of the gesture

y the participant.

Each participant performed three blocks of trials in the experiment.

e first checked the learning effect on menu selection over the three

locks of trials to see if the data collected had reached a level of stability.

e analyzed the error rates after each experiment block. Error rates

ecreased over blocks. The average error rates were 12.67 (SD = 1.31)

or block 1, 11.11 (SD = 0.87) for block 2, and 8.31 (SD = 0.83) for

lock 3. Significant differences in error rates were found between block

and block 3 (p < 0.05) and between block 2 and block 3 (p < 0.05). No

ignificant difference was found between block 1 and block 2 (p = 0.27).

esponse time also decreased over blocks. The average response times

83

ere 2.55 (SD = 0.33) for block 1, 2.43 (SD = 0.39) for block 2, and 2.03

SD = 0.14) for block 3. Significant differences in response times were

ound between block 1 and block 3 (p < 0.05) and between block 2 and

lock 3 (p < 0.05). No significant difference was found between block 1

nd block 2 (p = 0.33). Apparently, participant performance was steady

fter two blocks. Thus, taking the learning effect into account, we used

ata only from the third block for the rest of our analysis in this section.

In addition, we removed errors caused by “mental slip ”

Norman, 1981 ). “Mental slip ” is common human error caused by

omentary loss of concentration. For example, sometimes when in-

tructed to select the menu on the left, a participant would accidently

ove the hand to the right. We did not include these types of errors in

he data sets because they were not caused by selection inaccuracies.

e removed forty-one “mental slip ” errors from the data set. During

he experiment, the participants were given real-time instructions using

he custom software. This software was developed to give instructions

ased on thresholds set for vector speeds of participant hand move-

ents (e.g., start moving the hand for the gesture, stop moving the

and after the gesture, etc.). In a few cases, the software could not

ive the instruction and could not log the movement data because the

articipants ’ hand movements did not match the system ’s thresholds

e.g., the participants moved more slowly than the threshold speed).

n such cases, we recorded the trial number of those movements and

emoved those data from the data set.

A two-way repeated-measure ANOVA (analysis of variance) was used

ith the two factors of the number of items (breadth) and the number

f levels (depth). All tests were run at a significant level of alpha ( 𝛼) =.05.

.6.1. Errors

Using a two-factor repeated measures ANOVA, we did statistical

nalyses for error rates. The results of Mauchlys test of Sphericity in-

icated that there were no violations of the assumption of Sphericity

or breadth and the depth. However, Mauchlys test of Sphericity for

readth x depth interaction was violated, x 2 (44) = 88.8, p < 0.01. So,

he Greenhouse-Geisser procedure was applied to correct the degree of

reedom of the F-distribution, F(3.41, 37.51) = 3.76.

The results show that there is a significant main effect for the menu

readth, F(3, 33) = 118.1, p < 0.001; a significant main effect for the

enu depth, F(3, 33) = 25.5, p < 0.001; and a significant interaction

etween these two factors, F(3.41, 37.51) = 3.76, p < 0.02.

Post hoc pair-wise analysis with Bonferroni correction indicated that

rror rates were not significantly different among 4-item, 6-item and

-item menus, up to 2 levels in depth. However, error rates became

ignificantly higher for 6-item and 8-item menus than for 4-item menus

tarting from 3 levels (p < 0.01). No significant difference was found

etween 6-item and 8-item menus at any level. Error rates in 12-item

enus were significantly higher than error rates in other menu layouts

t any level (p < 0.01). Error rates in each menu layout are shown in

ig. 4 and Table 1 .

N.K. Dim et al. Int. J. Human-Computer Studies 109 (2018) 79–88

Fig. 4. Percentages of errors in each menu layout.

Fig. 5. Response times in each menu layout.

Table 2

Mean response times. Standard errors are shown in

parentheses.

Breadth Level

1 2 3 4

4 1.19 1.59 2.53 3.36

(0.17) (0.11) (0.30) (0.25)

6 1.41 1.85 2.73 4.03

(0.29) (0.21) (0.21) (0.29)

8 1.14 1.91 2.73 3.75

(0.11) (0.91) (0.22) (0.31)

12 1.36 2.16 3.30 3.87

(0.13) (0.22) (0.26) (0.24)

4

1

a

<

t

m

t

l

s

l

e

m

Table 3

Criteria for acceptable error rates and usable menu layouts (The error rates are shown in

parentheses).

Criteria Usable menu layouts

Error < 1% 4 × 1 (0.0), 4 × 2 (0.0), 4 × 3 (0.93), 6 × 1 (0.93), 8 × 1 (0.0)

1% < Error < 3% 6 × 2 (2.78), 8 × 2 (2.78)

3% < Error < 10% 8 × 3 (7.41)

4

v

m

w

n

o

c

l

t

g

t

o

a

f

(

o

i

l

d

i

a

p

l

b

(

f

l

a

r

v

a

o

1

(

a

u

e

a

l

p

s

g

l

a

u

o

w

.6.2. Response time

Response time was affected by both the menu breadth (F(3, 33) =1.99, p < 0.05) and the menu level (F(3,33) = 65.84, p < 0.05). Breadth

nd level also interacted to affect the response time (F(9, 99) = 3.04, p

0.05). Post hoc pair-wise analyses with Bonferroni correction indicated

hat 4-item menus were significantly faster than 6-item menus, 8-item

enus and 12-item menus. No significant differences were found be-

ween 6-item and 8-item menus. However, response times were slightly

onger in 6-item menus compared with 8-item menus due to the inclu-

ion of more off-axes items in 6-item menus. Response times increased

inearly as the number of menu levels increased ( Fig. 5 ), and the differ-

nces were significant at any level (p < 0.001). Response times in each

enu layout are shown in Table 2 .

84

.7. Discussion

The experiment results and qualitative data collected in the inter-

iew enabled us to answer our research questions relating to marking

enus for users with visual impairments.

Q1) How many directions can people with visual impairments distinguish?

From our study and preliminary interviews, we observed that people

ith visual impairments are less aware of directions in terms of cardi-

al directions or clockwise directions (e.g., 2 o ’clock point) than of body

riented directions (e.g., left, upper-left). Participants were able to suc-

essfully perform marking menus for 4-item, 6-item and 8-item menus

abelled for body oriented directions. Thus, in general, it was reasonable

o conclude that people with visual impairments can perform directional

estures in up to 8 directions.

The participants could distinguish the menu directions well when

he directions were labelled in relation to the human body (whether left

r right). The literature has also reported that, although people who

re congenitally blind have more difficulty in representing spatial in-

ormation allocentrically, tasks requiring egocentric frames of reference

relating to one ’s body) were similarly performed by early blind (early

nset of blindness), late blind and sighted people ( Iachini et al., 2014 ).

Q2) How many hierarchic levels can people with visual impairments nav-

gate in marking menu selections?

Error rates are the main limiting factors for the number of hierarchic

evels in marking menus ( Kurtenbach, 1993 ). Acceptable error rates vary

epending on the consequences of errors and the purpose of use. Thus,

n table Table 3 , we showed menu layouts with error rates less than 10%

s suggested in ( Kurtenbach, 1993 ).

A previous marking menu study which was performed with sighted

eople recommended using menus with a breadth of 4 up to 4

evels (maximum error rate 5.10%, SD = 4.20) and menus with a

readth of 8 up to level 2 (maximum error rate 8.82%, SD = 4.62)

Kurtenbach, 1993 ). In our study, the nearest error rate to 8.82% was

ound when the participants performed gestures for 8-item menus up to

evel 3 (error rate 7.41%, SD = 3.16). Comparing results from our study

nd those from ( Kurtenbach, 1993 ), our results indicated lower error

ates in all menu layouts. Thus, it is questionable whether people with

isual impairments have any advantage over sighted people in spatial

bility when navigating hierarchic marking menus.

In addition to the differences between participants with and with-

ut visual impairments from our study and those in (Kurtenbach,

993), there is also the difference regarding input modalities. In

Kurtenbach, 1993 ), the participants drew marks on the screen using

pen or a mouse. In our study, the participants performed markings

sing motion gestures. Physical movement with motion gesture is an

xpressive channel which has six degrees of freedom that can be easily

pplied for proprioception. Moreover, menu direction in our study was

abeled in relation to the human body (left, right) which allowed the

articipants to rely on kinesthetic cues (awareness of object positions in

pace with respect to one ’s body).

In our experiments, the used menus were largely divided into three

roups: one group consisted of entirely on-axis menus (i.e. up, down,

eft and right), another group consisted of a mixture of on-axis and off-

xis menus, and the last group consisted entirely of off-axis menus (i.e.

pper-left, lower-right, etc). To find out whether these different layouts

f menu configuration effect error rates among menu breadth and depth,

e analyzed user performance in these three different menu configura-

N.K. Dim et al. Int. J. Human-Computer Studies 109 (2018) 79–88

Fig. 6. The menu systems used in Study 2. (a) TalkBack - linear gestures, (b) TalkBack - spatial localization. Four menus were arranged horizontally. Each menu is about 1.75cm wide

and about 1.5cm high, (c) Motion Marking Menus.

t

=

i

r

h

m

t

m

1

c

d

f

l

f

5

f

t

s

w

p

b

m

c

p

g

t

p

A

m

U

t

5

m

l

a

t

p

F

o

p

c

m

t

=

o

e

5

t

o

r

$

(

W

i

w

a

p

s

u

d

i

n

I

l

i

t

A

i

r

o

5

e

f

t

g

l

p

a

p

t

o

ions. As we expected, off-axis had a significant effect on errors (F(2, 22)

16.2, p < .05). We also found that axis and menu levels had significant

nteractions on errors (F(8, 88) = 30.56, p < .05). This was particularly

eflected in 6-item menus. In 6-item menus, errors become significantly

igher in level 3 (up to 10.04). As the number of menu levels increased,

ore off-axis items were included in the combination because most of

he menus existed off-axis (60, 120, 240 and 300 degrees) in 6-item

enus. Similarly, in 8-item menus, errors increased from 7.41 up to

5.74 in level 4.

To evaluate the most challenging strokes/directions of motion, we

ompared performance in pairs of menus (i.e. left/right, up/down,

own-left/down-right and up-left/up-right). In general, less errors were

ound for right and up-right directions when compared to left and up-

eft. However, no significant effect from direction on performance was

ound for any direction.

. Study 2: efficiency of motion marking menus in smartphones

In Study 1, we investigated usable menu layouts for marking menus

or people with visual impairments. Study 2 was intended to investigate

he feasibility of proposed interactions in terms of user performance and

ubjective assessment using the motion marking menu system. Study 2

as intended to answer Q3: How receptive are people with visual im-

airments to motion marking menu systems for mobile interactions? To

etter understand the relative efficiency and user assessments of the

arking menu system, we compared our prototype to TalkBack TM , a

ommercial menu system on Android devices for users with visual im-

airments.

In TalkBack, users can browse the menus either by directional flick

estures, that is, right for the next content and left for the previous con-

ent ( Fig. 6 a) or by localizing the menu contents (i.e. by estimating the

osition of the menu on the screen and directly pointing on it ( Fig. 6 b).

fter each gesture or spatial localization, the system reads out the active

enu name. Users gesture or localize until they find the desired menu.

sers confirm the menu by performing a double-tap on the screen. Users

hen perform the same procedure to find the submenu.

.1. Experimental design

We developed a proof-of-concept prototype for motion marking

enus with a breadth of four and a depth of two, i.e., 4 × 2 menu

ayout. Two mobile phone applications, phone book and music, were

ssigned for the high-level menu items. Four submenus were assigned

o each high-level menu. That is, four contact names were assigned in the

hone book menu and four submenus were assigned in the music menu

ig. (6) . Before the experimental tasks, a pilot study was conducted with

ne participant with visual impairments and two blind-folded partici-

ants. We chose four by four menu selections in order to minimize the

ognitive load of the participants when remembering the menu layout.

85

The participants selected each submenu three times. The same eight

enu items were assigned in the TalkBack system. Therefore, each par-

icipant performed 48 (2 menu systems × 8 menu selections × 3 blocks

48) menu selections in total. The design was within-subjects and the

rder of experiment was counter balanced. Task completion times and

rror rates were measured.

.2. Participants and apparatus

Twelve participants (3 female, 9 male) took part in Study 2. Nine of

hem had participated in Study 1. Ages ranged from 26 to 78 years. Four

f them were smartphone users and they were experienced with screen

eader functions on smartphones. All were right-handed. Each was paid

10 for their participation.

For both menu systems, an Alcatel OneTouch Idol 2S smartphone

Processor - Quad-core 1.2 GHz, Dimension - 5.37 ×2.74 ×0.30 in,

eight - 1.6g) was used. In order to test the feasibility of the proposed

nteraction technique, a prototype of the motion marking menu system

as implemented using a built-in accelerometer sensor. The recognition

lgorithm used a dual-axis tilt calculation method that support com-

lete 360 ° tilt sensing. To reduce sensitivity mismatch, the offset and

ensitivity were calibrated and the calibrated acceleration output was

sed to calculate the angle of inclination. The angular threshold was 3

egrees. The system was implemented in Java language. For the mark-

ng menu system, both novice and expert modes were implemented. In

ovice mode, the system read out all menu items at each menu level.

n expert mode, only the final target menu was read out once it was se-

ected. Novice mode was used to let the participants discover the menu

tems and practice menu selections. After practicing in novice mode,

he participants used only the expert mode in the experimental trials.

lthough the motion sensing system was different from sensing method

n Study 1, through pilot studies, we made sure that the sensing algo-

ithms supported similarly acceptable accuracy for the menus used in

ur experiment.

.3. Procedure

Each participant ’s consent form was collected before the trials. The

xperimenter then demonstrated each system and explained how to per-

orm each task. The participants were allowed to practice with each sys-

em. For TalkBack, the participants were trained with both directional

estures ( Fig. 6 a) and spatial localization ( Fig. 6 b), and they were al-

owed to use either method during the experiment. Only two of the

articipants chose to use spatial localization. The practice session took

round 15 minutes. Once the practice session was completed, the ex-

eriment began. The order of menu systems was randomized among

he participants. Half of the participants started with TalkBack, and the

ther half started with the marking menu system. For each technique,

N.K. Dim et al. Int. J. Human-Computer Studies 109 (2018) 79–88

Fig. 7. Selection times using MMM and TalkBack systems.

t

t

r

r

a

e

“

t

(

w

W

T

5

c

t

s

c

r

0

b

a

r

f

r

b

f

t

o

F

5

s

<

s

0

i

t

b

m

a

u

w

(

f

(

a

f

Table 4

Mean selection time for Talkback TM in each

menu level after discarding speech feedback

times. Contacts/Playlists 1 to 4 were assigned

in ascendant order.

Main menu Submenu Selection time

Phone Contact 1 3.03 (0.88)

Contact 2 3.56 (0.81)

Contact 3 4.75 (1.20)

Contact 4 6.01 (1.89)

Music Playlist 1 3.89 (0.66)

Playlist 2 5.34 (1.90)

Playlist 3 5.46 (1.97)

Playlist 4 6.82 (1.92)

Fig. 8. Subjective assessment on the two menu systems. The lower number indicates less

fatigue in the Fatigue assessment. In other assessments, the higher number indicates the

more satisfaction and ease of use.

s

r

1

(

5

i

a

s

t

f

i

t

t

m

r

f

i

c

a

t

p

t

p

p

s

i

t

a

b

i

t

b

he participants performed 24 trials for each of the following applica-

ions: (i) phone book (ii) music. Menu selection times and errors were

ecorded. All the experimental trials were video recorded.

After the experimental trials, the participants were asked to rate the

espective systems in terms of ease of use, satisfaction with the systems

nd fatigue when performing the menu selections, using a 7-point Lik-

rt scale. They were also asked to give a rating based on the statement,

I would like to use them if marking menus and motion gesture in-

erfaces were available on smartphones, ” using a 7-point Likert scale

1 = strongly disagree and 7 = strongly agree). The experiment ended

ith the participants making questions or comments if they had any.

e recorded all the participants comments for qualitative data analysis.

he experiment took around one hour.

.4. Results

We analyzed menu selection times, error rates and the subjective

omments of our participants after they had used the two menu sys-

ems. Participants performed three blocks of trials and we checked to

ee whether there were any learning effects by repeating trials. We also

onfirmed that the data collected had reached a level of stability. Error

ates decreased over blocks. The average error rates were 0.35 (SD =.11) for block 1, 0.20 (SD = 0.09) for block 2, and 0.15 (SD = 0.07) for

lock 3. However, no significant differences in error rates were found

mong blocks. Response time also decreased over blocks. The average

esponse times were 20.78s (SD = 23.58) for block 1, 11.83s (SD = 10.91)

or block 2, and 6.89 (SD = 5.73) for block 3. Significant differences in

esponse times were found between block 1 and block 2 (p < 0.01) and

etween block 1 and block 3 (p < 0.01). No significant difference was

ound between block 2 and block 3 (p = 0.16). Like in Study 1, for selec-

ion times and errors, we analyzed the data from block 3. In addition, as

nly two participants selected the localization option in TalkBack (see

ig. 6 b), we removed their data from the analysis.

.4.1. Selection times and errors

A one-way repeated measures ANOVA test showed that the menu

ystems had a significant effect on selection times (F(1,9) = 86.10, p

0.05). The average selection time in TalkBack was 12.2 (SD = 3.74)

econds while the selection time in the marking menus was 1.88 (SD =.49) seconds. The menu selection times in the two systems are shown

n Fig. 7 .

We remark that, in TalkBack, the speech feedback after every ges-

ure or spatial localization was one cause of the slow selection times,

ecause users needed to wait for voice feedback after each gesture. To

ake selection times in the two menu systems more comparable, we also

nalyzed the data after discarding the audio feedback time in TalkBack

sing a one-way repeated measures ANOVA. Motion marking menus

ere significantly faster than for the TalkBack system without feedback

F(1,9) = 68.70, p < 0.05). We also analyzed the selection times ranging

rom the best cases to the worst cases, that is, where the target menus

main menu and submenu) were located at the top in the menu list,

nd vice versa. Table 4 shows selection times in TalkBack with voice

eedback times discarded.

86

Regarding selection errors, a one-way repeated measures ANOVA

howed that the menu systems had no significant effect on the error

ates (F(1,9) = 1.46, p = 0.26). The average error rate in TalkBack was

.99 (SD = 3.41) while the error rate in the marking menus was 4.64

SD = 6.38).

.4.2. Subjective assessment

After experimental trials, participants were asked to rate how easy

t was to complete the tasks, how tiring it was to complete the tasks

nd how satisfying the system was, for each system. Fig. 8 shows the

ubjective ratings of the participants. The participants scored higher for

he motion marking menu system in all dimensions except for fatigue. A

urther Wilcoxon signed-rank test showed significant differences in sat-

sfaction (z = -2.75, p < 0.05) and fatigue (z = -2.03, p < 0.05), however

his test revealed no significant difference for ease of use. When asked

o rank the statement regarding how much they want to use motion-

arking menus, 7 participants rated strongly agree and 3 participants

ated to agree. Two of the participants preferred to use the traditional

eature phones that they were currently using.

Most of the participants made positive comments about the mark-

ng menu system. One of the participants who was a smartphone user

ommented, “using the motion marking menu was like using shortcuts

nd it was very efficient. ” The participant also mentioned, “Sometimes,

he continuous voice guidance in current smartphones is frustrating in

ublic places. ” Another participant who did not use a smartphone men-

ioned, “Both TalkBack and marking menus were great, really enabling

eople with visual impairments to use smartphones. ” Ten out of twelve

articipants preferred the motion marking menu system to the TalkBack

ystem.

The participants ’ comments also showed that there are learnability

ssues in motion marking menu system. Our participants commented

hat it was hard to remember all the menu items in a short time, for ex-

mple, one of the participants stated, “The gestures are easy to perform

ut it ’s difficult to remember menu positions. But it would be different

f I could arrange the menus myself. ” The same participant commented

hat customizing gestures for calling frequently called contacts would

e very useful. For the TalkBack system, only two of the participants

N.K. Dim et al. Int. J. Human-Computer Studies 109 (2018) 79–88

u

f

p

m

p

d

m

o

c

w

s

t

p

5

t

t

s

s

p

p

o

a

m

t

i

w

s

t

r

o

M

t

s

t

e

a

i

t

t

g

m

t

t

b

f

s

6

c

m

d

p

6

(

B

t

t

d

t

t

6

t

c

b

i

a

m

e

s

b

6

i

t

t

a

m

s

l

t

g

t

i

w

n

a

i

w

b

6

t

l

u

i

m

s

t

w

f

6

a

a

F

g

b

w

n

a

sed spatial localization to select the menus stating that it was difficult

or them to remember the menu positions on the screen.

When doing the gestures for menu selections, most of the partici-

ants used small movements (i.e. moving only the wrist). Using small

ovements, gestures to left were particularly difficult for the partici-

ants. One of the participants stated, “Leftward movements are very

ifficult. I should move my arm (i.e., large movements including elbow

ovement for gestures to left. ” Another participant stated, “I want to use

nly one stroke motion for frequently used functions such as answering

alls. ” Comments from the participants indicated that the participants

ere very alert to the potential benefit of using motion marking menu

ystems. For example, one of the participants stated, “I wish motion ges-

ure interfaces would become very popular and every visually impaired

erson could use them. ”

.5. Discussion

Recalling Q3, “How receptive are people with visual impairments

o motion marking menu systems for mobile interactions?, ” the subjec-

ive comments of our participants and their ratings on 7-point Likert

cales confirmed that they were very receptive to MMM (e.g., higher

atisfaction and ease of use, and lower fatigue than with TalkBack. In

articular, the subjective comments of our participants confirmed that

eople with visual impairments are receptive to motion marking menus

n smartphones as a way of interacting with smartphone devices.

While the key advantage of motion gesture interfaces is that they en-

ble one-handed operations, we also observed two important features of

otion gesture interfaces throughout the study. (1) Familiarity of the in-

eraction space is particularly important. Compared to motion gesture

nterfaces, our participants had much more difficulty becoming familiar

ith and performing the tasks on touchscreens. Tasks on touchscreens

uch as gestures and taps were particularly difficult for some of our par-

icipants. One of the participants stated, “I feel the phone screen cannot

espond to my gestures, may be because my fingers are too dry? ” An-

ther participant stated, “Double taps are just too difficult for me. ” (2)

otion marking menus provide more freedom with regard to interac-

ion boundaries, i.e., users can start the interactions anywhere in 3D

pace, so they are not limited to the space available on a device (e.g.,

ouchscreen) in the case of motion marking menus.

Nevertheless, there are situations where motion gestures are not nec-

ssarily the most suitable form of interaction. Some users may not be

ble to remember infrequently used menu items. In those cases, navigat-

ng the menus on touchscreens would be easier for users than recalling

hose menus from a motion marking menu system. Also, motion ges-

ures may not be appropriate for use in crowded public places because

estures require users to wave their phones around (even though the

ovements do not need to be large). More studies need to be conducted

o test the use of this interaction approach in the wild. Thus, we regard

hat currently available menu systems and motion marking menus can

e good as supplements. For example, users can use MMM for calling

requently called contacts and TalkBack for more complicated functions

uch as changing the system settings.

. Design implications and guidelines

Throughout the studies, we collected the participants ’ subjective

omments about gesture preferences, menu selections and preferred

enu layouts. Based on comments from the participants, we provided

esign guidelines for motion marking menus in mobile devices for peo-

le with visual impairments.

.1. Preferred gestures

In Study 1, all participants commonly stated that diagonal directions

i.e. upper-left, upper-right, etc.) were difficult to instantly understand.

87

ecause all our participants were right-handed, they mentioned that ges-

ures to the lower-left and upper-left directions were the most tiring ges-

ures to perform. Thus, designers should avoid gestures in the forehand

irections of the dominant hand for frequently used functions. The par-

icipants also mentioned that downward gestures were particularly easy

o perform because they followed the force of gravity.

.2. Menu layout

Our participants suggested that one motion (i.e. single stroke ges-

ure) was desirable for functions such as answering or hanging up a

all. Also, gesture customization should be supported wherever possi-

le. The subjective comments informed us that end-user customization

s particularly desirable for functions such as arranging contact names

nd customizing playlists. User customization of menu layouts can be

ade available for people with visual impairments so they can navigate

fficiently and comfortably. However, for default menus assigned by de-

igners, menu items of 4, 6, and 8 with menu up to 2 levels deep would

e the most preferable.

.3. Menu learnability

There is a need to consider the learnability of the entire menu layout

n novice mode for motion marking menus. One advantage of the tradi-

ional marking menu is that it helps users make efficient transitions from

he novice to the expert modes because it provides visual information

bout the entire menu layout in the novice mode. Once in the expert

ode, marking menu provides shortcuts to the target via continuous

trokes. For motion marking menus, information about the entire menu

ayout should be provided to users using speech before the user switches

o the expert mode. Once in the expert mode, users may select the tar-

et by a series of continuous motions. Although participants were able

o learn menu layout and transit to the expert mode reasonably quickly

n our experiment, more learnability would be achieved if menu items

ere arranged as close as possible according to each user ’s priorities and

eeds, e.g., place the most frequently used menus in on-axis positions,

nd arrange the menu items in meaningful ways. This is particularly

mportant because mental mapping is a great aid to learning for people

ith visual impairments ( Ungar, 2000 ). End-user customization would

e helpful for learnability and memorability of menu layouts.

.4. Gesture recognition

When performing menu selections in Study 2, we found that the par-

icipants had different movement preferences regarding length and ve-

ocity. It was difficult for the system to recognize gestures performed

sing movements with very low velocity. Gesture recognition should be

mplemented to allow more freedom of movement to the users. Also, as

ost of the participants used small gestures, accuracy needs to be as-

ured for fine movement gestures. However, it is not always preferable

o have very sensitive gesture recognition. Thus, in real applications, it

ould be a good idea to allow the end-user personalization of thresholds

or gesture recognition (e.g. slow movement, small gesture).

.5. Feedback

In our experiment, feedback was provided with the menu names read

loud when selections were successfully completed. Feedback and guid-

nce should also be provided when menu selections are not successful.

or example, users may not perform enough movement necessary to trig-

er menu selection. In that case, vibrations or speech guidance should

e issued to request more significant movements form the user. Also, it

ould be desirable to give diverse feedback options. Users may prefer a

on-verbal sound or vibration feedback, once they become more expert

t using the system.

N.K. Dim et al. Int. J. Human-Computer Studies 109 (2018) 79–88

7

m

w

e

t

(

p

m

b

a

w

u

c

s

o

d

p

A

e

e

t

t

i

K

t

m

R

B

B

B

D

E

F

G

G

H

H

H

I

J

K

K

K

K

L

L

N

N

O

O

O

O

R

R

S

S

U

W

W

Y

Y

Y

Z

. Conclusion

This paper aims at mitigating the effects of visual impairments in

enu selection on mobile phones. We proposed that marking menus

orking together with motion gestures could offer more natural and

fficient interactions on smartphones. We investigated and presented

he number of menu items (breadth) and the number of menu levels

depth) for motion marking menus with which people with visual im-

airments could perform well. We also compared the efficiency of a

otion-based marking menu system compared to a commercial touch-

ased linear menu system, TalkBack.

Results from Study 1 indicate that people with visual impairments

re able to efficiently perform directional gestures in up to 8 directions

hich are consistently referenced to the users own body (i.e. left, right,

p, down, upper-left, upper-right, lower-left, lower-right).

Results from Study 2 indicate that marking menus are faster than the

urrent commercial menu system. We also found that people with vi-

ual impairments were very receptive to motion marking menu systems

n smartphones. Through the analyses of qualitative data, we provided

esign guidelines for motion marking menus for people with visual im-

airments.

cknowledgements

This study has been partially supported by the Grant-in-Aid for Sci-

ntific Research by MEXT ( Ministry of Education, Culture, Sports, Sci-

nce and Technology of Japan ), under Grant No. 25330241 . In addition,

his research was supported by Basic Science Research Program through

he National Research Foundation of Korea (NRF) funded by the Min-

stry of Education (Grant No. NRF-2017R1D1A3B03033353) and by the

eimyung University Research Grant of 2017. Authors would like to

hank the participants, Zhenxin Wang, Guanghui Chen, Qi Fang and

embers of CHEC for their great effort and support.

eferences

auer, J. , Ebert, A. , Kreylos, O. , Hamann, B. , 2013. Marking menus for eyes-free inter-

action using smart phones and tablets. In: Availability, Reliability, and Security in

Information Systems and HCI. Springer, pp. 481–494 .

ergstrom-Lehtovirta, J. , Oulasvirta, A. , 2014. Modeling the functional area of the thumb

on mobile touchscreen surfaces. In: Proceedings of the SIGCHI Conference on Human

Factors in Computing Systems. ACM, pp. 1991–2000 .

utler, A. , Izadi, S. , Hodges, S. , 2008. Sidesight: multi-touch interaction around small

devices. In: Proceedings of the 21st annual ACM symposium on User interface software

and technology. ACM, pp. 201–204 .

im, N.K. , Ren, X. , 2014. Designing motion gesture interfaces in mobile phones for blind

people. Journal of Computer Science and Technology 29 (5), 812–824 .

cker, R. , Broy, V. , Butz, A. , De Luca, A. , 2009. pietouch: a direct touch gesture inter-

face for interacting with in-vehicle information systems. In: Proceedings of the 11th

international Conference on Human-Computer interaction with Mobile Devices and

Services. ACM, p. 22 .

rancone, J. , Bailly, G. , Lecolinet, E. , Mandran, N. , Nigay, L. , 2010. Wavelet menus on

handheld devices: stacking metaphor for novice mode and eyes-free selection for ex-

pert mode. In: Proceedings of the International Conference on Advanced Visual Inter-

faces. ACM, pp. 173–180 .

uerreiro, T. , Gamboa, R. , Jorge, J. , 2008. Mnemonical body shortcuts: improving mo-

bile interaction. In: Proceedings of the 15th European conference on Cognitive er-

gonomics: the ergonomics of cool interaction. ACM, p. 11 .

ustafson, S. , Bierwirth, D. , Baudisch, P. , 2010. Imaginary interfaces: spatial interaction

with empty hands and without visual feedback. In: Proceedings of the 23nd annual

ACM symposium on User interface software and technology. ACM, pp. 3–12 .

arrison, C. , Hudson, S.E. , 2009. Abracadabra: wireless, high-precision, and unpowered

finger input for very small mobile devices. In: Proceedings of the 22nd annual ACM

symposium on User interface software and technology. ACM, pp. 121–124 .

88

arrison, C. , Hudson, S.E. , 2010. Minput: enabling interaction on small mobile devices

with high-precision, low-cost, multipoint optical tracking. In: Proceedings of the

SIGCHI Conference on Human Factors in Computing Systems. ACM, pp. 1661–1664 .

arrison, C. , Tan, D. , Morris, D. , 2010. Skinput: appropriating the body as an input surface.

In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.

ACM, pp. 453–462 .

achini, T. , Ruggiero, G. , Ruotolo, F. , 2014. Does blindness affect egocentric and allocen-

tric frames of reference in small and large scale spaces? Behavioural brain research

273, 73–81 .

ain, M. , Balakrishnan, R. , 2012. User learning and performance with bezel menus. In:

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM,

pp. 2221–2230 .

ane, S.K. , Bigham, J.P. , Wobbrock, J.O. , 2008. Slide rule: making mobile touch screens

accessible to blind people using multi-touch interaction techniques. In: Proceedings of

the 10th international ACM SIGACCESS conference on Computers and accessibility.

ACM, pp. 73–80 .

ratz, S. , Rohs, M. , 2009. Hoverflow: expanding the design space of around-device in-

teraction. In: Proceedings of the 11th International Conference on Human-Computer

Interaction with Mobile Devices and Services. ACM, p. 4 .

urtenbach, G. , Buxton, W. , 1994. User learning and performance with marking menus.

In: Proceedings of the SIGCHI conference on Human factors in computing systems.

ACM, pp. 258–264 .

urtenbach, G.P. , 1993. The design and evaluation of marking menus. Ph.D. thesis, Uni-

versity of Toronto .

i, F.C.Y. , Dearman, D. , Truong, K.N. , 2009. Virtual shelves: interactions with orientation

aware devices. In: Proceedings of the 22nd annual ACM symposium on User interface

software and technology. ACM, pp. 125–128 .

i, F.C.Y. , Dearman, D. , Truong, K.N. , 2010. Leveraging proprioception to make mo-

bile phones more accessible to users with visual impairments. In: Proceedings of the

12th international ACM SIGACCESS conference on Computers and accessibility. ACM,

pp. 187–194 .

egulescu, M. , Ruiz, J. , Li, Y. , Lank, E. , 2012. Tap, swipe, or move: attentional demands for

distracted smartphone input. In: Proceedings of the International Working Conference

on Advanced Visual Interfaces. ACM, pp. 173–180 .

orman, D.A. , 1981. Categorization of action slips. Psychol. Rev. 88 (1), 1 .

akley, I. , Park, J. , 2007. A motion-based marking menu system. In: CHI ’07 Extended

Abstracts on Human Factors in Computing Systems. ACM, pp. 2597–2602 .

akley, I. , Park, J. , 2009. Motion marking menus: An eyes-free approach to motion input

for handheld devices. Int. J. Human-Computer Studies 67 (6), 515–532 .

h, U. , Kane, S.K. , Findlater, L. , 2013. Follow that sound: using sonification and correc-

tive verbal feedback to teach touchscreen gestures. In: Proceedings of the 15th Inter-

national ACM SIGACCESS Conference on Computers and Accessibility. ACM, p. 13 .

liveira, J. , Guerreiro, T. , Nicolau, H. , Jorge, J. , Gonçalves, D. , 2011. Blind people and

mobile touch-based text-entry: acknowledging the need for different flavors. In: The

proceedings of the 13th international ACM SIGACCESS conference on Computers and

accessibility. ACM, pp. 179–186 .

omano, M. , Bellucci, A. , Aedo, I. , 2015. Understanding touch and motion gestures

for blind people on mobile devices. In: Human-Computer Interaction. Springer,

pp. 38–46 .

uiz, J. , Li, Y. , Lank, E. , 2011. User-defined motion gestures for mobile interaction. In:

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM,

pp. 197–206 .

ánchez, J. , Aguayo, F. , 2007. Mobile messenger for the blind. In: Universal Access in

Ambient Intelligence Environments. Springer, pp. 369–385 .

hinohara, K. , Wobbrock, J.O. , 2011. In the shadow of misperception: assistive technol-

ogy use and social interactions. In: Proceedings of the SIGCHI Conference on Human

Factors in Computing Systems. ACM, pp. 705–714 .

ngar, S. , 2000. 13 cognitive mapping without. Cognitive Mapping: Past, Present, Future

4, 221 .

ebAIM, 2015. Screen reader user survey #4 results.

olf, K. , Dicke, C. , Grasset, R. , 2011. Touching the void: gestures for auditory interfaces.

In: Proceedings of the fifth international conference on Tangible, embedded, and em-

bodied interaction. ACM, pp. 305–308 .

e, H. , Malu, M. , Oh, U. , Findlater, L. , 2014. Current and future mobile and wearable de-

vice use by people with visual impairments. In: Proceedings of the SIGCHI Conference

on Human Factors in Computing Systems. ACM, pp. 3123–3132 .

ee, K.-P. , 2003. Peephole displays: pen interaction on spatially aware handheld comput-

ers. In: Proceedings of the SIGCHI conference on Human factors in computing systems.

ACM, pp. 1–8 .

fantidis, G. , Evreinov, G. , 2006. Adaptive blind interaction technique for touchscreens.

Universal Access Inf. Soc. 4 (4), 328–337 .

hao, S. , Dragicevic, P. , Chignell, M. , Balakrishnan, R. , Baudisch, P. , 2007. Earpod: Eye-

s-free menu selection using touch input and reactive audio feedback. In: Proceed-

ings of the SIGCHI Conference on Human Factors in Computing Systems. ACM,

pp. 1395–1404 .