-
Master Thesis
Computer Science
Thesis no: MCS:2011:21
Sept. 2011
A Comparative Evaluation of Usabilityfor the iPhone and iPad
Muhammad Azam
Luqman Ahmad
School of Computing
Blekinge Institute of Technology
SE-371 79 Karlskrona
Sweden
1
-
This thesis is submitted to the School of Computing at Blekinge Institute of Technology in partial
ful�llment of the requirements for the degree of Master of Science in Computer Science. The
thesis is equivalent to 20 weeks of full time studies.
Contact Information:
Author(s):
Muhammad Azam
Address: Gyllenstjärnas Väg 18, 20
Post code 37140 Karlskrona,
Sweden
Email: [email protected]
Luqman Ahmad
Address: Gyllenstjärnas Väg 18, 20
Post code 37140 Karlskrona,
Sweden
E-mail: [email protected]
University advisor:
Dr. Veronica Sundstedt
School of Computing
School of Computing
Blekinge Institute of Technology Internet : www.bth.se/com
SE-371 79 Karlskrona Phone : +46 455 38 50 00
Sweden Fax : +46 455 38 50 57
2
-
ACKNOWLEDGMENTS
In the name of Allah, the Most Bene�cent, the Most Merciful.
First and foremost we would like to thank Allah the Almighty, Who blessed us courage
and devotion for this study.
We heartily thank our supervisor Dr. Veronica Sundstedt, who encouraged us and
gave guidance throughout this research work. Her constructive suggestions, directions
and invaluable advice made us capable to complete this thesis. We would also like to
express our appreciation to those who participated and dedicated their valuable time for
these studies, without their participation these studies would not have been feasible.
Last but not least, we express our gratitude to our parents and family members, for
providing us untiring support and prayers during this research work.
3
-
Abstract
Many everyday systems and products seem to be designed with little regard to
usability. This leads to the frustration, wasted time and errors. So the usability of
the product is important for its survival in the market.
In many previous studies the usability evaluation of the iPhone and iPad carried
out individually and very little work has been done on the comparative usability
evaluation. However, there was not any study conducted on the comparative us-
ability evaluation and measuring the performance of the iPhone versus iPad in a
controlled environment.
In this research work, the authors performed the comparative usability evaluation
and measured the performances of the iPhone and iPad on the selected applications
by considering the young users as well as the elderly users. Another objective of this
study is to identify the usability issues in performances of the iPhone and iPad.
A survey and experiment techniques were used to achieve the de�ned objectives.
The survey questionnaire consisted of 42 statements that presented the di�erent
usability aspects. The objectives of the survey study were to validate the identi�ed
issues from the literature study, identify new issues and measure the signi�cant
di�erence in user opinions for the iPhone and iPad. However, the experiment studies
helped to measure the performance signi�cances between the devices against the
three user groups (novice user, experienced user, elderly user) and among the groups
over the devices. Further, objective was to measure the satisfaction level of the
participated users against the iPhone and iPad.
The experiment was performed in a controlled environment. Total six tasks (two
tasks per application) were de�ned and each participant performed the same task on
both devices. Generally the authors found that the participants performed better
on the iPad with lower error rates as compare to the iPhone.
Keywords: Usability, Usability Evaluation, Touch Screen, Smart-phone, iPhone,
iPad, performance.
4
-
Contents
1 INTRODUCTION 10
1.1 Aim and Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.2 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.3 Study Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4 Thesis Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2 BACKGROUND 14
2.1 What is Usability? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2 Usability Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3 Performance Measuring . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4 Usability Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.5 Touch Screen and Smart Phones . . . . . . . . . . . . . . . . . . . . . . . . 17
2.6 The iPhone and the iPad . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.7 Features of the iPhone and iPad . . . . . . . . . . . . . . . . . . . . . . . 18
2.7.1 Multi-touching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.7.2 Operating System (iOS 4) . . . . . . . . . . . . . . . . . . . . . . . 19
2.7.3 Applications (Apps) Store . . . . . . . . . . . . . . . . . . . . . . . 19
2.8 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.9 Issues in the iPhone and iPad . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.9.1 Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.9.2 A�ordance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.9.3 Small Search Box and Missing Information . . . . . . . . . . . . . . 21
2.9.4 Invisible Control and Missing Instructions . . . . . . . . . . . . . . 22
3 RESEARCH METHODOLOGY 23
3.1 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.2 Survey Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.3 Experimental Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3.1 Participant Selection . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.3.2 Variables Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.3.3 Hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.3.4 Task De�nition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.3.5 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.3.6 Experimental Material . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.3.7 Post Test Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . 33
3.3.8 Pilot Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.4 Evaluation Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.4.1 Mann Whitney Test . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5
-
3.4.2 Normality Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.4.3 Analysis of Variances (ANOVA) . . . . . . . . . . . . . . . . . . . . 34
3.4.4 Unpaired T-test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.4.5 Paired T-test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.4.6 Wilcoxon Matched Paired Signed Ranks Test . . . . . . . . . . . . 35
3.4.7 Error Bars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.5 Validity Threats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.5.1 Internal Validity Threats . . . . . . . . . . . . . . . . . . . . . . . . 36
3.5.2 External Validity Threats . . . . . . . . . . . . . . . . . . . . . . . 36
3.5.3 Constructive Validity Threats . . . . . . . . . . . . . . . . . . . . . 36
3.5.4 Statistical Conclusion Validity Threats . . . . . . . . . . . . . . . . 36
4 STATISTICAL ANALYSIS, RESULTS ANDDISCUSSIONS 37
4.1 Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.2 Demographics Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.3 Usability and Preference Testing . . . . . . . . . . . . . . . . . . . . . . . . 37
4.3.1 System Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.3.2 Touch Screen Gestures Results . . . . . . . . . . . . . . . . . . . . . 41
4.3.3 Keypad Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.3.4 Applications Results . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.3.5 Participant Comments . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.4 Measuring Usability Attributes . . . . . . . . . . . . . . . . . . . . . . . . 51
4.4.1 Task Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.4.2 Accuracy Measurement . . . . . . . . . . . . . . . . . . . . . . . . 61
4.4.3 Satisfaction Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.4.4 Participant Comments . . . . . . . . . . . . . . . . . . . . . . . . . 68
5 CONCLUSIONS AND FUTURE WORK 70
5.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
6 APPENDICES 73
6.1 Appendix A Survey Questionnaire . . . . . . . . . . . . . . . . . . 73
6.2 Appendix B Survey Questionnaire Results . . . . . . . . . . . . . 76
6.3 Appendix C Survey Questionnaire p-values . . . . . . . . . . . . . 78
6.4 Appendix D Post Test Questionnaire . . . . . . . . . . . . . . . . 80
6.5 Appendix E Task Time on the iPhone and iPad . . . . . . . . . . 82
6
-
List of Figures
1 The Overall Study Process. . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2 Shackel's Usability De�nition [21]. . . . . . . . . . . . . . . . . . . . . . . . 14
3 Nielsen's Usability De�nition [21]. . . . . . . . . . . . . . . . . . . . . . . . 15
4 ISO 9241-11 Usability De�nition [21]. . . . . . . . . . . . . . . . . . . . . . 16
5 Di�erent Consistency Issues in SMS and Mobile Voip Applications. . . . . 20
6 A�ordance Issues in the Weather Applications of the iPhone and iPad. . . 21
7 Small Search Box Issues in the Applications of the iPhone and iPad. . . . . 21
8 Invisible Control and Missing Instructions Issues in the Twitter and Face-
book Application on the iPhone and iPad. . . . . . . . . . . . . . . . . . . 22
9 Presentation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
10 System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
11 Touch Screen. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
12 Keypad. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
13 Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
14 Participants Selection for the Experiment. . . . . . . . . . . . . . . . . . . 27
15 Laboratory Setup for User Trials. . . . . . . . . . . . . . . . . . . . . . . . 32
16 Participant Group Levels Comparison Across the Device. . . . . . . . . . . 34
17 Device Levels Comparison Across the Participant. . . . . . . . . . . . . . . 35
18 Percentage Graph Results of the Degree of Agreement Across Each System
Statement on the iPhone. St. Represents Strongly. . . . . . . . . . . . . . 41
19 Percentage Results of the Degree of Agreement Across Each System State-
ment of the iPad. St. Represents Strongly. . . . . . . . . . . . . . . . . . . 41
20 Graphs Show Percentage Results of the Degree of Agreement Across the
Each Statement of the Touch Screen Gestures on the iPhone and iPad
Respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
21 Graph Shows Percentage Results of the Degree of Agreement Across the
Each Statement of the Keypad on the iPhone. . . . . . . . . . . . . . . . 45
22 Graph Shows Percentage Results of the Degree of Agreement Across the
Each Statement of the Keypad on the iPad. . . . . . . . . . . . . . . . . . 46
23 Percentage Graph Results of the Degree of Agreement Across Each Appli-
cations Statement on the iPhone. . . . . . . . . . . . . . . . . . . . . . . . 49
24 Percentage Graph Results of the Degree of Agreement Across the Each
Statement of the Applications on the iPad. . . . . . . . . . . . . . . . . . 49
25 Device Levels Comparison Graph of the Performance Measurement of the
Participant Group Levels for the Task Facebook Login. . . . . . . . . . . . 54
26 Device Levels Comparison Graph of the Performance Measurement of the
Participant Group Levels for the Task Send a Message. . . . . . . . . . . . 55
7
-
27 Device Levels Comparison Graph of the Performance Measurement of the
Participant Group Levels for the Task Location Identi�cation. . . . . . . . 56
28 Device Levels Comparison Graph of the Performance Measurement of the
Participant Group Levels for the Task Location Close View. . . . . . . . . 58
29 Device Levels Comparison Graph of the Performance Measurement of the
Participant Group Levels for the Task New Note. . . . . . . . . . . . . . . 59
30 Device Levels Comparison Graph of the Performance Measurement of the
Participant Group Levels for the Task Evernote Logout. . . . . . . . . . . 60
31 Novice User Mistakes Results Graph on the iPhone and iPad. . . . . . . . 62
32 Experienced User Mistakes Results Graph on the iPhone and iPad. . . . . 63
33 Elderly User Mistakes Results Graph on the iPhone and iPad. . . . . . . . 63
34 The Graph Presents the Median Comparison of the Degree of Agreement
Against Each Question by Three User Groups on the iPhone. . . . . . . . . 67
35 The Graph Presents the Median Comparison of the Degree of Agreement
Against Each Question by Three User Groups on the iPad. . . . . . . . . . 68
8
-
List of Tables
1 iPhone and iPad Speci�cation Comparison. The Bold Values Show the
Di�erences in Speci�cation. . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2 Usability Issues on the iPhone and iPad. . . . . . . . . . . . . . . . . . . . 22
3 Advantages and Disadvantages of Qualitative Approach [52] . . . . . . . . . 23
4 Results of Surveyed Participants Comments Against the Di�erent Features
of the iPhone and iPad. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5 Results from the ANOVA Tests for Estimating the Signi�cance Among the
Group Levels and Between the Device Levels Across Each Task. The p-
value Shown the Signi�cance Among the Mean Values of Each Participant
Group Against the Each Task. However, the Bold (p-values) Values Shown
the Signi�cant Di�erences Among the Groups and Between the Devices. . . 51
6 Participant Group Levels Comparison Results. The p-values Showed the
Di�erences Between the Participant Group Levels on the iPhone and iPad.
However, the Bold p-values Shown that there was a Signi�cant Di�erence
Between the Group Levels. . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7 Device Levels Means Comparisons Results. The p-value Showed the Dif-
ferences Between the Means of the iPhone versus iPad Against each User
Group. However, the Bold p-values Shown the Signi�cant Di�erence. . . . 52
8 Means Results and Di�erences of Each Type of Mistake Made by Each
User Group on the iPhone and iPad. However, the Bold p-values Shown
the Signi�cant Di�erence. . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
9 Post Test Questionnaires Results in the Form of the Median, the Standard
Deviation and the Di�erences Between the Users Opinions Against Each
Question. However, the Bold p-values Shown the Signi�cant Di�erence. Q,
iP and id Represented the Question, iPhone and iPad Respectively. . . . . 64
10 Frequency and Percentage Results of the User Comments on the Di�erent
Aspects of Both the Devices. . . . . . . . . . . . . . . . . . . . . . . . . . . 68
9
-
Chapter 1
1 INTRODUCTION
Human Computer Interaction (HCI) is concerned with the joint performance of tasks by
the human and machines [1]. It has become valuable due to the fact that it involves the
design, implementation and evaluation of computer interactive systems that human beings
use in di�erent contexts to accomplish their tasks. The main goal of HCI is to produce
computer systems which are easy to use, well functional, e�cient, e�ective, and that are in
accordance with the requirements of the user [2]. The HCI goal would be ensured with the
usability of the product. According to the Webster dictionary, the usability originate from
the word �usable� which means �capable of being used � or �convenient and practicable for
use�.
We are living in a mobile society [3]. New information and communication technologies
(ICTs) have become a part of people's everyday lives. The mobile technologies are devel-
oping very quickly [5] and the trend is changing from conventional desktop computing to
the mobile computing. It facilitates the users to access and use their services anywhere
anytime [4] e.g. doing trading, shopping, banking and bill payment etc [6]. So in the
last few years, the developers redirect their design e�ort from a desktop perspective to a
small screen approach [7]. Under the constraint of small size, the usability of the mobiles
devices and their applications is a key factor for the success of mobile computing [8].
The mobile phone technologies have an impact on all age groups in our society. Studies
illustrate that young people are more active users of technology [9, 10]. They are the
�rst users who adopt the new technology and develop an actual mobile communication
culture [10]. On the other hand, the elderly people are also very keen to use the new
mobile technology but because of the small size, invisibility of text and complexity of the
functions, they avoid to use it [11]. The elderly population is growing rapidly in almost
all the economically developed countries [12]. It is noticed that elderly people take more
interest in games as compared to younger but because of the complexities and di�culties
of the user interfaces they cannot take part more e�ectively [13]. Earlier studies presented
that most of the services are designed by considering the young population and mistakenly
the elderly people are also considered as a part of the same age group [14].
In the mobile phone multi-touch technology, the iPhone, iPod Touch, and iPad are
sophisticated devices, which provide advanced and sensitive user interactions based on
gestures [15]. The iPhone and iPad support millions of the applications that designed and
developed by the Apple and the third party. It has been considered that the Evernote
is used by millions of the people in the world to remember their important notes and
10
-
synchronize them with their PCs [38]. Meanwhile, in the social network sites the Facebook
is one of the leading applications having more than 500 million active users in the world
[69]. Only in the Sweden more than 4 million people are using the facebook [71]. Similarly,
the Google web application (search engine) is also a famous site having the large number
of users [70]. The simplicity and consistency of the devices motivated the authors to
conduct the comparative usability research studies of the iPhone and iPad.
In order to �nd out whether the iPhone and iPad are performing according to user
expectations or not, the authors performed the survey and empirical usability evaluation
studies. The authors selected three frequently used applications (Facebook, Google Mobile
Application, Evernote) and measured the usability performances of three user groups
(novice user, experienced user, elderly user) as well as the iPhone and iPad. The age
range for the young people is 22-34 and for the elderly people is 60-75.
1.1 Aim and Objectives
The aim of this research work is to evaluate and compare usability performance of the
iPhone and iPad on three selected applications considering the elderly people as well as
the young people. To achieve this goal the following objectives are de�ned:
Identifying usability issues regarding the iPhone and iPad.
Investigating usability performance di�erences of the same applications on the iPhone
and iPad.
Investigating the performance di�erences between di�erent user groups using the
same applications and same devices.
1.2 Research Questions
1. What usability issues exist in the iPhone and iPad?
2. How does the performance di�er while using the same application on the iPhone
and iPad?
3. How does the performance di�er using the same application on the same device by
di�erent user groups?
11
-
1.3 Study Process
The study process presented a hierarchical �ow that was carried out during the the whole
research work.
Figure 1: The Overall Study Process.
1.4 Thesis Structure
Chapter 1 introduces the thesis and de�nes the research domain. In Section 1.1 aim and
objectives are given. In Section 1.2 research questions are given. In Section 1.3 the overall
research process is presented in the form of diagram .
Chapter 2 provides background knowledge about the usability. In Section 2.1 usability
and its di�erent attributes are discussed that de�ned by di�erent authors. Section 2.2
describes how to measure the usability. Section 2.3 describes the performance measuring.
Section 2.4 describes the usability evaluation. Section 2.5 describes the touch screen and
smart phones. Section 2.6 describes the iPhone and iPad. Section 2.7 is about the main
features of the iPhone and iPad. Section 2.8 presents the related work and Section 2.9
presents the usability issues concerning the iPhone and iPad.
Chapter 3 is about the research methodology adopted in the thesis. Section 3.1 presents
12
-
a literature review. Section 3.2 presents the survey design . Section 3.3 consists of the
experiment design . Section 3.4 presents the evaluation methods and Section 3.5 presents
the validity threats in research.
Chapter 4 presents the survey and experimental analysis results and discussion. Sec-
tion 4.1 presents the survey. Section 4.2 describes the demographics results of the survey.
Section 4.3 presents the usability and preference testing of the survey data. Section 4.4
describes the participants comments regarding the usability of the iPhone and iPad in the
survey questionnaire. Section 4.5 presents the measurement of usability attributes.
Chapter 5 contains conclusions and future work of the thesis. Section 5.1 presents the
conclusions of the research study and Section 5.2 presents the future work.
Chapter 6 contains the Appendices. In Section 6.1 Appendix A represents the survey
questionnaire. In the Section 6.2 Appendix B survey Questionnaire results are given. In
Section 6.3 Appendix C contains the p-values of the survey results. In the Section 6.4
Appendix D presents the post test questionnaire. Section 6.5 contains the Appendix E
that presents the experimental data i.e task time on both devices.
13
-
Chapter 2
2 BACKGROUND
2.1 What is Usability?
Usability in itself is not a new topic [17]. It is considered as a core term HCI [18] and has
become an important part in the design of products, software applications and devices.
However the user centered design and goal directed interaction design has increased us-
ability of products [17]. The role of usability is to make the system e�ective and easy to
use [19]. There are many de�nitions of the term usability. According to ISO 9241-11 the
term usability can be de�ned as �The extent to which a product can be used by speci�ed
users to achieve speci�ed goals with e�ectiveness, e�ciency and satisfaction in a speci�ed
context of use� [20].
Usability is the composition of the entities [21] e.g. e�ectiveness, e�ciency, learnability,
memorability etc. Some times these entities are de�ned as dimensions, components, scales
or factors of usability. All these forms have the same meaning, so the commonly used
term for them is usability attributes [21].
Shackel [19] is one of the famous authors who realized the importance of usability and
de�ned it as a relative property of a system. This property makes the relationship between
users and the systems; i.e. the subjective usability measure. The other side relates to
objective measures of interaction i.e. how easily and e�ciently users could achieve their
goals . Shackel de�ned the following attributes to measure the usability.
E�ectiveness: how e�ciently and with minimum errors the user completes his
goal.
Learnability: how much time the user required learning and how long this could
remember it.
Flexibility: adaptation to change in tasks.
Attitude: user satisfaction with the system.
Figure 2: Shackel's Usability De�nition [21].
14
-
Nielsen [22] is another usability expert, who de�ned usability by considering the following
attributes.
Learnability: the system should be simple and easy to learn.
E�ciency: the system should be capable to perform the users work within speci�ed
time.
Memorability: it should be easy to remember, so that the user can easily use it
after some period of not using the system, without having to learn all again.
Errors: the system should have low error rate and provide help to the user while
he/she make errors.
Satisfaction: the system should be attractive, so that the user feels comfort and
satisfaction while using it.
Figure 3: Nielsen's Usability De�nition [21].
Since from last 15 years, the International Standard Organization (ISO) has developed
di�erent standards for HCI and usability. The functionality of these standards is to
make the system simple and consistent. The ISO 9241-11 decompose usability into three
attributes [23] that are as discussed below.
E�ectiveness: the accuracy and time for speci�ed goal achievement by speci�ed
users.
E�ciency: utilization of resources for completion of speci�ed tasks.
Satisfaction: the user must be feeling comfort while using the system.
The software engineering community has also associated usability with interface design.
The ISO 9126-1 de�ned usability as �quality in use�. This de�nition has the same meaning
as the ISO 9241-11 de�ne usability but it is used in speci�c context. For the measurement
of usability, ISO 9126-1 speci�ed the following usability attributes [24].
Understandability: the software product must have the capability that the user
can easily understand and utilize it for the speci�ed task.
15
-
Figure 4: ISO 9241-11 Usability De�nition [21].
Learnability: the software system should have the capability that the user should
learn it easily.
Operability: the software system must have the capability that user could easily
operate and control it.
Attractiveness: the software system must have capability to attract the user.
The existing usability studies on mobile applications de�ne the nine usability attributes
such as: learnability, e�ciency, memorability, errors, user satisfaction, e�ectiveness, sim-
plicity, comprehensibility and learning performance. Selection of suitable attributes for
evaluation of mobile applications depends on the nature of the applications and the ob-
jectives of the usability study [25].
2.2 Usability Measurement
Many past studies presented that the usability is dependent on the context of use [18].
This context of use is formed by the user, task, equipment and environment [26]. Most
of the studies are conducted in controlled environment, where the user performed some
prede�ned tasks using the speci�ed system. The usability can be assessed by consider-
ing several measurable parameters. These parameters can be divided into two groups:
subjective preference measures and objective performance measures.
Subjective preference measures are being assessed by considering what the user likes
and dislikes about the system. It can be measured by using for example a questionnaires
[27]. An advantage of a questionnaires is that they can provide the evaluator with feedback
from the user's point of view. Another bene�t of the questionnaires is to provide the
comparable measures across the systems that being evaluated [28].
The objective performance measures relate to the e�ectiveness of system (e.g. time,
errors, and number of activities). Objective performance measures are being assessed by
giving the speci�ed tasks under a speci�ed time limit. Sometimes it can be measured by
counting the number of tasks per unit time [27]. The e�ciency measurement of the system
is also related to the total time on task, usages of the resources and mental resources used
to manipulate the system interfaces.
16
-
2.3 Performance Measuring
Performance is the degree to which the system accomplishes its designated function under
certain constraints [29]. In this study, the authors followed the usability attributes that
de�ned by ISO 9241-11 i.e. e�ectiveness, e�ciency and satisfaction. To test the usability
attributes following metrics were measured.
Task Time: How long it took a participant to complete a task in seconds [30].
Errors: Errors are an unintentional action, mistakes, slips and omissions, a user makes
while performing a task [30].
Satisfaction: It is the reaction of the participants about the overall performance of the
system.
2.4 Usability Evaluation
Usability evaluation is �the systematic process of collecting data, in ordered to have a
better understanding of the users and how the user groups use the product to perform a
speci�c task under speci�ed conditions� [31]. It has been required in di�erent stages of
the system development process. Its goal is to provide feedback that helps to improve
the quality and functionality of the system [32]. Zhang [33] de�ned three approaches
of usability evaluation methods such as: testing (e.g. Coaching method, performance
measurement, thinking aloud protocol), inspection (e.g. heuristic evaluation) and inquiry
(e.g. interviews, �eld observation, questionnaires). In the usability testing approach,
speci�ed users perform speci�c tasks using the system or prototype [21]. The usability
evaluator examines, how easily and e�ciently the users perform their tasks. The usability
inspection approach involves the usability experts or professional to examine whether the
system or each part of the system follows usability principles. In the usability inquiry
approach the evaluator collect information about user perceptions and understanding of
the system through interviews, surveys and verbal discussions [21].
2.5 Touch Screen and Smart Phones
Touch screen user interfaces have become more attractive in electronic devices [34]. There
are two types of touch screens: capacitive and resistive. Resistive touch screens are
made of a number of metallic and electronic conductive layers, separated with a small
gap. When a user tap on the touching surface, both layers make a connection and cause
the electric current that activate the touching event. Capacitive touch screens work by
sensing conductive object e.g. �nger. They allow the multi-touch functionality that can be
performed by using multiple bare �ngers [35]. The touch screen input method is becoming
17
-
popular in the smart-phones and other mobiles devices such as PDAs and tablet laptops
[36]. In 1993, IBM presented the �rst smart-phone and its name was Simon. The main
features of Simon were a calendar, a calculator, a world clock, and an email client. It had
a touch screen interface that let the users write text by using an on-screen keyboard or
a stylus (i.e. stick like a pen). There is no proper de�nition of the term smart phone.
However, it is considered as a mobile phone that has a greater computing power than the
normal cell phones and having the features like the PCs. After launching the iPhone, the
user interface design had great impact on the smart-phones with multi-touch technology
[37].
2.6 The iPhone and the iPad
Apple designs, manufactures and markets a range of digital products. These products
contain personal computers, mobiles devices, music players, related software and third-
party applications. In the past few years, the company has launched products such as the
iPhone, iPod, and iPad [38]. The iPhone and iPad are considered as revolutionary Apple
products [39] and both have some similar features and look like the smaller and larger
version of each other. However in some perspectives they have some di�erences such as
RAM, size, resolution and weight [40]. These similarities and di�erences are presented in
the Table 1.
Feature iPhone 4 iPad
Chip A4 A4
OS iOS 4 iOS 4
Processor 1 GHz ARM Cortex-A8 1 GHz ARM Cortex-A8
RAM 512 MB 256 MB
Connection Wi-Fi Wi-Fi
Size 4.5 x 2.31 x0.37 inch 9.56 X 7.47 X 0.52 inch
Resolution 960 X 640 pixel 1024 X 768 Pixel
Weight 137g 730g
Table 1: iPhone and iPad Speci�cation Comparison. The Bold Values Show the Di�er-ences in Speci�cation.
2.7 Features of the iPhone and iPad
The main features of the iPhone and iPad are described below:
2.7.1 Multi-touching
The multi-touch technology is used as an input method for the iPhone and iPad. It is
de�ned as the technology where the user can interact with system with multiple inputs
at a time e.g. two or more �ngers for a single task. The users can use their �ngers for
18
-
typing email, enlarging text, swapping through photos and for using zooming functions.
All input functions are performed by using the �ngers at touch screen interfaces.
2.7.2 Operating System (iOS 4)
The iOS 4 is an operating system that is being used by Apple for the iPhone 4 and iPad.
It is considered as a simple operating system. The user can perform various tasks (e.g.
writing email and zooming text) simply using the �ngers on the touch screen interface.
The user can use millions of di�erent built-in applications with a few �nger gestures such
as tap, drag, swipe and pinch [38].
2.7.3 Applications (Apps) Store
Apps Store is a database that is especially designed for Apple's products. It contains
business applications, education applications, entertainment applications, social network-
ing applications, news applications and so on. The applications are designed by Apple's
designers and developers as well as by a third party. Some applications are free but for
some applications you will have to pay [38].
2.8 Related Work
Tsung [41] performed an age based usability study on the iPhone. Three age groups
i.e. college-level youth, middle-aged professionals and senior citizens were taken. The
conclusion of this study was that the system should not only consider the �glamorous
animation and fancy utilities� but all human physical conditions e.g. hearing and eye
sight that will make the system more senior friendly.
Chaparro [42] evaluated the keyboard performance of the Netbook and iPad. They
measured the performance on three di�erent style keyboards. The iPad soft touch key-
board (portrait as well as in landscape) and the Acer Netbook physical keyboard. The
attributes and their metrics are given below.
Performance (task time and errors)
Perceived Mental Workload
Satisfaction
The study showed that there was a slight di�erence in the performance of the Netbook
physical keyboard and the iPad landscape and no di�erence in the performance of the
iPad portrait and landscape keyboards. The satisfaction level of the iPad landscape was
slightly higher than the physical keyboard and portrait was slightly lower than the physical
keyboard. The preference level of the landscape iPad keyboard and physical keyboard was
same the while portrait keyboard was less preferred by the participants .
19
-
In another study, six sta� members at Curtin University were supplied with the iPad
to test usability for 100 days. The iPad was handed over to each sta� member and at the
end of the third month these participants were interviewed. The research showed that
there was no problem for inexperienced users using the iPad. The consistency issues were
noted like the other studies and also some common functionalities were identi�ed [43].
The Nielsen and Norm group (NNGroup) [44] conducted a usability study of the iPad
applications and contents, a few weeks after Apple launched the device. They tested seven
users; all had at least three months iPhone experience and only one of them was an iPad
user. In this study they assessed the interface of applications, websites and speci�c iPad
applications, touch screen, changing orientation, and di�erent gestures. They criticized
and pointed out that the applications are designed without considering the importance of
usability e.g. read tab asymmetry, zoom the map or zoom the page, use of proper mental
models, small target area, accidental tapping and the back button etc. They highlighted
many usability issues e.g. inconsistencies and accidental actions etc.
2.9 Issues in the iPhone and iPad
From the literature [44], the authors found some usability issues that are discussed below:
2.9.1 Consistency
The consistency issue remain almost all the time with the iPhone and iPad applications.
Figure 5 shows the inconsistency in the button, keyboard appearance and writing envi-
ronment of the messages of SMS and MobileVoip application of the iPhone.
Figure 5: Di�erent Consistency Issues in SMS and Mobile Voip Applications.
20
-
2.9.2 A�ordance
The other issue in the iPhone as well as the iPad application was a�ordance. �Users don't
know that something is touchable unless it looks so� [44]. In the weather application of
the iPhone it is very di�cult to add new a location as the option lacks the a�ordance.
Figure 6 shows this a�ordance problem in the weather application of the iPhone. The �i�
at the right bottom side of the application is used for adding the new location but the
option is not very clear as well as not seems to be tappable.
Figure 6: A�ordance Issues in the Weather Applications of the iPhone and iPad.
2.9.3 Small Search Box and Missing Information
Figure 7 shows the small search box and missing search button at the bottom of the Apps
Store application of iPad. As there is a big screen on the iPad the search box needs to be
more clear and also users expect the search button at the bottom of the application.
Figure 7: Small Search Box Issues in the Applications of the iPhone and iPad.
21
-
2.9.4 Invisible Control and Missing Instructions
Figure 8 shows the invisible control and missing instructions in the iPhone and iPad
applications. In Figure 8 at the right side when the message is too long it create a scroll
bar but it remain invisible until a someone tap on it. Similarly in twitter there are four
pages that can swipe forward and backward but there are no instructions to move them.
Another miss leading button at the top of the twitter application that seems like a setting
button but in fact it is used for reply option.
Figure 8: Invisible Control and Missing Instructions Issues in the Twitter and FacebookApplication on the iPhone and iPad.
Table given bellows describe the list of other usability issues in the iPhone and iPad.
Usability Issues Detail
Inconsistency in interaction Nothing happens
Enlarging the picture
Hyper-linking to a more detail page about that item
Replace the picture with new picture
Popping up a set of navigation choices
Accidental touch Touch something that one did not mean to
False back button Button that look like back button, but when pressed
it lead to new page
Lack of a�ordance Tappable elements look like it not supposed to touch
and untappable look like tappable
Hyperlinks Missing of important hyperlinks like wired magazine
using back button instead of hyperlinks
Changing orientation Application looks di�erent in portrait and landscape
Gestures Di�cult to remember gestures
Multiple panels Small font size and crowded contents
Auto-correct Should turn into autocomplete
Table 2: Usability Issues on the iPhone and iPad.
22
-
Chapter 3
3 RESEARCH METHODOLOGY
The research methodology is the systematic way to solve the research problem. Commonly
two main approaches are used for the research studies such as quantitative and qualitative
[45]. However, Creswell [46] has also introduced a third type of the research approach
called a mixed method approach i.e. the combination of quantitative and qualitative
approaches.
The quantitative research approach is based on the measurement of quantity or amount.
It is used when one begins with a hypothesis and tests for con�rmation and dis-con�rmation
of that hypothesis [47]. It is applicable to phenomena that can be expressed in terms of
quantity [48]. Generally, the survey and experiment are used in the quantitative approach
to inquiry the hypothetical issues and participants of the studies [46]. Some advantages
of the quantitative approach [49] are listed below:
1. Data can be gathered from large user groups in a short period of time
2. Large sample data can be generalized to the entire population
3. Quantitative research is repeatable
The qualitative research approach is concerned with subjective assessment of attitudes,
opinions and behaviors [48]. It is an interpretative approach which uses the meaning of
phenomena as understood by the participants of the studies [50]. Generally, open ended
questions are used in qualitative research [51]. Some of the advantages and disadvantages
are listed in Table 3.
Advantages of Qualitative Approach Disadvantages of Qualitative Approach
It goes deep and takes into account all details in Finding cannot be generalized because they
the environment are particular environment dependent
No need to recruit a large number of participants The �nding can be biased according to the
Flexibility in time and place understanding of the researcher
No need to conduct interviews of all participants
at same time
Table 3: Advantages and Disadvantages of Qualitative Approach [52] .
In this research, the authors adopted the mixed research approach. The quantitative
research approach was used to perform the experiment and the qualitative research ap-
proach was used to conduct the literature review and analyze the open ended questions
and participants comments. These research studies were carried out in multiple steps such
as a literature review, survey design, experiment design, pilot tests studies, experiment
23
-
execution, selection of suitable evaluation methods, analysis of gathered data, results,
conclusions and future work.
3.1 Literature Review
In the �rst step a literature review was done. The objectives of a literature review was
to understand the usability, usability attributes, usability issues in smart phone, and the
usability issues in the iPhone and iPad. Another objective of a literature review was
acquiring the understandability of the di�erent statistical usability evaluation methods.
The authors utilized the all available information resources to carried out this re-
search work. The mentioned keywords are used to �nd out the related data from all
available databases. However some of the authentic knowledge resources were listed as
BTH Library's books, IEEE, ACM digital library, Engineering Village, Google Scholar,
Springer-link, Scopus, and Ebrary.
3.2 Survey Design
To �nd usability issues the authors used the survey technique. The objectives of the
surveys were:
1. To validate the issues regarding the usability of the iPhone and iPad that authors
�nd out during the literature review.
2. To get the input for the experimental design.
3. To �nd out the users point of views about the usability of the iPhone and iPad.
A survey is an e�ective way to gather information from a large number of the population
in a short period of time at relatively low cost. It is considered that the respondents are
less likely try to please the researcher and provide the social acceptable responses. The
survey technique gives time to the respondents to think about the question or statement
before answering and do not bind them to complete it in one setting [53].
The seven point Likert scale was used in a survey questionnaire that helps the users
to express their degree of agreement with a statement. The numbers (1-7) were used for
strongly disagree, disagree, near to disagree, neutral, near to agree, agree and strongly
agree respectively for each statement.
In order to collect feedback from the users about the iPhone and the iPad, the authors
divided the survey into demographics, systems (iPhone, iPad) information, applications
information, touch screen features and keypad functionality. Each part contained several
related statements, as well as a space for own comments. The detailed survey questionnaire
is given in Appendix A. However, here the small piece of each section of questionnaire
was presented and structured as follows.
24
-
1. Presentation This section presented the users demographics (age, sex, quali�cation,
using experience), purpose of the survey study, explain the questionnaire and provide the
general information. Following is the presentation section of the survey questionnaire.
Survey Questionnaire for Evaluating the Usability of the systems(iPhone/iPad)
Participant Number: ______________ Sex: ___________________Home Country: _____________________ Age: ________________Education level: 2 BSc 2 MSc 2 PhD 2 Other __________________This questionnaire gives you an opportunity to describe your experiences of the system.Your responses will help us to evaluate the usability of the system.To as great degree as possible, consider all the tasks that you have been performing withthe system while you answer these questions. Please read each statement and indicatehow strongly you disagree or agree with the statement by marking (X) in the circle. If astatement does not apply to you, leave it empty or use the word (N/A) at the commentsline.Please write comments to elaborate on your answers (if you have any).As you complete the questionnaire, please do not hesitate to ask any questions.
Thank you.
Figure 9: Presentation.
2. System (iPhone/ iPad) This section presented the overall impression of the de-
vices. The objective of this section was to obtain the feedback about the system design,
graphics, capabilities and somehow about the ease of use. A piece of the system section
of a questionnaire is presented in Figure 9 below.
Figure 10: System.
3. Touch Screen This section presented the usability of the touch screen gestures.
Figure 10 shows a few statements of such section.
25
-
Figure 11: Touch Screen.
4. Keypad This section presented to gain the user opinions about the performance
of the keypad. The statements contained the di�erent keypad aspects such small size of
keys, landscape style and single character keys. The Figure 10 shows a few statements
regarding the keypad section.
Figure 12: Keypad.
5. Applications This section presented the di�erent usability features related to the
applications simplicity, ease to learn, consistency, visibility of the text and recoverability
of the mistakes. A small piece of this section presented in the Figure 12.
Figure 13: Applications.
26
-
3.3 Experimental Design
Experimental design is the process of planning a study to meet speci�ed objectives. Proper
planning of an experiment is important in order to ensure that the right type of data,
a su�cient sample size and resources are available to answer the research questions of
interest as clearly and e�ciently as possible [54, 55]. The following aspects provide further
insight into the experimental design.
3.3.1 Participant Selection
The authors selected the three user groups such as young novice users (20), young expe-
rience users (20) and elderly users (20). All the young users were bth students and well
known with the English language. In the elderly users 12 were bth employees; �ve were
the Karlskrona public library employees and three were Ronneby public library employees.
In the beginning, the authors were interested to divide the elderly people into two groups
like the young people (novice, experienced) and also visited the old people apartments in
Karlskrona and Ronneby. However, after meeting with the management of the old people
apartments the authors realized that it is di�cult to collect reliable data from the elderly
people because of language communication problems. So the authors decided to make a
single group of the elderly people and involved all those people who would speak, write,
read and understand the English language. Figure 13 shows the participants selection for
the experiment.
Figure 14: Participants Selection for the Experiment.
Novice Participant The person who was never used the smart phone, touch screen
systems or the internet on small devices.
27
-
Experienced Participant The authors considered the person as an experienced par-
ticipant who was using the smart phone or the iPhone more than three months.
Elderly Participant The person who was at least using a computer system or smart
phone or having the experience about the touch screen system.
3.3.2 Variables Selection
A variable is any character or attribute that can vary across people or situation or thing
[56]. There are two basic types of variables; independent variable and dependent variable.
The independent variable is one which the experimenter controls and manipulates to see
the e�ect on the treatments (dependent variable). On the other hand, the dependent
variable is the responses of the independent variable [56].
In the experiment, the authors used two types of the variables, independent variable
(factor) i.e. �Participant group� and dependent variable i.e. �Device�. The �Participant
group� variable has three levels such as novice user, experienced user and elderly user.
Similarly, the �Device� variable also has two levels such as the iPhone and iPad.
3.3.3 Hypotheses
The �rst step in the research process is the formulation of hypothesis about a speci�c
issue [57]. It is a precise problem statement that can be directly tested through empirical
investigation [58].
In comparative usability study of the iPhone and iPad across the three groups of par-
ticipants, the authors were interested to measure the performance di�erences between the
device levels as well as the di�erences among the participant group levels. The hypotheses
statements were de�ned as as follows:
H01: There is no di�erence in group levels over the device variable.
Ha1: There is a di�erence in group levels over the device variable.
H02: There is no di�erence in device levels across the group variable
Ha2: There is a di�erence in device levels across the group variable.
Here H0 and Ha represent the null hypotheses and alternative hypotheses respectively.
3.3.4 Task De�nition
For the execution of usability tests, six tasks were de�ned. Each task and its description
were designed on a separate paper. The presentation of the tasks was as follows:
28
-
Facebook
Task 1: Login to Facebook.
Email address: [email protected]
Password: usbtest2k11
Description:
Tap on Facebook
Tap on the First Text box �Email� and enter the email address
Tap on the �password� text box and enter the password
Tap on �Go� at the keypad
To use capital text tap on
To use numbers tap on
29
-
Task 2: Compose a new message on Facebook and send it.
TEXT: This is a usability test message for the iPhone
Send
Description:
Tap on the Facebook Home button
Tap on messages
Tap on create message at the top right corner of the Facebook application
Enter email address to whom you want to send a text , i.e. Given above
Tap just below the line and Write a short message
Tap on Send Button at the top of application
Google Map
Task 3: using the Google mobile application �nd the location of Blekinge
Tekniska Högskola and tap on the sign to con�rm the location.
Description:
Tap on Google mobile application
Tap on Apps at middle of bottom
Tap on Maps
Write Blekinge Tekniska Högskola
Tap on search button
Tap on the sign to con�rm the location
30
-
Task 4: Make a close view of Blekinge Tekniska Högskola building with zoom
in function.
Description:
Tap on Layers at the bottom left corner
Select the satellite from list
Tap on search box
Write Blekinge Tekniska Högskola
Tap on sign for zoom in
Evernote
Task 5: Create a new note in the Evernote application
User name: azamjarral
Password: aS5AK9
Title: First Note
Text: My �rst note in the Evernote application
Save
Description:
Open Evernote application
Tap on Sign in button
Tap on text box username and enter username
Tap on the text box password and enter password
Tap on Sign in
Tap on New Note at the bottom left corner. (For iPhone at the middle of Facebook
application)
Tap on the Save button at the right top corner of the application
31
-
Task 6: Logout from Evernote
Description:
Open Evernote application
Tap on the setting icon
Find the Logout option
Tap on Logout
3.3.5 Procedure
The authors booked a silent room in BTH that was used as a usability laboratory. To
provide the same environment to all the participants, the authors �xed the devices and
chairs. Before starting the experiment, all the tasks were arranged at a separate table. it
was conducted with a single user at a time. The experimental setup is shown in Figure
14.
Figure 15: Laboratory Setup for User Trials.
3.3.6 Experimental Material
The experimental material contained the iPhone, iPad, documents for the tasks descrip-
tion, demonstration and post test questionnaire.
32
-
3.3.7 Post Test Questionnaire
To know the participants usability experience abut the systems, a post-test questionnaire
was designed. It contained close ended as well as open ended questions. The close ended
questions were designed by using seven points Likert Scale and open ended questions were
also part of the same questionnaire but they were in the form of participants comments.
The close ended questions limit the respondent to the set of alternatives being o�ered,
however the respondent could easily respond the maximum questions within the limited
time. On the other hand the open ended questions help to explore the respondent and
thus avoiding the bias which may occur in the case of close ended questions [59]. The
detailed questionnaire is given in Appendix D.
3.3.8 Pilot Studies
Three pilot studies were conducted i.e. an experiment, a survey and a post test question-
naire. Before starting the experiment pilot test were conducted to assure that the tasks
were properly de�ned and the participants would have no di�culty in understanding every
step in each task. Three tests were conducted, one for each group and improvements were
made on the basis of their feedback. At the start the authors decided to just mention the
task e.g. login to Facebook, however after the pilot test, the authors realized that it was
not easy for the novice users as well as for elderly user to successfully complete the given
tasks. Then the authors decided to write down the step by step description of each task
with the symbolic representation wherever required. In this way the ambiguities were
removed in the experimental tasks.
Similarly the pilot studies were also conducted for the survey questionnaire and post
test questionnaire. After those studies, the changes were made in both questionnaires
according to the requirement of the users.
3.4 Evaluation Methods
In order to evaluate the survey and experimental data, the authors use multiple statistical
test methods. The detail of each method is given below.
3.4.1 Mann Whitney Test
The Mann Whitney test is a nonparametric test that can be used for comparing two
unequal sample size and unpaired data [60, 61]. The key result is the �p� value that shows
the signi�cant di�erence between both samples [62]. In this research, the survey data was
unpaired, unequal samples size and consist on rating scale that is why the Mann Whitney
test used to measure the signi�cance between the iPhone and iPad.
33
-
3.4.2 Normality Test
After obtaining the data from the experiment, it was important to test the normality
of the data. The authors read three test techniques: Shapiro Wilk Test, Kolmogorov-
Smirnov (KS) Test, and Histogram. The KS test is used for more than 50 samples, so
the authors did not use this method. On the experimental data, the Shapiro Wilk Test
and Histogram techniques could be applied to check the normality. however the authors
used �Shapiro Wilk test� online software that directly show whether the data is normally
distributed or not [63].
3.4.3 Analysis of Variances (ANOVA)
ANOVA is the most powerful and robust parametric method used to measure the signif-
icance. It has some assumptions such as the homogeneity of variance or normally distri-
bution of the data. The ANOVA is used to measure the signi�cant di�erence between
two or more than two groups [64]. In the experiment, there were three user groups and
two devices. The purpose experiment was to measure the di�erences among the groups
and between the devices. So to analyze the data, the authors used two way ANOVA
with replication. Further to see where the signi�cant di�erences between the devices and
among the groups were, the unpaired t-test and paired t-test were used respectively.
3.4.4 Unpaired T-test
The unpaired t-test is used to see where the signi�cant di�erences among groups across
the each device were. The comparison of the groups was made as follows.
Figure 16: Participant Group Levels Comparison Across the Device.
3.4.5 Paired T-test
The paired t-test used to see where the signi�cant di�erences between the devices across
the each group were. The comparison between devices across groups was as follows.
34
-
Figure 17: Device Levels Comparison Across the Participant.
3.4.6 Wilcoxon Matched Paired Signed Ranks Test
The Wilcoxon Matched Paired Signed Ranks test is a nonparametric test that is an
alternative to the paired t-test. It should be used if the distribution of di�erences between
pairs may be non-normally distributed [65]. The authors applied this method to �nd out
the signi�cance between the post-test questionnaires of the iPhone and iPad.
3.4.7 Error Bars
The error bars are used to estimate the variation or distribution of a statistical data in
the graphs. They can represent the standard deviation, the standard error, the con�dence
interval, or some other measure of error or uncertainty. Here the authors only discussed the
standard deviation and standard error. The standard deviation is a measure of variation
in data about their mean and standard error is a standard deviation of the estimated
statistical data such as mean, and regression coe�cient etc. The error bars do not let you
decide whether the di�erence between the mean is statistically signi�cant or not when
the two samples error bars do overlap or not [66]. It indicates how closely the population
mean is likely to be estimated by the sample mean.
3.5 Validity Threats
Validity refers to the conceptual and scienti�c soundness of a research study and investi-
gation [67]. There are many potential issues that may a�ect the reliability of the research.
Four types of validity threats discussed in the literature [46] are as follows:
Internal validity threats
External validity threats
Constructive validity threats
Statistical conclusion validity threats
35
-
3.5.1 Internal Validity Threats
Internal validity threats refer to the research design that causes the interferences of the
researcher's ability to draw the correct inference from the gathered data [46]. They might
be the inadequate procedures, wrong participant's selection, and technical skill of the
researchers, ambiguities in the experiment, post-test questionnaire and survey design and
improper instrumentation of the experiment [67, 68]. To overcome such types of issues
the authors studied the core concepts of survey design, experiment design and ongoing
research studies about usability of the iPhone and iPad. In the survey and post test
questionnaire, the authors obtained the demographics that help to minimize the wrong
selection of the participants. Pilot tests were conducted for the survey questionnaire,
experiments and post-test questionnaire which helped to �nd out the ambiguities in all
of them. Furthermore the authors also used the counterbalancing technique to overcome
the bias.
3.5.2 External Validity Threats
External validity threats relate to the generalizability of the results of a research study [67].
They might be the sample characteristics, stimulus characteristics and settings, reactivity
of experimental arrangements, multiple treatments interference, novelty e�ect, reactivity
of assessment and timing of measurement [67]. To minimize the external validity threats,
the authors requested all of the participants to report their own experiences because the
purpose of the study is to evaluate the performances of devices not to the participants.
However there might be external validity risks, as people really do not report their personal
reality because they want to see themselves in a good light [68].
3.5.3 Constructive Validity Threats
Constructive validity threats relate to the selection of de�nitions and the measures of the
variables in a study [46]. Authentic data sources such as IEEE, ACM Digital Library,
ScienceDirect provide the correct de�nitions and measures of the variables. The authors
try to use the authentic data sources to minimize such types of validity threats.
3.5.4 Statistical Conclusion Validity Threats
Statistical conclusion validity threats relate to the statistical conclusion of a study. These
threats might be the cause of the inadequate selection of a statistical power such as
sampling and suitable methods [67, 68]. To reduce such type of validity threats, the
authors studies di�erent statistical methods in detail.
36
-
Chapter 4
4 STATISTICAL ANALYSIS, RESULTS AND DISCUSSIONS
This chapter presents the analysis, results and discussions of the results about the collected
data of the studies. There are two main sections of this chapter; the �rst one presents the
results and discussions on the results of the survey data and the second one presents the
results and discussions on the results of the experimental data.
4.1 Survey
In order to evaluate the survey data, the authors clustered the 42 survey statements into
subsections such as system results, touch screen results, keypad results and applications
results. The results of the survey responses are given in the Appendix B and the p values
are given in the Appendix C.
4.2 Demographics Results
This section of the survey was presented the demographics of the respondents. The
authors collected feedback from 14 (8M, 6F, age = 21-35 years) iPhone and 10 (7M, 3F,
age = 23-47 years) iPad users through the survey. 69% of the respondents had been using
the iPhone for more than six months. 8% of the respondents had been using the iPhone
for four to six months and the remaining 23% participants had been using it for the last
one to three months. On the other hand, the iPad responses showed that 20% of the
respondents were using the iPad for more than six months. 30% of the respondents had
been using the iPad for four to six months. 30% of the respondents had been using it for
the last one to three months and 20% of the respondents had been using the iPad for less
than one month.
4.3 Usability and Preference Testing
4.3.1 System Results
The �rst section of the usability testing questionnaire named �system� contained 15 state-
ments. Each statement was related to the overall satisfaction about the iPhone and iPad.
The results and signi�cances between the users opinions against each statement regarding
the iPhone and iPad are:
The system is easy to use for opening applications.7% (1) of the iPhone participants
disagreed with a statement and they considered it di�cult for opening applications. 21%
(3) were near to agree, 14% (2) were agree and 57% (8) were strongly agree that the
37
-
iPhone was easy to use in opening application. On the other hand, the iPad participant's
responses results presented that 10% (1) was near to agree, 70% (7) were agree and 20% (2)
were strongly agree with the above statement. However the comparative study regarding
this statement presented that there was no signi�cant di�erence (p = 0.183 > 0.05)
between the user views for the iPhone and iPad.
The system is easy to use in closing applications. The iPhone results regarding this
statement showed that 8% (1) of the respondent was disagree and he thought it is not easy
to use in closing applications; 25% (3) participants were neutral; 8% (1) participant was
near to agree, 25% (3) were agree and 33% (4) were strongly agree. The iPad participants
responses results presented as: 11% (1) was disagree, 11% (1) was near to disagree; 11%
(1) was neutral; 44% (4) were agree and 22% (2) were strongly agree. The comparative
study of the user's opinions concerning the iPhone and iPad presented that there was no
signi�cance (p = 0.404 > 0.05).
The system is easy to use in �nding help. The iPhone participant's responses results
concerning this statement were presented as: 14% (2) were disagree, 7% (1) was near to
disagree; 43% (6) were neutral; 14% (2) were near to agree, 7% (1) was agree and 14%
(2) were strongly agree. Similarly, the iPad participants responses results presented as:
10% (1) was disagree, 20% (2) were near to disagree; 40% (4) were neutral; 20% (2) were
near to agree and 10% (1) was agree. The comparative study of the user's views showed
that there was no signi�cant di�erence (p = 0.083 > 0.05) between the iPhone and iPad.
I can easily use the system in the sun light. The iPhone participants responses results
regarding this statement were presented as: 7% (1) was near to disagree; 21% (3) were
neutral; 36 % (5) were near to agree, 21% (3) were agree and 14% (2) were strongly
agree. The iPad responses results presented as: 20% (2) were near to disagree; 40% (4)
were neutral; 30% (3) were near to agree and 10% (1) was agree. The comparative study
concerning this statement showed that there is a signi�cant di�erence (p = 0.011 < 0.05)
in user's views regarding the iPhone and iPad. The statistical result showed that the users
preferred the use of iPhone in sun light than the iPad.
It is easy to �nd a new application in the Apple store. The iPhone responses results
showed that 7% (1) was near to disagree; 14% (2) were neutral; 14% (2) were near to
agree, 7% (1) was agree and 57% (8) were strong agree. Similarly the iPad responses
results regarding this statement showed that 10% (1) was disagree, 10% (1) was near to
disagree; 30% (3) were neutral; 30% (3) were near to agree, 10% (1) was agree and 10%
(1) was strongly agree. Comparative statistical result showed that there was a signi�cant
di�erence (p = 0.018 < 0.05). The users responses result showed that they would easily
use the iPad for �nding the new applications on the Apple store comparing the iPhone.
The layout of the applications on the system screen is clear. The results of the iPhone
respondents showed that 7% (1) was disagree; 21% (3) were neutral; 14% (2) were near to
agree, 21% (3) were agree and 36% (5) were strongly agree. Similarly, the iPad participants
38
-
responses results were presented as: 10% (1) was neutral; 60% (6) were agree and 30%
(3) were strongly agree across above statement. The comparison result showed that there
was no signi�cance (p = 0.322 > 0.05) in users views for the iPhone and iPad.
The graphics of the system are appealing for all age groups. Responses results regarding
the iPhone presented that 7% (1) was disagree; 7% (1) was neutral; 21% (3) were near
to agree, 29% (4) were agree and 36% (5) were strongly agree. Similarly the iPad results
presented that 11% (1) was near to agree, 56% (5) agree and 33% (3) were strongly agree.
The comparison result showed that there was no signi�cant di�erence (p = 0.341 > 0.05)
between the participants opinions about the iPhone and iPad.
I can easily change the background colour of the interface. The iPhone participant's
responses results presented as: 14% (2) were disagree, 14% (2) were near to disagree;
21% (3) were neutral; 14% (2) were near to agree and 36% (5) were strongly agree. On
the other side, the iPad participant's responses results showed that 10% (1) was near
to disagree; 30% (3) were neutral; 20% (2) were near to agree, 30% (3) were agree and
10% (1) was strongly agree. However the comparative study showed that there was no
signi�cance (p = 0.489 > 0.05) between the participants opinions regarding the iPhone
and iPad.
I think the system is designed for all age groups. The iPhone results showed that 7%
(1) was strongly disagree, 7% (1) was disagree; 21% (3) were neutral; 14% (2) were near
to agree, 21% (3) were agree and 29% (4) were strongly agree. Alike, the iPad results
presented that 20% (2) were near to disagree; 10% (1) was neutral; 10% (1) was near
to agree, 30% (3) were agree and 30% (3) were strongly agree. There was no signi�cant
di�erence(p = 0.467 > 0.05) in the user's opinions for the iPhone and iPad.
I like the �nishing of the interface. The iPhone responses results presented that 7%
(1) was disagree; 7 % (1) was neutral; 21% (3) were near to agree, 21% (3) were agree
and 43% (6) were strongly agree. Similarly the iPad results presented that 20% (2) were
neutral; 20% (2) were near to agree, 20% (2) were agree and 30% (3) were strongly agree.
The comparative study showed that there was no signi�cant di�erence (p = 0.382 > 0.05)
in the participants' views for the iPhone and iPad.
The interface of the system is pleasant. The results of the iPhone responses: 7% (1)
was disagree; 14% (2) were neutral; 21% (3) were near to agree, 29% (4) were agree and
29% (4) were strongly agree. Similarly, the results of the iPad responses: 10% (1) was
neutral; 40% (4) were near to agree, 20% (2) were agree and 30% (3) were strongly agree.
The comparative study showed that there was no signi�cant di�erence(p = 0.467 > 0.05)
between the users outlook for the iPhone and iPad.
It is easy to install the required application. The iPhone responses results showed
that 7% (1) was disagree; 7% (1) was neutral; 14% (2) were near to agree, 29% (4) were
agree and 43% (6) were strongly agree. Similarly, the iPad responses results showed that
11% (1) was near to disagree; 11% (1) was neutral; 33% (3) were near to agree, 22% (2)
39
-
were agree and 22% (2) were strongly agree. The comparison result showed there was no
signi�cant di�erence (p = 0.174 > 0.05).
It is simple to uninstall the application. The iPhone responses results showed that 7%
(1) was disagree, 7% (1) was near to disagree; 7% (1) was neutral; 7% (1) was near to
agree, 36% (5) were agree and 36% (5) were strongly agree. On the other side, the iPad
responses results showed that 11% (1) was near to disagree; 11% (1) was neutral; 11% (1)
was near to agree, 44% (4) were agree and 22% (2) were strongly agree. The comparative
study showed that there was no signi�cance(p = 0.279 > 0.05) in the iPhone and iPad
users views.
The system has all the functions and capabilities I expect it to have. The iPhone
responses results showed that 29% (4) were disagree, 7% (1) was near to disagree; 7% (1)
was neutral; 29% (4) were near to agree, 21% (3) were agree and 7% (1) was strongly
agree. Similarly, the iPad responses results showed that 30% (3) were disagree, 10% (1)
was near to disagree; 20% (2) were neutral; 20% (2) were near to agree and 20% (2) were
agree. The comparison of the users point of views about the iPhone and iPad was showed
no signi�cance(p = 0.265 > 0.05).
I like using the interface of the system. For such statement, the iPhone responses
results showed that 7% (1) was strongly disagree, 7% (1) was near to disagree; 14% (2)
were neutral; 7% (1) was near to agree, 43% (6) were agree and 21% (3) were strongly
agree. The iPad responses results showed that 13% (1) was near to disagree; 13% (1)
was neutral; 50% (4) were near to agree and 25% (2) was strongly agree. Similarly the
comparative study about the user´s point of views regarding the iPhone and iPad showed
no signi�cance (p = 0.233 > 0.05). Figures 17 and 18 showed the results graphs of the
users opinions across the each statement regarding the iPhone and iPad respectively.
40
-
Figure 18: Percentage Graph Results of the Degree of Agreement Across Each SystemStatement on the iPhone. St. Represents Strongly.
Figure 19: Percentage Results of the Degree of Agreement Across Each System Statementof the iPad. St. Represents Strongly.
4.3.2 Touch Screen Gestures Results
The second section of the usability testing questionnaire named �Touch Screen Gestures�
contained 16-21 statements. Each statement presented di�erent usability features such as
41
-
zooming functionality, gestures memorability and learnability. The results obtained from
the user responses for each statement presented as follows.
I can easily use the text �zoom in� and �zoom out� signs (gestures). The results of
the iPad responses presented that 7% (1) was strongly disagree; 7% (1) was neutral; 7%
(1) was near to agree, 14% (2) was agree and 64% (9) were strongly agree. The iPad
results showed that 10% (1) was near to disagree; 10% (1) was neutral; 40% (4) were
near to agree, 20% (2) were agree and 20% (2) were strongly agree. The analysis of
the relationship showed that there was signi�cant di�erence (p = 0.027 < 0.05). For this
statement the users views result showed that they could easily performed the zoom in/out
gestures using the iPad than the iPhone.
I like the gestures for selecting and zooming the text. The results of the iPhone re-
sponses showed that 14% (2) were near to disagree; 7% (1) was near to agree, 50% (7) were
agree and 29% (4) were strongly agree. Similarly, the results of the iPad responses showed
that 20% (2) was near to disagree; 20% (2) were neutral; 10% (1) was near to agree, 40%
(4) were agree and 10% (1) was strongly agree. The comparative study showed that there
was no signi�cance(p = 0.108 > 0.05) in users opinions for the iPhone and iPad.
I think the gestures are easy to learn. The iPhone results showed that 7% (1) was
disagree, 7% (1) was near to disagree; 7% (1) was neutral; 43% (6) were near to agree, 7%
(1) was agree and 29% (4) were strongly agree. Similarly, the iPad results showed that
20% (2) were near to disagree; 10% (1) was neutral; 20% (2) were near to agree, 50% (5)
were agree. There was no signi�cance (p = 0.424 > 0.05) in users opinions for the iPhone
and iPad.
I think the gestures are easy to remember. The iPhone results showed that 7% (1) was
near to disagree; 14% (2) were neutral; 29% (4) were near to agree, 21% (3) were agree
and 29% (4) were strongly agree. On the other side, the iPad results showed that 40% (4)
were neutral; 20% (2) were near to agree, 30% (3) were agree and 10% (1) was strongly
agree. The comparative study showed that there was no signi�cance(p = 0.214 > 0.05)
between users views for the iPhone and iPad.
I can easily change the location of the contents (icons) on the interface. The iPhone
results showed that 7% (1) was disagree, 7% (1) was near to disagree; 7% (1) was neutral;
7% (1) was near to agree, 29% (4) were agree and 43% (6) were strongly agree. On the
other hand, the iPad results showed that 10% (1) was near to disagree; 20% (2) were
neutral; 50% (5) were near to agree and 20% (2) were agree. There was a signi�cance(p =
0.015 < 0.05) in users opinions for the iPhone and iPad. The results showed that the user
could easily change the contents location using the iPad as compared to iPhone.
I can easily select the written text for deletion. The iPhone results presented that 7%
(1) was strongly disagree, 14% (2) were disagree, 14% (2) were near to disagree; 14%
(2) neutral; 21% (3) were near to agree, 14% (2) were agree and 14% (2) were strongly
agree. Similarly, the iPad results showed that 30% (3) were disagree, 20% (2) were near
42
-
to disagree; 10% (1) was neutral; 30% (3) were near to agree and 10% (1) was agree. The
analysis relationship result showed that there was a no signi�cance (p = 0.198 > 0.05).
In Figure 19 presented the graphs results of the users opinions against each statement on
both the devices.
Figure 20: Graphs Show Percentage Results of the Degree of Agreement Across the EachStatement of the Touch Screen Gestures on the iPhone and iPad Respectively.
4.3.3 Keypad Results
The third section of the survey questionnaire named �keypad� contained eight statements.
Each statement assessed the performance of the keypad of both devices. The results
obtained from the user's responses for the each statement are presented as follows.
I enjoy using the keypad while writing Emails. The results of the iPhone responses
showed that 7% (1) was strongly disagree, 21% (3) were near to disagree; 14% (2) were
neutral; 14% (2) were near to agree, 7% (1) was agree and 36% (5) were strongly agree.
The results of the iPad responses showed that 10% (1) was strongly disagree, 10% (1) was
disagree; 10% (1) was neutral; 20% (2) were near to agree, 40% (4) were agree and 10%
(1) was strongly agree. The comparative study of the users opinion showed that there
was no signi�cance(p = 0.446 > 0.05) for the iPhone and iPad.
I like the keypad in landscape style while writing Emails. The results of the iPhone
responses showed that 14% (2) were disagree; 21% (3) were neutral; 14% (2) were near
to agree, 29% (4) were agree and 21% (3) were strongly agree. Similarly, the iPad results
43
-
showed that 20% (2) were strongly disagree, 10% (1) was near to disagree; 30% (3) were
neutral; 20% (2) were near to agree and 20% (2) were strongly agree. The comparative
study of the user's responses showed that there was no signi�cance(p = 0.143 > 0.05)
regarding the iPhone and iPad.
It is easy to use the special characters (e.g., `' _ *). The iPhone user's responses
results showed that 7% (1) was strongly disagree, 21% (3) were near to disagree; 7% (1)
was neutral; 14% (2) were near to agree, 29% (4) were agree and 21% (3) were strongly
agree. On the other side, the iPad results showed that 22% (2) were disagree; 22% (2)
were neutral; 11% (1) were near to agree and 44% (4) were strongly agree. There was no
signi�cant di�erence (p = 0.206 > 0.05) between the user's opinions for the iPhone and
iPad.
The arrangement of alphabets on the keypad is in accordance to my choice. The iPhone
results showed that 14% (2) were strongly disagree, 7% (1) was near to disagree; 21% (3)
were near to agree, 14% (2) were agree and 43% (6) were strongly agree. The results of
the iPad showed: 10% (1) was strongly disagree, 10% (1) was disagree, 10% (1) was near
to disagree; 10% (1) was neutral; 30% (3) were near to agree and 30% (3) were agree.
The comparison between the user's point of views for the iPhone and iPad was showed
no signi�cance (p = 0.080 > 0.05).
Small keys can cause typing mistakes. The iPhone results showed that 7% (1) was
strongly disagree, 14% (2) were disagree, 7% (1) was near to disagree; 14% (2) were
neutral; 21% (3) were near to agree, 21% (3) were agree and 14% (2) were strongly
agree. Similarly the iPad results showed that 10% (1) was disagree, 20% (2) were near
to disagree; 50% (5) were neutral; 10% (1) was agree and 10% (1) was strongly agree.
The comparison of the user opinions about the iPhone and iPad showed that there was
no signi�cance (p = 0.183 > 0.05).
I can easily �nd the number (e.g. 1, 2, 3...) on the keypad. The iPhone results showed
that 8% (1) was disagree; 15% (2) were neutral; 15% (2) were near to agree, 8% (1) was
agree and 54% (7) were strongly agree. On the other side, the iPad results showed that
20% (2) were near to disagree; 30% (3) were near to agreed, 30% (3) were agreed and
20% (2) were strongly agreed. The relationship between the iPhone and iPad responses
showed that there was no signi�cance(p = 0.129 > 0.05).
The single character keypad improves my writing performance. The iPhone results
showed that 7% (1) was strongly disagree, 7% (1) was disagree, 14% (2) were near to
disagree; 7% (1) was neutral; 29% (4) were near to agree, 14% (2) were agree and 21%
(3) were strongly agree. Similarly, the iPad results showed that 10% (1) was strongly
disagree, 10% (1) was near to disagree; 40% (4) were neutral; 30% (3) were near to agree
and 10% (1) was agree. The comparative study showed that there was no signi�cance(p =
0.143 > 0.05) between user's opinions for the both devices.
I can easily move the cursor forward and backward in the editor while writing text.
44
-
The iPhone results showed that 14% (2) were strongly disagree, 7% (1) was disagree, 36%
(5) were near to disagree; 7% (1) was neutral; 14% (2) were near to agree, 14% (2) were
agree and 7% (1) was strongly agree. Alike, the iPad results showed that 22% (2) were
disagree, 22% (2) were near to disagree; 22% (2) were neutral and 33% (3) were near to
agree. The comparative study showed that there was no signi�cance (p = 0.500 > 0.05)
in user's opinions for the both devices. The Figure 20 represented the graphs results of
users opinions against each statement of the Keypad on the iPhone and iPad respectively.
Figure 21: Graph Shows Percentage Results of the Degree of Agreement Across the EachStatement of the Keypad on the iPhone.
45
-
Figure 22: Graph Shows Percentage Results of the Degree of Agreement Across the EachStatement of the Keypad on the iPad.
4.3.4 Applications Results
The fourth section of the questionnaire named �Applications� contained 13 statements.
The results of each statement for the both devices are presented as follows.
The applications (e.g. Evernote, calendar) are simple. The iPhone results presented
that 29% (4) were near to disagree; 7% (1) was neutral; 29% (4) were near to agree, 14%
(2) were agree and 21% were strongly agree. On the other side, the iPad results showed
that 11% (1) was near to disagree; 44% (4) were near to agree, 33% (3) were agree and
11% (1) was strongly agree. The relationship between the users opinions regarding the
iPhone and iPad was not signi�cant (p = 0.385 > 0.05).
The applications (e.g. Facebook, calendar) work very well in portrait style. The iPhone
results showed that 7% (1) was disagree, 7%