a scalable framework to quantitatively evaluate success ...€¦ · a scalable framework to...

Int. J. Mobile Learning and Organisation, Vol. 5, Nos. 3/4, 2011 299

Copyright © 2011 Inderscience Enterprises Ltd.

A scalable framework to quantitatively evaluate success factors of mobile learning systems

Ghassan F. Issa and Hussein Al-Bahadili* Faculty of Information Technology, Petra University, Amman, Jordan Email: [email protected] Email: [email protected] *Corresponding author

Maher Abuhamdeh Sales & Services Unit, Orange Jordan, Amman, Jordan Email: [email protected]

Abstract: There has been an enormous increase in the use of mobile learning (m-learning) systems in many fields due to the tremendous advancement in information and communication technologies. Although, there are many frameworks that have been developed for identifying and categorising the different components of m-learning systems, most of them have some limitations, drawbacks, and no support for quantitative assessment for the success factors (global weights) of the system criteria. In this paper, a new scalable hierarchal framework is developed, which identifies and categorises all components that may affect the development and deployments of cost-effective m-learning. Furthermore, due to the hierarchal structure of the framework, any of the analytic hierarchy process techniques can be used to quantitatively estimate the success factors of the system criteria. In order to demonstrate the benefits and flexibility of the new framework, we develop an interactive software tool for computing success factors of the different system criteria. The tool is referred to as SFacts, and it is used to compute success factors for different sets of preferences.

Keywords: m-learning; success factors; AHP; analytic hierarchy process; FAHP; fuzzy AHP; FEA; fuzzy extended analysis.

Reference to this paper should be made as follows: Issa, G.F., Al-Bahadili, H. and Abuhamdeh, M. (2011) ‘A scalable framework to quantitatively evaluate success factors of mobile learning systems’, Int. J. Mobile Learning and Organisation, Vol. 5, Nos. 3/4, pp.299–316.

Biographical notes: Ghassan F. Issa received his BET in Electronic Engineering from Toledo University, Ohio, 1983, and BSEE in Computer Engineering from Tri-State University, Indiana, 1984. He received his MS and PhD in Computer Science from Old Dominion University, Virginia, 1987 and 1992. He was a faculty member and Department Chair of Computer Science at

300 G.F. Issa, H. Al-Bahadili and M. Abuhamdeh

Pennsylvania College of Technology from 1992 to 1995. He also served as faculty member and Dean of Computer Science at the Applied Science University in Amman, Jordan, from 1995 to 2007. Currently, he is an Associate Professor and Dean of Faculty of IT, Petra University, Amman, Jordan.

Hussein Al-Bahadili received his BSc in Engineering from University of Baghdad in 1986. He received his MSc and PhD in Engineering from London University in 1988 and 1991. His field of study was parallel computers. He has published many papers and book chapters in different fields of science and engineering in numerous leading scholarly journals, and presented at leading world-level scholarly conferences. He has participated in publishing four novel algorithms in computer networks, network security, data compression and Web search engine. Currently, he is an Associate Professor at the Faculty of IT, Petra University, Amman, Jordan.

Maher Abuhamdeh is currently working as a Team Leader for fixed billing in Sales & Services Unit at Orange, Jordan. He received his MSc and PhD in Computer Information System (CIS) from the University of Banking and Financial Sciences, Amman, Jordan, in 2005 and 2010, respectively. He received his BSc in Computer Information System from Philadelphia University, Amman, Jordan, in 1998. The title of his PhD thesis was ‘A Hierarchal Framework to Quantitatively Evaluate Success Factors of Mobile Learning’. His research interests include software engineering, database, computer networks, network security, artificial intelligence, operating system and expert systems.

1 Introduction

Electronic learning (e-learning) comprises all forms of electronically supported learning and teaching. It first emerged in the late 1980s as a contender to classical face-to-face learning (Issa et al., 2010). It is also referred to as distance learning (d-learning) (Koole, 2006). In the 1990s, a new form of learning was emerged, namely, mobile learning (m-learning), which is generally defined as learning through relatively small-size, low-power consumption, low weight to accompany users anytime and anywhere, location independent, and context-aware devices, like laptops, mobile phones, smart phones, personal digital assistants (PDAs), and any other mobile microprocessor-based devices that may be use in learning (Roschelle, 2003; Wains and Mahmood, 2008; Ahmad, 2009; Laxman, 2009; Issa et al., 2011).

M-learning emerges mainly due to the impressive development in computer, communication, and internet technologies which led to the production of powerful mobile devices; the expansion of wireless communication systems, the tremendous advancement in internet protocols; the affordability of mobile devices; and the demands for continuous business and social communications (Trifonova and Ronchetti, 2003; Upadhyay, 2006; Yordanova, 2007; Williams, 2009). Hence, many terms have been used to define m-learning, for example web-based learning, internet-based learning, and online learning. Each of these definitions implies a ‘just-in-time’ instructional and learning approach.

A scalable framework to quantitatively evaluate success factors 301

The main elements of m-learning are mobile technologies, mobile devices, wireless protocols, wireless language, wireless applications, and learning materials. They all together allow mobile application to be developed (Amin et al., 2006; Li et al., 2008). These elements can be categorised into hardware and software components as shown in Figure 1. The hardware components may include a server, which acts as a source of information (also known as content or learning material) normally accessed using mobile devices through client-server architecture, clients’ mobile devices, and wireless communication systems. The software components may include a learning management system (LMS), a middleware to handle communication and interfacing between a content server and mobile client devices, and application software to run the learning session.

Figure 1 Hardware and software components of m-learning

M-Learning System

Software Hardware

Server (Source of Information)

Clients (Mobile Devices)

Wireless Communication Systems

Learning Management System (LMS)

Middleware

Application Software

The main advantage of m-learning is freeing learners from location-based learning and provides just-in-time learning. In this direction, many system frameworks have been proposed by many researchers around the world to utilise the concept of m-learning in spreading education in rural and diverse areas. Issa et al. (2010) developed an interactive satellite TV based mobile learning (STV-ML) framework, in which a satellite TV station is used as an integral part of a comprehensive interactive m-learning environment. Their proposed framework can assist in building a reliable, efficient, and cost-effective environment to meet the growing demands of m-learning all over the world, especially in developing countries. This concept was used later by Issa et al. (2011) as a cornerstone to propose a unified and interactive m-learning model, namely, IAESat for interactive Arab education satellite, to help with expanding and spreading education in the Arab homeland countries. However, to meet users’ satisfaction and to ensure its expansion, it has to accomplish the following requirements: convenience, collaboration, compatibility, portability, engaging, interaction, increase motivation, bridging of the digital divide, and openness to society (Desmond, 2002; Upadhyay, 2006).

Many frameworks have been developed during the last two decades to help with identifying and categorising the components of m-learning, such as the framework for the rational analysis of mobile education (FRAME) (Koole, 2006), the Helsinki University of Technology model (Hosseini and Tuimala, 2005), the conceptual framework for m-learning design requirements model (Barbosa et al., 2007). Most of these frameworks


have their limitation and they do not support quantitative or numerical evaluation of the success factors of their components due to the non-hierarchal structure of these frameworks. They can only evaluate the success factors through distributed questionnaire that includes a certain set of questions asked to the system users (e.g. learners). Such type of evaluation is not enough to enable the system stakeholders (researchers, developers, and managers) to identify the critical success factors neither it provides the impact of the variation in these factors.

This paper presents a new scalable hierarchal framework for identifying and categorising all components that may affect the development and deployments of cost-effective m-learning. Three distinct levels are considered in this framework; the first one includes the main system criteria, which are: mobile devices, quality, and learners’ requirements constraints. Each of them is subsequently accommodated in a number of sub-criteria denoted as level two criteria, and each sub-criterion is further alienated into sub-sub-criteria (level three criteria), except the learners’ requirements constraints, which is concluded at level two.

Due to the hierarchal structure of the framework, the analytic hierarchy process (AHP) based techniques [such as, the conventional AHP, fuzzy AHP (FAHP), fuzzy extend analysis (FEA), and α-cut-based] can be used to quantitatively evaluate the success factors of the system components. The detail derivation and implementation of these techniques in an interactive and user-friendly software package, SFacts, are given by Abuhamdeh (2010) and Al-Bahadili et al. (2011), who used it to demonstrate the benefits of the new framework, and evaluate the success factors (global weights) of the m-learning criteria for different sets of preferences (relative importance).

This section introduces the main theme of this paper. The rest of this paper is organised as follows: Section 2 describes three well-known frameworks for evaluating the success factors of m-learning. The proposed model is described in Section 3. Section 4 provides a brief introduction to SFacts. Evaluation procedure, results, and discussions are presented in Section 5. Finally, in Section 6, based on the results obtained, conclusions are drawn and recommendations for future work are pointed out.

2 Previous experiences

There are a number of m-learning frameworks that have been developed during the last decade, these are: the framework for the rational analysis of mobile education (FRAME) (Koole, 2006), the Helsinki University of Technology model (Hosseini and Tuimala, 2005), and the conceptual framework for m-learning design requirements (Barbosa et al., 2007). Descriptions of these frameworks are given in following sub-sections.

2.1 The framework for the rational analysis of mobile education (FRAME)

The framework for the rational analysis of mobile education (FRAME) model was developed as a basis for assessing the effectiveness of mobile devices for distance learning (Koole, 2006). The model is intended to establish a description of the m-learning process which, in turn, will allow the development of an operational definition of m-learning. Once this concept has been defined, it is then possible to more accurately ascertain the characteristics of mobile devices that will best support distance education. The model was proposed to guide the development of future mobile devices, the


development of learning materials destined for m-learning, and the selection of teaching and learning strategies for mobile education. The context for the FRAME model is information. Information may be internal or external to the learner; that is, it can be derived from personal, social, technological, or any other environmental stimuli. All such information constitutes the learning environment. Figure 2 outlines the components of the FRAME model.

Figure 2 The FRAME model (Koole, 2006)

(AC) Social

Computing

(C)Social Aspect

(A)Device

Usability Aspect

(ABC) M-LEARNING

(AB) Context Learning (B)

Learner Aspect

(BC) Interaction Learning

Information

Information

The three circles represent the device usability aspect (A), learner aspect (B), and social aspect (C). The regions where two circles overlap are called the secondary intersections and contain attributes that belong to both aspects. The attributes located inside the secondary intersections of context learning (AB) and social computing (AC) describe the affordances of mobile devices. The secondary intersection labelled interaction learning (BC) contains instructional and learning theories viewed through the philosophical lens of social constructivism. All three aspects overlap at the primary intersection of m-learning (ABC) which is located in the centre of the Venn diagram. The device usability aspect describes characteristics unique to electronic, networked mobile technologies; the learner aspect describes characteristics of individual learners; and, the social aspect describes the mechanisms of interaction among individuals.

2.2 The Helsinki University of Technology model

The Helsinki University of Technology model (Hosseini and Tuimala, 2005) is based on three main domains: mobile usability, wireless technology, and e-learning system. Figure 3 outlines the main domains of the model.


Figure 3 The Helsinki University of Technology model (Hosseini and Tuimala, 2005)

M obileU sability

W irelessTechnology

E -Lean ing System

The first domain is mobile usability, which involves mobile device type, mobile device features, and mobile content design methods. Mobile usability researches focus on identifying the requirements of each mobile device capability to offer services in a usable way, providing services for mobile devices without considering the usability issues as a useless effort. The second domain is the wireless technology which is related to network infrastructure and operators’ rolls. The users utilise the mobile devices if the network provides fast, secure and reliable network connections. The type of network infrastructure and the cost of the services are important factors that affect the overall m-learning. The third domain is e-learning system which is the requirements of the e-learning system and the type of utilised e-learning platform. The e-learning platform influences greatly the m-learning system, and user groups influences the selection of e-learning types and also the distribution of services to the devices. The framework consists of four steps (Hosseini and Tuimala, 2005):

1 Education components which defines the needs and services of the system.

2 Device network capabilities which identify the capabilities and boundaries of the network also determine the mobile devices and their usability requirements.

3 Concept development which writes scenarios, then express the detail services distribution to different utilised mobile devices with technology capability.

4 Prototyping of a sample system for concept and system evaluation so that a final decision can be taken.


2.3 The conceptual framework for m-learning design requirements

A conceptual framework was developed by considering four m-learning requirements and interactions among these requirements, namely, learning objective, learning experience, m-learning contexts, and generic mobile environment design issues (Barbosa et al., 2007). Figure 4 shows how these four requirements interact with each other. The framework is based on combination of a game metaphor and several studies of m-learning contexts. As an example of these interactions, consider m-learning for dynamic complex situations, such as rescue services which requires the collective learning objectives, particularly developing team skills. This learning objective would be supported by the learning experiences conflict, competition, challenge, and opposition and social interaction. These learning experiences would require the corresponding m-learning contexts; including activity, spatio-temporal, facility and collaboration, which in turn would map in a context specific way to generic mobile design requirements. The framework was tested and applied to studying m-learning environments that had differing characteristics.

Figure 4 The conceptual framework for m-learning design requirements

User role and profile

Mobility

Mobile interface design

Media types

Communication support

Identity

Learner

Activity

Spatial Temporal

Facility

Collaboration

Organized contentsBusiness rules, learning roles

Outcome and FeedbackTest scores, leagues

Goals and objectivesSkills and knowledge

Representation or storyCase studies, role plays

Conflict, Competition,Challenge, OppositionIndividual and team

development

Social InteractionBlogs, wikis, discussiongroups, tests, teamwork

Improvedskills

New skills

Social skills

Teamskills

Generic mobile environment issues

Mobile learningcontext issues

Learning experience Learning objectives


3 The proposed scalable hierarchal framework

This section presents the detail description of a new framework for m-learning. It can be easily recognised that there are a number of criteria (components, requirements, and constraints) that may constraint the design of a successful m-learning system. In developing this framework, first, all criteria that influence the efficiency and performance of the system are identified. Second, each group of criteria is related to a certain representative category. Finally, the categories are structured in a hierarchal framework as shown in Figure 5. The proposed framework can be explained as follows:

• The overall goal or objective, which is declared as m-learning system, is laid at the first (highest) level of the hierarchy.

• The main criteria that may constraint the design and performance of the system and consequently affect its success factors gathered at the second level.

• The sub-criteria for each of the second-level criteria are collected at the next level and so on for sub-sub-criteria.

• The lowest level contains the candidate alternatives.

As a result of the flexible hierarchical structure of the framework, it can be easily scale up/down to accommodate/eliminate any necessary/unnecessary components.

Figure 5 Standard hierarchical framework

Fourth Level

Goals or Objectives

Criterion 1 (C1)

Criterion c (Cc)

Sub criterion

(C11)

Sub criterion

(C1m)

Sub criterion

(Cc1)

Sub criterion

(Ccn)

Sub-sub criterion

(C111)

Sub-sub criterion (C11w)

Sub-sub criterion (C1m1)

Sub-sub criterion (C1mx)

Sub-sub criterion

(Cc11)

Sub-sub criterion

(Cc1y)

Sub-sub criterion

(Ccn1)

Sub-sub criterion

(Ccnz)

Alternative 1 (A1)

Alternative 2 (A2)

Alternative a (Aa)

First Level (Highest)

Second Level

Third Level

Lowest Level

Based on previous studies (Papanikolaou and Mavromoustakos, 2006; Parsons and Ryu, 2006), we believe that the main criteria that must be considered in developing a successful m-learning are:


1 Mobile device constraints.

2 Quality of services and applications.

3 Learners’ requirements.

Therefore, in the proposed framework, these criteria are laid at the second level in the system hierarchy as shown in Figures 6–9. In what follows, a description is given for each of the above criteria.

Figure 6 The main criteria of proposed m-learning framework

A B C

M-Learning

Quality (C2)

Learner’s Requirements

(C3)

Mobile Device (C1)

3.1 Mobile device constraints

Mobile devices interact with services offered by learning platforms through specially developed applications, the interaction between mobile learners and service providers is accomplished through different mobile devices and wireless networks. Thus, the mobile device constraints are decomposed into three sub-criteria: software, hardware, and network (Figure 7). A brief description is given for each sub-criterion and their sub-sub-criteria.

1 Software constraints: From system point of view software can be decomposed into three different types: operating system, development environment and applications. From user point of view the most important software constraint is the application, which, in turns can be decomposed into user interface and pedagogical materials. In this framework, the software constraints are considered as a combination of the two points of views and only two sub-criteria are considered for simplicity, these are: (a) standardised operating system, and (b) use-friendly interface.

2 Hardware constraints: These constraints mainly concerns with mobile device hardware. It can be decomposed into: (a) small screen, (b) small multifunction keypads, (c) limited computational power, (d) limited memory, (e) limited battery lifetime, (f) non-volatile capacity, (g) display resolution, (h) graphical limitation, and (i) complicated text input mechanisms.


3 Network constraints: M-learning involves the delivery of pedagogical material to learner through wireless technology. The network constraints can be decomposed into: (a) limited bandwidth, (b) low connection stability, (c) limited security, (d) converge, (e) interference, (f) high delay, (g) frequent disconnection, and (h) roaming capabilities.

Figure 7 The components of the mobile device constrains

A

Hardware (C12)

Network (C13)

Software (C11)

Small screen (C121)

Limited bandwidth (C131)

Standardised operating system

(C111)

Small multi-function keypad

(C122)

Low connection stability (C132)

User friendly interface (C112)

Limited memory (C124)

Limited security (C133)

Limited computing power

(C123)

Limited battery life

(C125)

Convergence (C134)

Interference (C135)

High delay (C136)

Frequent disconnection

(C137)

Roaming capabilities

(C138)

Non-volatile capacity (C126)

Display resolution (C127)

Graphical limitation

(C128)

Complicated text input mechanism

(C129)


3.2 Quality of services and applications

The quality constraints can also be discussed from three points of views. These are: hardware quality, software quality and network quality. In the context of software/ hardware engineering, although there are several different definitions for quality, the definition stated by the ISO 9126 standard is the most appropriate, in which software/hardware quality measures how well software/hardware is designed (quality of design), and how well the software/hardware conforms to that design (quality of conformance). Whereas quality of conformance is concerned with implementation, quality of design measures how valid the design and requirements are in creating a worthwhile product. Quality in the ISO 9126 standard concerns with the following characteristics: usability, functionality, system reliability, efficiency, maintainability and portability. Each of these sub-criteria is also disintegrated into a number of sub-sub-criteria as shown in Figure 8.

1 Usability: The system must be implemented in such a way to allow easy understanding of its functioning and behaviour. The usability decomposed into six sub-criteria, (a) understandability, (b) learnability, (c) friendliness, (d) operability, (e) playfulness and (f) ethics.

2 Functionality: The system must include all the necessary features to accomplish the required task. Functionality can be decomposed into a number of sub-criteria, these include: (a) accuracy, (b) suitability, (c) compliance, (d) interoperability and (e) privacy.

3 Reliability: The system must maintain a specified level of performance in case of software faults with the minimum crashes possible. Sensitive data such as student details, student grades, exams, etc. should be protected and correctly recovered. Reliability can be decomposed into a number of sub-criteria: (a) fault tolerance, (b) crash frequency, (c) recoverability, (d) maturity and (e) security.

4 Efficiency: System response-time performance must be fast enough to satisfy user needs. Long waiting times result in reduced user interest, de-motivation and boredom leading to unwillingness to use the system. Specifically, the system should be able to adapt to the different mobile devices and technologies. A balance between quality and performance should be maintained. Fast access to information must be examined also throughout the system’s life to ensure that user requirements are continuously met on one hand, and that the application remains useful on the other. Efficiency can be decomposed into a number of sub-criteria, these are: (a) response time, (b) different vendor, (c) quality, (d) network speed and (e) bandwidth.

5 Maintainability: Due to rapid technological changes especially in the area of internet and mobile engineering, the demanding user requirements for continuously updated material, and for easy system modifications and enhancements, maintainability is so important factors. It can be decomposed into a number of sub-criteria, these are: (a) changeability, (b) serviceability, (c) reparability and (d) testability.

6 Portability: Portability is defined as the ability of the mobile application to be installed and run by any mobile device as well as to be adaptive to different specified environments. It can be decomposed into: (a) different environment, and (b) different mobile.


Figure 8 The components of the quality constraints

Functionality (C22)

Reliability (C23)

Usability (C21)

Understand-ability (C211)

Learnability (C212)

Operability (C214)

Friendliness (C213)

Playfulness (C215)

Ethics (C216)

B

Efficiency (C24)

Maintainability (C25)

Portability (C26)

Accuracy (C221)

Suitability (C222)

Inter-operability

(C224)

Compliance (C223)

Privacy (C225)

Fault tolerance (C231)

Crash frequency

(C232)

Maturity (C234)

Recoverability(C233)

Security (C235)

Response time

(C241)

Different vendor (C242)

Network speed

(C244)

Quality (C243)

Bandwidth (C245)

Changeability (C251)

Serviceability(C252)

Testability (C254)

Reparability (C253)

Different environment

(C261)

Different mobile (C262)

3.3 Learners’ requirements

The individual communication and interaction among instructors and users is very important and can be easily created in real application where face-to-face communication exists. The sharing of information, experiences and views forms an indispensable part of the learning process and is very effective in disseminating knowledge and establishing a community of learners. Learners should be able to maintain their individuality and differentiation while participating in the online mobile community, thus allowing them to express their personal preferences and abilities, as well as motivating and inspiring them, learners’ requirements can be decomposed into a number of sub-criteria. These sub-criteria are shown in Figure 9 and discussed below (Schreurs, 2008).

1 Identification of learners’ needs: The m-learning environment should be shaped according to the predefined learners’ needs and course required pedagogical outcome. The learners’ needs vary depending on socio-cultural background, education level, skills and competences acquired from previous education and training.

2 Structuring of the pedagogical material: The pedagogical material should be constructed in a way that facilitates the successful transfer of the required knowledge. Customisation of the material according to its recipients increases and retains user interest and reinforces knowledge transfer.


3 Enhancement of the m-learning environment: The m-learning environment can be used either complimentary or in parallel to the e-learning environment. In either case, the m-learning environment should adhere to the basic mechanisms and functions of the real environment.

4 Motivation for learner participation: Learners are not always willing to use the virtual environment for a number of reasons, such as the difficulty of using the mobile device, the non-intuitive nature of the environment, the provision of reduced interactivity.

5 Tutorials: The m-learning environment should be able to offer the learners a basic problem-solving mechanism. Mechanisms such as online tutorials, contact with the instructor, references, resources materials and even access to a technical helpdesk would offer learners support and help.

6 Collaborative mechanisms: In the virtual environment, the learners can be easily isolated and separated from the rest of the class. This can be prevented in the real classroom by using face-to-face communication. In the virtual classroom, student isolation can be avoided, by organising and operating in an online collaborative basis, with the establishment of learners groups and the use of collaborative work. In this manner, the learners are encouraged to participate and communicate through the electronic environment.

7 Supporting tools: Vocational training requires different solutions than academic training and undergraduate training has different pedagogical targets than postgraduate training. Tools and components can be utilised to enhance the m-learning environment more efficiently.

8 Combination of learning processes: The most important learning processes are identified as follows: analysis, synthesis, reasoning, judging, problem solving, collaboration, simulation, evaluation, presentation and relation. These processes should be used dynamically for constructing the learning scene for each course and student.

Figure 9 The components of the learner’s requirements constraints

Structuring of the

pedagogical material (C32)

Enhancement of the

M-Learning environment

(C33)

Identification of learner’s

needs (C31)

C

Motivation for learner

participation (C34)

Tutorials (C35)

Collaborative mechanisms

(C36)

Supporting tools (C37)

Combination of learning processes

(C386)

4 The success factors (SFacts) tool

The success factor (SFacts) tool is an interactive software package developed to quantitatively evaluate the success factors of m-learning. SFacts utilises four evaluation AHP-based techniques, these are: the AHP, FAHP, FEA, and α-cut-based techniques.


The detail description and implementation of these techniques can be found in the works of Abuhamdeh (2010) and Al-Bahadili et al. (2011). The tool is developed using .Net technology, in particular using VB.NET of Microsoft Visual Studio 2008 (Version 9.021022.8 RTM). It can be used by m-learning decision makers to estimate the success factors of their systems and investigate the effect of the criteria that may affect developing a successful m-learning system. The main features of SFacts include:

1 Has a user-friendly (interactive) graphical user interface (GUI).

2 Build using an object-oriented approach so that it can be easily modified to incorporate new evaluation techniques or factors.

3 Quantitatively evaluate the success factors of any hierarchy structured process using AHP-based techniques.

4 Include enough help to help users make the right choices.

5 Demand little resource in terms of processing speed and memory.

6 Enter new project or retrieve an existing one.

7 Select the evaluation technique (AHP, FAHP, FEA, α-cut-based).

8 Enter the number of criteria at each level. The current version of the package can handle up to 99 criteria at the main (second) level.

9 Enter the name of each criterion (sub-criterion, sub-sub-criterion, and so on) and select value for its relative importance from drop-down menu.

10 Proceed to calculation to estimate the global weight for each criterion and rank the criteria accordingly from the one with the highest weight to the one with the lowest, i.e. from the most important to the least important.

11 All input and computed parameters are stored in a database using Microsoft Access format and can be retrieved for further processing any time later.

5 Results and discussions

In order to demonstrate the flexibility of the proposed comprehensive m-learning framework and the effectiveness of SFacts in evaluating the critical success factors of m-learning, we set a scenario for evaluating the success factors of system main criteria [mobile device constraints (C1), quality (C2), and learner’s requirements (C3)] for three sets of preferences (relative importance). The main objectives of this scenario are investigating the effects of the actual relative importance of each criterion on the global weights (ranking), and comparing the estimated global weights using different AHP-based evaluation techniques (AHP, FAHP and FEA).

The first step in performing any evaluation process using an AHP-based technique is to specify the pairwise comparison matrix depending on the relative importance of each criterion. In this work, we shall assume for the sake of demonstration the following relative importance of the three criteria (i.e. there are no practical reasons for selecting these relative importance and it is up to the user to practically set these values):


1 Case #1: C1 has a strong importance relative to C2 and moderate importance relative to C3.

2 Case #2: C1 has a very strong importance relative to C2 and moderate importance relative to C3.

3 Case #3: C1 has an extreme importance relative to C2 and moderate importance relative to C3.

The numerical values of the elements of the pairwise comparison matrix vary according to the evaluation technique. The pairwise comparison matrices of the three criteria (C1, C2, and C3) for the above cases for the AHP, FAHP, and FEA techniques are given in Table 1. Table 1 Comparison matrices: Pairwise comparison matrices for the AHP technique (Cases #1

to #3)

Case #1 Case #2 Case #3 Criteria

C1 C2 C3 C1 C2 C3 C1 C2 C3 C1 1 5 3 1 7 3 1 9 3 C2 1/5 1 1/3 1/7 1 1/3 1/9 1 1/3 C3 1/3 3 1 1/3 3 1 1/3 3 1

Fuzzified pairwise comparison matrices for the FAHP technique (Cases #1 to #3) C1 0.5 0.5 0.6 0.5 0.6 0.6 0.5 0.8 0.6 C2 0.5 0.5 0.4 0.4 0.5 0.4 0.2 0.5 0.4 C3 0.4 0.6 0.5 0.4 0.6 0.5 0.4 0.6 0.5

Fuzzified pairwise comparison matrices for the FEA technique (Cases #1 to #3) C1 1,1,1 3,5,7 1,3,5 1,1,1 5,7,9 1,3,5 1,1,1 7,9,11 1,3,5 C2 1/7,1/5,1/3 1,1,1 1/5,1/3,1 1/9,1/7,1/5 1,1,1 1,1/3,1/5 1/11,1/9,1/7 1,1,1 1,1/3,1/5 C3 1/5,1/3,1 1,3,5 1,1,1 1/5,1/3,1 5,3,1 1,1,1 1/5,1/3,1 5,3,1 1,1,1

Following the construction of the pairwise comparison matrices, SFacts was used to compute the relative weights of the main criteria using each of the three AHP-based techniques. The computed weights are listed in Table 2 and plotted in Figure 10. Table 2 Scenario #1: Comparing the computed weights for Cases 1 to 3 using AHP, FAHP,

and FEA techniques

Case #1 Case #2 Case #3 Criteria

AHP FAHP FEA AHP FAHP FEA AHP FAHP FEA C1 0.6333 0.3750 0.5734 0.6687 0.4286 0.6724 0.6923 0.5217 0.7582 C2 0.1062 0.2857 0.0512 0.0882 0.2500 0.0000 0.0769 0.1538 0.0000 C3 0.2605 0.3158 0.3754 0.2431 0.3158 0.3276 0.2308 0.3158 0.2418

Total weight 1.000 0.9765 1.000 1.000 0.9844 1.0000 1.0000 0.9913 1.0000


Figure 10 Variation of relative weights for Scenario #1 (see online version for colours)

The main outcomes of this scenario are discussed below. All the techniques showed that as the importance of C1 is increasing from strong importance to extreme importance, its individual weight increases relative to the other two criteria, which comes in line with the common sense of the decision maker. Furthermore, this increase is not only paid by one of the other criteria, but it is shared between them; despite the fact that the relative importance of C1 to C3 was remained unchanged. But in microscopic level analysis of the relative importance of the three criteria, C3 is positively changed relative to C2. This can be explained in this way: C3 keeps the same importance relative C1, which increases relative to C2, means the relative importance of C3 to C2 should increase.

Decision makers usually based their decisions on the differences between the computed weights. As these differences increase more decisive and comfortable decisions can be taken. Therefore, for this scenario, a better decision can be taken using FEA, then the AHP. For the FAHP, due to the way of constructing the fuzzified pairwise comparison matrices, where non-optimistic preferences for the importance were chosen, the computed weights have little differences between them (e.g. for Case #1, the weights for C1, C2, C3 are 0.3750, 0.1857, and 0.3158, respectively). Thus, difficult decision needs to be taken. The whole picture is changed when the fuzzified pairwise comparison matrices were constructed with optimistic preferences for the FEA techniques.

6 Conclusions

This paper presented a detail description of a new scalable hierarchal framework that can be used to accommodate and categorise all criteria affecting the development and deployments of cost-effective m-learning. The framework is so simple and straightforward as it can be easily used to add or remove any criteria that may affect system performance. Using the AHP-based evaluation tool (SFacts), the global weights for the system criteria can be estimated and ranked from the one with the highest weight (most important) to the one with the lowest (least important). This in fact what decision makers is looking for, so they can based their decisions on and make their planning according to the weight and


importance of the criterion. Ranking the system criteria is so important, because the estimated global weight can only give a limited indication on the actual importance of the criterion, and it will be more helpful if it is compared against the weights of other criteria.

In this paper, we only consider the analysis for the main level criteria, it is highly recommended to perform further investigations to analyse the performance of m-learning considering various sets of preference to the system low levels criteria.

References Abuhamdeh, M. (2010) A hierarchal framework to quantitatively evaluate success factors of

mobile learning, PhD Thesis, University of Banking and Financial Sciences, Amman, Jordan. Ahmad, N. (2009) Examining the Effectiveness of a Mobile Electronic Performance Support

System in a Workplace Environment, PhD Dissertation, Columbia University, New York, USA.

Al-Bahadili, H., Issa, Gh. and Abuhamdeh, M. (2011) ‘A hierarchical framework for evaluating success factors of m-learning’, The Research Bulletin of Jordan ACM, Vol. 2, No. 3, pp.46–51.

Amin, A.H.M., Mahmud, A.K., Zainal Abidin, A.I. and Abdul Rahman, M. (2006) ‘M-learning management tool development in campus-wide environment’, The Journal of Issues in Informing Science and Information Technology, Vol. 3, pp.423–434.

Barbosa, J., Hahn, R., Barbosa, D. and Geyer, C. (2007) ‘Mobile and ubiquitous computing in an innovative undergraduate course’, Proceedings of the 38th ACM Technical Symposium on Computer Science Education (SIGCSE), ACM Press, pp.379–383.

Desmond, K. (2002) The Future of Learning from E-learning to M-learning, Hagen, Germany. Hosseini, A.M. and Tuimala, J. (2005) ‘Mobile learning framework’, in Isaias, P., Borg, C.,

Kommers, P. and Bonanno, P. (Eds): Proceedings of the IADIS International Conference on Mobile Learning, pp.203–208.

Issa, G.F., Hussain, S.M. and Al-Bahadili, H. (2010) ‘A framework for building an interactive satellite TV based m-learning environment’, The International Journal of Interactive Mobile Technologies, Vol. 4, No. 3, pp.19–24.

Issa, G.F., Hussain, S.M. and Al-Bahadili, H. (2011) ‘Unified m-learning model through interactive education satellite: a proposal for an Arab Homeland education satellite’, The International Journal of Interactive Mobile Technologies, Vol. 5, No. 2, pp.26–33.

Koole, M.L. (2006) The Framework for the Rational Analysis of Mobile Education (FRAME) Model: An Evaluation of Mobile Devices for Distance Education, Master Thesis, Athabasca University, Alberta-Canada.

Laxman, K. (2009) ‘Facilitating adult mobile technology-based learning through problem solving’, International Journal of Mobile Learning and Organization, Vol. 3, No. 1, pp.15–24.

Li, Q., Lau, R.W.H., Shih, T.K. and Li, F.W.B. (2008) ‘Technology supports for distributed and collaborative learning over the Internet’, ACM Transactions on Internet Technology, Vol. 8, No. 2.

Papanikolaou, K. and Mavromoustakos, S. (2006) ‘Critical success factors for the development of mobile learning applications’, Proceeding of the 24th ASTED International Multi-Conference Internet and Multimedia Systems and Applications, ACTA Press, Track 516-068.

Parsons, D. and Ryu, H. (2006) ‘A framework for assessing the quality of mobile learning’, Proceedings of the International Conference for Process Improvement, Research and Education, pp.17–27.

Roschelle, J. (2003) ‘Keynote Paper: Unlocking the learning value of wireless mobile devices’, Journal of Computer Assisted Learning, Vol. 19, No. 3, pp.259–271.


Schreurs, J. (2008) ‘Mobile e-learning course scenario model on PDA’, International Journal of Mobile Learning and Organisation, Vol. 2, No. 4, pp.358–368.

Trifonova, A. and Ronchetti, M. (2003) ‘Where is mobile learning going?’, Proceedings of the World Conference on E-learning in Corporate, Government, Healthcare, & Higher Education, pp.1794–1801.

Upadhyay, N. (2006) ‘M-learning: a new paradigm in education’, International Journal of Instructional Technology and Distance Learning, Vol. 3, No. 2, pp.27–34.

Wains, S.I. and Mahmood, W. (2008) ‘Integrating m-learning with e-learning’, Proceedings of the 9th ACM Special Interest Group for Information Technology Education (SIGITE’08), ACM, NY, pp.31–37.

Williams, P. (2009) Assessing mobile learning effectiveness and acceptance, PhD Dissertation, The George Washington University, Columbia, USA.

Yordanova, K. (2007) ‘Mobile learning and integration of advanced technologies in education’, Exchange Organizational Behavior Teaching Journal, Vol. 285, pp.1–6.

a scalable framework to quantitatively evaluate success ...€¦ · a scalable framework to...

Documents