design, implementation and evaluation of mobile learning resources

Víctor Robledo-Rella, Luis Neri, Violeta Chirino, Julieta Noguez and Gerardo Aguilar

Tecnológico de Monterrey, Campus Ciudad de México

vrobledo@itesm.mx; neri@itesm.mx; vchirino@itesm.mx; jnoguez@itesm.mx; gerardo.aguilar@itesm.mx

Design, Implementation and Evaluation of Mobile Learning

Resources

Abstract

• The Tecnológico de Monterrey, Campus Ciudad de México (CCM) launched in 2008 its Mobile Learning (ML) model which involves a massive widespread and systematic use of ML resources among high school and undergraduate students. Each new first semester student was given cell phone with 24/7 data access.

• Mobile resources were designed and catalogued for specific courses according to the curricula by a group of professors and instructional designers.

• In order to measure its effectiveness, we describe here the methodology followed to design, implement and evaluate a ML resource for Physics I courses.

Abstract

• We designed and applied an Pre-Test / Pos-Test assessment tool for a total sample of N = 222 students enrolled in two consecutive semesters.

• We found that the integrated relative gain for the focus group, which used ML (Nf = 113) is about 30% larger than that for the control group, which did not used ML (Nc = 109).

• A student survey (N = 95) show that students consider that their experience using ML resources was good, and that the use of ML resources helped them to improve the comprehension of course content.

• These results encourage and support the use of ML resources for educational purposes.

Introduction• Mobile devices are here to stay, and we barely grasp the extend to which

they are changing the way we live and communicate.

• The Tecnológico de Monterrey is a world leading private institution regarding the use of innovation and technology for educational purposes.

• This work belongs to a wider project been carried out by the eLearning Research Group of the CCM regarding the impact that ML resources have on the teaching and learning process. Further results are reported by Aguilar et al. 2010.

• The contribution of this work is to present and discuss the main elements behind the design and implementation of a ML resource, anchored to an evaluation instrument to measure its pedagogical effectiveness.

Design of a ML resource• Educational resources were produced having in mind the following

considerations:a) Its content is aimed to achieve a given course learning objective.b) It meet basic quality criteria regarding graphics and sound

deployment through standard mobile devises.c) It includes variation of stimuli to engage the learner and promote

meaningful learning.

• A Mobile Learning Knowledge Management System was (SICAM) developed at the CCM to storage and classify ML resources’ metadata, so to evolve them towards Learning Objects.

Design of a ML resource• We re-designed a ML resource for the Physics I course for the

Dynamics of a Particle module, entitled “Free Body Diagram (FBD) and Newtons’s 2nd law”, being these central concepts for the course.

• We designed a 5 min video, and it was uploaded into a WAP portal hosting all ML resources produced by CCM faculty and Virtual University staff.

• The video explains how to build a FBD step by step, how to decompose forces into their Cartesian components and how to write down Newton’s 2nd law to study the dynamic evolution of a system. At the end, the video pose a challenge to the student about what as been explained.

Design of a ML resource• The aesthetic and graphic design of the ML resource was carried out by

CCM Virtual University staff (length and time management, clarity of presented elements, sound and meaningful content, adequate audio and video elements,…)

• ML resource is self-contained (framing, development and closure-evaluation)

• See WAP portal and video .

Evaluation process• We developed a Pre-Test / Pos-Test assessment tool consisting of a

short written quizzes (~ 15 min) that were answered during class hours. Their aim is to asses the degree to which the ML resource fulfills its course learning objectives. Both Pre-Test and Post-Test were basically the same.

• We applied the Pre-Test to a total sample of N = 222 students (6 Physics courses and 2 Math courses; with similar ML resources for the Math course – see Aguilar et al. 2010) during the Fall 2009 and Spring 2010 terms.

Evaluation process• The Pre-Test was applied to all students before using the ML resource.

We then split randomly the sample so that half the students were asked to study the ML video for about a week using their mobile devices. This defines the control group (Nc = 113).

• The remaining students did not used the mobile device, which constitutes the control group (Nc = 109).

• Finally, the Post-Test was applied to the whole sample.

Results and conclusions• For each group we defined and integrated relative gain (Hake 1988):

Where <Pos> and <Pre> are the average Pos-Test and Pre-Test grades for a given group, respectively.

• Our results are summarized in Table 1 and Figure 1.

Pre

PrePosG

100

Table 1. Average Pre-Test, Pos-test and integrated relative gains

.

N <Pre> <Pos> < g_i > < G >

Control Phys. A 28 34 ±18 43 ±18 0.09 ±0.30 0.14

Control Phys. B 17 30 ±17 44 ±18 0.14 ±0.38 0.20

Control Phys. C 18 25 ±18 33 ±22 0.13 ±0.22 0.11

Control Math A 24 1 ±4 63 ±25 0.64 ±0.25 0.64

Control Math B 22 0 ±0 36 ±14 0.36 ±0.14 0.34

< 0.29 >

Focus Phys. E 21 28 ±17 47 ±16 0.23 ±0.21 0.26

Focus Phys. F 29 22 ±17 46 ±20 0.29 ±0.27 0.30

Focus Phys. G 22 33 ±18 57 ±18 0.34 ±0.30 0.36

Focus Math C 21 1 ±3 81 ±12 0.81 ±0.13 0.81

Focus Math D 20 0 ±0 62 ±19 0.62 ±0.19 0.62

< 0.47 >

Integrated Relative Gain vs. Pre-Test

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 20 40 60 80 100Pre-Test

Inte

gra

ted

Rel

ativ

e G

ain

With - Phys Without - Phys With - Math Without - Math

0

5

10

15

20

Alumns

Question 9. My general experience using ML resources was

Very bad

Bad

Regular

Good

Very good

0

5

10

15

20

Alumns

Question 5: The content and activities in the ML resources helped me to improve my comprehension of the concepts

of the Physics course

Totally disagree

Disagree

Indiferent

Agree

Totally agree

Figure 2. Survey results (N = 95)

Results and conclusions• Although the total sample is still small and the uncertainties are

relatively large, we found that there is a higher integrated relative gain for the focus group as compared to that for the control group:

<G>control = 0.29, while <G>focus = 0.47.

• The difference between Math and Physics results is considerable and is due to differences in the Math Pre-Test/ Pos-Test and ML resource itself.

• A student survey for the Physics courses (N = 95) showed students consider that their experience using ML resources was good, and that the use of ML resources helped them to improve their comprehension of course concepts, as well as their ability to solve specific problems.

¡Gracias por su atención!