venkat sastry talking on mole projecta global technology initiative

Dr. Venkat V S S Sastry,

Jacob Hodges

13 November 2014 1ONRG Grant N62909-10-1-7121

21 countries

Chile

Bulg

aria

Canada

Azerb

aija

n

Fra

nce

Germ

any

Georg

ia

Jord

an

Mexic

o

Norw

ay

Pola

nd

Peru

Rom

ania

UK

US

Serb

ia

South

Afr

ica

Sin

gapore

sw

itzerland

Ukra

ine

UA

E

VIDEO/project_launch_movie.wmv


13 November 2014ONRG Grant N62909-10-1-71212

• MoLE Project was a two year initiative sponsored by

Coalition Warfare Program involving 24 countries

• Based on requirement by the Commander,

US Naval Forces Europe/Naval Forces Africa/Sixth Fleet

Deputy Director for Training and Readiness to

effectively operate in the largest maritime area of

operations where the most difficult

challenge is the ability to train and communicate

MoLE would leverage the global cellular

network infrastructure. Mobile

technologies and emerging mobile

applications/service models to build an m-

Learning capability



2mins 50 secs



Medical Content Working Group

Technology & Transition Group

Testing & Evaluation Working Group


MC

WG

: A

cco

mp

lish

men

ts-

To

ols

an

d C

on

ten

t


Tec

hn

olo

gy &

Tra

nsit

ion

W

G

Blending “task” and

“learning” Initially, content

grouped by moment of

need

Mission Tools

A collection of interactive job aids and performance

support tools

Learning

The Learning section focuses on informal and formal

training experiences.

Library

The Library contains a multimedia collection of indexed

materials in different media (text, eBook, video) covering

topics relevant to the organization.

Mission Packs

Mission packs allows download of new content

Standards

The Standards section contains critical references

Network

Crowd-sourcing information on local resources and key

contacts.

Place holder for

future content

Most work based m-learning happens in stolen moments, which is why a smooth blend between learning, performance-support and productivity-tools is so key

Compelling m-learning is not just about the content. Or the technology. It is also about good learning design, good usability, and an app that you want to use

“

”

11

Converting and originating new content for MSO mobile learning

11

Task

Locate the pre-deployment

checklist and check off at

least 3 items for your

personal preparations.

Add one new item that you

might need, and save

14

Usability review October 2011

Proof of Concept trials April to June 2012 using “baked in” evaluation

layer

All content needed significant

“mobilization”

Blend of tools + courses worked well

Benefits of strong UX / user interface

Cross platform = reuse (content still in use)

Rich media / video very effective

Similar responses regardless of age,

device type, nationality, background

Testi

ng

& E

va

luati

on

Gro

up

Data collection plan• Demographic data

• Formulate research questions

Compliance with Research Ethics

Data analysis and interpretation


The Testing & Evaluation Process adhered to the

research protocol guidelines established by the

Ethics

1. US Department of Defense Directive

(DODD) 5400.11 (DOD Privacy Program,

DODD 3216.2)

2. (Protection of Human Subjects in DoD-

Supported Research, SECNAVINST

3900.39D, 32CFR219 (Common Rule),

3. European Union (EU) Data Protection

Requirements

4. UK Ministry of Defence JSP 536 (Research

Ethics Committee).


All the investigators need

to complete Research

Ethics courses online

Testi

ng

& E

va

luati

on

Gro

up

(a) What is the effectiveness or

practicality of using mobile

technologies to provide

training?

0

10

20

30

40

50

600

1

2

3

4

5

6

7

All Users Responses

19

(b) What is the benefit or availability

of using mobile devices in providing

training?

Testi

ng

& E

va

luati

on

Gro

up

20

(c) Do persons who have used the

mobile device provided training

believe they are capable to

perform a desired outcome?

Testi

ng

& E

va

luati

on

Gro

up

21

(d) What is the degree to which a

mobile training application is

available in austere environments to

as many Role players as possible?

Testi

ng

& E

va

luati

on

Gro

up

22

EVALUATION QUESTIONS Category

1.1 Rate your confidence in using the checklist. Self-efficacy

2.1 How useful are mobile devices for this type of training? Utility

2.2 How easy was the CTIP content to navigate? Accessibility

3.1 How useful is access to digital reference materials on a mobile device? Utility

4.1 How easily could you explain to a colleague how to look up NGO logos? Self-efficacy

5.1 How useful are videos as a tool to enhance your understanding of USAID?Usefulness

6.1 How helpful are collaboratively updated contact details? Self-efficacy

7.1 How useful are mobile devices for training? Utility

7.2 How useful are mobile devices for refresher training? Utility

7.3 How useful are mobile videos to enhance understanding? Utility

7.4 How useful are digital reference materials on mobile devices? Utility

7.5 How useful are collaborative contact pages to access real time information?Utility

7.6 How easy was the Global MedAid app to navigate? Accessibility

7.7Please write up to five single words which best describe your overall

experience of using the mobile device as a tool for learning.

7.8 What did you like most about the mobile device as a tool for learning?

7.9 What did you like least about the mobile device as a tool for learning?

8 Any additional comments

23


Course

content

Short quiz

Collapsible

content


End of

module

exercises

Hyperlinked

terminology


0 – no response

1 – not confident

at all

2 – not confident

3 – not very

confident

4 – neither

confident nor

unconfident

5 – quite

confident

6 – confident

7 – very

confident

30

• A “Welcome Email” was initially sent to

approximately 600 volunteer on the first

day of the MoLE Proof of Concept

• The announcement email provided

additional information on installation, the

user guide and an introduction video.

31

• Launched: 252

• Started Proof of Concept: 166 (65.9%)

• Completed Proof of Concept: 126 (50%)

32

Question Responses

Age less than 20, 20-29, 30-39,

40-49, 50+, no answer

Gender Male, Female

How proficient are you in English Beginner . . . Advanced

Are you using your own personal smartphone for the purpose of

this trial?

Yes, No, No Answer

How comfortable are you with using the mobile device that's

running the MoLE app? [Beginner to Advanced]

Beginner . . . Advanced

Have you previously been involved in humanitarian assistance or

disaster relief operations?

Yes, No, No Answer

What is your professional expertise? Medical, Rescue, Training,

E-learning, Other

Have you taken the Trafficking in Persons (CTIP ) course within

the last two years?

Yes, No, No Answer

33

Age Group Breakdown

< 20 years 5 1.87%

20+ years 51 19.03%

30+ years 83 30.97%

40+ years 75 27.99%

50+ years 52 19.40%

No Response 2 0.75%

34

Gender Breakdown

Male 181 67.54%

Female 87 32.46%

35

Expert

Entering

37

NewbyExpert and

above

43

Likert Rating Question All-Users Android iPhone

N/A or Skipped 8 3 5

1 Not useful at all 9 1 8

2 Not useful 9 4 5

3 Not very useful 20 14 6

4 Neither useful nor not useful 18 6 12

5 Quite useful 33 15 18

6 Useful 60 23 37

7 Very useful 22 5 17

n: 179

Minimum: 0

Maximum: 7

Mean: 4.7 +/-0.3

Median 5.4

Mode: 6

Standard Deviation: 1.9

Variance: 3.5

0102030405060

0

1

2

3

4

5

6

7

All Users Responses

0

5

10

15

20

250

1

2

3

4

5

6

7

Android Users Responses

0

10

20

30

400

1

2

3

4

5

6

7

iPhone Users Responses

44




2 Not useful 4 2 2




6 Useful 47 26 21


n: 179

Minimum: 0

Maximum: 7

Mean: 5.1 +/-0.3

Median 6.0

Mode: 7


Variance: 4.5

0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

05

1015202530

Skipped orN/A

1

2

3

4

5

6

7


0

10

20

30

40

Skipped orN/A

1

2

3

4

5

6

7



A majority of the Android

participants were “confident “and

iPhone users were

“very confident” in using the

checklist to support pre-

deployment activities.

46




2 Not useful 2 0 2




6 Useful 45 17 28


n: 179

Minimum: 1

Maximum: 7

Mean: 5.1 +/-0.3

Median 6.0

Mode: 7


Variance: 5.4

0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

05

10152025

Skipped orN/A

1

2

3

4

5

6

7


0

10

20

30

40

Skipped orN/A

1

2

3

4

5

6

7


47




2 Not useful 5 1 4




6 Useful 54 30 24


n: 179

Minimum: 2

Maximum: 7.0

Mean: 4.8 +/-0.3

Median 6.0

Mode: 6


Variance: 5.2

0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

0

10

20

30

40

Skipped orN/A

1

2

3

4

5

6

7


05

1015202530

Skipped orN/A

1

2

3

4

5

6

7


48




2 Not useful 5 0 5




6 Useful 46 27 19


n: 179

Minimum: 1

Maximum: 7

Mean: 4.5 +/-0.4

Median 5.0

Mode: 6


Variance: 5.8

01020304050

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

05

1015202530

Skipped orN/A

1

2

3

4

5

6

7


05

1015202530

Skipped orN/A

1

2

3

4

5

6

7


49




2 Not useful 0 0 0




6 Useful 54 21 33


n: 179

Minimum: 3

Maximum: 7

Mean: 4.4 +/-0.4

Median 6.0

Mode: 6


Variance: 7.2

0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

0

10

20

30

40

Skipped orN/A

1

2

3

4

5

6

7


05

10152025

Skipped orN/A

1

2

3

4

5

6

7


50




2 Not useful 11 4 7




6 Useful 38 20 18


n: 179

Minimum: 0

Maximum: 7

Mean: 4.4 +/-0.3

Median 5.0

Mode: 6, 7


Variance: 4.2

0

10

20

30

400

1

2

3

4

5

6

7

All Users Responses

0

5

10

15

200

1

2

3

4

5

6

7


0

10

20

30

400

1

2

3

4

5

6

7


51




2 Not useful 22 8 14




6 Useful 58 29 29


n: 179

Minimum: 0

Maximum: 7

Mean: 4.8 +/-0.3

Median 5.3

Mode: 6


Variance: 3.4

0102030405060

0

1

2

3

4

5

6

7

All Users Responses

05

1015202530

0

1

2

3

4

5

6

7


05

1015202530

0

1

2

3

4

5

6

7


‘They can because they

think they can’

52




2 Not useful 2 0 2




6 Useful 45 17 28


n: 179

Minimum: 2

Maximum: 7

Mean: 4.7 +/-0.4

Median 6.0

Mode: 7


Variance: 6.3

0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

0

10

20

30

40

Skipped orN/A

1

2

3

4

5

6

7


05

10152025

Skipped orN/A

1

2

3

4

5

6

7


53




2 Not useful 4 2 2




6 Useful 47 26 21


n: 179

Minimum: 0

Maximum: 7

Mean: 4.7 +/-0.3

Median 5.6

Mode: 7


Variance: 4.6

0

10

20

30

40

50

600

1

2

3

4

5

6

7

All Users Responses

05

1015202530

0

1

2

3

4

5

6

7


0

10

20

30

400

1

2

3

4

5

6

7


54

280

113

42 33

59.8%

84.0%

92.9%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

0

50

100

150

200

250

300

350

400

450

Nu

mb

er

of

Refe

ren

ces

Categories

A mobile application framework that

consisted of (a) a mobile applications (for

iOS and Android mobile operating

systems) for the storage and display of

mobile learning content range of open

formats; and (b) an application

programming interface (API) describing

how interactive content (produced in

HTML) may interact with the application.


A web-based content management system

for the storage and management of

learning content, plus the associated

standards for 'packaging' and meta-data

describing learning content.



A 'mobile evaluation layer' for surveying

user feedback through the mobile

applications during the period of research

by (a) presenting users with contextually-

relevant questions during use of the mobile

applications; and

collecting and collating evaluation data in

a central repository, including techniques

for associating multiple evaluation

responses from single user and allowing

evaluations to be undertaken 'offline'

(without cellular or Wi-Fi internet access)

for asynchronous transfer to the repository.


All of the Mobile Learning Environment

(MoLE) source code for the iOS and Android

Apps and the online content repository has

been uploaded to Open Source Mobile

Learning Environment (OMLET) and is

available, free of charge, at

https://wss.apan.org/1539/JKO/mole/SitePa

ges/Home.aspx.


Mobile devices are practical and effective

Screen size is very important

Productivity tools enhance learning

experience

Mobile training applications are

appropriate in challenging environments


Book

The “Mobile Learning Environment (MoLE) Project: A Global

Technology Initiative”, is about a two-year training and education

project that was sponsored by the U.S. Department of Defense

(DoD)’s Coalition Warfare Program (CWP) and other research-

related organizations that focused on providing training/education in

areas with low-bandwidth and limited internet connectivity and

infrastructure.

This book is available in paperback and e-reader version via

Amazon. The paperback is also available via CreateSpace

(https://www.createspace.com/4174484)

It will be of interest to:

• Learning and Development Professionals

• Educators and Teachers

• E-Learning and m-Learning Professionals

• Evaluators

• Government and Non-Governmental Organizations (NGO)

• Information Technology (IT) Professionals

• Medical Professionals

• Quality Management Professionals

• Research, Development, Testing & Evaluation (RDT&E)

Professionals

• Scientist and Technology (S&T) Professionals

A blog has been created http://wp.me/p2ZlwE-4o for anyone

interested in reading the Preface.

A brief overview of the MoLE Project via Slideshare

(http://www.slideshare.net/jrhodges1972/tracking/new#)

https://www.createspace.com/4174484

http://wp.me/p2ZlwE-4o

http://www.slideshare.net/jrhodges1972/tracking/new


Backup Slides

(Not used in the presentation)

Learning is strongly influenced by

motivational factors and context

Measurement of learning is hotly debated

the extent to which technology enhanced

instruction facilitates learning adds to this

debate


Plan – 80 minutes

Play Video to begin with (3 mins)

Background to project; sponsors

Early designs (from Tribal slides)

Example content 15 mins

• Medical

• MATTs

• Evaluation

Challenges 10 mins

• Technological

• Cultural; expectations; linguistic

• Ethical compliance; IRBs

Data Collection Plan 15 mins

• Informed consent

• Questionnaire


• Data Analysis 15 mins

• Demographic data

• Exploratory data analysis

• Analysis of text; word co-

occurrence

• Final Report

• Major outputs

• Include Jake’s booklet on

Amazon

• Concluding Remarks

• Offer Live Demo

• Quiz NGO etc

67

• A “Welcome Reminder Email”

was initially bi-weekly to remind

the volunteers.

• The reminder email also

provided additional information

on installation, the user guide

and an introduction video.

69




2 Not useful 2 0 2




6 Useful 53 27 26


n: 179

Minimum: 2

Maximum: 7

Mean: 4.9 +/-0.4

Median 6.0

Mode: 7


Variance: 6.1

0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

0

10

20

30

40

Skipped orN/A

1

2

3

4

5

6

7


05

1015202530

Skipped orN/A

1

2

3

4

5

6

7


70




2 Not useful 2 0 2




6 Useful 46 25 21


n: 179

Minimum: 1

Maximum: 7

Mean: 5.0 +/-0.3

Median 6.0

Mode: 7


Variance: 5.2

01020304050

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

0

10

20

30

40

Skipped orN/A

1

2

3

4

5

6

7


05

10152025

Skipped orN/A

1

2

3

4

5

6

7


71




2 Not useful 2 0 2




6 Useful 45 17 28


n: 179

Minimum: 2

Maximum: 7

Mean: 4.7 +/-0.4

Median 6.0

Mode: 7


Variance: 6.3

0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

0

10

20

30

40

Skipped orN/A

1

2

3

4

5

6

7


05

10152025

Skipped orN/A

1

2

3

4

5

6

7


72




2 Not useful 0 0 0




6 Useful 47 21 26


n: 179

Minimum: 3

Maximum: 7

Mean: 4.7 +/-0.4

Median 6.0

Mode: 7


Variance: 7.5

0

20

40

60

80

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

01020304050

Skipped orN/A

1

2

3

4

5

6

7


05

10152025

Skipped orN/A

1

2

3

4

5

6

7


73




2 Not useful 0 0 0




6 Useful 34 11 23


n: 179

Minimum: 3

Maximum: 7

Mean: 4.5 +/-0.4

Median 5.0

Mode: 7


Variance: 7.3

0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

0

10

20

30

40

Skipped orN/A

1

2

3

4

5

6

7


0

5

10

15

20

Skipped orN/A

1

2

3

4

5

6

7


74




2 Not useful 0 0 0




6 Useful 46 24 22


n: 179

Minimum: 3

Maximum: 7

Mean: 4.6 +/-0.4

Median 6.0

Mode: 7


Variance: 7.5

0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

01020304050

Skipped orN/A

1

2

3

4

5

6

7


05

10152025

Skipped orN/A

1

2

3

4

5

6

7


75




2 Not useful 2 1 1




6 Useful 35 15 20


n: 179

Minimum: 1

Maximum: 7

Mean: 4.4 +/-0.4

Median 6.0

Mode: 7


Variance: 7.6

0102030405060

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

0

10

20

30

40

Skipped orN/A

1

2

3

4

5

6

7


05

10152025

Skipped orN/A

1

2

3

4

5

6

7


76




2 Not useful 3 2 1




6 Useful 44 23 21


n: 179

Minimum: 0

Maximum: 7

Mean: 4.0 +/-0.4

Median 5.0

Mode: N/A or

Skipped


Variance: 6.9

01020304050

Skipped orN/A

1

2

3

4

5

6

7

All Users Responses

05

1015202530

Skipped orN/A

1

2

3

4

5

6

7


05

10152025

Skipped orN/A

1

2

3

4

5

6

7



0

1

2

3

4

5

6

7

<20 20+ 30+ 40+ 50+ <20 20+ 30+ 40+ 50+

0

3.9

1.41.1

3.4

6.8

6.3

5.24.8

3.9

Series1

As a learning platform

As a collaborative platform

As a problem solving tool

As a data acquisition device

As a computational resource (e.g. run

simulations)

As a communication device


an evaluation strategy must be cognizant

of desired learning outcomes for the

intended learner, and should take into

account the combination of instructional

strategies that include delivery media


learning effectiveness is a function of

effective pedagogical practices

smartphone technologies facilitate

constructive engagement with the learner

that in turn influences the pedagogic

practice


The internal and cognitive information

processing events include temporary

storage, encoding and retrieval.

The external instructional events include

gaining attention, drill and practice, and

feedback


Individuals who proactively approach their learning tasks• Personal initiative, perseverance, adaptive skills

Self-regulated processes help learners acquire knowledge and skills more effectively

Self-regulation of learning has been suggested to refer to self-directed processes that help learners learn more effectively


Item Instrument; Remarks

Background knowledge,

prior skills, abilities

Conduct a pre-test

Self-regulation in learning

context(Toering, Elferink-

Gemser, Jonker, Van

Heuvelen, & Visscher,

2012)

Use of tools that support self-regulation; Collect Self-regulation of Learning and

Self-Report Scale prior and post learning intervention. The online form is available

at

https://docs.google.com/spreadsheet/viewform?formkey=dHFwNnBiRjY1QWM0eE5

ielA5X0pVdXc6MQ.

Learning content Analysis of learning content provided for the learner;

Learning support tools Use of auxiliary support tools online, on the device; usage by the learner;

measurement of learner engagement; learner analytics

Learning transfer Conduct post-tests, apply knowledge to specific tasks that map to learning

objectives; these tests include, quizzes, case studies, written assignments and peer

review

Learner feedback Qualitative feedback and textual analysis

Learning style/preferences Learning modality preference inventory and Kiersey Temperament Inventory. It is

anticipated that the learning style has second order effects on learning. (see also

(Neuhauser, 2002)). Learning style may have stronger influence on learner

engagement and attention span.

Time on task Record of tasks completed and the use of tools. This data may be collected on the

device (aka MoLE).

Instructional method Types used and the nature of the tasks

Teacher effects Controlled experiment


https://docs.google.com/spreadsheet/viewform?formkey=dHFwNnBiRjY1QWM0eE5ielA5X0pVdXc6MQ

venkat sastry talking on mole projecta global technology initiative

Education