Learning analytics for Virtual Learning Environments

Pedro J. Muñoz-Merino, José A. Ruipérez-Valiente, Derick Leony, Carlos Delgado Kloos, Héctor Javier

Pijeira Díaz, Javier Santofimia, Diego Redondo

Twiter: @pedmume email: pedmume@it.uc3m.es

Universidad Carlos III de Madrid

2

Our Learning Analytics Research lines

● Technological issues

Effective processing of the data

Compatibility between different sources

● Useful indicators

● Useful visualizations

● Know the relationships among indicators. Know

the causes

● Evaluation of the learning process

● Actuators: Recommenders, adaptive learning,

etc.

3

Inference of indicators

● Pedagogical theories

● Best practices

● Adapted models from other systems

● Indicators that can predict other interesting

indicators

4

Examples of indicators I

Learning profiles

Effectiveness, efficiency,

interest

Behaviours

Skills

Emotions

Pedro J. Muñoz-Merino, J.A. Ruipérez Valiente, C. Delgado Kloos, “Inferring higher level learning information from low level data for the Khan Academy platform.” Proceedings of the Third International Conference on Learning Analytics and Knowledge, pp. 112–116. ACM, New York, (2013)

Example II: Inference of optional activities

José A. Ruipérez‐Valiente, Pedro J. Muñoz‐Merino, Carlos Delgado

Kloos, Katja Niemann, Maren Scheffel, “Do Optional Activities Matter in Virtual Learning Environments?”,European Conference on Technology Enhanced Learning, pp 331-344, (2014)

6

Example III: Inference of more precise indicators: Effectiveness

Pedro J. Muñoz-Merino, José A. Ruipérez-Valiente, Carlos Alario-Hoyos, Mar Pérez-Sanagustín, Carlos Delgado Kloos, "Precise effectiveness strategy for analyzing the effectiveness of students with educational resources and activities in MOOCs", Computers in Human Behavior, vol. 47, pp. 108–118 (2015)

7

Example IV: Inference of emotions

Derick Leony, Pedro J. Muñoz-Merino, José A. Ruipérez-Valiente, Abelardo Pardo, David Arellano Martín-Caro, Carlos Delgado Kloos, "Detection and evaluation of emotions in Massive Open Online Courses", Journal of Universal Computer Science (2015)

Example V: HMM for emotion detection

Each emotion has its corresponding HMM.

Events are normalized and used as observations for a HMM.

Two current approaches:

Probability of emotion HMM is mapped to emotion level.

Emotion with highest probability in HMM.

Derick Leony, Pedro J. Muñoz-Merino, Abelardo Pardo, Carlos Delgado Kloos, “Modelo basado en HMM para la detección de emociones a partir de interacciones durante el aprendizaje de desarrollo de software”, Jornadas de Ingeniería Telemática, 2013

Level of relationship

•30.3 % hint avoider •25.8 % video avoider •40.9 % unreflective user •12.1% of hint abuser

Hint

avoid.

Video

avoid.

Unrefl.

User

Hint

abuser

Hint avoidance 1 -0.186

Video avoid. 1 0.289 0.096

Unrefl. user 0.289 1

Analysis of optional activities •76.81% of students who accessed, did not use any optional activities

•Difference of use per course and depending on type of optional activities

•55 goals (50.9% completed)

•40 votes (26 positive, 13 indifferent, 1 negative)

Type of activity Percentage of activities accessed

0% 1-33 % 34-66% 67-99% 100%

Regular learning

activities 2.48% 51.55% 23.19% 18.84% 3.93%

Optional activities 76.81% 18.43% 4.14% 0.41% 0.21%

Level of relationship: optional activities vs learning activities

Optional activities

sig. (2-tailed) N = 291

Exercises accessed:

0.429*

(p=0.000)

Videos accessed:

0.419*

(p=0.000)

Exercise abandonment:

-0.259*

(p=0.000)

Video abandonment:

-0.155* (p=0.008)

Total time:

0.491* (p=0.000)

Hint abuse:

0.089 (p=0.131)

Hint avoider:

0.053

(p=0.370)

Follow recommendations:

-0.002

(p=0.972)

Unreflective user:

0.039

(p=0.507)

Video avoider:

-0.051

(p=0.384)

Proficient exercises sig. (2-tailed)

N = 291

Optional activities:

0.553*

(p=0.000)

Goal:

0.384* (p=0.000)

Feedback:

0.205* (p=0.000)

Vote:

0.243* (p=0.000)

Avatar:

0.415* (p=0.000)

Display badges:

0.418*

(p=0.000)

Clustering depending on effectiveness

Grouping types of students

Prediction models

Learning gains= 25.489 - 0.604 * pre_test + 6.112 * avg_attempts + 0.017 * total_time + 0.084 * proficient_exercises

José A. Ruipérez-Valiente, Pedro J. Muñoz-Merino, Carlos Delgado Kloos, “A predictive Model of Learning Gains for a Video and Exercise Intensive Learning Environment”, Artificial Intelligence in Education conference, pp. 760-763 (2015)

Evaluation of the learning process

Pedro J. Muñoz-Merino, José A. Ruipérez-Valiente, Carlos Delgado Kloos, Maria A. Auger, Susana Briz, Vanessa de Castro, Silvia N. Santalla, “Flipping the classroom to improve learning with MOOC technology: A case study with engineering students using the Khan Academy platform”, Submitted for publication

15

ALAS-KA: Khan Academy

José A. Ruipérez-Valiente, Pedro J. Muñoz-Merino, Derick Leony, Carlos Delgado Kloos, “ALAS-KA: A learning analytics extension for better understanding the learning process in the Khan Academy platform”, Computers in Human Behavior, vol. 47, pp. 139-148 (2015)

16

LA for Google Course Builder:

● Links

Source code:

https://github.com/roderickfanou/Learning_analytics_on_

GCB

Video: http://www.youtube.com/watch?v=b5vOmWHgmqg

Miriam Marciel, Foivos Michelinakis, Roderick Fanou, and Pedro J. Muñoz-Merino, “Enhancements to Google Course Builder: Assessments Visualisation, YouTube Events Collector and Dummy Data Generator”, SINTICE conference (2013), Madrid, España

Visualization of emotions Time-based

visualizations Contextualized

visualizations

Timeline for the level of each emotion.

Daily average of emotion by day. Emotion levels vs student grades.

Emotion levels vs use of tools.

Derick Leony, Pedro J. Muñoz-Merino, Abelardo Pardo, Carlos Delgado Kloos, “Provision of awareness of learners’ emotions through visualizations in a computer interaction-based environment”, Expert Systems With Applications, vol. 40, no. 3 (2013), pp. 5093-5100

18

ANALYSE: LA support for open edX

● General information and Github

http://www.it.uc3m.es/pedmume/ANALYSE/

https://github.com/jruiperezv/ANALYSE

● Evaluation

● Demos

https://www.youtube.com/watch?v=3L5R7BvwlDM&fe

ature=youtu.be

Héctor Pijeira, Javier Santofimia, José A. Ruipérez-Valiente, Pedro J. Muñoz-Merino, Carlos Delgado Kloos, Demo ANALYSE, EC-TEL 2015

19

MOOC of maths

● MOOC of maths:

Available at: http://ela.gast.it.uc3m.es

Topics: Units of measurement, algebra, geometry

High school education for adults

Generation of educational materials: CEPA Sierra

Norte de Torrelaguna: Diego Redondo Martínez

28 videos, 32 exercises

Configuration, support and personalization of the

MOOC platform at Univ. Carlos III de Madrid

Open for everyone

Flipped the classroom methodology

20

ANALYSE in the MOOC of maths

● Uses of ANALYSE in the experience

Self-reflection for students

Support for the flipped classroom

Evaluation of educational materials

Evaluation of students

Evaluation of the course

21

Example I: Self-reflection

22

Example II: Evaluation of students

23

Example III: Flipped classroom, evaluation of materials

24

Example IV: Evaluation of the course

25

Present and future work in ANALYSE

● Design and implementation of higher level

learning indicators and their correspondent

visualizations

● Scalability. Work with a big amount of users

● Recommender based on previous data analysis

we have performed on other experiences ->

clustering, prediction, relationship mining

● Integration with Insights

26

ANALYSE in the Spanish TV

● Part of the mapaTIC project:

“La Aventura del Saber”, La 2, TVE,

http://www.rtve.es/alacarta/videos/la-aventura-del-

saber/aventuramapatic/3163463/

ANALYSE and the MOOC of maths: 5:30-7:11

27

Adaptation (I)

28

Adaptation (I): ISCARE

29

ISCARE competition tool

30

ISCARE: Publications

Pedro J. Muñoz-Merino, Manuel Fernández-Molina, Mario Muñoz-Organero, Carlos Delgado Kloos, “Motivation and Emotions in Competition Systems for Education: An empirical study”, IEEE Transactions on Education vol. 57, no. 3 (2014), pp. 182-187

Pedro J. Muñoz-Merino, Carlos Delgado Kloos, Manuel Fernández Molina, “Analyzing Learning Gains in a Competition Intelligent Tutoring System”, Intelligent Tutoring Systems Conference (2014)

Pedro J. Muñoz-Merino, Manuel Fernández Molina, Mario Muñoz-Organero, Carlos Delgado Kloos, “An Adaptive and Innovative Question-driven Competition-based Intelligent Tutoring System for Learning”, Expert Systems With Applications, vol 39, no 8 (2012), pp. 6932-6948

ADAPTATION (II)

Repetici

ones

4

4

Aprobado

Suspenso

No realizado

No cuenta

Master Thesis: Alma Collado

Learner profile is augmented with affective information (AI).

Fila 1 Fila 2 Fila 3 Fila 4

0

2

4

6

8

10

12

Columna 1

Columna 2

Columna 3

Implementation in the ROLE Personal Learning Environment.

Adaptation (III): Document recommendation based on emotions

Adaptation (IV)

Pedro J. Muñoz-Merino, Carlos Delgado Kloos, Mario Muñoz-Organero, Abelardo Pardo, "A Software Engineering Model for the Development of Adaptation Rules and its Application in a Hinting Adaptive E-learning System", Computer Science and Information Systems, vol. 12, no. 1, pp. 203-231(2015)

34

Adaptation (V): FEST Tutor

35

Adaptation (V): FEST Tutor

Master Thesis: Belén del Campo

Learning analytics for Virtual Learning Environments

Pedro J. Muñoz-Merino, José A. Ruipérez-Valiente, Derick Leony, Carlos Delgado Kloos, Héctor Javier

Pijeira Díaz, Javier Santofimia, Diego Redondo

Twiter: @pedmume email: pedmume@it.uc3m.es

Universidad Carlos III de Madrid