learning analytics applied to serious games, invited talk at ecgbl 2013 porto, portugal
TRANSCRIPT
![Page 1: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/1.jpg)
Learning Analytics Applied to Serious GamesEuropean Conference in Game-Based Learning (ECGBL 2013), Porto, Portugal
Baltasar Fernandez-Manjon, [email protected] , @BaltaFMe-UCM research group, www.e-ucm.es
![Page 2: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/2.jpg)
2
About me and context
‣ CS Professor at Complutense U.• Director of e-UCM
‣ e-UCM research group about Learning technologies
• 15 researchers• Serious games
- Application to the medical domain• European projects
- GALA- SEGAN- CHERMUG
• www.e-ucm.es
![Page 3: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/3.jpg)
3
Learning analytics and SG?The NMC Horizon
Report: 2013 Higher Education Edition
‣ new and emerging technologies on teaching, learning, and research
Time-to-Adoption Horizon: Two to Three Years
‣ Games and Gamification
‣ Learning Analyticshttp://www.nmc.org/publications/2013-horizon-report-higher-ed
![Page 4: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/4.jpg)
4
Serious Games use?
‣ Serious Games have probed to be educationally effective in several domains
• Medicine, military, business, corporate training‣ But still is a low adoption of Serious Games
• Cost? ROI?‣ Serious Games considered usually as a
complementary content• Mainly used for motivational purposes• No actual impact on the final mark
‣ Difficult to include Serious Games in the learning curriculum
• Assessment of acquired learning?
![Page 5: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/5.jpg)
5
Black box model‣ Games as
independent pieces of content
‣ No information about what is happening during the in-game play
‣ Or very simple
• Completed or not completed
• Time used
del Blanco et al (2013). Using e-Learning standards in educational video games. Computer Standards & Interfaces 36 (1) pp. 178–187
![Page 6: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/6.jpg)
6
Learning Analytics
‣ Improving education based on data analysis
• Data driven
‣ Evidence-based education
‣ Related with …• Educational data mining
• Business intelligence
• Visual analytics
www.ed.gov/edblogs/technology/files/2012/03/edm-la-brief.pdf
![Page 7: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/7.jpg)
7
Different uses
‣ Learning analytics• Data analysis that helps students improve
learning outcomes.‣ Academic/program analytics
• Data analysis that provides information of what is happening in a specific program and how to plug holes or otherwise adjust.
‣ Institutional analytics• Data analysis that helps make decisions about
how to improve at the institutional level.
Learning Impact Blog, Big data: Cool; Small data: Cooler http://www.imsglobal.org/blog/?p=258
![Page 8: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/8.jpg)
8
When data is analyzed?
‣ Off-line• Analyzing data after use• Discovering patterns of use• Allows to improve the experience for future
‣ Real time• Analyzing data while the system is in use to
improve/adapt the current learning experience• Allows to also use it in actual presential classes
‣ Mixed approach
![Page 9: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/9.jpg)
9
Extensive Data
‣ Large number of participants‣ Relatively limited number of variables‣ Usually very little demographic information‣ Relatively few observations for each user‣ Wide but shallow data set
Adapted from: Learning Analytics and Educational Data Mining WorkshopNew York University – CREATE Lab April 4–5, 2013
![Page 10: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/10.jpg)
10
Intensive Data
‣ Relatively low number of participants‣ Large number of observations for each
variable‣ Large number of variables for each
participant, such as• User actions, In-Game Events• Survey responses; Extensive demographic
information• Video Observations• Biometric Data (HR, RESP, GSR, EEG, EKG)• Eye-tracking
‣ Narrow but deep data set
‣ Correlations among different data?Adapted from : Learning Analytics and Educational Data Mining WorkshopNew York University – CREATE Lab April 4–5, 2013
![Page 11: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/11.jpg)
11
But there is a parallel world …
http://us.battle.net/wow/en/media/screenshots/races?keywords=&view#/goblins04
![Page 12: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/12.jpg)
12
Game Analytics‣ Application of
analytics to game development and research
‣ Telemetry
• Data obtained over distance
• Mobile games, MMOG
‣ Game metrics
• Interpretable measures of data related to games
• Player behaviour
![Page 13: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/13.jpg)
13
Game metrics
‣ User metrics
• Customer
• Player
‣ Performance metrics
• Technical infrastructure
‣ Process metrics
• Development of the game
‣ User metrics
• Generics metrics
• Genre specific metrics
• Game specific metrics
![Page 14: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/14.jpg)
14
Game metrics
Springer, Game Analytics Maximizing the Value of Player Data, pag 22
![Page 15: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/15.jpg)
15
Game requirements for LA‣ Most of games are black boxes.
• No access to what is going on during game play.
‣ We need access to game “guts”• Game state, game variables
‣ Or the game must communicate with the outside world
• Using some logging framework• Not applicable to COTS games (yet)
‣ Mozila Open Badges? http://openbadges.org/
![Page 16: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/16.jpg)
SESSION VARIABLES
PLAYER GAME
VARIABLES
GAME VARIABLES
TRACES
GOALS
From game data to educational information
PLAYER VARIABLES
User Information
Educational System Information
SERIOUS GAME
![Page 17: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/17.jpg)
Sessions
GAME PLAYER SESSIONS+ =
A player plays a game. Produces a game session.
One game session is a set of traces of ONE specific player in ONE specific game.
SESSIONS =
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
player starte
d phase
1
player scored
200in
phase 1
player clicked
in Help
button
![Page 18: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/18.jpg)
VARIABLESGENERATO
R
SESSION
Analyzing game sessions
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
VAR
VAR
VAR
VAR
VAR
VAR
VAR
VAR
VAR
VAR
GOAL RESULT
GOAL RESULT
GOAL RESULT
GOAL RESULT
GOAL RESULT
GOALSGENERAT
OR
VAR DEFINITI
ONVAR
DEFINITION
VAR DEFINITI
ONVAR
DEFINITION
VAR DEFINITI
ON
VAR DEFINITI
ON
GOALDEFINITI
ON
GOALDEFINITI
ONGOAL
DEFINITION
GOALDEFINITI
ON
![Page 19: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/19.jpg)
Variable generator: User sessions
VAR DEFINITION VAR+ =
TRACE
TRACE
varName:‘beatgame‘, operation: { type:'trace_present‘,traceType: ‘logic', trace: { event:‘end_game’ }}
event: ‘end_game',timeStamp: 1380181610150
usersession.beatgame = true;+ =
![Page 20: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/20.jpg)
Goal Generators: User sessions
VAR
VAR
GOALDEFINITION
GOAL RESULT=
usersession: { beatgame: true, score: 4747}
id: ‘finalResult’result: ‘beatgame && score > 3000’}
usersession: { beatgame: true, score: 4747, goals: { finalResult: true }}
=
+
+
![Page 21: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/21.jpg)
PLAYER * GAME
Analyzing game results
SESSION
SESSION
VAR VAR
VAR VAR
GOAL RESULT
GOAL RESULT
SESSION
SESSION
VAR VAR
VAR VAR
GOAL RESULT
GOAL RESULT
VARIABLESGENERATO
R
GOALSGENERATO
R
VAR DEFINITI
ONVAR
DEFINITION
VAR DEFINITI
ONVAR
DEFINITION
VAR DEFINITI
ON
VAR DEFINITI
ON
GOALDEFINITI
ON
GOALDEFINITI
ONGOAL
DEFINITION
GOALDEFINITI
ON
![Page 22: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/22.jpg)
eAdventure game platform
Open code authoring environment for the production of point-and-click adventure games & immersive learning simulations
Easy to include Learning Analytics in eAdventure games
![Page 23: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/23.jpg)
http://www.chermug.eu
With MGH-Harvard Universityhttp://first-aid-game.e-ucm.es
With ONT, educ@ONT
![Page 24: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/24.jpg)
eAdventure + Learning Analytics
Game Engine
CommunicationAPI
Logic
Input
Input
Logic
Input
{ type: 'input', timeStamp: some_timestamp, device: 'some_device', action: 'some_action', target: 'target_id', data: { key1: value, ...}}
{ type: 'logic', timeStamp: some_timestamp, event: 'some_event', target: 'some_id', data: { key1: value, ...}}
Learning Analytics Database
![Page 25: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/25.jpg)
25
GLEANER
‣ GLEANER: Game Learning Analytics for education research
• Open code framework to capture game traces
Reference model in the EU NoE GALA
![Page 26: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/26.jpg)
GLEANER Collector
‣ Implemented: Nodejs + Mongodb‣ A web server ready to receive traces‣ Cleans and sorts the data
![Page 27: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/27.jpg)
GLEANER Analysis
‣ Reporter has access to the database, and presents its data through reports
• graphics, heat-maps, relational tables…
‣ Evaluator has access to the database and checks the educational defined goals in the assessment model
![Page 28: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/28.jpg)
Example: A game to learn XML
http://gleaner.e-ucm.es/lostinspace/play/index.html
![Page 29: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/29.jpg)
About the game...
‣ A basic platform game to acquire familiarity with XML documents
• Learn the syntax• Understand nesting and attributes• Gain agility writing and reading XML documents
‣ Designed as a complementary activity in a Web Programming Course
• For undergraduate computer science students
‣ Not developed with eAdventure
![Page 30: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/30.jpg)
Understanding the game…
Goal
Main characte
r
Power-ups (new syntax elements)
Write XML snippets here to move the
main character to the goal
![Page 31: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/31.jpg)
Educational goals
‣ Basic understanding of XML Syntax• Markup syntax• Basic documents• Attributes• Complex documents
‣ Basic programing skills• Sequencing• Loops
![Page 32: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/32.jpg)
LA perspective
‣ What does GLEANER trace?• Higher level events• Generic traces & game specific traces• Only those relevant for our learning objectives
‣ The aggregator will filter and transmit:• Level completion events• Each XML fragment submitted by the player
![Page 33: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/33.jpg)
Main dashboard (GLEANER Report)
33
![Page 34: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/34.jpg)
Progress report
34
User Session
Phases complete
d
Current score
Learning achievements
Achievements are updated and
highlighted in real time
![Page 35: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/35.jpg)
Detailed traces
35
Filters XML snipets
The fragments are updated in real time as submitted by the student. The instructor can filter to see only
mistakes, or all the fragments submitted by a specific students.
![Page 36: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/36.jpg)
La Dama Boba : The Game
Based on The Foolish Lady by Lope de VegaThe game is available at http://damaboba.e-ucm.es/ (in Spanish)
![Page 37: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/37.jpg)
Experiment game vs class‣ 757 students
‣ From 8 middle and high schools in Madrid region
‣ Control group and experimental group
![Page 38: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/38.jpg)
![Page 39: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/39.jpg)
39
Formal evaluation pre-post‣ Similar results with
Learning Analytics than with pre-post test?
![Page 40: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/40.jpg)
learning analytics outcomes
![Page 41: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/41.jpg)
41
![Page 42: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/42.jpg)
42
Other issues in LA
‣ Ethical and legal aspects
‣ Security model
‣ Ownership of information• Informing the user
‣ Anonymization of information
‣ Aspects specially relevant if you are working with kids!
![Page 43: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/43.jpg)
43
But there are new especifications and
develpments that could sistematize the work
![Page 44: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/44.jpg)
44
ADL eXperience API (xAPI)‣ Result of Project Tin Can
‣ Tracks experiences, informal learning, real-world experiences (not just completions)
‣ Allows data storage AND retrieval (ex. 3rd party reporting and analytics tools)
‣ Enables tracking mobile, games, ITS, and virtual worlds experiences
‣ Developed by open source communityFrom Damon Regan (ADL) presentation at SINTICE2013
![Page 45: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/45.jpg)
45
Activity Streams
‣ http://activitystrea.ms
‣ Collaboration between Google, Facebook, Microsoft and others
‣ Allows reporting of experiences, not just completions‣ Format: <Actor> <Verb> <Object> (I did this):
Simple Statement: I (actor) watched (verb) a video on protecting employee data (object)
Complex: in the context of [information assurance certification training] with result [timestamp:2013-0618T18:30:32.360Z ].
From Damon Regan (ADL) presentation at SINTICE2013
![Page 46: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/46.jpg)
Reporting
Systems
Assessment
Services
Semantic Analysis
Statistical Services
xAPI Learning Record Store (LRS)
46
From Damon Regan (ADL) presentation at SINTICE2013
![Page 47: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/47.jpg)
‣ Raw data can feed several systems• An LRS• A Learning Analytics System
eAdventure + LA with xAPI
Logic
Input
Logic
Input
Input
Input
Raw data
LRS
Learning Analytics System
StatementsAnalyzer
StatementsAnalyzer
EXPERIENCEAPI
EX
PE
RIE
NC
EA
PI
![Page 48: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/48.jpg)
IMS Global Learning Analytics Interoperability Framework
http://www.imsglobal.org/IMSLearningAnalyticsWP.pdf
![Page 49: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/49.jpg)
![Page 50: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/50.jpg)
50
Conclusions
‣ LA in Serious Games has a great potential from the application and research perspective
• Simplify more complex and complete experiments
‣ LA in Serious Games should benefits from Games Analytics experience and work
‣ Still complex to implement LA in SG• Increases the (already high) cost of the games
‣ Frameworks and new standards specifications could greatly simplify LA implementation and adoption
![Page 52: Learning analytics applied to serious games, invited talk at ECGBL 2013 Porto, Portugal](https://reader038.vdocuments.mx/reader038/viewer/2022103016/55495278b4c905e94e8b4b65/html5/thumbnails/52.jpg)
Our current projects
Increasing patient safety using games
Modelling/teaching medical procedures atNational Transplant Organization