observational and log analysis methods for assessing engagement and affect in educational games
DESCRIPTION
TRANSCRIPT
![Page 1: Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games](https://reader036.vdocuments.mx/reader036/viewer/2022062615/547cf5f9b479597f508b47cc/html5/thumbnails/1.jpg)
Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational
Games
Ryan S.J.d. BakerAssistant Professor of Psychology, Learning Science, and Computer Science
Worcester Polytechnic Institute
![Page 2: Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games](https://reader036.vdocuments.mx/reader036/viewer/2022062615/547cf5f9b479597f508b47cc/html5/thumbnails/2.jpg)
2
Many ways to assess engagement and affect
• I’ll discuss two methods our lab uses
![Page 3: Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games](https://reader036.vdocuments.mx/reader036/viewer/2022062615/547cf5f9b479597f508b47cc/html5/thumbnails/3.jpg)
3
Quantitative Field Observations (Expert Judgments)
• Repeated 20 second observations of students’ engagement and affect as they use serious game or other learning environment in genuine learning setting– Conducted using peripheral vision/side glances– Good inter-rater reliability: k 0.6-0.8– Include engaged behaviors (collaboration with other students) and
disengaged behaviors (off-task behavior)– Include positive affect (delight, engaged concentration) and negative
affect (boredom, frustration)
• Ecologically valid assessments of how much and when – Students are disengaged– Students experience specific affect
![Page 4: Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games](https://reader036.vdocuments.mx/reader036/viewer/2022062615/547cf5f9b479597f508b47cc/html5/thumbnails/4.jpg)
4
Automated Detectors• Models that assess student engagement and affect in real-time or
retrospectively from behavior within software• In our approach, no sensors used
– Improves scalability – lots of data being automatically collected these days– Reduces predictive power for some affective states, relative to sensor-based
detectors• Successful at detecting disengaged behaviors such as off-task
behavior, carelessness, gaming the system• Successful at detecting engaged concentration and boredom in two
learning systems– Plus sensor-free affect detectors for AutoTutor by D’Mello et al. (2008)
• Used in interventions that improve learning outcomes(Baker et al., 2006)
![Page 5: Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games](https://reader036.vdocuments.mx/reader036/viewer/2022062615/547cf5f9b479597f508b47cc/html5/thumbnails/5.jpg)
5
Ongoing Project (NSF PSLC)
Quantitative Field
Observation
Affect Basic Research
Comparative Analysis
Completed for intelligent tutors; in process for serious games
Detector Development
![Page 6: Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games](https://reader036.vdocuments.mx/reader036/viewer/2022062615/547cf5f9b479597f508b47cc/html5/thumbnails/6.jpg)
6
Use in Research
![Page 7: Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games](https://reader036.vdocuments.mx/reader036/viewer/2022062615/547cf5f9b479597f508b47cc/html5/thumbnails/7.jpg)
7
How does student affectdiffer between games and ITS?
(Rodrigo & Baker, 2011)
![Page 8: Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games](https://reader036.vdocuments.mx/reader036/viewer/2022062615/547cf5f9b479597f508b47cc/html5/thumbnails/8.jpg)
8
Aplusix .vs. MathBlaster
Matched mathematical content between systems
Student affect assessed using quantitative field observationswith real students in real classrooms
![Page 9: Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games](https://reader036.vdocuments.mx/reader036/viewer/2022062615/547cf5f9b479597f508b47cc/html5/thumbnails/9.jpg)
9
Interesting differences in affect
Condition Engaged Concentration Delight
Aplusix 76% 6%
MathBlaster 63% 12%
Proportions of each affective state shown
![Page 10: Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games](https://reader036.vdocuments.mx/reader036/viewer/2022062615/547cf5f9b479597f508b47cc/html5/thumbnails/10.jpg)
10
How does social behavior influence affective dynamics in games?(Baker, Moore, et al., under review)
![Page 11: Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games](https://reader036.vdocuments.mx/reader036/viewer/2022062615/547cf5f9b479597f508b47cc/html5/thumbnails/11.jpg)
11
Students compete to be first to identify a substance chosen by their opponent
Chemistry Game (Yaron et al., 2010)
Student affect assessed using quantitative field observationswith real students in real classrooms
![Page 12: Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games](https://reader036.vdocuments.mx/reader036/viewer/2022062615/547cf5f9b479597f508b47cc/html5/thumbnails/12.jpg)
12
Without Social Behavior(D’Mello et al., 2007; Baker et al., 2010)
Bored
Confused
Gaming the
System
![Page 13: Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games](https://reader036.vdocuments.mx/reader036/viewer/2022062615/547cf5f9b479597f508b47cc/html5/thumbnails/13.jpg)
13
With Social Behavior(Baker, Moore, et al., under review)
Off-Task Behavior Bored
Confused
Gaming the
System
On-Task Conversation
![Page 14: Observational and Log Analysis Methods for Assessing Engagement and Affect in Educational Games](https://reader036.vdocuments.mx/reader036/viewer/2022062615/547cf5f9b479597f508b47cc/html5/thumbnails/14.jpg)
14
Bottom-Line
• Field observations and detectors are powerful tools
• For assessing and understanding student engagement and affect during learning
• Including in serious games