evaluating the student activity meter: two case studies
DESCRIPTION
presented at ICWL 2011, Hong Kong, 8-9 December 2011.TRANSCRIPT
![Page 1: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/1.jpg)
EVALUATING THE STUDENT ACTIVITY METER:
TWO CASE STUDIES.Sten Govaerts, Katrien Verbert & Erik Duval
http://www.role-project.eu
1
Thursday 8 December 11
![Page 2: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/2.jpg)
OVERVIEW
• Problem statement
•Objectives
•Design & Implementation
• Evaluations
• Conclusion & Future work
2
Thursday 8 December 11
![Page 3: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/3.jpg)
PROBLEM
3
Thursday 8 December 11
![Page 4: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/4.jpg)
PROBLEM
3
Thursday 8 December 11
![Page 5: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/5.jpg)
OBJECTIVES
• self-monitoring for learners
• awareness for teachers
• time tracking
• learning resource use and recommendations
4
Thursday 8 December 11
![Page 6: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/6.jpg)
STUDENT ACTIVITY METER (SAM): DEMO.
5
Thursday 8 December 11
![Page 7: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/7.jpg)
THREE EVALUATIONS
• with CS students• ICWL 2010 paper: “Visualizing Activities for
Self-reflection and Awareness”
• with CGIAR courses and teachers
• with Learning and Knowledge Analytics course participants
6
Thursday 8 December 11
![Page 8: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/8.jpg)
2010 PAPER
• usability and user satisfaction evaluation with 12 CS students
• in general, people understand the visualizations well!
• solved some issues and requests since
Thursday 8 December 11
![Page 9: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/9.jpg)
2010 PAPER
• usability and user satisfaction evaluation with 12 CS students
• in general, people understand the visualizations well!
• solved some issues and requests since
Thursday 8 December 11
![Page 10: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/10.jpg)
2010 PAPER
• usability and user satisfaction evaluation with 12 CS students
• in general, people understand the visualizations well!
• solved some issues and requests since
Thursday 8 December 11
![Page 11: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/11.jpg)
2010 PAPER
• usability and user satisfaction evaluation with 12 CS students
• in general, people understand the visualizations well!
• solved some issues and requests since
Thursday 8 December 11
![Page 12: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/12.jpg)
METHODOLOGY.
• design-based research: give students & teachers a tool and evaluate
• usability studies
• paper mockups
• surveys
8
Thursday 8 December 11
![Page 13: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/13.jpg)
CASE STUDY
• 20 persons: 3 CGIAR, 2 Law, 8 CS teachers & 7 CS TA’s.
• An online survey about usefulness, teacher issues and how the tool can resolve these.
• on average: 40 mins are spent using SAM.
9
Thursday 8 December 11
![Page 14: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/14.jpg)
CGIAR CASE STUDY
10
Provide feedback to the students
Being aware of what students are doing
Knowing about collaboration and communication
Knowing which documents are used and how much
Knowing how and when online tools have been used
Finding the students who are not doing well
Finding the best students
Knowing how much time students spent
Knowing if external learning resources are used
issue for teacher addressed
✔
✗
?✔
✔
✔?
✔
??
✔
✔✔??✔?
✔
Thursday 8 December 11
![Page 15: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/15.jpg)
CGIAR CASE STUDY
10
Provide feedback to the students
Being aware of what students are doing
Knowing about collaboration and communication
Knowing which documents are used and how much
Knowing how and when online tools have been used
Finding the students who are not doing well
Finding the best students
Knowing how much time students spent
Knowing if external learning resources are used
issue for teacher addressed
✔
✗
?✔
✔
✔?
✔
??
✔
✔✔??✔?
✔
Thursday 8 December 11
![Page 16: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/16.jpg)
CGIAR CASE STUDY
11
Bar chartParallel coordinatesLine chart
✔✔✔
Thursday 8 December 11
![Page 17: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/17.jpg)
CGIAR CASE STUDY
12
Thursday 8 December 11
![Page 18: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/18.jpg)
CGIAR CASE STUDY
good indicator for effortunderstand the workload
use for course design optimization
obtain course overview
increase awareness
detect outliers
progress evolutioncompare students
want better 1 to 1 comparison tool
more details on student
want to search for students
more metricswanted
12
Thursday 8 December 11
![Page 19: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/19.jpg)
CGIAR CASE STUDY
good indicator for effortunderstand the workload
use for course design optimization
obtain course overview
increase awareness
detect outliers
progress evolutioncompare students
want better 1 to 1 comparison tool
more details on student
want to search for students
more metricswanted
12
Thursday 8 December 11
![Page 20: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/20.jpg)
CASE STUDY
• open course on learning and knowledge analytics, http://bit.ly/dWYVbX
• 12 visual analytics enthousiasts + experts (who also teach)
• almost identical survey to CGIAR case.
13
Thursday 8 December 11
![Page 21: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/21.jpg)
CASE STUDY
• open course on learning and knowledge analytics, http://bit.ly/dWYVbX
• 12 visual analytics enthousiasts + experts (who also teach)
• almost identical survey to CGIAR case.
13
Thursday 8 December 11
![Page 22: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/22.jpg)
LAK CASE STUDY
14
Provide feedback to the students
Being aware of what students are doing
Knowing about collaboration and communication
Knowing which documents are used and how much
Knowing how and when online tools have been used
Finding the students who are not doing well
Finding the best students
Knowing how much time students spent
Knowing if external learning resources are used
issue for teacher addressed
✔
✗✔
✔
✔?
?
?
✔?
✔?
✔?
✗
✗?
?
Thursday 8 December 11
![Page 23: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/23.jpg)
LAK CASE STUDY
14
Provide feedback to the students
Being aware of what students are doing
Knowing about collaboration and communication
Knowing which documents are used and how much
Knowing how and when online tools have been used
Finding the students who are not doing well
Finding the best students
Knowing how much time students spent
Knowing if external learning resources are used
issue for teacher addressed
✔
✗✔
✔
✔?
?
?
✔?
✔?
✔?
✗
✗?
?
Thursday 8 December 11
![Page 24: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/24.jpg)
LAK CASE STUDY
15
Bar chartParallel coordinatesLine chart
???
Thursday 8 December 11
![Page 25: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/25.jpg)
LAK CASE STUDY
16
Thursday 8 December 11
![Page 26: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/26.jpg)
LAK CASE STUDY
chronological course dwell time
verify the status of the classroom activity
self-reflection to measure progress and increase
motivation find students experiencing problems and low
engagement
more metrics
more data16
Thursday 8 December 11
![Page 27: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/27.jpg)
LAK CASE STUDY
chronological course dwell time
verify the status of the classroom activity
self-reflection to measure progress and increase
motivation find students experiencing problems and low
engagement
more metrics
more data16
Thursday 8 December 11
![Page 28: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/28.jpg)
IDEA’S FROM EXPERTS
17
detailed information per student
the used resource types
detailed information of 2 students
detailed usage stats of resources
stats or vis. on content creation
Thursday 8 December 11
![Page 29: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/29.jpg)
IDEA’S FROM EXPERTS
1
2
3
4
5
17
detailed information per student
the used resource types
detailed information of 2 students
detailed usage stats of resources
stats or vis. on content creation
Thursday 8 December 11
![Page 30: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/30.jpg)
CONTINUED WORK...
18
Thursday 8 December 11
![Page 31: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/31.jpg)
CONTINUED WORK...
18
Thursday 8 December 11
![Page 32: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/32.jpg)
FUTURE WORK
• extend SAM with feedback of experts
• evaluate the usefulness in a larger group of students
• evaluate with teachers during and after a course.
19
Thursday 8 December 11
![Page 33: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/33.jpg)
YOUR DATA IN SAM ?!
20
Thursday 8 December 11
![Page 34: Evaluating the Student Activity Meter: Two Case Studies](https://reader035.vdocuments.mx/reader035/viewer/2022070313/5549b6f5b4c905fc7f8b4a0d/html5/thumbnails/34.jpg)
2
slides will appear on http://www.slideshare.net/stengovaerts
slides will appear on http://www.slideshare.net/stengovaerts 21
Thursday 8 December 11