the student activity meter for awareness and self-reflection
DESCRIPTION
presentation of the student activity meter at SigCHI 2012 in Austin, TX.TRANSCRIPT
THE STUDENT ACTIVITY METER FOR AWARENESS AND
SELF-REFLECTIONSten Govaerts, Katrien Verbert, Erik Duval, Abelardo Pardo.
1
SigCHI 2012, Long Case Study.
our team:Human-Computer Interaction
•technology enhanced learning•music•research 2.0
2
OBJECTIVES
• self-monitoring for learners
4
http://free-extras.com/images/obama_hope_progress-2848.htm
OBJECTIVES
• self-monitoring for learners
• awareness for teachers
4
http://free-extras.com/images/obama_hope_progress-2848.htm
OBJECTIVES
• self-monitoring for learners
• awareness for teachers
• time tracking
4
http://free-extras.com/images/obama_hope_progress-2848.htm
OBJECTIVES
• self-monitoring for learners
• awareness for teachers
• time tracking
• learning resource use and recommendations
4
http://free-extras.com/images/obama_hope_progress-2848.htm
OBJECTIVES
• self-monitoring for learners
• awareness for teachers
• time tracking
• learning resource use and recommendations
• part of Learning Analytics research [ACM LAK conf., Siemens 2011, Duval 2011]
4
http://free-extras.com/images/obama_hope_progress-2848.htm
STUDENT ACTIVITY METER (SAM): DEMO.
5
http://dilbert.com/strips/comic/2000-02-24/
http://research.uow.edu.au/learningnetworks/seeing/snapp/index.html SNAPP
GISMO http://www.noe-kaleidoscope.org/pub/researcher/resources/video/aied/mazza.html
http://www.cs.uu.nl/docs/vakken/uem/Janssen.pdfThe Participation Tool
METHODOLOGY.
• design-based research [Obrenović 2011]: design a tool, give students & teachers and evaluate
• iterative process of design & evaluation
• paper mockups
• usability studies
• surveys on usefulness
9http://funnytogo.com/pictures/evolution/timeline.htm
10
iter. evaluation goal methodology demo-graphics data
I.usability, satisfaction,
preliminary usefulness
task-based interview with think-aloud, SUS
& MSDT
12 CS students at KULeuven
timetracking via Twitter
II.assessing teacher
needs, use & usefulness
online survey with Likert items, yes/no and open questions
19 teachers & TA’s with
CGIAR coursesMoodle logs
III.
assessing teacher needs, expert
feedback, use & usefulness
online survey with Likert items, yes/no and open questions
12 participants from LAK MOOC
Moodle logs
IV. use, usefulness & satisfaction
structured face-to-face interview with tasks &
open questions
11 teachers & TA’s at Carlos III
Uni. Madrid
VM with extensive
tool tracking
11
11
demographics
evaluation goal
design changes negative positive
I. 12 CS students
usability, satisfaction, preliminary usefulness
1st iteration small usability issues
•↑learnability•↓errors•good satisfaction•usefulness positive
ITERATION TWO & THREE
• CGIAR: 19 teachers - LAK: 12
• An online survey about usefulness, teacher issues and how the tool can resolve these.
13
ITERATION TWO & THREE
• CGIAR: 19 teachers - LAK: 12
• An online survey about usefulness, teacher issues and how the tool can resolve these.
13
Provide feedback to the students
Being aware of what students are doing
Knowing about collaboration and communication
Knowing which documents are used and how much
Knowing how and when online tools have been used
Finding the students who are not doing well
Finding the best students
Knowing how much time students spent
Knowing if external learning resources are used
issue for teacher
✔
✗
?✔
✔
✔?
✔
??
✔
✔✔??✔?
✔
14
add-ressed
✔✔
✔
✔?
?
✔
✗
?
issue for teacher
add-ressed
✗
?
✔?
✔?
?
✗
?
CGIAR LAK1 2 3 4 5
Issues
Provide feedback to the studentsBeing aware of what students are doingKnowing about collaboration and communication *Knowing which documents are used and how muchKnowing how and when online tools have been usedFinding the students who are not doing well
Finding the best students
Knowing how much time students spentKnowing if external learning resources are used
Addressed by SAM
1 2 3 4 5
Provide feedback to the students
Being aware of what students are doing
Knowing about collaboration and communication
Knowing which documents are used and how much
Knowing how and when online tools have been used
Finding the students who are not doing well
Finding the best students
Knowing how much time students spent
Knowing if external learning resources are used
issue for teacher
✔
✗
?✔
✔
✔?
✔
??
✔
✔✔??✔?
✔
14
add-ressed
✔✔
✔
✔?
?
✔
✗
?
issue for teacher
add-ressed
✗
?
✔?
✔?
?
✗
?
CGIAR LAK
Provide feedback to the students
Being aware of what students are doing
Knowing about collaboration and communication
Knowing which documents are used and how much
Knowing how and when online tools have been used
Finding the students who are not doing well
Finding the best students
Knowing how much time students spent
Knowing if external learning resources are used
issue for teacher
✔
✗
?✔
✔
✔?
✔
??
✔
✔✔??✔?
✔
14
add-ressed
✔✔
✔
✔?
?
✔
✗
?
issue for teacher
add-ressed
✗
?
✔?
✔?
?
✗
?
CGIAR LAK
CGIAR CASE STUDY
15
CGIAR CASE STUDY
good indicator for effortunderstand the workload
use for course design optimization
obtain course overview
increase awareness
detect outliers
progress evolutioncompare students
want better 1 to 1 comparison tool
more details on student
want to search for students
more metricswanted
15
CGIAR CASE STUDY
good indicator for effortunderstand the workload
use for course design optimization
obtain course overview
increase awareness
detect outliers
progress evolutioncompare students
want better 1 to 1 comparison tool
more details on student
want to search for students
more metricswanted
15
LAK CASE STUDY
16http://maranash.blogspot.com/2011/01/conference-shout-out.html
LAK CASE STUDY
chronological course dwell time
verify the status of the classroom activity
self-reflection to measure progress and increase
motivation
find students experiencing problems and low
engagement
more metrics
more data
16http://maranash.blogspot.com/2011/01/conference-shout-out.html
LAK CASE STUDY
chronological course dwell time
verify the status of the classroom activity
self-reflection to measure progress and increase
motivation
find students experiencing problems and low
engagement
more metrics
more data
16http://maranash.blogspot.com/2011/01/conference-shout-out.html
demographics
evaluation goal
design changes
negative positive
I.12 CS
students
usability, satisfaction, preliminary usefulness
1st iterationsmall usability
issues
•↑learnability•↓errors•good satisfaction•usefulness positive
II.19
teachers & TA’s
assessing teacher
needs, use & usefulness
help functionresource
recomm. not useful
•provides awareness•all vis. useful•many uses•90% want it
III.12
participants
assessing teacher
needs, expert feedback, use & usefulness
re-orderable parallel
coordinates with
histograms
most addressed needs are indecisive
•provides awareness and feedback•many uses•66% want it•recomm. can be useful
ITERATION FOUR
18
• a CS course on C++ programming
• 11 people: 7 teachers, 2 TA’s & 1 course planner
• richer data set: tracking from programming environment
• qualitative study using a structured face-2-face interview with 25 open questions.
ITERATION FOUR
18
• a CS course on C++ programming
• 11 people: 7 teachers, 2 TA’s & 1 course planner
• richer data set: tracking from programming environment
• qualitative study using a structured face-2-face interview with 25 open questions.
ACHIEVABLE TASKS
19
identify students doing well and at risk
understand time spent better
better course overview
detect students for assistance
course planning
monitor tool and resource use
get a time summary
conflicting visions on students doing well
and at risk
http://www.tdpri.com/forum/telecaster-discussion-forum/48431-ic503z-cap-52-ri-blackguard-book.html
ACHIEVABLE TASKS
19
identify students doing well and at risk
understand time spent better
better course overview
detect students for assistance
course planning
monitor tool and resource use
get a time summary
conflicting visions on students doing well
and at risk
http://www.tdpri.com/forum/telecaster-discussion-forum/48431-ic503z-cap-52-ri-blackguard-book.html
USABILITY & SATISFACTION
20
• average SUS score: 69,69%
[Bangor et al. 2008]
USABILITY & SATISFACTION
20
• average SUS score: 69,69%
all: want to continue using it9/11: give it to students
[Bangor et al. 2008]
demo-graphics
evaluation goal
design changes negative positive
I.12 CS
students
usability, satisfaction, preliminary usefulness
1st iterationsmall usability
issues
•↑learnability•↓errors•good satisfaction•usefulness positive
II.19
teachers & TA’s
assessing teacher needs, use & usefulness
help functionresource
recomm. not useful
•provides awareness•all vis. useful•many uses•90% want it
III.12
participants
assessing teacher needs, expert
feedback, use & usefulness
re-orderable PC with
histograms
most addressed needs are indecisive
•provides awareness and feedback•many uses•66% want it•recomm. can be useful
IV.11
teachers & TA’s
use, usefulness & satisfaction
filter & search, icons,
zooming in line chart, editing PC
axes
conflicting visions of students
doing well or at risk
•provides time overview•provides course overview•PC assist with detecting problems•many uses & insights•100% want it
CONCLUSION
• SAM is perceived useful and contributes to better awareness.
22
CONCLUSION
• SAM is perceived useful and contributes to better awareness.
• SAM enables:
22
CONCLUSION
• SAM is perceived useful and contributes to better awareness.
• SAM enables:
• to detect potential students doing well and at risk
22
CONCLUSION
• SAM is perceived useful and contributes to better awareness.
• SAM enables:
• to detect potential students doing well and at risk• a better course overview
22
CONCLUSION
• SAM is perceived useful and contributes to better awareness.
• SAM enables:
• to detect potential students doing well and at risk• a better course overview• understanding student time spending
22
CONCLUSION
• SAM is perceived useful and contributes to better awareness.
• SAM enables:
• to detect potential students doing well and at risk• a better course overview• understanding student time spending• to find a wide variety of new insights
22
CONCLUSION
• SAM is perceived useful and contributes to better awareness.
• SAM enables:
• to detect potential students doing well and at risk• a better course overview• understanding student time spending• to find a wide variety of new insights
• almost all want to continue using SAM22
LESSONS LEARNED
• worked well:
23http://slurmed.com/fanart/dexus5/005_professor.png
LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early
23http://slurmed.com/fanart/dexus5/005_professor.png
LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early• Use real-world settings
23http://slurmed.com/fanart/dexus5/005_professor.png
LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early• Use real-world settings• Visualisations are powerful tool for creating
awareness
23http://slurmed.com/fanart/dexus5/005_professor.png
LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early• Use real-world settings• Visualisations are powerful tool for creating
awareness
• worked less well:
23http://slurmed.com/fanart/dexus5/005_professor.png
LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early• Use real-world settings• Visualisations are powerful tool for creating
awareness
• worked less well:
• finding real-world settings
23http://slurmed.com/fanart/dexus5/005_professor.png
LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early• Use real-world settings• Visualisations are powerful tool for creating
awareness
• worked less well:
• finding real-world settings• getting tracking data
23http://slurmed.com/fanart/dexus5/005_professor.png
LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early• Use real-world settings• Visualisations are powerful tool for creating
awareness
• worked less well:
• finding real-world settings• getting tracking data
• longitudinal study would be useful23
http://slurmed.com/fanart/dexus5/005_professor.png
24
http://www.fitbit.com/
http://media.cnbc.com/i/CNBC/Sections/News_And_Analysis/_News/_SPECIAL_REPORTS/
BetterYourBusiness_2012/BYB-Digital%20-%20April/Slideshows/Digital_Appliances_ab/10-
Smart-Dig-Appliances-Wifi-scale-.jpg
http://quantifiedself.com/
http://nikerunning.nike.com/nikeos/p/nikeplus/en_EMEA/sportband
LAK13: Learning Analytics and Knowledge Conf.8-12 April 2013 Leuven, Belgium
http://lakconference.org/
2
slides will appear on http://www.slideshare.net/stengovaerts
slides will appear on http://www.slideshare.net/stengovaerts 26
contact: [email protected]@erikduval
2
slides will appear on http://www.slideshare.net/stengovaerts
slides will appear on http://www.slideshare.net/stengovaerts 26
If you are interested in visualisations and tabletops, we have a PhD position available.
contact: [email protected]@erikduval