usability evaluation of the course management features of sakai jonathan howarth {[email protected]}...
TRANSCRIPT
Usability Evaluation of the Course Management Features of Sakai
Jonathan Howarth {[email protected]}Rex Hartson {[email protected]}
Aaron Zeckoski {[email protected]}
Virginia Tech
Sakai Conference, Atlanta, 2006
2 | | | | | | |
Overview Introduction Usability Setup Positive Feedback Performance and Satisfaction Usability Problems Next Steps
3 | | | | | | |
Introduction - Scholar is Virginia Tech’s implementation
of Sakai Using Scholar in a series of dissertation studies Usability evaluation in exchange for input from
Scholar developers Summary report
{http://issues.sakaiproject.org/confluence/download/attachments/3115
0/ScholarUsabEvalReportSummary.pdf?version=1} Full report
{http://issues.sakaiproject.org/confluence/download/attachments/3115
0/ScholarUsabEvalReportFull.pdf?version=1}
4 | | | | | | |
Usability Examples (1) Examples from the real world
5 | | | | | | |
Usability Examples (2) Examples from the digital world
6 | | | | | | |
Usability Examples (3) Some additional contrived examples
World Usability Day posters provided by the Usability Professional’s Association {http://www.worldusabilityday.org/tools/world-usability-day-posters}
7 | | | | | | |
Usability Examples (4) Usability in the news
8 | | | | | | |
Usability Definition (1) Usability is a characteristic of an interactive
system that indicates ease of use, usefulness
Usability includes Effectiveness Efficiency Satisfaction
9 | | | | | | |
Usability Definition (2) It is NOT "dummy proofing” “Doing usability” is NOT (just) usability testing Usability is NOT “user friendliness” Misconceptions:
Peanut butter theory Priest in a parachute theory
10 | | | | | | |
Usability Motivation Why should you care
For the user, the UI is the system Usability can make or break your product Reduces training and help-desk costs Increases productivity Stand out among competition
An example Shutterfly increased print orders by 200% by
redesigning their website {http://
www.jaredresearch.com/why_it_worked.html}
11 | | | | | | |
Setup (1) Comparison with other Sakai usability evaluations
{http://bugs.sakaiproject.org/confluence/display/UI/Usage+Reports} Scholar version 2.2.x Course management features of Scholar Goals
Satisfaction Learnability Performance
12 | | | | | | |
Setup (2) Walkthrough Usability testing
5 participants with different backgrounds Sessions lasted 2 hours 17 tasks 2 questionnaires Gratuity
13 | | | | | | |
Positive Feedback Variety of tools Help system Mechanisms for supporting collaboration
14 | | | | | | |
Performance and Satisfaction (1) Performance
Time on task Long amounts of time required for initial tasks
Example – 10 minutes average to add lecture and lab events to the schedule
Much variance among participants Improvement for repeated tasks
Assists As a group, the participants were only able to complete 2 initial
tasks Task of adding an assignment resulted in 14 assists
15 | | | | | | |
Performance and Satisfaction (2) Satisfaction
Scores on questionnaire Average around a 5 (scale of 1 to 10, 1 = poor, 10 = excellent) No change in scores between 1st and 2nd questionnaire
Summary Need to decrease time on task Results suggest that participants were able to learn
quickly Performance/perceived performance did not have a
substantial effect on satisfaction
16 | | | | | | |
Usability Problems (1)
Group P1 P2 P3 P4 P5 Total
Terminology 9 11 15 4 9 48
Consistency 6 9 10 5 7 37
Sections and groups 2 10 6 5 4 27
Conformance with standards 2 8 5 1 5 21
Assignments and gradebook assignments 5 1 6 3 4 19
Workspaces and worksites 2 4 4 5 1 16
Distribution of functionality 2 3 4 5 1 15
Miscellaneous 11 12 12 9 8 52
Usability problems – verbal feedback, hesitations, deviation from correct sequence of actions
17 | | | | | | |
Usability Problems (2) Terminology
Roster {video – explaining what a roster is}, Resources {video – what are resources?}
18 | | | | | | |
Usability Problems (3) Consistency
Lists, confirmation, feedback, . . .
19 | | | | | | |
Usability Problems (4) Sections and Groups
Sections are tools (appear in the menu bar) and are exclusive
Groups are not tools (appear in site info) and are not exclusive
Conformance with standards Partial saving of page state {video – trying to get to the
main Site Info page} Functioning of the browser’s back button
20 | | | | | | |
Usability Problems (5) Assignments and gradebook assignments {video
– working in both tools to correct a point value}
21 | | | | | | |
Usability Problems (6) Workspaces and Worksites {video – trying to
explain the difference}
22 | | | | | | |
Usability Problems (7) Distribution of functionality
Adding and removing students {video – separation of add and remove causes problems on repeated task}
Editing assignments {video – ability to access assignment from multiple locations}, etc
Example – RSmart.com
23 | | | | | | |
Usability Problems (8) Miscellaneous problems
Reoccurring schedule entries, system-centered permissions, clean Word text {video – difficulties pasting text}
24 | | | | | | |
Next Steps Usability evaluation must be iterative
It’s not possible to get it right the first time (in any complex design domain)
Need:
Ready, Fire, Aim!
25 | | | | | | |
Questions
?