Download - Workshop 1 (analysis and Presenting)
Analyzing and Presenting
Performance metrics
Analyzing and Presenting
Time-on-Task Data
Common way
• TIME-ON-TASK is usually to measure – Efficiency
• Look at the average amount of time spent on– A particular set of tasks.
• Solution – Create ranges or discrete time intervals
• To find patterns among participants – report the
» frequency of participants per time interval» Find out who took long to finish and if they share
common characteristics
Other solution
• Thresholds– Set a time for completion
• What matter is whether users can complete certain tasks in that time
– calculate the percentage of users above or below the threshold • e.g. show the percentage of participants who completed
each task in less than one minute.
• Aims is to minimize the number of users who – need an excessive amount of time to complete a task.
Issue to know • Look at all tasks or just the successful tasks?
– Successful tasks • provide cleaner measure of efficiency; • but no information on Unsuccessful performance
– Without that you can (Unsuccessful performance)• Measure can be less accurate• It can be harder to reflection of the overall user experience
– Solution • Use only the times for the successful tasks• And errors for unsuccessful task
• Shall we use a think-aloud protocol?– This helps you contextualize your data and not to misinterpret the data
• But should not influence the time-to-task measurement– Solution
• you can ask participants to hold most of comment to the between tasks period
• Shall we tell participants that time is being measured?– Not directly
Analyzing and Presenting
Number of mistakes
Errors
• Or mistakes– Help to understand possible usability issues– how a specific action or set of actions • result in task failure
• Errors vs usability issues– A usability issue is the underlying cause of a
problem; and – a error is the outcome
Errors can tell
• How usable something really is.– Number of mistakes made,
• How they were made,
– Type and frequency of errors and it correlation with • Product design
– Loss in efficiency• e.g. filling in a form and a error results in lost of time to
complete the task• e.g. influence cost effectiveness or influence task failure
Common way
• Organize by task– Define the correct set of actions to do the task
• Define the correct and incorrect possible number of actions
• Then…– Collect number or errors
• By user and by task
• How to collect– Observe the participant
• During a lab study• During a video record
Error analysis
• Look at the frequency of the error for each task– calculate the average number of errors made by each
participant for each task.– frequency of errors for each task.
• This helps you to– find out which task is associated with the most errors – Tell in which (task) participants made more error
• From that results you get an idea what were the most significant usability issues.
Other approach
• If your concern is not with how participants perform a specific task – but about how participants performed overall
• Then…– Find out the overall error rate for the study.• By averaging the error rates for each task into a single
error rate
Issues to know
• Don’t double-count the errors– Double-counting happens when you assign more
than one error to the same event• Test the error counting before start the procedure and • Define a clearly what errors you need to count
• If you need to know not just error rate but– Also why this errors occurs then…• You need to add a code name to each possible error
Analyzing and Presenting
Number of steps taken
Number of steps taken
• TASK-SETPS also measure – Efficiency
• Efficiency metrics should be concerned not only – With the time-to-task, but also– With the amount of cognitive and physical effort
involved
How to measure - Common Way
• Identify the action(s) to be measured– More actions taken VS more effort is involved• Type of efforts
– Cognitive» involves finding the right place to perform an action
– Physical» Involves the physical activity required to take action
How to identify- Common Way
• Identify the action(s) to be measured– Define the start and end of an action• Count the actions
– Easier way is to count while participants are doing the task– If not possible use video recordings
• Actions should be meaningful – represent an increase in cognitive and/or physical effort
» More action more effort
• Look only at successful tasks
Analysis
• Calculate an average for each task (by participant) – to see how many actions are taken– This will help you to identify which task • Requires the most amount of effort
• Another way is to use a method called Lostness
Combine of metrics• Task success and time-on-task to measure
– Efficiency• How
– Ratio of the task completion to the task time in minutes
To read
• Measuring the User Experience.pdf– Dropbox folder• Workshops/W1
• Have a look at – CHAPTER 4• Pages 63-92