soft performance - measuring
TRANSCRIPT
Soft PerformanceMeasuring
me
Dimiter Simov - Jimmy usability practitioner
product experience for SAP HANA Cloud Platform @ SAP
goal make you think about metrics when designing and testing
Performance
a task or operation seen in terms of how successfully it is performed
pay increases are now being linked more closely to performance
the capabilities of a machine, product, or vehiclethe hardware is put through tests which assess the performance of the processor
Source: Google define
recall ISTA 2013: performance has a soft sidetext and formattingFitts’ lawlayout and structureuser success and engagementpresentation of progressaesthetics 1 x 1.618 – the golden ratio
1. avoid messages, especially modal onesDON’T WRITE MESSAGES, DESIGN INTERACTIONS
2. if you have to give a message, make sure it is obvious who shows itwhat happenedwhywhat users can do about it
3. be practical
recall ISTA 2014: about messages
about measuring soft performance
look at how soft performance can be measured
success rates, satisfaction scores, orientation
higher scores predict better performance; not always
big data – a glimpse
trinity
effectiveness
efficiency
satisfaction
effectiveness | efficiency | satisfaction
effectiveness
effectiveness = result = success
shows whether users accomplish their tasks
the basic metric
1 or 0: users either succeed or not (could as well be a fraction)
measure success rate break down user journey into tasks and subtasks
assign 1 = success, 0 = failed
count and sum for each user and task
average across tasks, users, and user groups
report success rate aim for 100% but 70-80% is a great score
4 users out of 5 succeeded ≠ 80% success
confidence interval calculator[1]
[1] calculator: http://www.measuringu.com/wald.htm
sample
report
effectiveness | efficiency | satisfaction
efficiencyorientation
time on task
effort
errors
learnability
efficiency = productivity
ratio of what you get to what you invest
goal = maximum result with minimum effort and time
time
on
task
time on task = speed
shows how quickly usersperform tasks
measure by the clock relative: compare to another system or version
time on task
subjective time matters
example 10 websites: users performed their own personal tasks no two users performed the same tasks on any site measured perceived speed and actual speed per task
the faster website
about.comactual speed: 8 secondsperceived speed: slowest
amazon.comactual speed: 36 secondsperceived speed: fastest
perceived speed related to success
when a site is slow, if users succeed, they feel it fast
when a site is fast, if users fail, they feel it slow
estimate task times
rule of thumb
measure your timemultiply by 3 to 5
effort
effort
shows the consumption of resources needed to perform the tasks
involves counting: clicks,steps, calories, etc. effort
clicks and performance
fewer clicks usually mean better performance:nobody likes “clicky” UIs
clicks predict about 25% of task time [1]
[1] http://www.measuringu.com/blog/click-clock.php
KLM = Keystroke-Level Model
“predicts how long it will take an expert user to accomplish a routine task without errors” [1]
covers the straight case
[1] https://en.wikipedia.org/wiki/Keystroke-level_model
KLM example: dropdown versus radio buttons
m thinkp point the dropdown arrowb click the arrowr dropdown expands = 0m look at the listp point the item neededb click the itemr dropdown collapses = 0m look at the value to see it is selectedp move to the Submit buttonb click
m look at the listp point the item neededb click the itemp move to the Submit buttonb click
m p b r m p b r m p b = 7.2 sec m p b p b = 3.6 sec
m thinkp pointb click in the boxm look at the list againk type (between 8 and 24 characters)m look at the value to see it is correctp move to the Submit buttonb click
m p b m kkkkkkkk m p b = 8.24 – 12.72 sec
KLM calculator: http://courses.csail.mit.edu/6.831/2009/handouts/ac18-predictive-evaluation/klm.shtml
orientation
orientation = ability to navigate
shows how much users deviate from an optimal path
0 to 100 (could be percent)
orientation
measure coefficient of orientation define optimal path of steps for each task = 100% (O)
count the actual steps performed by users (A)
CoR = (1 – A/O)%
CoR = (1 - 3/6) = 50%optimal path actual path
1
2
3
1
3
2
1’
2’ 2’’
learnability
learnability
shows how fast users get to use the product
fast to slow (to never)
learnability
errors
errors = wrong UI
show why soft performance suffers
two types of errors to count slips = I know the correct way mistakes = I am guessing the correct way
errors
slip
I want to save the changesbut click Don’t Save instead of Save
my friend who has never before driven an automatic car is pressing the break pedal with his left foot
mistake
I wanted to change the background of a survey to be the same color as the orange color used on the ISTA website
R:G:B 220:110:15
the color I need is not in the palette
so I prepare an image with the desired color
and click the only option that looks different
…I dragand dropthe image file…
The photo you uploaded is too small! It must be at least 800 pixels wide and 200 pixels tall.
I edited the fileto 800 by 200 pixelsuploaded it and got this header
not the same as color I uploaded
effectiveness | efficiency | satisfaction
task 1
task 2
task 3
overall satisfaction
satisfaction = sense of accomplishment
shows how much users like the product from “not at all” to “completely”
caveat – users might be evaluating: look and feel self
task
satisfaction
task-level satisfaction
one question after each task
the single ease question (SEQ)
SEQ: http://www.measuringu.com/blog/single-question.php
topic-level satisfaction
overall
satisfaction
overall satisfaction defines the overall experience variety of methods and metrics
SUS = System Usability Scale NPS = Net Promoter Score SUPR-Q = Standradized Universal Percentile Rank Questionaire
…
SUS = System Usability Scale1. I think that I would like to use the system frequently2. I find the system unnecessarily complex3. I think the system is easy to use4. I think that I would need the support of a technical person to be able to use
the system5. I find the various functions in the system are well integrated6. I think there is too much inconsistency in the system7. I would imagine that most people would learn to use the system very quickly8. I find the system very cumbersome to use9. I feel very confident using the system10. I needed to learn a lot of things before I could get going with the systemSUS: https://en.wikipedia.org/wiki/System_usability_scale
scoring SUS
1. do this in Excel2. scale all values from 0 to 4 (four is the most positive)
odd items: user response - 1even items: 5 - user response
3. convert to range 0 to 100 sum the scaled responses for each user and multiply the total by 2.5
SUS shows three things [1]
satisfaction – the full set of items
learnability – items 4 and 10
usability – the other 8 items
[1] http://www.measuringu.com/sus.php
SUS explains likelihood to recommend [1]
NPS score ≈ 0.1* SUS score
SUS score of 80 ≈ NPS of 8
[1] http://www.measuringu.com/blog/nps-sus.php
average SUS
68
0 100
> 82promoters
[1] http://www.measuringu.com/products/SUSpack
overallbusiness apps
websites 75consumer apps
65cellphones
58average
41lowest
79highest
my studies in the last 2-3 years
ISTA website SUS
0 100
66satisfaction
60.9usability 86.2
learnability
as of 11:00, 19 Nov 2015, 68 participants
about the ISTA website, we can be: 95.0% confidentthe population SUS Scoreis between 61.78 and 70.14
97.5% confidentthe population mean SUS Scoreis above 61.78
mean = 66
margin of error = 4.18
promoters: 5.88%
passives: 32.35%
detractors: 61.76%NPS = -55.88%
trinity?
analytics
big data
internet of things
quality out of quantity data is out there (or in there) data is neither good, nor bad what matters is how we use data
can we use data to promote wellbeing and happiness?
things you might want to check quantified self PERMA happiness index – GNH = gross national happiness Sonja Lyubomirsky’s 12 steps to happiness
soft performance metrics to take home
effectivenesssuccess rate
satisfactiontask-level: SEQoverall: SUS
efficiencytime on taskeffort: Keystroke-Level Modelorientation: CoRerror rate
that’s
folks
@dsimovabout.me/
dsimov
all
all illustrations in this presentation are by Iliya Beshkov from the book „Тошко Африкански“ (Toshko Afrikanski)