Juan E. Gilbert, Ph.D.IDEaS Professor & Chair Human-Centered Computing Division

School of Computing

Professor Automotive EngineeringClemson Universityjuan@clemson.edu

http://www.JuanGilbert.com/http://www.HumanCenteredComputing.org/

http://www.clemson.edu/computing

Holistic Usability Measure (HUM)

2 of 18

Usability! “If you could build a system

that resulted in world peace, but no one could use it ... it would be useless. Usability matters.” – Gilbert

However, “there has been no generally agreed-upon method of measuring usability, which makes it difficult to compare among different findings.” - Polkosky

Holistic Usability What is Holistic Usability?

Usability Effectiveness, Efficiency, User Satisfaction

What really counts? Effectiveness, Efficiency, or User Satisfaction?

Holistic Usability A single measure of the overall usability

System Usability Scale (SUS) 10 statements to which

users rate their level of agreement

Half positive, half negative

5-pt scale of agreement

Tullis and Albert’s Figure 6.8, based on Brooke 1996

Taken from http://jldrury.googlepages.com/8IssuesSelf-ReportedMetrics10-26-08.ppt

System Usability Scale (SUS) Each item's score contribution will range from 0 to 4. For items 1,3,5,7,and 9 the score contribution is the

scale position minus 1. For items 2,4,6,8 and 10, the contribution is 5 minus

the scale position. Multiply the sum of the scores by 2.5 to obtain the

overall SUS score.

System Usability Scale (SUS) For items 1,3,5,7,and 9 the

score contribution is the scale position minus 1.

For items 2,4,6,8 and 10, the contribution is 5 minus the scale position.

Multiply the sum of the scores by 2.5 to obtain the overall SUS score.

Scores are on 100 point scale

Taken from http://jldrury.googlepages.com/8IssuesSelf-ReportedMetrics10-26-08.ppt

System Usability Scale (SUS) SUS SCORE 0-60 GRADE = F

SUS SCORE 60-70 GRADE = DSUS SCORE 70-80 GRADE = CSUS SCORE 80-90 GRADE = BSUS SCORE 90-100 GRADE = A

System Usability Scale (SUS) Nice for comparing systems

Short questionnaire

Easy to calculate

Doesn’t work across all systems

Holistic Usability Measure

10 of 18

Usability Goals

What is the goal of your interface/system?

Who decides the primary goal of the system? Designer Client Manager

11 of 18

Usability Goals

Did you build the right system/interface? Who decides?

How do you know if your system is better than another that does the same tasks?

You must test it!

12 of 18

Background During requirement analysis, before design

Information design; what is to be built What is the system/interface suppose to accomplish? Who are the users? Etc.

These are typical, but most people miss Usability metrics These are typically an afterthought As a result, usability gets cut

13 of 18

Background

Quantitative Metrics Speech recognition accuracy Task completion time

Subjective Metrics User friendliness Ease of use

14 of 18

Background

Which metrics are most important? It depends on your system/interface Decide how you will test your system before

you design it

Given several different metrics How do you measure the overall effectiveness

of the system/interface?

15 of 18

Holistic Usability Measure

HUM Assign each metric a weight (1 – 99%) based

on its importance

Design and implement the system/interface

Evaluate it Convert all metrics to a similar numerical scale

16 of 18

Holistic Usability Measure

HUM = W1*Metric1 + W2*Metric2 + ... + Wn*Metric n

where W1 + W2 + ... + Wn = 1

and 0 ≤ W1, W2, ..., Wn ≤ 1

HUM yields a percentage measure rating the usability according to the designer’s intent.

17 of 18

Conclusions HUM is a tool for testing

It reports usability with respect to the usability goals

Multiple ways to design same system. Which one is the best? HUM is ideal for comparing more than one implementation of the

same VUI

HUM is easy to use and it makes sense

Recall, “there has been no generally agreed-upon method of measuring usability, which makes it difficult to compare among different findings.” – Polkosky (SpeechTECH Magazine November/December 2005)

Juan E. Gilbert, Ph.D.IDEaS Professor & Chair Human-Centered Computing Division

School of Computing

Professor Automotive EngineeringClemson Universityjuan@clemson.edu

http://www.JuanGilbert.com/http://www.HumanCenteredComputing.org/

http://www.clemson.edu/computing

Thank You