quality of web usability evaluation methods: an empirical study on
TRANSCRIPT
Quality of Web UsabilityEvaluation Methods:
an empirical study on MiLE+Davide BolchiniUniversity of Lugano (Switzerland)
Franca GarzottoPolitecnico di Milano (Italy)
2
Outline
MotivationGoalsMiLE+Empirical study:
Experiment 1: „quick inspection“Experiment 2: „usability project“
Results and Discussion
3
Motivation
A proliferation of Usability Evaluation Methods (UEMs)Different philosophies - conception of „quality“, „usability“ and their interrelationships -, thus approaches and techniques
Every method should support the improvement of theusability of the application
However, ongoing discussion (and little agreeement) on the salient „quality attributes“ for UEMs…
4
Motivation - 2Quality attributes concerning the output of the evaluation
Effectiveness?Number of usability problems discovered …Thoroughness (found problems vs existing problems) …Reliability (consistency of results across different inspectors)…Validity (correcting predicting user‘s behaviour, no or minimized false positives…)Productivity…Scope…
Quality attributes concerning acceptability and adoptionLearnabilityApplicability and Compatibility in current practiceVerticalization on domainsReusabilityCost-effectiveness
Supporting arguments with empirical data
5
Goals
Evaluate MiLE+Ongoing development from 2000Used in consultancy projects, in training professionals + ca. 200 students a year.
Focus on few key attributes that we could measure in a realistic setting and that support effective adoption:
PerformanceEfficiencyCost-effectivenessLearnability
6
Goals
Verifying key „acceptability“ requirementsfor a UEM:
Become able to use the method after a reasonable time (1-3 person/days) of „study“Being able to detect the largest amount of usability problems with the minimum effortProducing a first set of results in a fewhours and a complete analysis in a few days
7
MiLE+ : Milano-Lugano Evaluation
MiLE+ is a usability inspection methodEvolution of two previous usability methods:
SUE (Systematic Usability Evaluation)MiLEBorrow general concepts from mainstream usabilityinspection approachesFoster a systematic, structured approach to the analysis, yet aimed at being particularly suitable to noviceevaluators
8
MiLE+ Activities Framework
TECHNICAL
INSPECTION
USER TESTING
EXPERT
EXPERT
APPLICATION
TechnicalHeuristics
NAVIGATION
CONTENT
TECHNOLOGY
SEMIOTICS
GRAPHICS
COGNITIVES
APPLICATION INDEPENDENT
USABILITY
APPLICATION DEPENDENT USABILITY
Validate / Invalidate
User ExperienceIndicators
(UEIs)
SC
EN
AR
IOS
Scenarios(as a tool,
not mandatory)
USER EXPERIENCE INSPECTION
UEIs
90% Framework
10% Specific
Library of Heuristics
Library of Scenarios
Library of UEIs
9
MiLE+ technical heuristics
(82) Technical Heuristics, coupled by a set of operational guidelines that suggest the inspection tasks to undertake in order to measure the various heuristics.Organized by design dimensions
Navigation: (36) heuristics addressing website info architecture and navigation structure Content: (8) heuristics addressing the general principles of content qualityTechnology/Performance: (7) heuristics addressing technology-dependent features of the applicationInterface Design: (31) heuristics that address the semiotics of the interface, the graphical layout, and some “cognitive” aspects
10
Operationalized Attributes - 1
Performance:Performance indicates the degree at which a method supports the detection of all existing usability problems for an application.It is operationalized as the average rate of the number of different problems found by an inspector (Pi) in given inspection conditions (e.g. time at disposal) against the total number of existing problems (Ptot)
Performance = avrg (Pi)/Ptot
11
Operationalized Attributes - 2
Efficiency:Efficiency indicates the degree at which a method supports a “fast” detection of usability problems.It is operationalized as the rate of the number of different problems identified by an inspector in relation to the time spent, and then calculating the mean among a set of inspectors:
Pi is the number of problems detected by the i-th inspector in a time period ti.
)(i
i
tPavrgEfficiency =
12
Operationalized Attributes - 3
Cost-effectiveness:Cost-effectiveness denotes the effort - measured in terms of person-hours - needed by an evaluator to:
carry out a complete evaluation of a significantly complex web applicationproduce an evaluation documentation that meets professional standards, i.e., a report that can be proficiently used by a (re)design team to address the usability problems.
13
Operationalized Attributes - 4
Learnability:Learnability denotes the ease of learning a method.We operazionalize it by means of the following factors:
the effort, in terms of person-hours, needed by a novice to become “reasonably expert” and to be able to carry on an inspection activity with a reasonable level of performancethe novice’s perceived difficulty of learning, i.e., of moving from “knowing nothing” to “feeling reasonably comfortable” with the method and “ready to undertake an evaluation”the novice’s perceived difficulty of applying the method in a real case.
14
Empirical study: general conditions
The overall study involved 42 participantsStudents from HCI course, Politecnico di Milano (Milano and Como campus)„novice inspectors“Preconditions:
No previous exposure to usabilityBasic background in web developmentEtherogeneous profile in terms of age and technical background
Preparatory conditions:5 hours classroom training on usability and MiLE+Assignment of learning material to study (MILE+ overview, technical heuristics library, 2 real-life case studies, excerptsfrom „USABLE“ - an online course on usability - and MILE+)
15
Exp.1: „quick inspection“Inspectors
16 graduate students (Como)Purpose
measure efficiency and performanceLearnability hypothesis: study effort to become proficient <= 2 days
Assigned Inspection Goals:Inspect website (Cleveland Museum of Modern Art) with MiLE+ technical inspection
Setting:Synchronus and syntopic individual inspection3 hour timeLimited inspection scope (2 main sections, around 300 page instances)One week after MiLE+ classes
Output produced:inspection notes including, for each usability problem, name, designdimension, description (max 3 lines), page URL
16
Key Results: from zero to hero…
Experiment 1Avg number of problems discovered: 14.8Hourly efficiency: avg 4.9 problems per hourExisting usability problems (team of experts): 41Performance: 36%
After 6 hours of training and a maximum of 15 hours of study of MiLE+, a novice can become able to detect more than one third of the existing usability problems.
17
Exp.2: „usability project“
Inspectors26 graduate students (Milano)
Purposemeasure perceived difficulty in learning and using MiLE+ andeffort needed to produce a professional evaluation report
Assigned Inspection Goals:Inspect website at choice (full application)
Setting:Asynchronous, team inspectionTwo months period
Output produced:Complete usability evaluation report
18
Key Results: Learning Effort
8%
75%
10% 7%
0%
10%
20%
30%
40%
50%
60%
70%
80%
<10 hours 10-13 hours 14-17 hours >18 hours
% students
Time invested in learning MiLE+ before and during the project
19
Key Results: Learning Difficulty
1%
73%
25%
1%3%
74%
22%
1%0
61%
37%
2%0%
10%20%30%40%50%60%70%80%
Very simple Simple Diff icult Very diff icult
Overall dif f iculty Technical Inspection User Experience Inspection
% students
Perceived difficulty in learning the various MiLE+ tasks
20
Key Results: Difficulty in Using theMethod
% students
0
47% 44%
9%0
79%
21%
0%0
19%
57%
24%
0
45% 41%
14%
0%10%20%30%40%50%60%70%80%90%
Very simple Simple Difficult Very difficult
Overall difficulty Technical InspectionScenario definition User Experience Inspection
Perceived difficulty in using the various MiLE+ tasks
21
Key Results: Individual Effort
32%
54%
12%
2%
0%
10%
20%
30%
40%
50%
60%
20-30 30-40 40-50 50-60
% students
Person/hours
Estimated effort for the entire evaluation process
22
Key Results: Individual Effort per Task
0%
20%
40%
60%
80%
100%
3-5 5-10 10-1515-2020-25
3-5 94% 8%
5-10 3% 69% 1% 4% 51%
10-15 39% 31% 23% 41%
15-20 40% 0% 34%
20-25 18% 0% 42%
Technical Inspection
Scenario definition
User Experience Inspection
Infra-team agreement
Documentation
% students
Person/hours
Estimated effort for the entire evaluation process BY task
23
Conclusions
Promote usability evaluation methods for adoptionFostering learnability and cost effectiveness
We have empirically substantiated the adoptionsuitability of MILE+, with encouraging results
Performant, efficient, cost-effective, easy to learn and use
Next steps:Comparative evaluation
Nielsen‘s heuristics, more unstructured approachesOngoing effort for transfering of MiLE+ to the industry:
Courses for professionalsGoogle - CH