statistical user experience measurement 1
DESCRIPTION
TRANSCRIPT
User Experience Measurement
Prototype Comparison
Sharon Harper - Taxonomist & User Experience Consultant
Sharon Harper - Taxonomist & User Experience Consultant
User Experience Measurement
Taxonomy Structure, Organizational, Navigational Difference In Prototype Designs
Sharon Harper - Taxonomist & User Experience Consultant
Statistical Measurement Strategy Compare Means - task completion time
3 or more prototypes
2 prototypes
Across different participants
(independent samples)
Within same group of
participants(paired samples)
< 30 participants
t-test
≥ 30 participants
z-test
Sharon Harper - Taxonomist & User Experience Consultant
1. Null Hypothesis . there is no difference in the usability (performance & satisfaction) of the new user interface and the previous interface. Neither the taxonomy integration, organization or navigation efforts have improved the user experience.
1.(New user interface) – (Earlier user interface) = 0
2. Measure. What difference in the values of the means of two interfaces can be obtained that would disprove this hypothesis? The difference of the paired samples means by the same participants is relative to the spread or variability of their scores (time on successful task completion between prototype interfaces). This is the t value
T – test: Paired Two Samples for Means Comparison
Sharon Harper - Taxonomist & User Experience Consultant
2. Compare this result, the t-value, to determine whether the result is statistically significant. Assign risk factor of 5% (α= 0.05)to ensure 95% significance.
3. Number of paired measurements (participants) minus 1 is the degree of freedom, df.
4. With the acceptable risk factor of 5% or 95% significance, df, and t-value the hypothesis is tested.
5. The resulting p value is less than 0.05 then the null hypothesis is NOT TRUE The time on task for the two interfaces is significantly different, that is the new interface is better
T – test: Paired Two Samples for Means Comparison (cont.)
Sharon Harper - Taxonomist & User Experience Consultant
Excel™ Data Analysis Add In
Other software packages available : SPSS, SAS , MiniLab, http://usablestats.com …
Explanatory resources: The Cartoon Guide to Statistics by Gonick & SmithIntroductory Statistics by Ross
Web Center for Social Research Methodshttp://www.socialresearchmethods.net/kb/stat_t.php
Sharon Harper - Taxonomist & User Experience Consultant
Install Excel Data Analysis Add In
Sharon Harper - Taxonomist & User Experience Consultant
Time on Task t-Test paired sample for means
Sharon Harper - Taxonomist & User Experience Consultant
Result
Sharon Harper - Taxonomist & User Experience Consultant
Misinformation about statistical measurement is strongly ingrained, ‘math anxiety’
‘school’ marketing presenters, anticipate KIS(s) , use examples from others domains: medicine, linguistics, UX development
NOT pages of formulas, jargon
Do not show p-value = .001293051, use .00129 or .001
Usability analysts should have rudimentary statistical knowledge and able
to read power tables identifying the minimum sample size and acceptable
risk from that sample size.
Other methods should accompany quantitative methods
Report/Presentation Elements
Sharon Harper - Taxonomist & User Experience Consultant
UX Resources
Tullis & Albert. (2008) Measuring The User Experience: Collecting, Analyzing, and Presenting Usability Metrics.
Sauro. A Practical Guide to Measuring Usability: 72 Answers to the Most Common Questions about Quantifying the Usability of Websites and Software
Sauro & Lewis. (2012) Practical Statistics for User Research.
http://www.measuringusability.com
Questions?
Sharon Harper - Taxonomist & User Experience Consultant
Questions?