questionnaires judy kay chai: computer human adapted interaction research group human centred...
Post on 29-Dec-2015
213 Views
Preview:
TRANSCRIPT
Questionnaires
Judy Kay
CHAI: Computer Human Adapted Interaction research group
Human Centred Technology Cluster for Teaching and Research
School of Information Technologies
2
Landscape of methods to understand user and work
Interviews, Surveys, Focus Groups
Ask people what they believe they want
Many limitations:• Human memory• Politeness• Self-awarenessMany benefits:• User involvement, commitment,
buy-in• Access to what users know, think,
perspectives, contexts that matter
Contextual Inquiry
Observe what people actually do
Many limitations:• Can you be there at the right
time?• Observer effect• IntrusivenessMany benefits:• Insights into actual behaviour• And contexts• And stakeholder perspectives
Questionnaires
• Can be easy to administer– Closed– Open Likert
• Difficult to design them well• Standard Questionnaires
– SUS– SEQ
SUS (on tasks completion)
SUS: A Quick and Dirty Usability Scale by John Brooke 3,000 Scholar citations
Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale
by Kortum Bangor and Miller May *
From usability.gov
• The scoring system is somewhat complex• There is a temptation, when you look at the scores,
since they are on a scale of 0-100, to interpret them as percentages, they are not
• The best way to interpret your results involves “normalizing” the scores to produce a percentile ranking
• SUS is not diagnostic - its use is in classifying the ease of use of the site, application or environment being tested
Please check the box that reflects your immediate response to each statement. Don’t think too long about each statement. Make sure you respond to every statement. If you don’t know how to respond, simply check box “3.” *
1. I think that I would like to use this system frequently.2. I found the system unnecessarily complex.3. I thought the system was easy to use.4. I think that I would need the support of a technical person to be able to
use this system.5. I found the various functions in this system were well integrated.6. I thought there was too much inconsistency in this system.7. I would imagine that most people would learn to use this system very
quickly.8. I found the system very cumbersome (*awkward) to use.9. I felt very confident using the system.10. I needed to learn a lot of things before I could get going with this system.
* System … product
Likert scales, semantic differential
• From SUS:• Please check the box that reflects your
immediate response to each statement. Don’t think too long about each statement. Make sure you respond to every statement. If you don’t know how to respond, simply check box “3.”
Semantic differential scales
• Granularity of the scale– Number of discrete points– Including anchors – The number of choices we allow users
Copyright MKP. All rights reserved. 8
Semantic differential scales
• Typical labeling of a point on a scale is verbal – Often with associated numeric value– Labels can also be pictorial
• Example, smiley faces• Helps make it language-independent
Copyright MKP. All rights reserved. 9
Example, semantic differential scale
• To assess participant agreement with this statement– “The checkout process on this Website was
easy to use.”
• Might have these anchors: Strongly agree and strongly disagree
Copyright MKP. All rights reserved. 10
Examples
• In between scale : – Agree, neutral, disagree
• Could have associated values – +2, +1, 0, -1, and -2
• What do these mean for the person?
Getting the SUS score (/100)
1. Convert each score into a number/41. For items 1,3,5,7,and 9 the score contribution is the
scale position minus 1 2. For items 2,4,6,8 and 10, the contribution is 5 minus
the scale position.
2. Multiply the sum of the scores by 2.5 .
Empirically – overall 68 is “average”
SUS: A Quick and Dirty Usability Scale by John Brooke
More subtlety in scoring• We have used this version of the SUS in almost all of the surveys we
have conducted, which to date is nearly 3,500 surveys within 273 studies. It has proven to be a robust tool, having been used many times to evaluate a wide range of interfaces that include Web sites, cell phones, IVR, GUI, hardware, and TV user interfaces. In all of these cases, participants performed a representative sample of tasks for the product (usually in formative usability tests) and then, before any discussion with the moderator, completed the survey.
Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale by Kortum Bangor and Miller May
More detail in SUS pragmaticshttp://www.measuringu.com/products/SUSpack
• Score SUS properly, accounting for errors, omissions, inconsistencies tested by internal reliability test (Cronbach's alpha)
• Percentile Ranks and Grades A+ .. F: Compare against 500 studies by application type
• Compares SUS Scores Statistically• Computes Sample Sizes, based on desired
margin of error
Design decisions
• The questions to ask– Challenges of biasing the user– Asking all questions in term of a positive view
• eg I found the LearnWell site easy to use
– Or all in terms of a negative view• eg I found the LearnWell site hard to use
• User choices:– Note we previously used a different approach Words to describe each
case (as in the learning scale for the concept inventory: never heard of it … could teach a friend)
– Just give end points• Even or odd?• What makes sense to ask users?
Why standard questionnaires• Reliability : how consistent responses are to the questions. How would you assess
that?
• Validity : correlates with other evidence eg completion rates.
• Sensitivity: discriminates between good interfaces and bad interfaces. The ability to detect differences at even small sample sizes (<10) was one of the major reasons why the System Usability Scale (SUS) and the Single Ease Question (SEQ) are recommended.
• Objectivity: Give independent instrument.
• Quantification: Fine grain of reporting and statistical analysis.
• Economy: Cheap to design!http://www.measuringu.com/blog/standardized-usability.php
SEQ
Single Ease Question Sauro, J., & Dumas, J. S. (2009, April). Comparison of
three one-question, post-task usability questionnaires. In Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (pp. 1599-1608). ACM.
Adapting questionnaires
• You can modify an existing questionnaire– Choosing a subset of questions – Changing the wording in some questions – Adding questions to address specific areas of
concern – Using different scale values
• Warning: Modifying a questionnaire can damage its validity
Copyright MKP. All rights reserved. 24
• “The model separates the four essential aspects:– The product quality intended by the designer. – The subjective perception of quality and
subjective evaluation of quality [by the user]. – The independent pragmatic and hedonic qualities. – Behavioural and emotional consequences. “
http://translate.google.com.au/translate?hl=en&sl=de&u=http://www.qu.tu-berlin.de/menue/forschung/laufende_projekte/joyofuse/joy_of_use/joy_of_use/measurement_methods/attrakdiff/&prev=/search%3Fq%3DAttrakDiff%26client%3Dfirefox-a%26hs%3DiG9%26rls%3Dorg.mozilla:en-US:official%26channel%3Dsb
top related