usability testing
DESCRIPTION
TRANSCRIPT
Usability Testing
Chapter 6
Reliability
• Can you repeat the test?
Reliability
• Huge difference between test subjects– Therefore, only 2 subjects is not a good
reliability check
• Significance levels (e.g., p = .20)• Confidence levels
– When should an usability problem be fixed?
• Error rates have the highest variability– Therefore, more test subjects are needed
Validity
• Does this test reflect what usability issues you want to test?
• Reliability measured with stats but validity requires methodological understanding
• Typical problems– Wrong users or wrong tasks– Confounding effects – text based to GUI
system
Test Plans
• What’s the goal of the test?
• What is it you are trying to prove or show?
• Pilot tests – show the problem areas of the test. E.g., the instructions
Test Plans
• The goal of the test?
• Where and were will the test take place?
• How long is each test session expected to take?
• What computer support will be needed for the test?
• What software needs to be ready for the test?
• Who will serve as experimenters for the test?
• How many test users are needed?
• …
Getting Test Users
• Test subjects (users) should be representative of who will use the system.
• Sales (“demoability”) – is it easy to show
• Asking managers to choose subjects– They will pick the best or the worse
• Formative evaluation– Help improve the interface as part of an
iterative design process
• Summative– Assessing the overall quality of an interface
Novice and expert users
• Most tests should test novice users
• If possible, test for expert users. But test separately from novice users.
• What are the effects of training or not training before a test?
Between-Subject test
• Simplest and most valid
• Different tests subject using different systems
• Possible problems:– Individual differences– Assignment to test groups (volunteer early or
late)
Within-Subject test
• All users are tested on all systems
• Problem: no longer a set of novice users when testing a secondary system
Choosing experimenters
• It’s better to get an usability person to conduct the test but you, the developer, can also run a test
• In fact, it’s good to know the system well• From Usability Engineering (Nielsen)
– “It is possible for computer scientistws to learn user test methods and apply them with good results.”
• However, designers try to explain away problems.
Ethical aspect of testing human beings
• Users should feel as comfortable as possible• Tell the test subjects:
– No information will be revealed– Explain the testing environment. E.g., computer, etc.
• The tester should not allow observers because they tend to influence results.
• The tester should not interfere with the user. Let the user find the solution themselves
Emphasize that it is the system that is being tested, not the user
Test tasks
• Based on a task analysis. What is it the user needs to do.
• Tasks should be small enough to be completed in time that the experiment takes.
• Test tasks should be in writing. The user may refer to the steps.
Stages of a Test
• 1. Prep – is everything in working order?
• 2. Intro - tell the user the purpose of the test.
• 3. Test
• 4. Debrief
Stages of a Test
• Intro - tell the user the purpose of the test.• A reminder that the test is confidential and
should not be discussed with others• A statement that participation is the test is
voluntary and the user may stop at any time• At FIT a human test subject document
needs to be signed by the tester and returned to the university.
Performance Measurement
• Has the usability goals been met? (see page 194)
• Test a set of tasks• The data collected: time to perform and
error rate• Goals are abstract, so break them down.• Make sure you know the start and finish of
a measured test
Thinking aloud
• “verbalizing their thoughts”• Shows how users interpret each interface
item
• Users performed 9 % better using thinking aloud.
• Constructive interaction (co-discovery learning) and coaching methods.
Usability Labs
• Is there really a need?
• Why videotape? Impact analysis – you look after taping for a known problem.
• Convince the manager and developers
• Usability kiosks