UX Research test scenarios
KP Ludwig John
UX Researchtest scenarios
UX Research test scenarios
KP Ludwig John
Before we startDelivery
Hypothesis + Description of MVP
UX Research test scenarios
KP Ludwig John
Hypothesis + Description of MVPCriteria
Description of individual Functions • specific and detailed• testable (Put it In or Leave it Out)
Clearly related to the needs and goalsof one or more of your Personas
Traceable (measurable) measurements
based on a clear research question?Which one?
Simplicity of the MVP focussed on research question / hypothesis
MVP
Hypothesis
UX Research test scenarios
KP Ludwig John
Feedback
• Hypothesis functions: specific, detailed, testable (Put it In or Leave it Out)
• Measurements traceable?
• Research question clear, focussed and understandable?
• Simplicity of the MVP Could it be done even simpler?
MVP
Hypothesis
Provide your fellow students withRecommendations and Suggestions
UX Research test scenarios
KP Ludwig John
Delivery
Hypothesis + Description of MVP
PDF per E-Mail
Deadline: Tonight midnight
UX Research test scenarios
KP Ludwig John
Usability TestPreconditions
Already existing
• PersonasDefinition of target groups
• Context of usewhere, when, how long, equipment
• Test goalsWhat do we want to learn?!
UX Research test scenarios
KP Ludwig John
• What do you want to learn by the test?Which elements of the IA design have been discussed controversial?Which elements seam to be critical for the general success of the application?
• Which elements of your application have to be coveredMain functions / use casesCrucial functions of the applicationUse Cases with many Alternate Flows
Usability-Test PredefinitionsTest goal
UX Research test scenarios
KP Ludwig John
Usability-Test
Comparison of 3 print Layouts Brochure with information about companies
IMS.UX - 2017
Test goal
Source: Team UXperts, IMS.UX 2017
Which Layout is best?
UX Research test scenarios
KP Ludwig John
Usability-Test
Source: Team UXperts, IMS.UX 2017
Aspects to consider
Extracting Information How easy is it to get the information? What information can you get?
Memorizing InformationCan you memorize the information after reading? Is it easy to remember
ClarityIs the structure obvious to the user? (Naming, Positioning, Design of elements)
ComprehensibilityDoes the user understand the content?
Readability - Is the text easy to follow?
Look & Feel - How is the first impression? Do you have a good feeling reading the paper?
Which Layout is best?
UX Research test scenarios
KP Ludwig John
Usability-Test
Source: Team „Group Title“, IMS.UX 2017
Aspects to consider
How well is information communicated?
Which Layout is best?
How fast can information be perceived and processed?
How much attention does relevant information get?
Measuring units!
Related questions
UX Research test scenarios
KP Ludwig John
Usability-Test
Source: Team „Group Title“, IMS.UX 2017
Aspects to consider
How well is information communicated?
Which Layout is best?
How fast can information be perceived and processed?
How much attention does relevant information get?
Measuring unit: Time to complete tasks.
Measuring unit: Time a test subject’s focus rests on relevant information.
Measuring unit: Amount of misunderstandings and missed information
Measuring units
UX Research test scenarios
KP Ludwig John
paper based software based
Usability-Test
• Which functionality has your prototype to providein order serve your test goals?
PredefinitionsTest object
• What kind of Prototype is needed in order serve your test goals
UX Research test scenarios
KP Ludwig John
Usability-Test
• Diary study (remote) • Thinking aloud (with note taking) • Video observation • Screen recording • Eye tracking • Post test interview
PredefinitionsTest methodWhich test method do you want to apply?
UX Research test scenarios
KP Ludwig John
Usability-Test
• What kind of equipment serves best the test goals? • Consider testing under of real context of use conditions
(Screen size, Space, time, mobile, distraction, sound)
PredefinitionsTest equipment
UX Research test scenarios
KP Ludwig John
Usability-Test PredefinitionsTest scenario
Create a test scenario which defineseach single step of the test in detail,
including
•Introduction •Tasks •and Expression of Thanks
UX Research test scenarios
KP Ludwig John
Test location • Where do you want to test? • Availability, Influence of Weather or Light?
Usability-Test PredefinitionsTest space
Test equipment • Where do you get it? • How to handle it?
UX Research test scenarios
KP Ludwig John
• Test conductor • Technical manager • Observer • Analyst
Usability-Test PreparationTest team
UX Research test scenarios
KP Ludwig John
• Where can the test persons be recruited from? (do they represent the target group, are they available (time) ?) • Do they get some expense allowance (Aufwandsentschädigung)? (which?) • recruit test persons
Usability-Test PreparationTest persons
Find test persons make appointments and a test schedule
• How many Test persons are needed? (about 7-8 per Test version (!) )
UX Research research methods
overview
KP Ludwig John
UsabilityTest persons
UX Research test scenarios
KP Ludwig John
It’s best to test with representatives of your main target group.But don’t be obsessed about it!If its difficult to recruit them, test with people you can get hold of. Take it in account when evaluating the test results:Would our users have that problem? Or was it only a problem because the participant wasn’t familiar with the field?
Rather few but frequentlystart early in the development processand repeat and repeat and repeat
Who do the test with?
Usability Test Test persons
How many test persons?
UX Research test scenarios
KP Ludwig John
reference:Nielsen (2000) https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
Test personsUsability TestHow many test persons?
The best results come from testing no more than 5 users and running as many small tests as you can afford.
UX Research test scenarios
KP Ludwig Johnreference:Nielsen 2018: https://www.youtube.com/watch?time_continue=31&v=RhgUirqki50
Test personsUsability Test
Jacob Nielsen N/N Group
UX Research test scenarios
KP Ludwig John
Reference: Steve Krug, Don‘t make me think! Redline, 2006
Test personsUsability Test
Test with 8 usersfound problems: 5 major ones
How many test persons?
UX Research test scenarios
KP Ludwig John
Test personsUsability TestHow many test persons?
2 Tests with 3 users each = 6 Test personsfound problems: 5 major + 4 minor = 9 problems
Reference: Steve Krug, Don‘t make me think! Redline, 2006
UX Research test scenarios
KP Ludwig John
Wen testen?
Datenquelle: Faulkner (2003): Beyond the five-user assumption: Benefits of increased sample sizes in usability testing; in Sarodnik/Braun: Methoden der Usability Evaluation (Huber Verlag 2011)
Test personsUsability Test
UX Research test scenarios
KP Ludwig John
Test personsUsability Test
source: www.nngroup.com/reports/how-to-conduct-eyetracking-studies PDF, page 19
UX Research test scenarios
KP Ludwig John
Most significant problems will be discovered by small number of test users already.
Finding three participants is easy to handle:test and debrief (Nachbesprechen) the same day! Design small tests with focussed tasks.Learn, move on and test again.
Test personsUsability Test
Reference: Steve Krug: „Rocket Surgery Made Easy“, New Riders 2010, chapter 5
How many test persons?
Can you use the same participants again?Not for the same task or application.They know too much already. ?
UX Research test scenarios
KP Ludwig John
Test scenarios in detail
UX Research test scenarios
KP Ludwig John
1. Test person works on given tasksoften under direct observation(screen recording software, video, human observer)
2. Interview with the userto better understand critical situations during the test.Possibility to discuss improvements together
Usability-TestTwo Steps
UX Research test scenarios
KP Ludwig John
Content and style
Test scenario
Identify at least 3 essential activitiesto be covered by the test
Create 1 or more tasks per activity to be followed by the test person
Example:
„Become a new member of this online communityand create your individual account!“
„Which of the dishes offered at Mensa next Monday you prefer? Note it on paper.“
UX Research test scenarios
KP Ludwig John
Content and style
• complex enough (not to simple) • not too complex either (shouldn’t take too long to work on each single task) • cover all important issues of your test
Test scenario
Tasks have to be
UX Research test scenarios
KP Ludwig John
First steps
Determine:
What do you want to find out?Which elements of the application are crucial?
Formulate the tasks straight forward, short and precise
Each task should have a defined goal(that defines, when the task is done)
Test scenario Content and style
!Start with simple, easy to handle tasks!More difficult (or more complex) tasks are possible,but not good at the very beginning of the test)
UX Research test scenarios
KP Ludwig John
Test scenario Content and style
UX Research test scenarios
KP Ludwig John
Test conductor although positioned next to the proband,remains silent / interferes only in critical situations.
Don’t get into conversations with the test person during the test itself. That can be done after the test.
Test scenario Content and style
Let the test person act rather independently and on its own
UX Research test scenarios
KP Ludwig John
Test scenario Content and style
Try out the tasks with team members or random usersbefore starting the official test
Set up test conditions close to the real context of use
UX Research test scenarios
KP Ludwig John
Test scenario Structure
Source: Team „Group Title“, IMS.UX 2017
• My name is Martin and I am guiding you through this test • First, we would like to thank you for participating in our test, we really appreciate your help and the time you took for this.
• Very important before we get started: We are not going to test you in person, we try to test different layouts for a magazine, so don’t worry, you can’t do or say anything wrong.
1. Welcoming and short introduction by test leader
UX Research test scenarios
KP Ludwig John
Test scenario
Source: Team „Group Title“, IMS.UX 2017
• If there are any questions or misunderstandings, just ask.
• Vivi, Kate, Daniel and Chris are here to observe the whole testing sequence. Ruben is our technical manager.
• You will have to do some tasks here in the UX-Lab, which I am going to explain later in detail.
• Bastian is a member of Academica. He will guide you to another room where you are going to get your last task,which is answering some questions in a questionnaire.
2. Introduction of the team members and their functions
Structure
UX Research test scenarios
KP Ludwig John
Test scenario
Source: Team „Group Title“, IMS.UX 2017
For the following test here in the UX-Lab, my team members will be observing the test.
We would like to do audio recording during the test. We are not going to provide this data to 3rd parties, but if you don’t want your voice being recorded, feel free to tell us.
3. Declaration of consent
Structure
UX Research test scenarios
KP Ludwig John
Test scenario
Source: Team „Group Title“, IMS.UX 2017
There will be 3 different PDFs which contain several information.
• You have to do some tasks, they will be displayed on the screen. Pressing space on the keyboard will start the first question,
• We would like to encourage you to tell us what you are thinking while doing the tasks. It helps us analyzing your final answers, but don’t feel forced to comment if you don’t feel like it.
• If there are no more questions from your side about the test procedure itself, you can start by pressing the SPACE key
• After a short calibration of the eye-tracking software the actual test is going to start immediately.
4. Definition of tasks
Structure
UX Research test scenarios
KP Ludwig John
oder:
Testszenarien entwickeln
Aufbau
Beispiel: IA6 2013 A Serious Game (Kinect)
Begrüßungstext
UX Research test scenarios
KP Ludwig John
Tasks
Source: IA6 2018, Team Gravecare
Test scenario Content and style
Each task should have a clear goal(that defines, when the task is done)
More specific = better
UX Research test scenarios
KP Ludwig John
Tasks
Source: IMS.UX 2017, Compare 3 Print-Layouts, Team UXperts
Test scenario Content and style
What’s the address of the company?Layout is shown, subject starts task.Subject gives answer
Does the company offer topics for Bachelor Thesis?Layout is shown, subject starts task.Subject gives answer
How many employees work at the company?Layout is shown, subject starts task.Subject gives answer
Each task should have a clear goal(that defines, when the task is done)
More specific = better
UX Research test scenarios
KP Ludwig John
Test scenario
source: https://www.nngroup.com/articles/priming/
When tasks use terms already present in the interface, participants are primed by the task formulation and tend to look for those exact terms.
When asking people to search for an “iPad keypad.” Although the word “keypad” is not as frequent as “keyboard”, most users typed “keypad” instead of “keyboard” in the search box.
… the test may have yielded the wrong conclusion that the product was easy to find.
Task wording
Content and style
UX Research test scenarios
KP Ludwig John
Tasks
Source: IMS.UX 2017, Compare 3 Print-Layouts, Team Xperience
Test scenario Content and style
Questions indicates meaning of UI element
Look for the rating points in the pdf. Find one criteria which has a bad rating and tell us.
Subject gets 2-3 minutes to look closer at the layout.We ask the user to talk about the parts he is looking at.
- Some critical examples
No clear goal, unnatural situation
Why critical?
UX Research test scenarios
KP Ludwig John
Expression of Thanks
Post test Interview to collect impressions and to discuss critical situation observed during the test
probably:
Test scenario Content and style
or probably
Thank you very much for participating in our test.
We would like to offer you this little gift to thank you.
After completing the tasks
UX Research test scenarios
KP Ludwig John
Test scenarioDuration
Rule of Thumb:Between 10 -30 minutes per person is OK
• less than 10 minutes seems rather short as well to the test person as for the team
• longer than 30 minutes requires a lot of effort from both sides
depends on test goal and circumstances
UX Research research methods
overview
KP Ludwig John
PracticePrepare your test
Formulate your test scenario in detail!
Plan and organize your test:
Appoint Functions and responsibilities in the test team Find and appoint test persons Schedule the test (incl. equipment and location)
UX Research research methods
overview
KP Ludwig John
PracticePrepare your test
• Test goals • Criteria of Success and Measurements for that criteria • Target group • Prototype • technical equipment • scenario • post test interview (if applicable)
Summarize your test design into one document, including
Be prepared to discuss it in detailnext week
hs-augsburg.de/homes/john