ux-research 4 results-evaluation qualitative-tests...
TRANSCRIPT
UX-Research Evaluation of test results
KP Ludwig John
Test team
Test leader: - guides through the test - exclusive contact person for test subject - introduces the other persons present in the room
Test Procedure
- provides and replaces the printed pages of the paper prototype - remains silent! - Very well prepared! - Avoid endless searching for next page
in case of paper based test: Assistant (Page provider):
UX-Research Evaluation of test results
KP Ludwig John
Test leader: - guides through the test - exclusive contact for test subject
Observer: - placed in a bit of distance, with good view to the test happenings - remain silent!! - take notes of special incidences
Technician: - takes care of equipment and operates it - starts/stops recording
Test team
UX-Research Evaluation of test results
KP Ludwig John
Test team
Technician: - takes care of equipment and operates it - starts/stops recording
Observer: - placed in a bit of distance, with good view to the test happenings - remain silent!! - take notes of special incidences
UX-Research Evaluation of test results
KP Ludwig John
Test team
Test leader: - guides through the test - exclusive contact for test subject
UX-Research Evaluation of test results
KP Ludwig John
• Engage 7 - 8 Test subjects.
• Pre-Test your equipment (record, storage, replay)
• Pre-Test your test procedure Are the task comprehensible? Does the prototype function properly? (readability, Internet connection, Links etc.)
Preparation
Test Procedure
UX-Research Evaluation of test results
KP Ludwig John
• Welcome the test subject and introduce the persons present
• Hand out the tasks (on paper) to the test subject.
Ask the test subject to process the tasks autonomously.In case of situations, when s/he would abort the given task,s/he should tell you.
Implementation
Test Procedure
• Now the test subjects processes the tasks. The test leader sits next to him/her, observers stay in distance.All team members stay silent.
UX-Research Evaluation of test results
KP Ludwig John
DurchführungTestablauf
Usability Test Ablauf
UX-Test AqualogicIMS 2012 Screen recording with video insert
Style of conversation
UX-Research Evaluation of test results
KP Ludwig John
• Encourage the test subject to verbally comment at any given situation throughout the test But avoid getting into a conversation with the test subject
Test ProcedureImplementation
• In case of silence … use impulses like these: • „What do you expect when clicking here?“ • „What would you do next?“ • „How would you proceed from here?“
• „Just try it!“
UX-Research Evaluation of test results
KP Ludwig John
• Only in cases of „emergency“ help the test person
Bear up against detours and errors.Do not comment on it!(That’s how you learn most from the test situation.)
Test ProcedureImplementation
• Finally thank the test subject for his/her time and contribution
UX-Research Evaluation of test results
KP Ludwig John
Gather your entire test team:
Test ProcedureEvaluation
Test again!
- Browse your notes and observations, accumulate and structure them
- Identify and describe found problems
- Evaluate them and decide which to solve (first) and how
UX-Research Evaluation of test results
KP Ludwig John
Which twoapproaches are generally distinguished??Usability testing
Discuss and collect1 minute
UX-Research Evaluation of test results
KP Ludwig John
• Processing speed for certain tasks • Number of autonomous completed tasks • Duration of task processing • Frequency of Errors
Vgl.: Prof. Dr. Burmester, Hochschule der Medien, Script 5, WS 2007/08 + Alan Cooper et al., Face 3, Willey 2007
Quantitative (summative) TestUsability Test Concepts
Based on Information that is measurable
Evaluation by means of software tools like: INTERACT (www.mangold.de) or Tobii Studio
Used to compare versions or to control norms
UX-Research Evaluation of test results
KP Ludwig John
Vgl.: Prof. Dr. Burmester, Hochschule der Medien, Script 5, WS 2007/08 + Alan Cooper et al., Face 3, Willey 2007
- Quantification of results - presentable as statistics Like - Duration - Error rate - Perception of certain interface elements Allows for version comparison
Results:
Usability Test ConceptsQuantitative (summative) Test
UX-Research Evaluation of test results
KP Ludwig John
Summativer Test
Ergebnis:
Summative Test
UX-Test User Interfaces in CarsUX-Lab 2013 Measured Values Tables
UX-Research Evaluation of test results
KP Ludwig John
Summativer Test
Ergebnis:
Test-ArtenUX-Test User Interfaces in CarsUX-Lab 2013 Measured Values Diagrams
Summative Test
UX-Research Evaluation of test results
KP Ludwig JohnUX-Test User Interfaces in CarsUX-Lab 2013 Comparison Table of Results
UX-Research Evaluation of test results
KP Ludwig John
UX-Test Navigation using Icon or TextIMS 2013 Result Tables + Diagrams
Analysis of EyeTracking: Gaze Time in Sec.
UX-Research Evaluation of test results
KP Ludwig John
- Element of the User Centered Design Process - observes (instead of: measures) Test Users while processing tasks - Goal: Discover and understand Usability problems in the real context of use
Vgl.: Prof. Dr. Burmester, Hochschule der Medien, Script 5, WS 2007/08 + Alan Cooper et al., Face 3, Willey 2007
Qualitative (formative) TestUsability Test Concepts
Popular Methods
- Note taking (while testing) - Video recording and subsequent Note taking - Capturing of verbal utterances of test users - Questionnaires and Surveys
UX-Research Evaluation of test results
KP Ludwig John
Logs contain all significant events throughout each of the Test sessions in chronological order:
- Tasks and Questions - Answers / Reactions of Test user - Observations by team members
Vgl.: Prof. Dr. Burmester, Hochschule der Medien, Script 5, WS 2007/08 + Alan Cooper et al., Face 3, Willey 2007
Evaluation starts by summarizing the test events as written logs
Usability Test ConceptsQualitative (formative) Test
>> but NO INTERPRETATION yet!!! >> Just facts and observations
UX-Research Evaluation of test results
KP Ludwig John
• Accumulate data from individual logs • Identify patterns • structure and interpret them
Evaluation procedure
Usability Test ConceptsQualitative (formative) Test
This entire process is known as
Edit it into a presentationunderstandable for partners (intern / extern)
UX-Research Evaluation of test results
KP Ludwig John
Goal: Identification of specific Usability problems distilled out of the collected data
Qualitative Analysis
Result: One Table per Usability problem
UX-Research Evaluation of test results
KP Ludwig John
Vgl: Prof. Dr. Burmester, Hochschule der Medien, Script 5, WS 2007/08
Qualitative Auswertung
5. Interpretation
Usabilityprobleme einzeln zusammen fassen
Quelle: HS-Augsburg | IMS | WS2011/2012 Anika Sanwald, Gregor Jaruga, Christian Langenmair, Wilfried Pfilf, Jakob Walkobinger
UX-Research Evaluation of test results
KP Ludwig John
Evaluation Steps and Elements:
To what extend did the test users represent the target group? Did this have influence on the recorded results?
a) Assessment of test conditions
Qualitative AnalysisGoal:
Identification of specific Usability problems distilled out of the collected data
1. Target group proximity (Zielgruppennähe)
UX-Research Evaluation of test results
KP Ludwig John
Qualitative Analysis
Assessment presentalthough rather generalmore specific = better
UX-Test IAM 2017 3D UI
1. Target group proximity (Zielgruppennähe)
UX-Research Evaluation of test results
KP Ludwig JohnUX-Test IAM 2015 PC Game
Assessment presentSome Test Subjects no real usersto be considered for interpretation of recorded data
Qualitative Analysis1. Target group proximity (Zielgruppennähe)
UX-Research Evaluation of test results
KP Ludwig John
UX-Test IAM 2015 Smart Phone App „asia“
Assessment OKPrimary Persona: „Student“ Technical Platform: smart phone
Qualitative Analysis1. Target group proximity (Zielgruppennähe)
UX-Research Evaluation of test results
KP Ludwig John
Evaluation Steps and Elements:
2. Quality of Prototype (Paper based, software, hardware etc.)
Did the Test object allow users to perform the test tasks properly?
3. Quality of Test scenario and circumstances
To what extend did the test situation and procedure influence the collected data? (Realistic context of Use?) Did the tasks represent real user needs and goals?
Qualitative Analysis
1. Target group proximity (Zielgruppennähe)
a) Assessment of test conditions
UX-Research Evaluation of test results
KP Ludwig JohnUX-Test IAM 2015 Smart Phone App „asia“
!
Clear Statement!to be considered for interpretation
2. Quality of Prototype3. Quality of Test scenario
Qualitative Analysisa) Assessment of test conditions
UX-Research Evaluation of test results
KP Ludwig John
4. Analysis of written Logs (Auswertung der Protokolle) Problem definition based on these logs 5. (if applicable) Evaluation of questionnaires Assessment of quality of use
6. Valuation of severity (Schweregrad) of the problems Team assigns a degree of severity to each problem
Some aspects in detail:
Evaluation Steps and Elements:
Qualitative Analysis
b) Systematization and Assessment of observations
UX-Research Evaluation of test results
KP Ludwig John
•chronological sequence of events observed •complemented by remarks of the observers • in Tabular form!
Written Log
UX-Test IAM 2015 Virtuelle Architektur „ArchiVision“
Qualitative Analysis(Beobachtungsprotokoll)
•one incident per numbered row
UX-Research Evaluation of test results
KP Ludwig JohnNr: number of incident Tn: number of participant A: number of task
Ereignis: specific events, questions by moderator, Observations Kommentar: Comments, explanations, further additions Video: TimeCode of video recording
Qualitative Analysis Written Log
•one incident per numbered row
UX-Research Evaluation of test results
KP Ludwig JohnUX-Test IAM 2015 Virtuelle Architektur „ArchiVision“
Qualitative Analysis Written Log•Next: label and assign problem areas (Indexing)
UX-Research Evaluation of test results
KP Ludwig John
OK, but„Interaktionsproblem“ to be further diversified
Qualitative Analysis
Index TablesGroup the problem areas found and describe them in form of
UX-Research Evaluation of test results
KP Ludwig John
• Create a detailed description of each Usability problem found • Think of possible changes to solve that problem
Beispiel:
schlecht: Benutzer haben Probleme bei Eingabe der Belüftungsdauer besser: Der Teilnehmer T2 will die minimale und die maximale Belüftungsdauer anpassen. Dabei stellen sich Verzögerungen / Schwierigkeiten bei der Eingabe mit Hilfe des Doppelsliders ein.
Interpretation
Qualitative Analysis
Condense each problem statementinto an individual Issue table
UX-Research Evaluation of test results
KP Ludwig John
Vgl: Prof. Dr. Burmester, Hochschule der Medien, Script 5, WS 2007/08
Qualitative Auswertung
5. Interpretation
Usabilityprobleme einzeln zusammen fassen
UX-Test IMS.mobile 2013AR at the museum - Movin’ KleeBritta Diehm, Xiaomeng Jiang, Yue Ma, Kerstin Vierthaler
Issue tableElements
How often did it appear? (Share of test users)
Where did it appear?
Description
Possible Causes
(Screen) Shot of situation
Which task produced the problem?
Which users produced the problem?
UX-Research Evaluation of test results
KP Ludwig John
Vgl: Prof. Dr. Burmester, Hochschule der Medien, Script 5, WS 2007/08
Qualitative Auswertung
5. Interpretation
Usabilityprobleme einzeln zusammen fassen
Quelle: HS-Augsburg | IMS | WS2011/2012 | Bernd Hacker, Boris Heißerer, Manuel Hörman , Moritz Schwind, Nico Thiebes
Confusing DesignBetter Use the table format
UX-Research Evaluation of test results
KP Ludwig John
Vgl: Prof. Dr. Burmester, Hochschule der Medien, Script 5, WS 2007/08
Qualitative Auswertung
5. Interpretation
Usabilityprobleme einzeln zusammen fassen
Quelle: HS-Augsburg | IMS | WS2011/2012 Anika Sanwald, Gregor Jaruga, Christian Langenmair, Wilfried Pfilf, Jakob Walkobinger
Severity Rating
UX-Research Evaluation of test results
KP Ludwig John
reference: https://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/
Qualitative Analysis Severity Rating
Rank the found problems by considering these factors:
Irrelevant for this rating: How much effort is required to fix the problem?!
• The frequency with which the problem occurs: Is it common or rare? (Share of test users ; Local or general problem)
• The impact of the problem if it occurs: Will it be easy or difficult for the users to overcome?Can users still accomplish their goals (although with difficulties)?
• The persistence of the problem: Is it a one-time problem that users can overcome once they know about it or will users repeatedly be bothered by the problem?
UX-Research Evaluation of test results
KP Ludwig John
0 I don't agree that this is no usability problem at all. 1 Cosmetic problem only: need not be fixed unless extra time is available on project
2 Minor usability problem: fixing this should be given low priority
3 Major usability problem: important to fix, so should be given high priority
5 Usability Desaster(!) imperative to fix this before product can be released
reference: https://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/
Severity Scale
Severity RatingQualitative Analysis
UX-Research Evaluation of test results
KP Ludwig John
Vgl: Prof. Dr. Burmester, Hochschule der Medien, Script 5, WS 2007/08
Severity Rating
Each team member rates each usability problemindividually
The over all mean valuerepresents the Severity of the problem
Qualitative Analysis
UX-Research Evaluation of test results
KP Ludwig John
Qualitative AnalysisRecommendations for improvementsNot an essential part of UX-Tests,but very common and often desired
Frequently derived from
Possibly leads to
• Observations while testing • general design principles • References to standards and user habits • Best practice experiences (from comparable products)
• Changes in design or conceptual aspects • Suggestions for further user studies with different focus • Development and tests of variations (a/b/n) for better solutions
UX-Research Evaluation of test results
KP Ludwig John
Highlight Video Example
Bachelor Interaktive Medien 2013iOS IAM Desk
UX-Research Evaluation of test results
KP Ludwig John
Highlight Video
Bachelor Interaktive Medien 2013iOS IAM Desk
UX-Research Evaluation of test results
KP Ludwig John
Highlight VideoContent and Style
• Compilation of representative test sequences • Goal: to demonstrate and prove the found problems • Accompany and Illustrate the problem tables
• Use Split screen(View to test user, View to prototype)
• Insert test task for each scene
UX-Research Evaluation of test results
KP Ludwig John
Examples: Master Interaktive MediensystemeA/B Test of Navigation concepts
Highlight Video
Bachelor Interaktive Medien 2013Installation Lumenear
Content and Style
• Stay short, focussed and objective
• NO MUSIC! • NO Making of (That’s how we’ve tested)
•Illustrate your findings • Video should speak for itself • External partners should understand your results
by watching the Highlight Video
UX-Research Evaluation of test results
KP Ludwig John
Highlight Video
MA Interactive Mediensystems 2013A/B Test Mobile Stand + EyeTracking
UX-Research Evaluation of test results
KP Ludwig John
Highlight Video
Bachelor Interactive Media 2013Installation Lumenear
Test User comments Recording
of test
UX-Research Evaluation of test results
KP Ludwig John
TASK
Organize and Run your usability testsbased on that scenarios
• 7 - 8 Test subjects per Design and Method • Video document your tests • If applicable:
Utilize software tools for statistic evaluationprovided by Mangold Interact or Tobii Studio
Develop a second set of UX-Test Scenarios based on new research questionsderived from your recent experiences and findings
Analyse, Evaluate, Document and Communicate itaccording to the guidelines of this presentation
UX-Research Evaluation of test results
KP Ludwig John
TASKStart with Index tables
ProvideSuggestions for improvements / next steps
Then CreateIssue tables and a Highlight Video to summarize and communicate your findings
IncludeResearch Question, test scenario and methodinto your report
Content of delivery
UX-Research Evaluation of test results
KP Ludwig John
TASK
Test design clearly based on research questions Focussed test effort (MVP, test method + procedure)
Criteria
Clear Communication of the entire process:• Assessment of procedure, methods and test implementation• HighLight Video illustrates the issues found• Tables clearly outline issues found
Deadline and details:
hs-augsburg.de/homes/john