ux-research 4 results-evaluation qualitative-tests...

56
KP Ludwig John UX Research evaluation of qualitative tests

Upload: trinhbao

Post on 25-Aug-2019

218 views

Category:

Documents


0 download

TRANSCRIPT

UX-Research Evaluation of test results

KP Ludwig John

UX Researchevaluation of qualitative tests

UX-Research Evaluation of test results

KP Ludwig John

UsabilityTest Procedure

UX-Research Evaluation of test results

KP Ludwig John

Test team

Test leader: - guides through the test - exclusive contact person for test subject - introduces the other persons present in the room

Test Procedure

- provides and replaces the printed pages of the paper prototype - remains silent! - Very well prepared! - Avoid endless searching for next page

in case of paper based test: Assistant (Page provider):

UX-Research Evaluation of test results

KP Ludwig John

Test leader: - guides through the test - exclusive contact for test subject

Observer: - placed in a bit of distance, with good view to the test happenings - remain silent!! - take notes of special incidences

Technician: - takes care of equipment and operates it - starts/stops recording

Test team

UX-Research Evaluation of test results

KP Ludwig John

Test team

Technician: - takes care of equipment and operates it - starts/stops recording

Observer: - placed in a bit of distance, with good view to the test happenings - remain silent!! - take notes of special incidences

UX-Research Evaluation of test results

KP Ludwig John

Test team

Test leader: - guides through the test - exclusive contact for test subject

UX-Research Evaluation of test results

KP Ludwig John

• Engage 7 - 8 Test subjects.

• Pre-Test your equipment (record, storage, replay)

• Pre-Test your test procedure Are the task comprehensible? Does the prototype function properly? (readability, Internet connection, Links etc.)

Preparation

Test Procedure

UX-Research Evaluation of test results

KP Ludwig John

• Welcome the test subject and introduce the persons present

• Hand out the tasks (on paper) to the test subject.

Ask the test subject to process the tasks autonomously.In case of situations, when s/he would abort the given task,s/he should tell you.

Implementation

Test Procedure

• Now the test subjects processes the tasks. The test leader sits next to him/her, observers stay in distance.All team members stay silent.

UX-Research Evaluation of test results

KP Ludwig John

DurchführungTestablauf

Usability Test Ablauf

UX-Test AqualogicIMS 2012 Screen recording with video insert

Style of conversation

UX-Research Evaluation of test results

KP Ludwig John

• Encourage the test subject to verbally comment at any given situation throughout the test But avoid getting into a conversation with the test subject

Test ProcedureImplementation

• In case of silence … use impulses like these: • „What do you expect when clicking here?“ • „What would you do next?“ • „How would you proceed from here?“

• „Just try it!“

UX-Research Evaluation of test results

KP Ludwig John

• Only in cases of „emergency“ help the test person

Bear up against detours and errors.Do not comment on it!(That’s how you learn most from the test situation.)

Test ProcedureImplementation

• Finally thank the test subject for his/her time and contribution

UX-Research Evaluation of test results

KP Ludwig John

Gather your entire test team:

Test ProcedureEvaluation

Test again!

- Browse your notes and observations, accumulate and structure them

- Identify and describe found problems

- Evaluate them and decide which to solve (first) and how

UX-Research Evaluation of test results

KP Ludwig John

UsabilityTest Concepts

UX-Research Evaluation of test results

KP Ludwig John

Which twoapproaches are generally distinguished??Usability testing

Discuss and collect1 minute

UX-Research Evaluation of test results

KP Ludwig John

• Processing speed for certain tasks • Number of autonomous completed tasks • Duration of task processing • Frequency of Errors

Vgl.: Prof. Dr. Burmester, Hochschule der Medien, Script 5, WS 2007/08 + Alan Cooper et al., Face 3, Willey 2007

Quantitative (summative) TestUsability Test Concepts

Based on Information that is measurable

Evaluation by means of software tools like: INTERACT (www.mangold.de) or Tobii Studio

Used to compare versions or to control norms

UX-Research Evaluation of test results

KP Ludwig John

Vgl.: Prof. Dr. Burmester, Hochschule der Medien, Script 5, WS 2007/08 + Alan Cooper et al., Face 3, Willey 2007

- Quantification of results - presentable as statistics Like - Duration - Error rate - Perception of certain interface elements Allows for version comparison

Results:

Usability Test ConceptsQuantitative (summative) Test

UX-Research Evaluation of test results

KP Ludwig John

Summativer Test

Ergebnis:

Summative Test

UX-Test User Interfaces in CarsUX-Lab 2013 Measured Values Tables

UX-Research Evaluation of test results

KP Ludwig John

Summativer Test

Ergebnis:

Test-ArtenUX-Test User Interfaces in CarsUX-Lab 2013 Measured Values Diagrams

Summative Test

UX-Research Evaluation of test results

KP Ludwig JohnUX-Test User Interfaces in CarsUX-Lab 2013 Comparison Table of Results

UX-Research Evaluation of test results

KP Ludwig John

UX-Test Navigation using Icon or TextIMS 2013 Result Tables + Diagrams

Analysis of EyeTracking: Gaze Time in Sec.

UX-Research Evaluation of test results

KP Ludwig John

- Element of the User Centered Design Process - observes (instead of: measures) Test Users while processing tasks - Goal: Discover and understand Usability problems in the real context of use

Vgl.: Prof. Dr. Burmester, Hochschule der Medien, Script 5, WS 2007/08 + Alan Cooper et al., Face 3, Willey 2007

Qualitative (formative) TestUsability Test Concepts

Popular Methods

- Note taking (while testing) - Video recording and subsequent Note taking - Capturing of verbal utterances of test users - Questionnaires and Surveys

UX-Research Evaluation of test results

KP Ludwig John

Logs contain all significant events throughout each of the Test sessions in chronological order:

- Tasks and Questions - Answers / Reactions of Test user - Observations by team members

Vgl.: Prof. Dr. Burmester, Hochschule der Medien, Script 5, WS 2007/08 + Alan Cooper et al., Face 3, Willey 2007

Evaluation starts by summarizing the test events as written logs

Usability Test ConceptsQualitative (formative) Test

>> but NO INTERPRETATION yet!!! >> Just facts and observations

UX-Research Evaluation of test results

KP Ludwig John

• Accumulate data from individual logs • Identify patterns • structure and interpret them

Evaluation procedure

Usability Test ConceptsQualitative (formative) Test

This entire process is known as

Edit it into a presentationunderstandable for partners (intern / extern)

UX-Research Evaluation of test results

KP Ludwig John

Qualitative Analysis

UX-Research Evaluation of test results

KP Ludwig John

Goal: Identification of specific Usability problems distilled out of the collected data

Qualitative Analysis

Result: One Table per Usability problem

UX-Research Evaluation of test results

KP Ludwig John

Vgl: Prof. Dr. Burmester, Hochschule der Medien, Script 5, WS 2007/08

Qualitative Auswertung

5. Interpretation

Usabilityprobleme einzeln zusammen fassen

Quelle: HS-Augsburg | IMS | WS2011/2012 Anika Sanwald, Gregor Jaruga, Christian Langenmair, Wilfried Pfilf, Jakob Walkobinger

UX-Research Evaluation of test results

KP Ludwig John

Evaluation Steps and Elements:

To what extend did the test users represent the target group? Did this have influence on the recorded results?

a) Assessment of test conditions

Qualitative AnalysisGoal:

Identification of specific Usability problems distilled out of the collected data

1. Target group proximity (Zielgruppennähe)

UX-Research Evaluation of test results

KP Ludwig John

Qualitative Analysis

Assessment presentalthough rather generalmore specific = better

UX-Test IAM 2017 3D UI

1. Target group proximity (Zielgruppennähe)

UX-Research Evaluation of test results

KP Ludwig JohnUX-Test IAM 2015 PC Game

Assessment presentSome Test Subjects no real usersto be considered for interpretation of recorded data

Qualitative Analysis1. Target group proximity (Zielgruppennähe)

UX-Research Evaluation of test results

KP Ludwig John

UX-Test IAM 2015 Smart Phone App „asia“

Assessment OKPrimary Persona: „Student“ Technical Platform: smart phone

Qualitative Analysis1. Target group proximity (Zielgruppennähe)

UX-Research Evaluation of test results

KP Ludwig John

Evaluation Steps and Elements:

2. Quality of Prototype (Paper based, software, hardware etc.)

Did the Test object allow users to perform the test tasks properly?

3. Quality of Test scenario and circumstances

To what extend did the test situation and procedure influence the collected data? (Realistic context of Use?) Did the tasks represent real user needs and goals?

Qualitative Analysis

1. Target group proximity (Zielgruppennähe)

a) Assessment of test conditions

UX-Research Evaluation of test results

KP Ludwig JohnUX-Test IAM 2015 Smart Phone App „asia“

!

Clear Statement!to be considered for interpretation

2. Quality of Prototype3. Quality of Test scenario

Qualitative Analysisa) Assessment of test conditions

UX-Research Evaluation of test results

KP Ludwig John

4. Analysis of written Logs (Auswertung der Protokolle) Problem definition based on these logs 5. (if applicable) Evaluation of questionnaires Assessment of quality of use

6. Valuation of severity (Schweregrad) of the problems Team assigns a degree of severity to each problem

Some aspects in detail:

Evaluation Steps and Elements:

Qualitative Analysis

b) Systematization and Assessment of observations

UX-Research Evaluation of test results

KP Ludwig John

•chronological sequence of events observed •complemented by remarks of the observers • in Tabular form!

Written Log

UX-Test IAM 2015 Virtuelle Architektur „ArchiVision“

Qualitative Analysis(Beobachtungsprotokoll)

•one incident per numbered row

UX-Research Evaluation of test results

KP Ludwig JohnNr: number of incident Tn: number of participant A: number of task

Ereignis: specific events, questions by moderator, Observations Kommentar: Comments, explanations, further additions Video: TimeCode of video recording

Qualitative Analysis Written Log

•one incident per numbered row

UX-Research Evaluation of test results

KP Ludwig JohnUX-Test IAM 2015 Virtuelle Architektur „ArchiVision“

Qualitative Analysis Written Log•Next: label and assign problem areas (Indexing)

UX-Research Evaluation of test results

KP Ludwig John

OK, but„Interaktionsproblem“ to be further diversified

Qualitative Analysis

Index TablesGroup the problem areas found and describe them in form of

UX-Research Evaluation of test results

KP Ludwig John

• Create a detailed description of each Usability problem found • Think of possible changes to solve that problem

Beispiel:

schlecht: Benutzer haben Probleme bei Eingabe der Belüftungsdauer besser: Der Teilnehmer T2 will die minimale und die maximale Belüftungsdauer anpassen. Dabei stellen sich Verzögerungen / Schwierigkeiten bei der Eingabe mit Hilfe des Doppelsliders ein.

Interpretation

Qualitative Analysis

Condense each problem statementinto an individual Issue table

UX-Research Evaluation of test results

KP Ludwig John

Vgl: Prof. Dr. Burmester, Hochschule der Medien, Script 5, WS 2007/08

Qualitative Auswertung

5. Interpretation

Usabilityprobleme einzeln zusammen fassen

UX-Test IMS.mobile 2013AR at the museum - Movin’ KleeBritta Diehm, Xiaomeng Jiang, Yue Ma, Kerstin Vierthaler

Issue tableElements

How often did it appear? (Share of test users)

Where did it appear?

Description

Possible Causes

(Screen) Shot of situation

Which task produced the problem?

Which users produced the problem?

UX-Research Evaluation of test results

KP Ludwig John

Vgl: Prof. Dr. Burmester, Hochschule der Medien, Script 5, WS 2007/08

Qualitative Auswertung

5. Interpretation

Usabilityprobleme einzeln zusammen fassen

Quelle: HS-Augsburg | IMS | WS2011/2012 | Bernd Hacker, Boris Heißerer, Manuel Hörman , Moritz Schwind, Nico Thiebes

Confusing DesignBetter Use the table format

UX-Research Evaluation of test results

KP Ludwig John

Vgl: Prof. Dr. Burmester, Hochschule der Medien, Script 5, WS 2007/08

Qualitative Auswertung

5. Interpretation

Usabilityprobleme einzeln zusammen fassen

Quelle: HS-Augsburg | IMS | WS2011/2012 Anika Sanwald, Gregor Jaruga, Christian Langenmair, Wilfried Pfilf, Jakob Walkobinger

Severity Rating

UX-Research Evaluation of test results

KP Ludwig John

reference: https://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/

Qualitative Analysis Severity Rating

Rank the found problems by considering these factors:

Irrelevant for this rating: How much effort is required to fix the problem?!

• The frequency with which the problem occurs: Is it common or rare? (Share of test users ; Local or general problem)

• The impact of the problem if it occurs: Will it be easy or difficult for the users to overcome?Can users still accomplish their goals (although with difficulties)?

• The persistence of the problem: Is it a one-time problem that users can overcome once they know about it or will users repeatedly be bothered by the problem?

UX-Research Evaluation of test results

KP Ludwig John

0 I don't agree that this is no usability problem at all. 1 Cosmetic problem only: need not be fixed unless extra time is available on project

2 Minor usability problem: fixing this should be given low priority

3 Major usability problem: important to fix, so should be given high priority

5 Usability Desaster(!) imperative to fix this before product can be released

reference: https://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/

Severity Scale

Severity RatingQualitative Analysis

UX-Research Evaluation of test results

KP Ludwig John

Vgl: Prof. Dr. Burmester, Hochschule der Medien, Script 5, WS 2007/08

Severity Rating

Each team member rates each usability problemindividually

The over all mean valuerepresents the Severity of the problem

Qualitative Analysis

UX-Research Evaluation of test results

KP Ludwig John

Qualitative AnalysisRecommendations for improvementsNot an essential part of UX-Tests,but very common and often desired

Frequently derived from

Possibly leads to

• Observations while testing • general design principles • References to standards and user habits • Best practice experiences (from comparable products)

• Changes in design or conceptual aspects • Suggestions for further user studies with different focus • Development and tests of variations (a/b/n) for better solutions

UX-Research Evaluation of test results

KP Ludwig John

Usability Test Evaluation

Highlight Video

UX-Research Evaluation of test results

KP Ludwig John

Highlight Video Example

Bachelor Interaktive Medien 2013iOS IAM Desk

UX-Research Evaluation of test results

KP Ludwig John

Highlight Video

Bachelor Interaktive Medien 2013iOS IAM Desk

UX-Research Evaluation of test results

KP Ludwig John

Highlight VideoContent and Style

• Compilation of representative test sequences • Goal: to demonstrate and prove the found problems • Accompany and Illustrate the problem tables

• Use Split screen(View to test user, View to prototype)

• Insert test task for each scene

UX-Research Evaluation of test results

KP Ludwig John

Examples: Master Interaktive MediensystemeA/B Test of Navigation concepts

Highlight Video

Bachelor Interaktive Medien 2013Installation Lumenear

Content and Style

• Stay short, focussed and objective

• NO MUSIC! • NO Making of (That’s how we’ve tested)

•Illustrate your findings • Video should speak for itself • External partners should understand your results

by watching the Highlight Video

UX-Research Evaluation of test results

KP Ludwig John

Highlight Video

MA Interactive Mediensystems 2013A/B Test Mobile Stand + EyeTracking

UX-Research Evaluation of test results

KP Ludwig John

Highlight Video

Bachelor Interactive Media 2013Installation Lumenear

Test User comments Recording

of test

UX-Research Evaluation of test results

KP Ludwig John

TASK

Organize and Run your usability testsbased on that scenarios

• 7 - 8 Test subjects per Design and Method • Video document your tests • If applicable:

Utilize software tools for statistic evaluationprovided by Mangold Interact or Tobii Studio

Develop a second set of UX-Test Scenarios based on new research questionsderived from your recent experiences and findings

Analyse, Evaluate, Document and Communicate itaccording to the guidelines of this presentation

UX-Research Evaluation of test results

KP Ludwig John

TASKStart with Index tables

ProvideSuggestions for improvements / next steps

Then CreateIssue tables and a Highlight Video to summarize and communicate your findings

IncludeResearch Question, test scenario and methodinto your report

Content of delivery

UX-Research Evaluation of test results

KP Ludwig John

TASKContent of delivery

UX-Research Evaluation of test results

KP Ludwig John

TASK

Test design clearly based on research questions Focussed test effort (MVP, test method + procedure)

Criteria

Clear Communication of the entire process:• Assessment of procedure, methods and test implementation• HighLight Video illustrates the issues found• Tables clearly outline issues found

Deadline and details:

hs-augsburg.de/homes/john