Sponsors

User Testing: Adapt to Fit Your Needs#DenStartupWeek #EffectiveUI

Agenda

1. Intro 2. What is evaluative research? 3. How To 4. Engaging the whole team 5. What to do next

Ari WeissmanLead Experience Architect

888.310.5327

• 10+ years global experience

• Responsible for all things UX, from research through experience design

• Project lead for TimeWarner Cable, AMEX, FreemanCo, and Securian

@ EffectiveUI

Lys Maitland Senior Experience Planner

888.310.5327

• 15+ years global experience

• Responsible for all things UX, from research through experience design

• Research lead for Davita, Scottrade, NextGear, Securian, CenturyLink

@ EffectiveUI

Insight-driven digital product company

Locations

2162 Market StreetDenver, CO 80205

DENVER

274 North Goodman St, Unit B264Rochester, NY 14607

ROCHESTER

85 Broad Street, 18th fl.New York, NY 10004

NEW YORK CITY

Our clients

What is evaluative research?

[a.k.a. user testing]

Generative v. Evaluative

Generative research

Helps define the problem.

The goal of generative research is to look to the activities, environments, interactions, objects and people to find opportunities for solutions and innovation. These solutions could be new products or experiences or they could be an update or improvement to an existing one.

Evaluative research Evaluates an existing design (in prototype form or in final form).

The goal of evaluative research is to test the existing solution to see if it meets people’s needs, is easy to access and use, and is enjoyable. This type of research should be conducted throughout the development lifecycle, from early concept design (think rough sketches or prototypes) to the final site, app, or product.

DEFINE DISCOVER DESIGN DEVELOP DEPLOY

EVALUATIVE RESEARCH

Field Research Iterative Concept Testing Usability Testing

Define Problem

Problem Approach

Gather Data

Observational Research

Benchmark Current Site

Distinguish Insights

Analytics Assessment

Create Concepts

Concept Test

Explore Solutions

Build Prototypes

Test & Refine

Collaborate On Options

Discover Gaps & Problems in Experience

Analytics Strategy

Implementation/ Change Management

Continuous Measurement

Test & Refine

Some examples of evaluative research

Testing an IA or design learnabilityCLICK TEST

Testing flows and contentUSABILITY TESTING

Testing and measuring current state for redesign

BENCHMARKING

Closed card sort to validate an IACARD SORTING

Iterative testing of flows and concepts, some generative research

ITERATIVE DESIGN TESTING

Benefits of User Testing

1 Validate assumptions 4 Provide justification for additional effort

2 Decide between options 5 Reveal how users think about the problems your solution solves

3 Discover hidden issues 6 Removes opinion from the design process

How To

How To

1. Define the goals

2. Define the audience

3. What to test

4. $ -> $$$

5. Debrief

6. Analysis

7. Outcomes/Deliverables

What are your goals?

888.310.5327

Some questions to ask: - Why are you testing and what will

you do with the data?

- What do you need to know / learn / validate? Why?

Who is your audience?

888.310.5327

Some questions to ask: - Who will use of the thing you’re

building?

- Are there different groups of users? If so, which have the highest priority?

- Where are they?

Recruiting

Screener • Script used to recruit

• Intro to the study

• Questions to find participants who demonstrate desired behaviors

• Offer an incentive

Schedule 1 hour interviews with at least 30 minutes between sessions

Recruit • 5-7 participants per

segment

• 5 participants finds 85% of usability issues

1 2 3

Participants are good to the extent they represent your target. If your participants don't match your target, your study will be useless. You can learn valuable things by asking the right people the wrong questions. If you are talking to the wrong people it doesn't matter what you ask. Bad participants undermine everything you’re trying to do.

— Erika Hall, author of Just Enough Research

“”

Run the first session as a pilot to work out the kinks and refine the moderators guide. Schedule an hour

after the first to re-tool as needed.

PRO TIP

What can I test?

ANYTHING

- CONCEPT

- HAND SKETCHED MESSY

- HAND SKETCHED NEAT

- WIREFRAMES

- VISUAL DESIGN

- CLICK-THROUGH PROTOTYPE

- CODED PROTOTYPE

- FINAL PRODUCT

The quick and dirty

Why quick and dirty?

- NO TIME

- NO MONEY

- NO FACILITIES

- TESTING SOMETHING SMALL

Quick and dirty

Location Anywhere!

Logistics Find a place

Find some people

Ask some questions

Write down answers

Costs Minimal to none

Fidelity to test Any

1 2 3 4

Middle of the road

Why middle of the road?

- NEED SOME DOCUMENTATION TO

SHARE WITH TEAM OR SAVE

- TOO MUCH CONTENT FOR QUICK

AND DIRTY

- WANT TO HAVE OBSERVERS

- DESIRED USERS ARE NOT READILY

AVAILABLE

Middle of the road

Location Preferably in context,

though may be remote

In a quiet room at a table

Logistics Schedule a location and

time for one hour

Recruit participants and consider incentive

Prep discussion guide

Create data capture sheet

Audio/Video recording

Costs Location

Recruiting and incentives

Screen sharing/Recording tool

Fidelity to test Typically wireframe or

above

1 2 3 4

Formal

Why formal testing?

- REQUIRES FORMAL DOCUMENTATION

- DESIRED USERS ARE NOT READILY

AVAILABLE

- STRICT DISCUSSION GUIDE

REQUIREMENTS

- LOTS OF THINGS TO TEST

- NEED TO INCLUDE NOTETAKERS AND

OBSERVERS

- MULTIPLE DEVICES OR COMPLEX

TECHNOLOGY REQUIREMENTS

Formal

Location Quiet room with a table

Separate location for observers

Quality phone if remote

Logistics Schedule a location and

time for two hours

Recruit participants and distribute incentive

Prep/share/revise discussion guide

Create data capture sheet

Audio/Video recording

Costs Location

Recruiting and incentives

Screen sharing/Recording tool

Snacks and beverages

Transcriptions

Fidelity to test Typically wireframe or

above

1 2 3 4

FacilitationDo you have 3 more hours?

Facilitation: The highlights

Asking questions Ground your questions

• Will you How have you…?

• In general In this specific instance…?

• Tell me how Show me how…?

Ask why? A lot!

Rhythm • Ask

• Listen

• Probe

• Validate

3 Types of tasks • Specific verb based

tasks (do, send, complete)

• Scavenger hunt (find, discover)

• Interview (tell me about)

1 2 3

So what I heard was, you do X and Y because of A

and B, but a little bit of 123 would make it better. Is

that right?

PRO TIP

The Debrief888.310.5327

Debrief after EVERY session

It’s a quick assessment of a session:

- Summary of the participant

- What went well

- What needs improvement

- Insights or surprises

- Opportunities

Analysis and Synthesis

Trends Look for trends across all participants or within a participant segment

Good and Bad Document what doesn't work AND what does work

Prioritize Choose the most important findings to share and focus on solving

Think big picture Is the issue identified a symptom of a bigger problem?

Quotes Identify quotes that capture the essence of your insights

Solutions What solutions were identified by participants or observers?

Future Opportunities What do the outcomes suggest would be a good next step?

Centralize and share All data in one place for access and redundancy. Shit happens.

Examples of analysis

OUTPUTS

Spreadsheet

39

Post-its

40

Live

Examples of evaluative research results

OUTPUTS

$

quick and dirty

$

quick and dirty

$$

middle of the road

$$$$

formal

Engaging the Whole Team

Engage your team at every step

- CHOOSING GOALS AND AUDIENCE

- REVIEWING DISCUSSION GUIDE

- OBSERVING AND NOTE TAKING

- DEBRIEFING

- PROBING DURING ANALYSIS

- PRIORITIZING RESULTS

- DETERMINING NEXT STEPS

What to do with outcomes

What to do with outcomes

1 4

2 5

3 6

Share your findings

(email, present, print, posterize)Create a roadmap with priorities

Identify opportunities Support ideation

Stay engaged!

Be available to answer questions.Make recommendations

7 Test Again!

Questions?

Vragen Preguntas 問題 Des questions שאלותвопросов คำถาม frågor

Thank you.Ari + Lys

P.S. We’re hiring! http://www.effectiveui.com/company/careers/

www.linkedin.com/company/effectiveuiLINKEDIN

https://twitter.com/effectiveui TWITTER

www.facebook.com/InsideEffectiveUIFACEBOOK

www.instagram.com/effectiveuiINSTAGRAM

https://medium.com/@effectiveui MEDIUM

Stay connected with EffectiveUI

https://abookapart.com/products/just-enough-researchJUST ENOUGH RESEARCH BY ERIKA HALL

http://rosenfeldmedia.com/books/interviewing-users INTERVIEWING USERS BY STEVE PORTIGAL

https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users WHY YOU ONLY NEED TO TEST WITH 5 USERS BY JAKOB NIELSEN

Mentioned References