defining user research methodologies: a pragmatic approach

Monday, April 30, 2012

DEFINING USER RESEARCH METHODOLOGIESA PRAGMATIC APPROACH

M I N N E W E B C O N 2 0 1 2


DEFINING USER RESEARCH METHODOLOGIESA PRAGMATIC APPROACH

M I N N E W E B C O N 2 0 1 2

#mwcresearch


ZACKSENIOR USER EXPERIENCE DESIGNERUSER RESEARCH PRACTICE LEAD

@zacknaylor


DAVEUSER EXPERIENCE DESIGNER

PHD CANDIDATE Tech Comm & User Experience

@Dave_L_Jones


WHAT ABOUT YOU?


WHAT WE’LL COVER


WHAT WE’LL COVER

Common user research methods


WHAT WE’LL COVER

Common user research methods

Identifying the appropriate use and combination of methods


WHAT WE’LL COVER

Identifying the appropriate use and combination of methods


WHAT WE’LL COVER


WHAT WE’LL COVER

Goals in research planning


WHAT WE’LL COVER


WHAT WE’LL COVER

Research data analysis


WHAT WE’LL COVER


USER RESEARCH

IS NOT

MARKETING RESEARCH


USER RESEARCH

IS NOT

MARKETING RESEARCH

...but there is some overlap


USER RESEARCH

IS NOT

MARKETING RESEARCH


We do this


USER RESEARCH

IS NOT

MARKETING RESEARCH


We do this

We DON’T do this


MARKETING RESEARCHUSER RESEARCH


MARKETING RESEARCH

Preferences

Opinions

Likes

Desires

USER RESEARCH


MARKETING RESEARCH

Preferences

Opinions

Likes

Desires

USER RESEARCH

Behavior

Needs

Goals

Tasks

Mental & Physical Context


RESEARCH SPECTRUM

http://www.flickr.com/photos/rosenfeldmedia/2159500714/in/set-72157603511616271/ Indi Young - Mental Models: Rosenfeld Media


http://www.flickr.com/photos/rosenfeldmedia/2159500714/in/set-72157603511616271/


RESEARCH SPECTRUM

http://www.flickr.com/photos/rosenfeldmedia/2159500714/in/set-72157603511616271/ Indi Young - Mental Models: Rosenfeld Media

overlap




WHY DO USER RESEARCH?



Remember when we said that design solves problems?



Remember when we said that design solves problems?

Remember when we said that a UX process starts with defining the problem?



Remember when we said that a UX process starts with defining the problem?


The systematic investigation into and study of materials and sources in order to establish facts and reach new conclusions.



Research tells us what the problem is.




Research tells us why it’s a problem.




Research shows us how to fix it.

Research tells us why it’s a problem.


BENEFITS OF USER RESEARCH



Throughly defines the problem.




Informs design decisions.





Provides direction & priority.






SHOWS US THE “WHAT”







TEACHES US THE “WHY”







TEACHES US THE “WHY”

GUIDES US TO THE “HOW”


EXAMPLE PROJECT NEEDS

“WHAT”

“WHY”

“HOW”

Stated: What Research Tells Us:



“WHAT”

“WHY”

“HOW”

“We need a website redesign”




“WHAT”

“WHY”

“HOW”

“We need a website redesign” The website doesn’t meet users’ expectations




“WHAT”

“WHY”

“HOW”


“The current site isn’t working for us”

The website doesn’t meet users’ expectations




“WHAT”

“WHY”

“HOW”




There are usability flaws in the design




“WHAT”

“WHY”

“HOW”



“We should update the look & feel”






“WHAT”

“WHY”

“HOW”






We need to design clearercall to action buttons.




“WHAT”

“WHY”

“HOW”








Vague & Undefined



“WHAT”

“WHY”

“HOW”








Vague & UndefinedClear & Actionable


vs. QUANTITATIVEQUALITATIVE

BEWARE OF THE DATA TYPE


vs. X QUANTITATIVEQUALITATIVE



vs.

Just remember

X QUANTITATIVEQUALITATIVE



vs.

Just remember

We need both!

X QUANTITATIVEQUALITATIVE




vs. X QUALITATIVEQUANTITATIVESubjective

“Non-measurable”NOT Numerical

Concepts

Objective“Measurable”

Numerical DataStatistics



QUALITATIVEQUANTITATIVE



QUALITATIVE

QUANTITATIVE


BEWARE OF THE DATA TYPEQUALITATIVEQUANTITATIVE


Page Views

Bounce Rate

Time On Site

Yes/No

True/False



Page Views

Bounce Rate

Time On Site

Yes/No

True/False

Expectations

Reactions

Confusion

Comprehension

Behavior



BEWARE OF THE DATA TYPEQUANTITATIVE QUALITATIVE



QUANTITATIVE

QUALITATIVE



QUANTITATIVE QUALITATIVE



informs QUANTITATIVE QUALITATIVE



informs

“WHAT”




informs

“WHAT” “WHY”




informs “WHAT” “WHY”




informs i.e. Sign-up page bounce rate it high.





informs i.e. Sign-up page bounce rate it high. i.e. Call to action text is confusing.




WHEN SHOULD YOU DO USER RESEARCH?

Timeframe Beginning of the project During Design Production Project Completion

- inform direction & scope- throughly defines the problem- provides insights for next steps

- re-enforce direction- validate design decisions- acquire design feedback at significantly lower cost

- gauge progress/success- guides product direction- discover areas for improvement

- Contextual Inquiry- Field Study/Ethnography- Stakeholder Interviews- Surveys

- Usability Testing- Card Sorting- Personas- Mental Models

- Usability Testing- Site Search Analytics- Web Analytics- A/B Testing/Multivariate Testing

Benefit

Example Method/Use


BONUS ROUND


BONUS ROUND

ONGOING RESEARCH


BONUS ROUND

ONGOING RESEARCH

How:


BONUS ROUND

ONGOING RESEARCH

How: - Create a solid and sustainable research plan for a continuous feedback loop from your customers/users


BONUS ROUND

ONGOING RESEARCH

Why:



BONUS ROUND

ONGOING RESEARCH

Why: - Your audience changes and evolves over time



BONUS ROUND

ONGOING RESEARCH


- Your product/service/website will attract new audience segments



BONUS ROUND

ONGOING RESEARCH


- Your product/service/website will attract new audience segments

-Other products/services/websites introduce new expectations for interacting with your information



COMMON RESEARCH SKILLS

INTERVIEWING

OBSERVATION

LISTENING

ANALYSIS (SENSE-MAKING)Monday, April 30, 2012

COMMON RESEARCH SKILLSESSENTIAL

INTERVIEWING

OBSERVATION

LISTENING

ANALYSIS (SENSE-MAKING)Monday, April 30, 2012

NERDERY RESEARCH METHODS

Usability Testing

Surveys

Card Sorting

Stakeholder Interviews

User Interviews

Contextual Inquiry

Personas

Field Study/Ethnography

Site Search Analytics

Web Analytics

A/B Testing : Multivariate Testing

Mental Models



Usability Testing

Surveys

Card Sorting


User Interviews

Contextual Inquiry

Personas



Web Analytics


Mental Models

“WHAT”



Usability Testing

Surveys

Card Sorting


User Interviews

Contextual Inquiry

Personas



Web Analytics


Mental Models

“WHY”



Usability Testing

Surveys

Card Sorting


User Interviews

Contextual Inquiry

Personas



Web Analytics


Mental Models

“HOW”


STAKEHOLDER INTERVIEWS


WHAT ARE STAKEHOLDER INTERVIEWS?

Definition:


WHAT ARE STAKEHOLDER INTERVIEWS?

one on one conversations with client champions of a project aimed at gaining understanding of three overarching themes of information:

1. Project Context

2. Target Audience

3. Project Success

Definition:


THE “HOW-TO” OFSTAKEHOLDER INTERVIEWS

4. Analyze

5. Insights Report

1. Recruiting

3. Conduct Interviews

2. Research Plan

Review what you learned. Did it match your hypothesis? What patterns emerged?

Create appropriate documentation to communicate what you found to the team.

Identify the stakeholders on the project, schedule ONE HOUR with each stakeholder separately.

Watch users completing relevant goals & tasks in their own context.

Establish a clear focus for what you expect to find & what you hope to learn.


WEB ANALYTICS


WHAT ARE WEB ANALYTICS?

Definition:


WHAT ARE WEB ANALYTICS?

Web analytics is the measurement, collection, analysis and reporting of internet data for purposes of understanding and optimizing web usage. (Wikipedia)

Definition:


THE “HOW-TO” OFWEB ANALYTICS

4. Insights Report

5. Pinpoint knowledge gap(s)

1. Identify data needs

3. Analyze

2. Gather data


Target unknowns and craft a qualitative research plan to fill known gaps in understanding. Outline how you plan to uncover the “why” of the problem.

What information and data are useful for defining the problem?

Segment where relevant and cross reference the raw data to find patterns and/or correlations.

Collect that data and information within a determined time period for analysis.


CONTEXTUAL INQUIRY


WHAT IS CONTEXTUAL INQUIRY?

Fancy Definition:



a field data-gathering technique that studies a few carefully selected individuals in depth to arrive at a fuller understanding of the work practice across all customers.

Fancy Definition:




Fancy Definition:

- Hugh Beyer and Karen Holtzblatt - Contextual Design: Defining Customer-Centered Systems




Fancy Definition:

Non-Nerd Version:





Fancy Definition:

go watch people work in their own context.Non-Nerd Version:





Fancy Definition:

go watch people work in their own context.Non-Nerd Version:


- Us


THE “HOW-TO” OF CONTEXTUAL INQUIRY

4. Analyze

5. Report

1. Recruiting

3. Observe

2. Research Plan


Create appropriate documentation to communicate what you found to the team.

Contact & schedule the people you will observe. THIS IS THE MOST IMPORTANT STEP.

Watch users completing relevant goals & tasks in their own context.

Establish a clear focus for what you expect to find & what you hope to learn.


WHAT’S THE BENEFIT OF CONTEXTUAL INQUIRY?

Environment

Applications

Issues

Tasks

Goals

What is their physical location like? How does it impact the design or how they use it?

What other hardware or software are they using to do their work? Can (or should) they be integrated? Can the new design eliminate the need of these factors?

Are there current frustrations or problems with the existing design? What about with their physical environment or other systems and processes? Can the new design support those?

What are the steps people are taking to accomplish those goals?

What are the high level priorities of the people using the current design? What are they trying to accomplish?


WHAT’S THE BENEFIT OF CONTEXTUAL INQUIRY?

Who does the person work with to accomplish a goal or task?

Do people have “cheat-sheets” or other materials to help them accomplish goals and tasks?

Are people creating ways to work around a poor design now?

Are there several ways in which people are accomplishing the same goal or task? Should the design support one? Both?

What causes someone to begin down a path of completing a goal?

Work-Arounds

Triggers

Variation(s)

Partners

“Crutches”


RECOMMENDATIONS FOR CONTEXTUAL INQUIRY

Recruiting

Timeline

Be sure to observe an appropriate, representative sample of your target audience.Aim for 3-5 participants separately.

Allow 1-2 weeks for recruiting effort. (Varies depending on the project & participant availability)Allow 1-2 weeks for conducting the research. (Assuming 5 participants)Allow AT LEAST 1 week for analysisAllow 1 week to create a report.

Approximately 4-8 weeks total


CARD SORTING


WHAT IS CARD SORTING?

Definition:


WHAT IS CARD SORTING?

method of gathering data to inform the information architecture, navigation, taxonomy and labeling of a design

Definition:


THE “HOW-TO” OFCARD SORTING

4. Analyze

5. Report

1. Recruiting

3. Conduct Sorts

2. Research Plan

Collect your findings and perform exploratory analysis or statistical analysis (or both).

Create appropriate documentation that conveys the findings from the research.

Determine who you will conduct card sorts with, write a screener to ensure you recruit your target audience from the responses & schedule the participants.

Conduct the card sort(s) with the target audience and selected data/content.

Where will the sorts take place? (remote or in person? with how many?) As before, Establish a clear focus for what you expect to find & what you hope to learn.

2.1 Logistics Choose your data, expected analysis method and sort method (open/closed sort). Single or group sorts?


WHAT’S THE BENEFIT OFCARD SORTING?

Labeling

Categorization

Navigation

Grouping

Hierarchy


WHAT’S THE BENEFIT OFCARD SORTING?

Informed Information Architecture


USABILITY TESTING


WHAT IS USABILITY TESTING?

Definition:


WHAT IS USABILITY TESTING?

a form of gathering feedback from actual users of a design by having them attempt to complete intended goals and tasks with said design.

Definition:


THE “HOW-TO” OFUSABILITY TESTING

4. Analyze

5. Report

1. Recruiting

3. Conduct Tests

2. Research Plan

Discuss test results with any participating team members & review recordings if available.

Create appropriate documentation that conveys the findings from the research.

Determine who you will conduct usability testing with, write a screener to ensure you recruit your target audience from the responses & schedule the participants. Again, a CRITICAL step.

Conduct the test (ideally with an experience moderator) & observe, while taking notes.

Where will the tests take place? What will the research cover? As before, Establish a clear focus for what you expect to find & what you hope to learn.


WHAT’S THE BENEFIT OF USABILITY TESTING?

Path Taken

Impression

Expectations

Level of difficulty

Task Completion

What were the steps involved in completing a given task?

Did the user(s) understand the overall message and intent that the design meant to convey?

Is the solution designed meeting expectations of those intended to use it?

If so, how difficult was it for them to complete the task(s)? Why?

Can the users complete the available tasks of the design?


VARIATIONS OFUSABILITY TESTING

Guerilla

In-Person, Moderated

Remote, Un-moderated

Remote, Moderated

Usability testing done with minimal recruiting effort and logistical planning. Common locations are coffee shops, bars, offices, etc.

Sessions are conducted at a physical location with a live, in-person moderator leading the session(s) with participants.

Sessions are conducted using an online service that allows users to participate at their convenience, without a moderator.

Research is conducted via “live recruiting” from an existing website. Participants are immediately connected with a moderator from a remote location.



Guerilla



Remote, Moderated





Good



Guerilla



Remote, Moderated





Better



Guerilla



Remote, Moderated





Best


RECOMMENDATIONS FORUSABILITY TESTING

Recruiting

Timeline

Do your own recruiting if possible. If this is not possible, work closely with a recruiting agency to ensure the participants being recruited match your target audience.(especially for the particular research you’re doing)

Aim for 5-10 participants. (dependent on the study)

Allow 1-2 weeks for recruiting effort. (Varies depending on the recruiting method)Allow 1 week for conducting the tests. (Assuming 5 participants)Allow 3 days to 1 week for analysis.Allow 3 days to 1 week to create a report.

Approximately 3-6 weeks total


WHERE TO BEGIN


STEP 1:UNDERSTANDING BUSINESS GOALS


What are the stated problems?

What impact (and implications will our design have on those goals?

How do we arrive at the core problem or root cause?

STEP 1:UNDERSTANDING BUSINESS GOALS


STEP 2:DEFINING YOUR INFORMATION NEEDS


What info is necessary for you to meet those business goals?

What information do you need to successfully design?

What information do you have available?

Where are you knowledge “blind spots”?

STEP 2:DEFINING YOUR INFORMATION NEEDS


STEP 3:CRAFT YOUR APPROACH


Which methods are applicable to reach your information needs?

How will you gather information about the target audience?

How do you plan to use that information?

What are the impacts of that data in design?

STEP 3:CRAFT YOUR APPROACH


ACTIVITY:USER RESEARCH PLAN


30-40 minutes



30-40 minutes


GO!


PUTTING IT ALL TOGETHER


FICTIONAL PROJECT: WEB APP REDESIGN


“We need to redesign our web application to increase customer engagement”



“We need to redesign our web application to increase customer engagement”


GO!


Let’s get started:





THE “WHAT”




Targeted SurveysTHE “WHAT”




Targeted Surveys Web AnalyticsTHE “WHAT”



THE “WHY”



User InterviewsTHE “WHY”



User Interviews Contextual InquiryTHE “WHY”



THE “HOW”



Usability TestingTHE “HOW”



Usability Testing

“HOW”

Project Overview:

Targeted Surveys

Web Analytics

User Interviews

Contextual Inquiry

Time

Prototype[design & development}

“WHY”“WHAT”

iteration(s)


DATA ANALYSIS


STEP 1:REVISIT YOUR RESEARCH GOALS


Did you collect the information necessary to design?

Were the goals of your research met?

Have you gained an understanding of the problem space?

STEP 1:REVISIT YOUR RESEARCH GOALS


STEP 2:BUILD A MODEL - DATA INTO KNOWLEDGE


Craft a visual representation of all the raw data you collected.

Pull out patterns, problems and context.

Prioritize the issues and patterns.

Brainstorm an approach to design from your new found knowledge.

STEP 2:BUILD A MODEL - DATA INTO KNOWLEDGE


ACTIVITY:DATA ANALYSIS


30-40 minutes



30-40 minutes


GO!


QUESTIONS?

@zacknaylor

@Dave_L_Jones

Fire away...

or

Keep the party goin’...


THANKS


defining user research methodologies: a pragmatic approach

Design

user research methodologies

notmarketing research

thismarketing research

research planningmonday

user researchis

guides us

website doesnt

whatinforms design decisions