choosing the right research methods for your project (webinar)
DESCRIPTION
It’s very easy for User Experience researchers to get stuck in the rut of using your favorite research methods for gathering information and getting user feedback. But, are you really gathering the best information that you can? Or are there other methods that are better suited for your project’s specific needs? Or, if you’re just starting out – how do you know whether you should conduct interviews, run a survey or a card sort, or something different all together? Don’t stress – in this webinar, we’ll cover the most popular user research methods and discuss their strengths and weaknesses. Each method shines in different circumstances, and we’ll highlight the factors that will make each successful. We will also present a structured approach to helping you choose the best method or methods for a particular situation.TRANSCRIPT
Susan Mercer – Senior Experience [email protected]@susanamercer
October 29, 2013
Choosing The Right Research Methods For Your Project
Hello, I’m Susan Mercer
BA and MSc in Geophysics
19 years in software and web UI and UX design Developer
Designer
Web Producer
Product Manager
Researcher
MS Human Factors, Bentley University
Twitter: @susanamercer
2
Introduction
“It is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.”
– Abraham Harold Maslow
3
Agenda: Market Research & User Research
Study Goals
User Research Methods
Choosing the Right Methods
Marketing Research & User Research
4
Marketing Research User Research
Preferences
Attitudes
Purchasing Behaviors
Use Behaviors
Context of Use
Actions
Usefulness
Emotional Reactions
Engagement
STUDY GOALS
5
Defining Goals
“The most serious mistakes are not being made as a result of wrong answers. The truly dangerous thing is asking the wrong questions.”
– Peter Drucker
6
Defining Goals
“The most serious mistakes are not being made as a result of wrong answers. The truly dangerous thing is asking the wrong questions.”
– Peter Drucker
7
The first key step of a good study is to determine your goals
Good Study Goals: Provide focus to the study
Become more specific as the project progresses
Focus on user behavior, not on interface specifics
Help drive actionable results to impact business or design decisions
Defining Goals
Consider: Where are you in the project?
Identifying problems with current behaviors to identify opportunities?
Evaluating the appetite for a product feature?
Gathering feedback on an early design?
Testing an existing product?
What are you trying to learn?
Is there an existing artifact to test?
How much time do you have?
How much of a budget do you have?
8
USER RESEARCH METHODS
9
User Research Methods
Generative
Interviews
Ethnography
Collaging
Surveys
Card Sorting
Diary Studies
Focus Groups
10
Evaluative
In-Person Usability
Remote Usability
Unmoderated Usability
Desirability Testing
Usefulness Testing
Surveys
Information Architecture Testing
Eye-Tracking
Biometrics
USER INTERVIEWS
11
User Interviews
What is a User Interview:
A focused one-on-one conversation
In-person or via telephone
Familiar format for both participant and interviewer
12
User Interviews
How to ask good interview questions:
Open-Ended Questions Who, What, When, Where, Why, How
Start with broad questions, then use prompts to fill in gaps Let the participant guide the discussion at first to what they think is important
Example: Tell me about sharing the printer and copier with others.
What works well?
What doesn’t work?
Do you need an ID, card, or code to use the shared printer/copier?
Does your printer keep track of how many copies/dollars you have left on your “budget”?
Do you know how much it costs per page to print?
Is your company actively trying to manage printing costs? If so, how?
13
User Interviews
14
Pros Cons
Moderator can ask follow on questions to focus on topics of interest
Qualitative data takes longer to analyze than quantitative
Gather rich, detailed data Questions may not completely consistent from participant to participant
Good for exploratory, formative research
Need to manage talkative and quiet participants
Small sample size yields effective data Not as efficient when there are many different demographics with truly different needs
Can be done remotely via telephone
Relatively inexpensive to conduct
User Interviews
Additional Resources:
Understanding Your Users: A practical guide to user requirements. Catherine Courage & Kathy Baxter, Morgan Kaufmann, 2005. Chapter 7
Interviewing Users: How to uncover compelling insights. Steve Portigal, Rosenfeld Media, 2013.
Mental Models: Aligning Design Strategy with Human Behavior. Indi Young, Rosenfeld Media, 2008. Chapter 7
15
ETHNOGRAPHY
16
Ethnography
What is ethnography?
Observing users in their natural environment while they perform the tasks of interest
Big “E” Ethnography “A research strategy that allows researchers to explore and examine the cultures and
societies that are a fundamental part of the human experience” (Murchison)
Generally involves immersion of the researcher within the culture to be studied
Longitudinal and can take many years
Little “e” ethnography UX Researchers perform little “e” ethnography
Typically involves a period of observation followed by an interview
Focuses on how users perform the tasks of interest and their overall environment
17
Ethnography
Good to examine:
What information users look for and where they find it
Who and what they interact with to get the task done
The order in which things happen, and whether that is important
Distractions, barriers, and interruptions that the participants encounter
18
Ethnography
19
Pros Cons
See interactions with other people and other products/systems
Logistics of setting up observations/interviews in someone’s home/office
Users will interact with their own items, including cheat sheets or other “coping mechanisms”
May be difficult to gain access to individuals or certain environments, or for extended periods of time
Good to understand how things are really done vs. how they are supposed to be done
Observing/interviewing many people requires more time
Can yield insightful “aha” moments and innovative ideas
On-site recording logistics can be challenging
Observe natural interruptions and other events which participants forget to report in interviews
Ethnography
Additional Resources:
Ethnography in UX, Nathanael Boehm, uxmatters.com, http://www.uxmatters.com/mt/archives/2010/06/ethnography-in-ux.php
Practical Ethnography: A guide to doing ethnography in the private sector, Sam Ladner, Expected soon. http://www.practicalethnography.com/
Ethnography Essentials: Designing, conducting, and presenting your research, Julian M. Murchison. Josey-Bass, 2010.
20
COLLAGING
21
Collaging
What is Collaging:
A creative activity to help uncover thoughts and attitudes.
Participants are given photographs, large paper and art supplies and asked to create a collage about a central theme. They then explain their collage.
Can be done one-on-one or as a focus group activity
Can make some participants conscience of their lack of creative skills
Their storytelling about the college is the key
Followed by a short interview
22
Collaging
How to do it:
1. Prepare materials
2. Define focus
3. Create the collage
4. Have them tell their story
23
What does “Saving Energy” mean to you?
Collaging
24
Pros Cons
Gather rich, detailed data Qualitative data takes longer to analyze
Small sample size Questions are not consistent from participant to participant
Good for exploratory research – when you don’t know the questions to ask
Some participants are not as open to creative exercises as others
Good for uncovering values, motivations, attitudes that are not top-of-mind – participants may not even be consciously aware of them
Preparation can be time-consuming
Good seeding activity for focus groups
Can help some participants verbalize thoughts
Collaging
Additional Resources:
Collaging: Getting answers to the questions you don’t know to ask. Kyle Soucy, Smashing Magazine, February 6, 2012.
http://uxdesign.smashingmagazine.com/2012/02/06/collaging-getting-answers-questions-you-dont-know-ask/
McKay, D., Cunningham, S. J., Thomson, K. Exploring the user experience through collage. CHINZ ’06 Proceedings of the 7th ACM SIGCHI New Zealand’s chapter’s international conference on Computer-human interaction: design centered HCI, p. 109-115.
25
DIARY STUDIES
26
Diary Studies
What is a Diary Study:
A semi-longitudinal study used gather user behavior over time
Users are asked to keep a diary for a certain topic When they interact with a device/website/etc.
Health, food, study habits, banking habits, etc.
When to use a diary study:
When a single research session will not truly capture users’ interactions
When the users’ environment plays a role in how they use the interface
If there is plenty of time in the project timeline
27
Diary Studies
28
Pros Cons
Captures users’ data in their natural setting
Reliant on participants keeping up with their diaries
Users will be interacting with their own items
Participants self-select content
Can capture data over timeManagement of the data collection mechanism
Must have time in the project timeline
Diary Studies
Additional Resources:
Observing the User Experience: A Practitioner’s Guide to User Research, Elizabeth Goodman, Mike Kuniavsky, & Andrea Moed, Morgan Kaufman, 2012.
Diary Studies in HCI & Psychology, Demetrios Karis, UPA Boston 2011 Conference. http://www.slideshare.net/UPABoston/diary-studies-in-hci-psychology
29
User Research Methods
Generative
Interviews
Ethnography
Collaging
Surveys
Card Sorting
Diary Studies
Focus Groups
30
Evaluative
In-Person Usability
Remote Usability
Unmoderated Usability
Desirability Testing
Usefulness Testing
Surveys
Information Architecture Testing
Eye-Tracking
Biometrics
EVALUATIVE METHODS
31
Evaluative Testing
Usable
UsefulEngaging
32
Usability Testing
Usefulness TestingDesirability Testing
DESIRABILITY TESTING
33
Desirability Testing
What is Desirability Testing?
A set of tools for exploring participants’ emotional reactions to different visual designs.
Use this method to:
Select a visual design which most closely aligns with desired traits
Evaluate a product’s visual design vs. competition to uncover differences in brand perceptions or user experience
Evaluate a single design for emotional response
34
Desirability Testing
Open-Ended Questions:
Ask for initial reactions Show a design to a participant and ask for their immediate reactions
Let them tell you whatever they want to tell you
Sentence Completions: Start 5-10 sentences and ask them to finish them based on their reactions to the
design
Product Reaction Words: Show them ~ 60 words – both positive and negative connotations
Ask them to select a small number which best represent the design
35
Sentence Completion
Have participants complete sentences
Gather focused feedback without being leading
Examples: This page makes me feel…
Reading information on this page is…
If I used this website all day, I would…
I want to use this website because…
I do not want to use this website because…
36
Product Reaction Words
37
Accessible Dated Hard to Use Simplistic
Appealing Dull Helpful Sophisticated
Approachable Easy to Use Impersonal Sterile
Boring Effective Innovative Stimulating
Busy Efficient Inspiring Straight Forward
Clean Energetic Intimidating Stressful
Clear Engaging Intuitive Time-consuming
Comfortable Enthusiastic Inviting Time-saving
Compelling Exciting Motivating Too Technical
Complex Familiar Organized Trustworthy
Confident Fast Overwhelming Understandable
Confusing Flexible Patronizing Usable
Convenient Fresh Predictable Useful
Creative Friendly Professional Valuable
Cutting Edge Frustrating Reliable
Product Reaction Words
Word Cloud: http://www.wordle.net
38
Product Reaction Words
Existing Design
39
Customers Customers & Non-Customers Non-Customers
Reliable Accessible Appealing Confusing
Comfortable Organized Professional Busy
Confident Straight Forward Approachable Complex
Effective Understandable Efficient Dull
Friendly Easy to Use Flexible Hard to Use
Inviting Clear Helpful Impersonal
Time-saving Convenient Overwhelming
Trustworthy Clean Boring
Useful Familiar Frustrating
Simplistic Intimidating
Valuable Stressful
Usable Time-consuming
Product Reaction Words
Number of positive vs. negative words
40
Existing Design A Design B Design C0
10
20
30
40
50
60
70
Number of Words
Positive Negative
Product Reaction Words
Words that the client desires the look and feel to portray
41
Existing Design A Design B Design C
Appealing Appealing Appealing Appealing
Clean Clean Clean Clean
Efficient Efficient Efficient Efficient
Engaging Engaging Engaging
Intuitive
Professional Professional Professional
Useful Useful Useful Useful
Total 9 20 16 9
Desirability Testing
42
Pros Cons
Gather qualitative feedback on emotional response to designs
Choice of adjectives could introduce subtle bias
Gather focused feedback in a neutral, open-ended fashion
Generally, relatively small sample size
Product reaction words provides some level of quantitative measurement, but should be used carefully
Qualitative feedback can take longer to analyze
Desirability Testing
Additional Resources:
Benedeck, J. & Miner, T. (2002) Measuring Desirability: New methods for evaluating desirability in a usability lab setting. Proceedings of Usability Professionals Association, 2003, 8-12.
http://www.pagepipe.com/pdf/microsoft-desirability.pdf
Rapid Desirability Testing: A Case Study, Michael Hawley, uxmatters.com, http://www.uxmatters.com/mt/archives/2010/02/rapid-desirability-testing-a-case-study.php
Users play cards We keep score Magic Results, presentation by Carol Barnum and Laura Palmer
http://www.slideshare.net/cbarnum/barnum-palmer-stc-2011-presentation
Kuhala, S. & Nurkka, P. (2012) Sentence Completion for Evaluating Symbolic Meaning. International Journal of Design, 6(3), 15-25.http://www.ijdesign.org/ojs/index.php/IJDesign/article/view/1166/523
43
USEFULNESS TESTING
44
Usefulness Testing
What is Usefulness Testing?
Usability testing identifies issues that prevent users from performing tasks
Usefulness testing evaluates the match between user needs and system features
1. Investigate how users do tasks today
2. Have them use the system or prototype
3. Ask questions about usefulness
45
Usefulness Testing
1. Background Questions:
Investigate how users do things today – what works well and what doesn’t
Sample Questions:
Tell me about your process for ordering equipment and supplies from XYZ Co. What do you typically need to order?
How do you determine whom to contact?
How often do you order supplies?
What happened last time you felt your resolve to eat well sagged? What sources of support did you rely on?
How did they help you?
How did you get “back on track”?
46
Usefulness Testing
2. Show them the product / prototype
Watch for:
Comments about usefulness “Wow. This would really make it easier to reorder the same things.”
“I don’t like this. It would take me too long to do this part.”
Neutral reactions “Eh. This is OK.”
Probe: “Might this help you get your work done? Or not?”
Be wary of “This isn’t how we do it today” comments Business processes may need to change in the future, so try to find out why something
would not work.
47
Usefulness Testing
3. Follow up Questions:
After they have seen the system/prototype, how do they think it might work for them?
Sample Questions:
What are your overall impressions of the website?
Based on what you saw today, how helpful would the system be to you? Why?
What could we do to make this more helpful to you?
48
Usefulness Testing
Additional Resources:
Hawley, M. & Berlin, D. Beyond Usability Testing: Assessing the Usefulness of Your Design. UPA Boston Conference, 2011.
49
CHOOSING THE RIGHT METHODS
50
How to Choose the Right Method(s)
1. Make sure your research goals are clear Know what you want to learn
Know what you will do with the information you gather
Know what decisions the business needs to make
2. Know your constraints Timeline?
Resources?
Budget?
Access to Users?
3. Create a Methods Chart The answers will become clear
51
Methods Chart
52
Method Pros Cons Candidate
Method 1 • List the advantages for this method for this specific project
• List the disadvantages for this method for this specific project
Yes / No – Would this method be a good candidate for this specific project
Method 2
Method 3
Etc…
Methods Chart Example
53
Method Pros Cons Candidate
Interviews • Good for capturing motivations for behavior change
• Can get good qualitative details on why users prefer each concept or not
• Good to explore issues to fine tune survey questions
• Can be done via phone (get broad geographic sample)
• Can be done quickly
• Small numbers – client wants large numbers
Y – would be good as a qualitative method to pair with a larger quantitative method
Focus Groups • Good for qualitative information gathering
• Could generate some interesting conversations about behavior change motivations
• Wouldn’t get as much detail as interviews
• Concern about group think when evaluating concepts
• Concern about not sharing details of personal goals in front of others
N – interviews would be better for qualitative
Survey • Good for large numbers• Easily replicated across
different countries/languages
• Can be done online for broad geographic distribution
• Can be done quickly
• Unclear what exact questions to ask
• Doesn’t provide detailed insights into qualitative topics
Y – good paired with qualitative method
Want to learn more?
Attend our Workshop
Choosing The Right Methods For Your Research Project
Susan Mercer & Dan Berlin
Full Day Workshop, IA Summit – San Diego, CA
March 26, 2014
http://www.iasummit.org
Contact Us
54
Susan MercerSenior Experience Researcher
[email protected]@susanamercer
Dan BerlinExperience Researcher Director
[email protected]@banderlin
DISCUSSION
55