beyond usability testing: getting a holistic view of the user experience
DESCRIPTION
Talk given to Atlanta Web Design Group, August 2013TRANSCRIPT
+Beyond usability testingGetting a holistic view of the user experience
Amanda Nance, User experience researcher
Atlanta Web Design GroupAug. 29, 2013
+About me
In UX field since 2002
Studied human-centered communication & psychology, Mercer University
M.S. in Human-computer interaction, Georgia Tech
Women’s barbershop quartet
@amandaux #beyondutest
+Context
Large, international corporation (14,000 employees)
Accounting products for small businesses Desktop and web-based No analytics
My product Product Management team - 5 Developers - 16 Dev Operations - 5 QA - 6 Interaction & visual designers - 3 Writers - 2 part-time Researcher - 1
+Context
31 studies since Jan 2010
120+ users
+About you
+My goal
Broaden your toolkit for including users
Convince you to start right away
+Usability testing is great. Try it!
See “Rocket Surgery Made Easy” by Steve Krug
+An ideal usability test would tell you everything, but…
Ideal In practice
Real users yes!
Identify unknown needs
Natural environment
User's own data
Real motivation debatable
Non-core tasks
Major usability problems apparent yes!
Minor usability problems apparent yes!
Quantitative information sometimes
+So we have these other methods
+
Field visits
+Watch and interview users in their natural habit
+Use field visits before designing to learn what people need
Before designing can mean: Before anything Before finalizing the design After launch to improve product
+
Used under CC licenseLeah Gregg
Asking users about problems limits the problems you’ll uncover
+The problem we solved was one users hadn’t thought about
“I hadn’t even thought that far out of the box.”-User reaction to our product concept
+Observation gives you details about what people need
+Use field visits for learning about sensitive topics
+Tips: Before the visit
Ask about pets
Talk to the participant yourself before the visit Even if using an external recruiter Ask to save up some work Tell them not to clean up!
Ask for driving directions / verify Google Maps accuracy
+Tips: During the visit
Say “Show me” and ask for specific examples
+Tips: During the visit
Verify your understanding in the moment
Always respond professionally
+Field visits: general guidelines
We pay $300 for 3 hours
Limit is 3 hours
Can divide into one-hour sessions amongst 3 users
+
Informal interviewsCalls & emails, screen sharing
+Use informal interviews for (perceived) requirements
Email and phone calls: easy and quick!
+Use screen sharing interviews to learn how the product is (really) used
We didn’t account for users who repeatedly subcontract for the same company.
+Screen sharing tips: recruiting
Find customers who are interested in the topic Analytics Survey responses Email
Explain logistics: ear buds or speakerphone required
Get participant’s phone number
+Screen sharing tips: tools
Join.Me - recommended by colleague
+Screen sharing tips: tools
Turn off the beeps when people join
Turn off chat for observers
Turn off list of meeting attendees
+Screen sharing tips: during the session
“Can you show me…” the issues you’re having how you _______
Help with issues at end
In case of emergency: let participant control your screen
+Screen sharing: general guidelines
We pay $75/hour
$25 for 20-30 min
Limit to 1 hour
+
Live recruiting testAKA intercept test
+Live recruit test = watching more natural behavior
CC licensehttp://www.flickr.com/photos/evilpeacock/6423510787/
+Use a live recruit test when motivation is key
Pretend that you are looking for bookkeeping software for your company…
+Live recruiting tips: Ethnio
Use Ethnio to recruit
Read all demographics (unqualified people are listed)
Answers may reveal unclear screener questions
Pause the recruiting if you leave
+Live recruiting tips
Decide how to handle people who have already started using the site
Have sales or relevant phone numbers handy
See www.remoteresear.ch for templates
+
Surveys
+Surveys
Questionnaire completed by users to learn about demographics, opinions, etc.
Use when you need quantitative data
+TipsFOCUS Keep it short
CC licensehttp://www.flickr.com/photos/44456430@N04/6825380102/
+A goal-less survey
1. How did you first become aware of Awesome Accounting?
2. To the best of your knowledge, what does Awesome Accounting do?
3. To the best of your knowledge, what is the price of Awesome Accounting?
4. What software were you using in the running of your business when you first considered Awesome Accounting?
5. What software were you looking to replace when you first considered Awesome Accounting, if any?
6. Why did you first consider Awesome Accounting?
7. Before subscribing to Awesome Accounting, did you do a free trial?
8. Are you currently using Awesome Accounting in running your business?
9. Which of the following parts of Awesome Accounting have you used, or do you intend to use, in running your business?
10. How satisfied are you with the following aspects of Awesome Accounting?
11. Please use the spaces below to comment on each of the aspects of Awesome Accounting.
12. How likely are you to continue to be a Awesome Accounting subscriber 12 months from now?
Used with CC permissionhttp://www.flickr.com/photos/stovak/
+Tip: Survey process
Ask questions that people can accurately answer What did you eat for breakfast 4 days ago? vs. Do you typically organize your emails into folders?
Include an “N/A” option
Avoid mandatory questions
Ask permission to contact for follow-up questions
Pilot test your survey
+
Diary studies
+Users record activities, problems, or feelings over time
CC license http://www.flickr.com/photos/bitchcakes/5457865373/sizes/o/in/photostream/
+Diary studies: when to use
Seeing experience over time Researching and buying a car
Infrequent tasks / non-core tasks
To identify places to dig deeper with other studies
+Diary studies will show you the bigger issues
+Diary studies: Tips
Reserve time to follow up on unclear entries
Interact with participants to maintain engagement
Continue participants who are useful
Ask for screenshots
+Diary studies: general guidelines
Compensation At least $25/week Can increase over time
Length: Can keep people engaged for several weeks if not months
Recruit: At least 10 More depending on your goals
+
General research tips
+General research tips: planning and logistics
Start with goals
Pilot
Consider recording with permission
Have someone else take notes if possible
+General research tips: facilitating
Ask open-ended questions
Ask neutral questions
Probe for more information
Ask for an example
Be genuine
Show that you are on the user’s side
+
How to get started
+What is keeping you from getting started?
+Getting started
Start lean and build up
Ask forgiveness, not permission from stakeholders
Find users who will help for free
Get your stakeholders involved
Don’t worry about being a research expert
+
More infoBooks, links, and stuff
+Quick read to get started
UX for Lean Startups: Faster, Smarter User Experience Research and Designby Laura Klein
Short. No UX background required.
+General references on qualitative research
Observing the User Experience, Second EditionElizabeth Goodman, Mike Kuniavsky, Andrea Moed
500+ pages. Each method = 1 chapter
Understanding Your Users: A practical guide to user requirementsby Catherine Courage & Kathy Baxter
700+ pages. Each method = 1 chapter.
+On field visits
User and task analysis by Hackos and Redish
Contextual Design by Beyer and Holtzblatt
+Live recruit testing and remote research
Remote Research: Real Users, Real Time, Real Researchby Nate Bolt, Tony Tulathimutte
The general reference books I mentioned don’t seem to cover this topic.
Look at www.remoteresear.ch for templates
+On interviewing
Steve Portigal’s “Interviewing users”
+On surveys
Coming in 2014: “Surveys that work” by Caroline Jarrett
Caroline’s podcast with UIE on surveys
