remote testing methods & tools webinar
DESCRIPTION
On Tuesday, December 13th, Bill Albert, co-author of Beyond the Usability Lab, and one of the most knowledgable experts in Remote Usability Testing, presented an update on Remote User Testing methods and tools to optimize your UX Roadmap. If you were not able to attend the live webinar please feel free to view the slideshare below! The webinar was co-hosted with 3 special guests: Dave Garr, Co-Founder, UserTesting.com Andrew Mayfield, CEO, Optimal Workshop Matt Paulus, Director of US Sales, UserZoomTRANSCRIPT
![Page 1: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/1.jpg)
Webinar:
Remote Testing Methods & Tools
December 13th, 2011
Bill Albert, Director of Design & Usability Center at Bentley University presents a complimentary webinar on Remote Testing Methods and Tools.
With special guests:
![Page 2: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/2.jpg)
2
Webinar Overview
1. Brief introductions & goal of the webinar
2. Updated overview of competitive landscape, updated figures, etc
3. The different methods / categories / tools. When to use what?
4. Types of research goals
1. Categories of tools available
2. Overview of some of those tools
5. Examples:
– Usertesting.com
– Optimal Workshop
– UserZoom
6. Takeaways, Q&A
![Page 3: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/3.jpg)
3
• Speakers today:o Bill Albert, Director of Design & Usability Center, Bentley Universityo Dave Garr, Co-Founder, UserTesting.como Andrew Mayfield, CEO, Optimal Workshopo Matt Paulus, Director of US Sales, UserZoom
• Goals of today’s session
Introduction
![Page 4: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/4.jpg)
[email protected] 781.891.2500 | www.bentley.edu/usability
Bill AlbertDecember 13, 2011
Remote Testing Methods and Tools
![Page 5: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/5.jpg)
5
• Introduction to remote usability testing
• Quantitative-based tools
• Qualitative-based tools
Agenda
![Page 6: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/6.jpg)
6
• Director, Design and Usability Center, Bentley University• Experience with remote testing tools• I am data driven!
About me
![Page 7: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/7.jpg)
7
I am a big fan of usability testing in the lab!
![Page 8: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/8.jpg)
Introduction to remote usability testing
![Page 9: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/9.jpg)
9
What do we mean by “remote testing tools”?• A remote testing tool is any technology that allows a researcher to collect
data from users about their experience in using an interface, without direct, face-to-face contact.
There are two ways of organizing remote testing tools• Moderated vs. Unmoderated• Qualitative vs. Quantitative-based
Defining Remote Testing Tools
Moderated Unmoderated
Qualitative-based Qualitative-based
Quantitative-based Quantitative-based
![Page 10: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/10.jpg)
10
Unmoderated Testing Tools
Unmoderated tools
Qualitative-based
Video tools
Reporting
Expert Reviews
Quantitative-based
Full-service
Self-service
Card Sorting/IA
Surveys
Click/Mouse
![Page 11: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/11.jpg)
11
Our UX Toolkit
![Page 12: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/12.jpg)
12
• Saves timeo Lab study takes 2-4 weeks from start to finish, unmoderated typically takes
hours to a few days*
• Saves moneyo Participants compensation typically a lot less ($10 vs. $100)o Tools are becoming very inexpensive
• Reliable metricso Only (reasonable) way to collect UX data from large sample sizes
• Geography is not a limitationo Collect feedback from customers all over the world
• Greater Customer insighto Richest dataset about the customer experience
Why Should You Care?
![Page 13: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/13.jpg)
13
• Questions from senior management are becoming more complex and time sensitive
• Traditional usability is no longer enough
• Push to measure the UX
• Convergence with market research and web analytics to paint a more complete picture of the UX
• Budgets/timeline constraints
Why is this important NOW?
![Page 14: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/14.jpg)
14
Which one goes first?
Lab first, then Unmoderated Unmoderated first, then Lab
Identify/fix “low hanging fruit”, then focus on remaining tasks with large sample size
Identify the most significant issues online through metrics, then use lab study to gather deeper qualitative understanding of those issues
Generate new concepts, ideas, questions through lab testing, then test/validate online
Collect video clips or more quotes of users to help bring metrics to life
Validate attitudes/preferences observed in lab testing
Gather all the metrics to validate design – if it tests well, then no need to bring users into the lab
![Page 15: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/15.jpg)
Quantitative-based tools
![Page 16: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/16.jpg)
16
Unmoderated Testing Tools - Quantitative
Unmoderated tools
Qualitative-based
Video tools
Reporting
Expert Reviews
Quantitative-based
Full-service
Self-service
Card Sorting/IA
Surveys
Click/Mouse
![Page 17: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/17.jpg)
17
Common Research Questions:• What are the usability issues, and how big?• Which design is better, and by how much?• How do customer segments differ?• What are user design preferences? • Is the new design better than the old design?• Where are users most likely to abandon a transaction?
Types of Studies:• Comprehensive evaluation• UX benchmark• Competitive evaluation• Live site vs. prototype comparison• Feature/function test• Discovery
Overview
Typical Metrics:• Task success• Task time• Self-report ratings such as ease of
use, confidence, satisfaction• SUS• Click paths• Abandonment
![Page 18: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/18.jpg)
18
Strengths:• Comparing products• Measuring user experience• Finding the right participants• Focusing on design improvements• Insight into the user’s real
experience
Limitations:• Not well suited to rapid, iterative
design• Need a deep understanding of
issues• Studies that require long sessions• Lose control over prototypes• Internet access
Strengths / Limitations / Myths
Myths:• Only test with websites• Very expensive• Only gather quantitative data• A lot of noise in the data• Same as any market research study• Comparing products
![Page 19: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/19.jpg)
19
Full Service Tools – Online Usability Testing
![Page 20: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/20.jpg)
20
Self Service Tools – Online Usability Testing
![Page 21: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/21.jpg)
21
Card Sorting / IA Tools
![Page 22: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/22.jpg)
22
Online Surveys
![Page 23: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/23.jpg)
23
Click and Mouse Tools
![Page 24: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/24.jpg)
Qualitative-based tools
![Page 25: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/25.jpg)
25
Unmoderated Testing Tools - Qualitative
Unmoderated tools
Qualitative-based
Video tools
Reporting
Expert Reviews
Quantitative-based
Full-service
Self-service
Card Sorting/IA
Surveys
Click/Mouse
![Page 26: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/26.jpg)
26
OverviewCommon Research Questions:• What are the big pain usability pain points, and
wins?• How do users think, and feel about the design?• Are we on the right (design) track? • What is the overall experience like? • Why might users abandon a transaction?
Types of Studies:• Low fidelity prototype• High fidelity prototype• Competitive evaluation• Comprehensive evaluation• Feature/function test• Discovery
Common Metrics:• Frequency of issues• Verbatim comments• Task success (via video)• Task time (via video)
![Page 27: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/27.jpg)
27
Video Tools
![Page 28: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/28.jpg)
28
Report Tools
![Page 29: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/29.jpg)
29
Expert Review
![Page 30: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/30.jpg)
30
Online:www.measuringUX.comwww.measuringusability.comhttp://usability.bentley.eduhttp://www.usefulusability.com/24-usability-testing-tools/http://remoteresear.ch/tools/
Follow me:@UXMetrics@BentleyDUC
LinkedIn Group:Bentley Design and Usability Center
Additional Resources
![Page 31: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/31.jpg)
Dave GarrCo-Founder
UserTesting.com
![Page 32: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/32.jpg)
In Cyberspace, no one can hear your visitor scream.
What’s your visitor thinking?
![Page 33: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/33.jpg)
33
Solution?
![Page 34: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/34.jpg)
How It Works
![Page 35: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/35.jpg)
“I don’t know why it keeps doing this.This is definitely very confusing and frustrating.”
![Page 36: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/36.jpg)
Write an Annotation
![Page 37: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/37.jpg)
“I can’t figure out which cartridge goes with my printer”
![Page 38: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/38.jpg)
How to share clips
![Page 39: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/39.jpg)
View Tester’s Answers
![Page 40: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/40.jpg)
Follow up with users
![Page 41: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/41.jpg)
Export to Excel
![Page 42: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/42.jpg)
42
Mobile Testing
• Test mobile apps
• Test mobile-friendly sites
![Page 43: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/43.jpg)
43
What do you want to test?
“I’d like to look at these three star reviews, … it doesn’t let me do that… I’d have to go through all of the
reviews.”
![Page 44: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/44.jpg)
44
Presenting Your Findings
![Page 45: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/45.jpg)
Andrew MayfieldCEO
Optimal Workshop
![Page 46: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/46.jpg)
Quantitative analysis to assess the
effectiveness of an Information
Architecture is
hard work.
![Page 47: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/47.jpg)
Peter Morville’sUX Honeycomb
![Page 48: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/48.jpg)
Optimal Workshop
![Page 49: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/49.jpg)
The Suite
by Optimal Workshop
![Page 50: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/50.jpg)
Qualitativevs
Quantitative
www.optimalworkshop.com/blogSee latest blog post on this topic
![Page 51: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/51.jpg)
![Page 52: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/52.jpg)
![Page 53: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/53.jpg)
Par
ticip
ant
Cen
tric
Ana
lysi
s
![Page 54: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/54.jpg)
![Page 55: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/55.jpg)
Safe orexistingoptions
Radicaloption
?
BENCHMARK
Remote studies enable near unlimitediteration and participation
![Page 56: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/56.jpg)
Matt PaulusDirector of US Sales
UserZoom
![Page 57: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/57.jpg)
UserZoom’s Software Solutions
• On-demand, web-based SaaS
• 100% self-service
• All-in-one quantitative unmoderated online research
• Enterprise level software
• Feature rich & intuitive
![Page 58: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/58.jpg)
• Quantitative remote testing
• Natural context
• International/domestic testing
• No moderation needed
• Behavior (analytics) + Feedback (online Surveys)
Unmoderated Remote Testing Methodology
![Page 59: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/59.jpg)
UserZoom Products
![Page 60: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/60.jpg)
UserZoom Products
60
Live Websites Wireframes & Prototypes
Multimedia content
Mobile Interfaces & Apps
Compatible Platforms
![Page 61: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/61.jpg)
UserZoom Task-Based Usability Testing
![Page 62: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/62.jpg)
UserZoom Results – Open Ended/Text Clouds
![Page 63: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/63.jpg)
UserZoom Results – Net Promotor Score
![Page 64: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/64.jpg)
UserZoom Results – Filters/Segmentation
![Page 65: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/65.jpg)
UserZoom Results – Effectiveness & Efficiency
![Page 66: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/66.jpg)
UserZoom Results – Clickstreams
![Page 67: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/67.jpg)
UserZoom Results- Click Heatmaps
![Page 68: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/68.jpg)
68
• Remote testing tools are becoming more advanced every day, and are a critical part of the UX toolkit
• Find the right type of tool based on your research question
• Demo a select number of remote testing tools to find the one that will deliver the results you need, within budget and timeframe
Key Takeaways
![Page 69: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/69.jpg)
69
Quick Poll!
![Page 70: Remote Testing Methods & Tools Webinar](https://reader033.vdocuments.mx/reader033/viewer/2022061300/54c779804a79590c788b4576/html5/thumbnails/70.jpg)
Thank You!
Floor open to Q&A
Bill AlbertDirector of Design & Usability CenterBentley University
Dave GarrCo-Founder
UserTesting.com
Andrew MayfieldCEO
Optimal Workshop
Matt PaulusDirector of US
SalesUserZoom