defining a product vision


Defining a Product Vision using

Remote Research Methodologies presented to

The 14th AnnualUsability Professionals’ Association

ConferenceJuly 1st, 2005

Mark SafireDirector of User Experience Research

Sachs Insights


Presentation• Who am I• The Products & The Vision

• The Company – NPD Group • The Software Products – Solution Folders & PowerView• The User Profiles• The Research Challenge• Remote Moderated Usability

» Definition; Benefits; Challenges» Audience Participation – Experiences with Remote Research

• The Transformation of PowerView through Research• Research Road Map• PowerView Research

» Phase 1: Remote Contextual Inquiry» Phase 2: Remote Moderated Usability» The Upshot

• Q&A


Who Am I?

• Mark Safire• Director of User Experience Research,

Sachs Insights» Qualitative research provider for NPD Group online/offline

software products since 2002.

• Spoken at UPA 2003 (Presentation) and UPA 2004 (Idea Market) on Remote Moderated Usability.

» Also at NYC and Dallas UPA chapters, and for Nielsen Norman Group conferences in Las Vegas and Amsterdam.

• 2003 Presentation is online at www.sachsinsights.com/services/remotemoderatedusability.html


Qualitative research consultancy founded in 1987

• Serve automotive, fashion, financial services, media, pharmaceutical, technology and travel industries.

Practice Areas Include: Brand & Advertising Research New Product & Service Development User Experience Research

In our User Experience practice, we conduct:• Concept & Process Focus Groups• Taxonomy & Nomenclature Focus Groups• Usability• Contextual Inquiry

Sachs Insights


The Company

• A leading provider of sales and marketing information based on combined point-of-sale and consumer data for over a dozen major industries.

» Apparel, appliances, automotive, beauty, cellular, consumer electronics, food and beverage, foodservice, footwear, housewares, imaging, information technology, movies, music, software, toys, travel, video games, ...*

» Supplemented by consumer panel data to give a more complete picture – demographics, etc.

» Data from “over 230 retail partners, representing over 90,000 doors worldwide”*

• Clients: Manufacturers, Retailers, Analysts• Quoted in media regularly

*Source: www.npd.com


The Software Products

• Goals of Software• Allow customers to analyze data that NPD has collected and

organized.

• Usage Scenarios• Monthly reports: Same reports, different numbers• Ad-hoc queries: Different reports each time

• Sample Queries for Reports • What is my market share vs. my competitors?• How are inkjet printer sales trending over the past 3 mo.’s?• What were the top ten products in $ sold last month?



• Solution Folders • PowerView



Solution Folders

Make Reports from Templates

PowerView

Make Reports from Scratch

FlexibilityLess

Less

Shorter

Little (Online)

Room for Error

Learning Curve

Setup Required

More

More

Longer

A Lot (Offline)



More Flexible

EasierTo Learn

Less Flexible

Harder To Learn

PowerView

SolutionFolders


The User Profiles – Ingoing Assumptions

• Disseminators• Market Research experts – the “gatekeepers”• Experienced in performing quantitative analysis• Pull, analyze, develop reports for others

• Users• Support personnel, bus. analysts & owners, sales & mktg.• Less experienced in performing quantitative analysis• Pull, analyze, developer reports for themselves

• Viewers• Bus. analysts & owners, sales & mktg. • Less experienced in performing quantitative analysis• Review reports created by others


Research Challenge – Finding Respondents

• Some had low incidence• “User” profile not yet a significant percentage.

• Geographical dispersion• Key retailers not clustered• Expensive and time consuming to fly them in

• 2nd Round of Research – Timing• Week before Christmas

Solution: Remote Moderated Usability


Usability in a Lab

Respondent Moderator Observer(s)

4


Remote Moderated Usability

Respondent Moderator Observer(s)

7


Remote Moderated Usability• Differences from In-Lab Usability

• Different Locations• Software Tool• Teleconference• Observers usually apart from Moderator

8


Remote Moderated Usability• Benefits

• Recruiting benefits» Reach geographically dispersed populations» Reach hard to recruit populations – don’t leave home/office

• Avoid cost and inconvenience of travel for all parties involved.

• Get “buy-in” from stakeholders in distant locations

9

NPD Research Conditions:• All had access to high-speed Internet connections• Participants & their companies were open to the technology• NPD Group had a WebEx license and in-house expertise,

used primarily for remote demos and help desk support


Remote Moderated Usability• Challenges

• Body Language

• Rapport• Interruptions (phone, visitors, email, IM, …)• Logistics (time zones, ...)• Security & Privacy • Observer dynamics• Broadband connection recommended• Not for newbies & less coordinated

10


Audience Participation

• Any Experiences? 1-2 Min. Summary:• Your Name, Title, & Company• Research Description: Industry, (Client),

Audience, & Objective• Why Remote Moderated Usability?• Challenges you faced & Tips for dealing

with them


Remote Moderated Usability Tools & Methods

• Many tools available• NPD done with WebEx Meeting Manager

• Others commercial apps/ASPs to consider:» IBM Lotus Web Conferencing (f.k.a. IBM Lotus Same Time)» Microsoft LiveMeeting (f.k.a. Placeware)» Citrix GoToMeeting (f.k.a. DesktopStreaming)» Macromedia Breeze

• Other free or open source solutions to consider:» Microsoft Messenger/MSN Messenger (f.k.a. Microsoft NetMeeting)» VNC (open source – various implementations, including RealVNC)

• Others kinds of tools» Full-time Remote Control Apps (See pcmag.com)

• pcAnywhere, GoToMyPC, Timbuktu, etc.

11


Research Roadmap

• Solution Folders• Initial Launch

» In-Lab Interviews

• Optimize Navigation» Remote Moderated

Usability

• PowerView• Needs Assessment

» Internal Focus Groups

» Remote Contextual Inquiry

• Prototype before Relaunch (version 5.0)

» Remote Moderated Usability


PowerView Research

Phase I

Remote Contextual Inquiry

• How do they use it?• They define & demo tasks.

• Does it meet actual needs?• See Role in Work Process

» Ultimate reports delivered

• How easy is it to use?• First Use / Experienced Use

• Things to change /Things NOT to change

Phase 2

Remote Moderated Usability

• Can they use it?• Use their pre-defined tasks

• Do fixes meet needs?• Role-play work process

• Observe ease of use.• First Use / Experienced Use

• Points of pain improved? / Have to relearn a lot?


Phase 1, Remote Contextual Inquiry (RCI)

• First time & “Ad Hoc use”• Even old-timers stumble when trying to create new reports.

• Drag & drop interface – good idea, don’t change!

• Inscrutable 8 character field names

• Hard to envision report layout from interface» Especially for “nested” reports

• Expert & Repeated use• Gargantuan monthly reports: Same thing, different numbers

» Each report is run individually, not as a “batch”» Typically formatted each time in Excel – not saved in tool.

• Often requires specific colors for company and competitors in for tables and charts.

• Extra columns created during export to show “suppression.”

Everyone: Look & Feel “dated” – want standard Windows conventions


Phase 2, Remote Moderated Usability (RMU)

• First time & “Ad Hoc use”• Drag & drop approach still used.• Comprehensible, full text, categorized field names

» But can change back to 8 char. abbrev’s in preferences...

• Preview of report layout

• Expert & Repeated use• Run as a batch

» But “Favorites” menuv – confusing metaphor

» Final: “MyReports”

• Save formatting • No extra columns!

» gray or red italicized numbers with asterisks


PowerView, Phase 1, RCI

2) Inscrutable 8 character field names, without any categorization

1) Drag & drop a good idea: don’t change!

3) Hard to envision report layout from interface

4) Look & Feel “dated” – not standard Windows conventions.



Widget Production Quarterly Trend

What they need

Q1 ’04 Q2 ’04 Q3 ’04 Q4 ‘04

Us 12.3 16.2 27.7 29.5

Them 32.0 28.1 20.1 18.2

Channel 70.1 89.3 76.2 88.4

0

10

20

30

40

50

60

70

80

90

Q1 ’04 Q2 ’04 Q3 ’04 Q4 ‘04

Us

Them

Channel



What they get

Q1 ’04 Q2 ’04 Q3 ’04 Q4 ‘04

Us 20.4 27.4 90 20.4

Them 30.6 38.6 34.6 31.6

Channel 45.9 46.9 45 43.9

5) Extra columns

6) Formatting not saved


PowerView, Phase 2: RMU1) Drag & drop not changed!

2) Full text, categorized field names

3) Preview of Report Layout

4) Overall Look & Feel is Contemporary


PowerView, Phase 2: RMU

5) No extra columns 6) Formatting can be saved


PowerView Version 5.0

• Validation• Changes in Prototype were well received.

• Tweaks• “My Favorites” was not an intuitive menu name

» Worked for Solution Folders, but not here. Since PowerView is not a Web application, the “My Favorites” web metaphor was not effective.

» Solution: Changed menu name to MyReports for launch.


The Upshot

More Flexible

EasierTo Learn

Less Flexible

Harder To Learn

PowerViewSolutionFolders


Contact Information

Mark Safire

Director of User Experience Research

Sachs Insights

200 Varick Street, Suite 500

New York, NY 10014

www.sachsinsights.com

msafire @sachsinsights.com

212-924-1600 x168

www.remoteusability.com

defining a product vision

Technology

npd research conditions

remote demos

company research description

different reports

powerview solution folders

developer reports

user experience practice

technology npd group