more research, more frequently: how to sell your stakeholders on research

RESEARCH

FREQUENTLY

I

oooooooooooooooooop

noooooooooooooooooomore

more

TUUUUUUUUUUUUUUUUUUUUUUVChris Avore @erova #confabMN

#WOCINTECH CHAT

rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr

• Why we should talk to our customers, users, clients, etc

• Best practices to conduct research

• Research deliverables

What we won’t be talking aboutINTRO

rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrWhat we won’t be talking aboutINTRO

RESEARCH

FREQUENTLY

I

oooooooooooooooooop

noooooooooooooooooomore

more

TUUUUUUUUUUUUUUUUUUUUUUVChris Avore @erova #confabMN

#WOCINTECH CHAT

During Investor Day, we also said that at NASDAQ our fundamental mission first and

foremost is to serve our customers. We accomplished this through intensive engagement and feedback with

these customers.

We also received much feedback from the early adopters, most of it

overwhelmingly positive, but this feedback loop is also helping us to develop and quickly

deploy innovative new features.

Bob Greifeld CEO, Nasdaq Inc.

NASDAQ INC Q1 2016 EARNINGS CONFERENCE CALL APRIL 27, 2016 8:00 AM ET

The first thing we have to do is listen to our customers, and in fact, great design …

starts with a really deep listening to the customer.

What are the biggest problems they have in doing their job today, or what do they wish they could do better, or where do they wish they

could take their relationships with their own clients?

We’ve found things that when we’ve shown it to the client, they said, oh, that’s exactly what I need, but I didn’t even know I needed it.

So that’s great design.

Adena Friedman COO & President, Nasdaq Inc.

OPENING REMARKS, PRODESIGN CONFERENCE JANUARY 30 2015

RESEARCH

FREQUENTLY

I

oooooooooooooooooop

noooooooooooooooooomore

more

TUUUUUUUUUUUUUUUUUUUUUUVChris Avore @erova #confabMN

#WOCINTECH CHAT

SURVEY SAYS…

SURVEY RESULTS rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

Location RoleEmployer Team size

SURVEY RESULTS rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

Do you have access to artifacts, transcripts, and other assets from any research activities?

SURVEY RESULTS rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

How frequently is research included in your process or projects?

SURVEY RESULTS rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

Select the barriers that prevent you from conducting research:

SURVEY RESULTS rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrsDo you have a governance strategy in place for research outcomes, assets, artifacts, and transcripts?

LAGGARD MATUREPROGRESSING

Exec Attitude

EARLY

• Limited to user feedback • Market research

• Ad-hoc • Late-stage testing existing products w/

clients • User/advisory groups • Log analysis • Limited to one business or product

• Discovery research w/ client base • Consistent, iterative usability testing • Monitored Beta & Pilot programs • Comprehensive analytics & data collection • Participatory design w/ clients, users, etc

• Discovery research w/ clients, non-clients • Consistent evaluation of current products • Explores new markets, audiences • Multi-channel touchpoints • Spans across organization • Not limited to functionality: includes customer

experience, recruiting & onboarding, etc

• Find what customers don’t like • Usability test to predict the future (CYA) • Confirm current product decisions are sound • Establish shared understanding of customers

and markets by product team

• Prioritize where to focus new innovations • Inform customer/market segmentation • Learn where to adjust product strategy • Identify unmet needs

• No dedicated research team • Product management or marketing

interacts with clients • May hire perception/market research

firm

• Designers lead guerrilla usability tests • May outsource research to agency

• Dedicated research person or team within organization

• Product management & design support the research team

• Management level position to lead alongside product, marketing, commercial etc

• Resources available for non-research teams to conduct research, report findings

• Individual product team • Product management • Marketing

• Unconvinced • Cautious • Enabling • Expecting

• Product management • Marketing • Design • Engineering

• Research processes are managed, consistent • Follow best-practices • Findings are available for wider teams

• Little pre-research planning • Reports are emailed to stakeholders • Little awareness of best practices

• none • Decisions are made consulting existing design research or create need for more research

• Strategy, tactics, findings are consistently applied across projects, orgs, businesses

• C-suite (CEO, CMO, etc) • Commercial leaders • Corporate Strategy • External partners

What does research cover?

What are some of the methods that are being used?

How far does research reach in the organization?

Why is the organization investing in design research?

Who makes up the research team in the organization?

Who uses the findings that come from design research?

What are the policies to how research will be conducted and consistently applied across teams, projects, and the organization?

+

+

+

+

+

Scope

Staffing

Purpose

Governance

Audience

WHAT DO WE DO AS DESIGNERS THAT PREVENT US FROM DOING ?MORE RESEARCH

MORE FREQUENTLY

NO CLEAR RESEARCH STRATEGY

NO CLEAR RESEARCH STRATEGY

ASSUME ONLY WE CAN DO THE RESEARCH

ASSUME ONLY WE CAN DO THE RESEARCH

ASSUME ONLY NEW RESEARCH WILL TELL US WHAT WE NEED

TO KONW

ASSUME ONLY NEW RESEARCH WILL TELL US WHAT WE NEED

TO KNOW

DOGMADOGMA

SELF-SERVING PRIORITIESSELF-SERVING PRIORITIES

NO USEFUL CONCLUSIONNO USEFUL CONCLUSION

IGNORE WHAT CUSTOMERS TELL YOU WHAT THEY WANTIGNORE WHAT CUSTOMERS

TELL YOU WHAT THEY WANT

• No clear strategy why to research • Assume only we can do the research • Demand new research • dogma

• self-serving priorities • No useful conclusions • ignore customers

methods are lemonsRESEARCH

Some

You’re Doing It

Well, actually,

WRONG

MAKING THE BEST OF IT rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrOutsource research to 3rd parties

• Little knowledge sharing beyond hiring

client team

• Difficult to adjust logistics, script,

scope, reach on the fly without

incurring new costs, contracts, etc

• Much insight leaves with the agency

when the project is over

• Check-the-research-box mentality:

one shot to do everything

• Make a case to reallocate funds in-

house or to *your* team

• Create a cross-project, cross-team

repository of reports/assets/etc

• Build elasticity into SOW

Yes,

and…

MAKING THE BEST OF IT rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrOther internal teams do all the research

• Research and findings exist in silos

• Teams who need research the most

may not be able to contribute

• Research priorities may be different

than yours

• Show connections between their

research and your work

• Invite those teams to your meetings

where design decisions, prioritization,

etc are made

• Find ways to help—reporting, recruiting,

managing research assets

Yes,

and…

MAKING THE BEST OF IT rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrFocus groups

• Groupthink

• Self-conscious or deliberately

secretive around strangers

• Little exploration or elaboration 

• No additional research necessary

• Get in the room

• Influence the script

• Get leads for more interviews

• No inadmissible evidence:Don’t fully discount what you hear

Yes,

and…

MAKING THE BEST OF IT rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrSurveys

• May give false sense of security

• Can be written to skew one way

• Harder to write than they seem

• No observable human behavior

• Provide opt-in to be contacted for more

comprehensive interviews later

• Mine for verifiable information

• Write questions that are appropriate for

surveys (duh)

Yes,

and…

MAKING THE BEST OF IT rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrOld Research

• Org may think research has been done

and doesn’t need more

• May not address evolving marketplace,

expectations, reality

• May have been discovery research, but

you need usability testing

• Show gaps in understanding or

where things have changed

• Don’t discount research just

because it’s old

• If available, talk to people who

did or paid for the research to

ask what’s missing

Yes,

and…

• Be involved

• Influence the script

• Use any outreach as an opportunity to

establish a relationship for later

• Don’t dismiss the value just if the Internet

doesn’t think it’s the best way

Make THe best of it

DChange

MANAGEMENT- -

klllll lllllm

1Find an

advocate

#WOCINTECH CHAT

IMPO

RTAN

CE Political support

Control over resources

Expertise

Same team

#WOCINTECH CHAT

Selecting the right challenge is paying attention to

who else cares about it

klllll lllllm

ESTABLISH A SENSE OF

URGENCY

klllll lllllmEstablish a sense of urgency

• Too much past success

• Lack of visible crises

• Low performance standards

• Insufficient feedback from

external, trusted sources

Traits of complacent orgs

klllll lllllm

Find external data to refute a comfortable status quo

Talk to unhappy or former customers

Show how profitable future opportunities are unobtainable

Establish a sense of urgency

klllll lllllmEstablish a sense of urgency

There were 70,000 searches for that topic last

quarter—people can’t find this content!

klllll lllllmCelebrate short-term wins

klllll lllllm

• Visible to outsiders

• Unambiguous

• Tied directly to change effort, vision

Celebrate short-term wins

klllll lllllm

• Provide evidence you’re on the right track

• Help hone the vision and long-term strategies

• Build momentum

• Keep bosses/advocates on your side

Celebrate short-term wins

klllll lllllmCelebrate short-term wins

We interviewed over 50 customers this quarter!

klllll lllllmCelebrate short-term wins

We researched patterns across 80% of our product portfolio—almost

everyone is on board!

klllll lllllmCelebrate short-term wins

Our beta program uncovered over 50 bugs we never

found in QC! Good thing we caught them when we did!

klllll lllllmCelebrate short-term wins

Our prototypes saved over $1MM of at-risk

business after those clients saw what was coming!

DChange

MANAGEMENTFind Advocates Create UrgencY

Celebrate The Wins- -

tactics

Tactics

rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrPlan for iterative researchTACTICS

rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr

• Address research in the project plan

• Plan to iterate, redo, refine, based on the research

• Align research with what’s getting built and when

• Be as consistent as makes sense across projects

Plan for iterative researchTACTICS

rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr

• Frame why you’re

researching for yourself,

your team, and stakeholders

• Applies focus

• Sets expectations

Transparent research strategyTACTICS

rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr

• Who’s who

• Themes to explore

• Duration of campaign

• Qualifying criteria for

participants

Transparent research strategyTACTICS

• Confirmed participants

• Methods to be used

• Script, if available

• Estimated date to deliver

findings

rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr

What do we want to learn

and what will we do with it?

Transparent research strategyTACTICS

rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr

• Most effective research doesn’t happen in a vacuum

• If dev informs the campaign upfront, they’re more prepared

for outcomes of campaign

• Marketing teams will be more receptive to your findings if

they know ahead of time

Encourage Dev/Product/Marketing collaborationTACTICS

rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr

• Critical these teams know about your efforts so you don’t

contact same people

• Gold mine for referrals/leads for recruiting

• Expect to educate these teams on why you want to talk to

“their” clients

Service, sales, and support are alliesTACTICS

rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr

• Show you’re on the right track

• Share recordings & video clips

• Don’t sweat cherry-picking

Show execs the good stuffTACTICS

rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr

• Before/After case study

• Show evolution of features/products based on user research

• Include quotes/metrics showing success

• Consider audience skeptical and unaware; don’t use jargon

or assume buy-in by default

Keep a case-study nearbyTACTICS

rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr

• Big data

• Machine Learning

• Network analysis

• Perception studies

Tie campaigns to larger business projectsTACTICS

}• Listening campaigns

• Sentiment analysis

• New people to talk to

rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr

• Benefits of research

• How to:

• conduct an interview

• synthesize research

• prepare a report

Coach your organizationTACTICS:

WORKSHOPS RESOURCES

• Script templates

• Mad-Libs cheat sheet

• Report templates

• Persona templates

tt tt tt tt

rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrs

qrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr

Market segmentation | Buyers | Promoters, saboteurs, etc

Expand the scopeTACTICS:

Plan for iterative research

Transparent research strategy

Encourage Product/Dev involvement

Sales, Service, & Support are allies

Show execs the good stuff

Keep a case study nearby

Tie campaigns to larger business projects

Train & coach to scale research

Expand the scope

Rethink your presentation

abab

abab

ITransparency Collaboration

Planning PromotionI

ONE MORE THING

NO RESEARCH GOVERNANCE STRATEGY

MOSAIQFree Wordpress theme to organize & manage all

your research activities and assets

Perfect for teams and agencies

Tie transcripts, notes, and participants to findings,

personas, and more

Download today at NasdaqDesign.com/Mosaiq

RESEARCH

FREQUENTLY

I

oooooooooooooooooop

noooooooooooooooooomore

more

TUUUUUUUUUUUUUUUUUUUUUUVChris Avore @erova #confabMN

#WOCINTECH CHAT