CX NYC 2018
Kelly Price, Analyst
Research for Different Levels of Design
2© 2018 Forrester Research, Inc. Reproduction Prohibited
Agenda
• The role of research in CX
• How to get the insights you need to succeed
• How to adjust your research plan
• Discussion and Q&A
3© 2018 Forrester Research, Inc. Reproduction Prohibited
Mastering six competencies leads to great CX
4© 2018 Forrester Research, Inc. Reproduction Prohibited
But getting to great starts with research
5© 2018 Forrester Research, Inc. Reproduction Prohibited
What do we mean when we say
“research”?
6© 2018 Forrester Research, Inc. Reproduction Prohibited
Source: Chart adapted from Tricia Wang :https://medium.com/ethnography-matters/why-big-data-needs-thick-data-b4b3e75e3d7
Data-driven
Thick data
Depth of insight
N
Insights derived from
quantitative methods using
large samples
Analytics, ML,
Big Data…
7© 2018 Forrester Research, Inc. Reproduction Prohibited
Source: Chart adapted from Tricia Wang :https://medium.com/ethnography-matters/why-big-data-needs-thick-data-b4b3e75e3d7
Data-driven
Thick data
Depth of insight
N
Human-insights
led
Insights derived from
quantitative methods using
large samples
Analytics, ML,
Big Data…
8© 2018 Forrester Research, Inc. Reproduction Prohibited
Source: Chart adapted from Tricia Wang :https://medium.com/ethnography-matters/why-big-data-needs-thick-data-b4b3e75e3d7
Data-driven
Human-insights
ledThick data
Depth of insight
N
Deep understanding derived from
qualitative research conducted with
smaller sample sizes
Insights derived from
quantitative methods using
large samples
Analytics, ML,
Big Data…
9© 2018 FORRESTER. REPRODUCTION PROHIBITED.
“Good inventors and designers deeply
understand their customer. They spend
tremendous energy developing that
intuition. They study and understand many
anecdotes rather than relying only on the
averages you'll find in surveys.”
- Jeff Bezos’ 2017 Letter to Amazon Shareholders
10© 2018 Forrester Research, Inc. Reproduction Prohibited
Where does this actually come
into play?
11© 2018 Forrester Research, Inc. Reproduction Prohibited
Typically people start with methods…
Website redesign
12© 2018 Forrester Research, Inc. Reproduction Prohibited
…but this is jumping the gun
Website redesign
What problem
am I solving?
13© 2018 Forrester Research, Inc. Reproduction Prohibited
Design happens at different levels of scale
Source: Peter Merholz and Kristin Skinner’s “Org Design for Design Orgs”
14© 2018 Forrester Research, Inc. Reproduction Prohibited
Asking the right questions starts with identifying the level of design
Example Challenges Level
Identify what opportunities exist to improve our
customers’ lives – what should we be designing?
Big picture
Reimagine a customer journey (e.g. onboarding) Strategy, Structure,
Surface
Design or redesign a website or mobile app Strategy, Structure,
Surface
Evaluate whether our site navigation matches how
users model content
Structure
Align our website design with our company’s new
brand guidelines
Structure, Surface
15© 2018 Forrester Research, Inc. Reproduction Prohibited
Research basically answers four core questions
What problems or
opportunities
exist?
What’s the best
way to solve this
problem?
Are our solutions
correct?
What’s happening
with this
experience?
16© 2018 Forrester Research, Inc. Reproduction Prohibited
Challenges map to specific questions
What problems or
opportunities
exist?
What’s the best
way to solve this
problem?
Big Picture Strategy Structure | Surface
Are our solutions
correct?
What’s happening
with this
experience?
17© 2018 Forrester Research, Inc. Reproduction Prohibited
Challenges map to specific questions
What problems or
opportunities
exist?
What’s the best
way to solve this
problem?
Big Picture Strategy Structure | Surface
UNCERTAINTY/PATTERNS/INSIGHTS CLARITY/FOCUS
Are our solutions
correct?
What’s happening
with this
experience?
18© 2018 Forrester Research, Inc. Reproduction Prohibited
Once we know where we’re starting we can get on the path to the right insights
Design Challenge Core research question
Research goalsResearch questions
Methods Informed design decisions
19© 2018 Forrester Research, Inc. Reproduction Prohibited
What problems or
opportunities exist?What’s the best way to
solve this problem?
Are our solutions
correct?
What’s happening
with this experience?
Example: Research on Retirement SavingsCore
question
related to
design
challenge:
20© 2018 Forrester Research, Inc. Reproduction Prohibited
What problems or
opportunities exist?What’s the best way to
solve this problem?
Are our solutions
correct?
What’s happening
with this experience?
Research
goal:
Understand
customers’
retirement goals
Identify behaviors that
prevent customers from
achieving their goals
Example: Research on Retirement Savings
Understand what features
increase customers’
retirement contributions
Get insight into whether
customers are meeting
their goals
Core
question
related to
design
challenge:
21© 2018 Forrester Research, Inc. Reproduction Prohibited
What problems or
opportunities exist?What’s the best way to
solve this problem?
Are our solutions
correct?
What’s happening
with this experience?
Core
question
related to
design
challenge:
Research
goal:
Understand
customers’
retirement goals
Identify behaviors that
prevent customers from
achieving their goals
Example: Research on Retirement Savings
Understand what features
increase customers’
retirement contributions
Get insight into whether
customers are meeting
their goals
Specific
question(s):What does “retirement”
mean to our customers?
Are people able to find
the information they
need?
How are my
experience KPIs
tracking over time?
What information are people
looking for when they interact
with us about retirement?
22© 2018 Forrester Research, Inc. Reproduction Prohibited
What problems or
opportunities exist?What’s the best way to
solve this problem?
Are our solutions
correct?
What’s happening
with this experience?
Core
question
related to
design
challenge:
Research
goal:
Understand
customers’
retirement goals
Identify behaviors that
prevent customers from
achieving their goals
Example: Research on Retirement Savings
Understand what features
increase customers’
retirement contributions
Get insight into whether
customers are meeting
their goals
Specific
question(s):What does “retirement”
mean to our customers?
What information are people
looking for when they interact
with us about retirement?
Are people able to find
the information they
need?
How are my
experience KPIs
tracking over time?
Methods: Discovery Definition Evaluation Measurement
23© 2018 Forrester Research, Inc. Reproduction Prohibited
Discovery Definition Evaluation Measurement
Exa
mp
le M
eth
od
s
Field observation/ethnography
Diary study
Contextual inquiry
Cultural probe
Experience sampling
Cognitive
mapping
Card sorting/tree testing
Participatory design/co-creation
Concept mapping
Usability testing
Surveys
Competitive analysis
Interviews
Benchmarking
WoZ testing
Cognitive walkthroughs
A/B/ multivariate testing
Analytics data
KPI tracking
Search log review /FAQ review
Support call logs
Eye-tracking
VoC feedback
Usability bug review
Heuristic evaluation
Existing research synthesis
Biometrics (EMG, GSR)
What problems or
opportunities exist?What’s the best way to solve
this problem?Are our solutions correct? What’s happening with this
experience?
24© 2018 Forrester Research, Inc. Reproduction Prohibited
What happens when we jump
ahead?
25© 2018 Forrester Research, Inc. Reproduction Prohibited
26© 2018 Forrester Research, Inc. Reproduction Prohibited
The Result?
$1.85BEstimated $1.85 billion
losses in same store sales
27© 2018 Forrester Research, Inc. Reproduction Prohibited
Common barriers to “ideal” research
$ Budget
Time
Buy-in
Regulations
Access to
participants
28© 2018 Forrester Research, Inc. Reproduction Prohibited
Example: Overcoming Challenges
Design challenge:
Improve the journey of researching and
applying for a checking account, to drive sales
What the research team wanted to do:
Diary study
Challenges:
3 weeks to do research,
< $3K budget,
compliance not comfortable with diary tools
What research did instead:
• Mapped using existing data
and insights, focused
research on filling gaps
• Remote ‘retrospective’
interviews over Web Ex with
target customers
• Total cost: < $2500
29© 2018 Forrester Research, Inc. Reproduction Prohibited
Example: Overcoming Challenges
Design challenge:
Improve the journey of researching and
applying for a checking account, to drive sales
What the research team wanted to do:
Diary study
Challenges:
3 weeks to do research,
< $3K budget,
compliance not comfortable with diary tools
What research did instead:
• Mapped using existing data
and insights, focused
research on filling gaps
• Remote ‘retrospective’
interviews over Web Ex with
target customers
• Total cost: < $2500
Key: Be willing to get “scrappy”, but do
so within the appropriate research
“swim lane”
STAKEHOLDERS
STUDY BACKGROUND RESEARCH QUESTIONS METHODS
PARTICIPANTS
Business Leaders• Who’s buy-in are you going to need to bring this
research plan to life?
• How will you frame the value and importance of
this research to get them on board?
• How will you involve them in the research process
to continue to prove the value?
Subject Matter Experts• Who has subject matter expertise for the
questions you are answering?
• What information might they have to help you form
stronger hypotheses?
• Are there places you can pull them into the
research process to help design better questions,
find better participants etc.?
Facilitators• Who will help plan, organize, and execute the
research?
• Who can help prove value creation or help link
research to success metrics?
Implementers• Who are the decision makers who will use the
research to make changes to the experience?
• How will you involve them in the process to
increase their buy-in?
• What is the context of this research? Why is it being done? • What are the 5-10 questions you want to have answered by the end of this
project?
• Why are these questions important?
• Do you have any existing research or data that provides insight into any of
these questions?
Identification• What methods will be used?
• How will each method apply to the research questions
you’ve identified?
Planning• What methods will you manage internally? Will you work
with any outside research providers?
• Where will the research take place? Will you need to
arrange transportation, or book space?
• How much budget do you have to allocate, and how will
that be divided across the different methods employed?Attributes • What attributes will you use to identify and recruit? Why are these attributes
important?
Recruitment• How will you locate and schedule participants? Will you work with an outside
provider?
Incentives• Will you offer incentives for participation? If so, how much? How will the
incentives be delivered?
Created by: Version:Date:
Source: Forrester Research, Inc. Unauthorized reproduction, citation, or distribution prohibited.
Project Tittle:
• Why are we conducting research?
• What are the 2-4 goals we have for this project? Why are
these goals important/justified?
RESEARCH GOALS
SCHEDULE
• What is the timeline for this project?
• What milestones will you meet along the way, and at what
dates? (e.g. recruitment, running methods, analysis)
SUCCESS METRICS
• What behavioral indicators will you use to provide
evidence that research findings are being used?
• What metrics will you use to provide evidence that
research lead to tangible experience improvements?
31© 2018 Forrester Research, Inc. Reproduction Prohibited
Discussion & Q&A
Thank youKelly Price
33© 2018 Forrester Research, Inc. Reproduction Prohibited
Appendix
34© 2018 FORRESTER. REPRODUCTION PROHIBITED.
How do we get from design challenge to design answer?
We start with a
challenge or a
problem
35© 2018 FORRESTER. REPRODUCTION PROHIBITED.
How do we get from design challenge to design answer?
We have a goal we
want to achieve
36© 2018 FORRESTER. REPRODUCTION PROHIBITED.
How do we get from design challenge to design answer?
Solving the problem
and achieving our goal
requires the right
human insights
37© 2018 Forrester Research, Inc. Reproduction Prohibited
Questions guide our research approach
What is the right design? Is this design right?
Discovery Definition Evaluation Measurement