optimizely and digital operative: optimization and testing

STEVEN SHYNE, DIGITAL OPERATIVE

CARA HARSHMAN, OPTIMIZELY

Consumer Listening:Tools and Tactics for Letting Your

Customers Guide You & Your Testing Plan

Gone are the Days of the One-Way Marketing Communication

•Consumers talk about and to you

•Great reach

•High speed

•Little-to-no cost

It’s Your Turn To Listen

•Understand and Sympathize with your Customers

•Discover Voices

•Understand Wants

• Improve Experience

Customer Experience vs. Voice of Customer

Your customers’ feedback about their experiences with your brand

How customers perceive their interactions with your brand

Pitfalls of Not Listening to Customers


“Our customers only want whatever

we are selling.”

“I don’t need to understand my

customers for them to understand our

offering.”


•Drive Away Customers

•Nine in ten (89%) consumers began doing business with a competitor following a poor experience.


•Lose Customers

•3 in 5 Americans (59%) would try a new brand or company for a better service experience.


•Leave Money On the Table

•86% of consumers claimed they would pay more for a better customer experience.


•Half of the Story

•Google Analytics provides great quantitative data

•Without qualitative data, only seeing WHAT not WHY.

TOOLS FOR LISTENING TO VOICE OF CUSTOMER

Consumer Listening

Tools for Listening to VoC

•Customer Listening Tools:

•Customer service channels

•Social listening

•Product reviews

•Blogosphere & microblogs

•Website communication


•Customer Asking Tools:

•Surveys & questionnaires

• Interviews/focus groups

•Usability studies

•Customer service channels


•Yes, Asking Does Work:

•According to SurveyMonkey, response rate for unsolicited customer feedback surveys is of 15-20%.

• In another study, 1 out 10 people say they will provide a product review when simply asked.

Tactics for Continual Customer Listening

UTILIZE NET PROMOTER SCORE (NPS)


Continual Customer Listening

•Net Promoter Score (NPS) Benefits:

•Implement easily across channels

•Segment customers by loyalty

•Identify unsatisfied/at-risk customers

•Benchmark against industry scores

•Uncover customer loyalty drivers

APPLY ACROSS CUSTOMER JOURNEY


Continual Customer Listening

•Apply Across Customer Journey:

•Discovery Phase

•Research Phase

•Consideration/Action Phase

•Post-Purchase Phase

Congratulations, you have a ton of data! Now what do you do?

@CARAHARSHMAN

Data Action

Our Goal:

@CARAHARSHMAN

CUS.

@CARAHARSHMAN

4 Steps to Turn Data into Action

@CARAHARSHMAN

Categorize

Prioritize

Hypothesize

A/B Test & Personalize

234

1

@CARAHARSHMAN

Categorize

Categorize your data into themes.

1

@CARAHARSHMAN

+

NPS =

@CARAHARSHMAN

@CARAHARSHMAN

Positive Themes

@CARAHARSHMAN

Negative Themes

@CARAHARSHMAN

Dashboards

@CARAHARSHMAN

Categorize

Categorize your data into themes.

1

Prioritize a theme and gather more data.

Prioritize2

@CARAHARSHMAN

Prioritize, then gather more data.

•Criteria for prioritizing:

•Business metrics

•Volume of feedback

•Urgency

•How to gather more data:

•5-8 customer interviews

•Behavior via website analytics

@CARAHARSHMAN

What people do and what they say

are rarely the same…A short story from Netflix

@CARAHARSHMAN

May 2011 Netflix:

@CARAHARSHMAN

June 2011: Netflix introduces new Watch Instantly experience

@CARAHARSHMAN

The new Netflix interface is complete crap.

…totally unusable.

The most terrible change I’ve ever seen.

“

@CARAHARSHMAN

So why did they do it?

Retention & engagement

A B

@CARAHARSHMAN

BRYAN GUMM

MANAGER OF EXPERIMENTATION

NETFLIX

“We were looking at the metrics and people were

watching more…they were more engaged…both tests

proved it.”

@CARAHARSHMAN

Prioritize a theme and gather more data.

Prioritize2

@CARAHARSHMAN

Identify the problem and hypothesize solutions.

Hypothesize3

@CARAHARSHMAN

Optimizely Stats Engine: A project inspired by user feedback

Theme: Data Trust Problem: Statistical Model

@CARAHARSHMAN

If ______ , then ______ because _______.

VARIABLE

RESULT

RATIONALE

Components of a Hypothesis

@CARAHARSHMAN

Identify the problem and hypothesize solutions.

Hypothesize3

@CARAHARSHMAN

A/B test your hypotheses and personalize experiences.

A/B Test & Personalize4

@CARAHARSHMAN

A/B Testing vs. Personalization

The best experience for specific audiences.

The average best experience for everyone.

@CARAHARSHMAN

A/B Testing

B C

A

Personalization

Detractors:“We listened to your feedback

and built a whole new statistics model”

B2B:“Statistics that marketers can understand is here”

Everyone else

A/B test your hypotheses and personalize experiences.

A/B Test & Personalize4

@CARAHARSHMAN

4 Steps to Turn Your Data into Action

1. Categorize your data into themes.2. Prioritize a theme and gather more data.3. Identify the problem & hypothesize solutions.4. A/B test your hypotheses and personalize.

@CARAHARSHMAN

QUESTIONS?

Thank You