building a testing playbook by andrew richardson
TRANSCRIPT
I CAN’T EAT LUNCH
YET
THIS GUY KEEPS
TALKING…
This is your
BRAIN
An attention span of
8 seconds
*https://advertise.bingads.microsoft.com/en-us/insights @analyticandrew #eMetrics
IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION.
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics
IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION.
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics
2.6%
2.8%
2.8%
3.2%
3.3%
3.4%
5.0%
5.0%
5.5%
6.0%
Higher Education
Real Estate
Health
Legal
Home Improvement
Business Services
Travel
Business Consulting
Credit & Lending
Vocational Studies & Job Training
*Unbounce Conversion Benchmark Report – March 2017
HOW DO YOU COMPARE?
Never stop
testing,
and your
advertising will
never stop
improving.
- David Ogilvy -
IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION.
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics
DAYLIGHTYour ideas see
No more
HIPPO
5xHigher ROI than
PPC
Up to
IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION.
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics
Not enough
TRAFFIC
We have
NO IDEAwhat to test
IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION.
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics
Nothing should be
OFF LIMITS
T-shirt
TEST
size your
ING
Testing is often about
CHANGINGpolicy & perception
IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION.
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics
Ok, I am
ALL INNow what?
Tool Selection:
BUILD or BUY
7 TOOLSMarTech 2011:
175 TOOLSMarTech 2017:
THE PEOPLEThe most important thing
THE PEOPLEThe most important thing
BUT!
THE PEOPLEThe most important thing
one: focus efforts where they matter most
two: avoid conflict-put opinions to the test
three: empower people to optimize & be data-driven
four: align teams around shared goals
five: collaborate around hypothesis generation,
test
ideas & execution
PROCESS
IS
CRUCIAL
FOR
SUCCESS
RESEARCH1.
PRIORITIZE2.
EXPERIMENT3.
ANALYZE, LEARN,
REPEAT
4.
RESEARCH1.
“If A/B tests are used in lieu of research, the variations
are essentially guesses. We can improve outcomes of
A/B tests by incorporating UX research to:
•improve cause identification
•develop more realistic hypotheses
•identify more opportunities for experimentation”
Nielsen Norman Group: www.nngroup.com
RESEARCH1.
RESEARCH1.
RESEARCH1.
RESEARCH1.
RESEARCH1.
A theory can be
proved by
experiment; but no
path leads from
experiment to the
birth of a theory.
RESEARCH1.
Issue
• What is the conversion problem?
Cause Theory
• What do you think is causing the problem?
Variation Hypotheses
• What do you propose changing?
Goals
• What outcomes are expected?
RESEARCH1.
Issue
• What is the conversion problem?
Cause Theory
• What do you think is causing the problem?
Variation Hypotheses
• What do you propose changing?
Goals
• What outcomes are expected?
RESEARCH1.
RESEARCH1.
User-Experience to Improve Optimization Testing
• Defining user intent and objections• Why did you visit? Were you successful? If not, why?
• Exposing interface flaws• Changing colors vs. changing form fields
• Measuring findability• Tree testing, site search, heatmaps
• Cleaning up design variations• Reduce friction, maximize testing
RESEARCH1.
Bug tracking Brainstorming Site Search
Web Analytics Site Surveys Chat
PRIORITIZE2.
• TEST• Where you put stuff for testing
• INSTRUMENT• Fixing, adding, improving tagging, events, etc.
• HYPOTHESIZE• Found something, but no clear solution
• JUST DO IT (JDI)• No brainers
• INVESTIGATE• Need to dig further, ask more questions
Craig Sullivan: @optimiseordie
PRIORITIZE2.
PRIORITIZE2.
Craig Sullivan Hypothesis Kit
one: Because we saw (data/feedback)
two: We expect that (change) will cause (impact)
three: We’ll measure this using (data metric)
https://medium.com/@optimiseordie/hypothesis-kit-2-eff0446e09fc
PRIORITIZE2.
Craig Sullivan Advanced Kit
one: Because we saw (qual and quant data)
two: We expect that (change) for (population)
will cause (impact(s))
three: We except to see (data metric(s)
change) over a period of (x business
cycles)
https://medium.com/@optimiseordie/hypothesis-kit-2-eff0446e09fc
EXPERIMENT3.
EXPERIMENT3.
EXPERIMENT3.
EXPERIMENT3.
ANALYZE, LEARN,
REPEAT
4.
Insert random cycle chart here
IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION.
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics
• Headlines
• Sub headlines
• Paragraph Text
• Testimonials
• Call to Action text
• Call to Action Button
• Links
• Images
• Content near the
fold
• Social proof
• Media mentions
• Awards and badges
DESKTOPSegment
From
MOBILE
SWEATDon’t
the small stuff
IS MY CONVERSION
RATE GOOD OR BAD?
WHY DO I NEED TO
TEST?
BUT NO ONE WANTS TO
DO TESTING AT MY
ORGANIZATION.
PLATFORM, PEOPLE,
PROCESS
WHAT DO I TEST AND
HOW DO I TRACK IT?
TIPS AND
SUGGESTIONS FOR
GETTING STARTED
TAKEAWAYS
@analyticandrew #eMetrics
audience.ask(questions);
@analyticandrew #eMetrics
Never stop
testing,
and your
advertising will
never stop
improving.
- David Ogilvy -