a step-by-step guide to a/b testing research
TRANSCRIPT
A Step-By-Step Guide to
A/B Testing ResearchSIDDHARTH DESWAL
Thue is the Kissmetrics Webinar Wizard and Marketing Ops Manager. Before joining forces with Kissmetrics, he was a Lyft driver in SF,
which is also how he ended up as a KISSmetrics marketer. Whenever Thue is not trying to automate everything around him, you can find him
hiking in the Sierras.
THUE MADSENMarketing Operations Manager, Kissmetrics
@ThueLMadsen
Siddharth runs demand generation at VWO. In addition to being interested in all things CRO, he’s also passionate about B2B SaaS
marketing, Sales, growth hacking and human resource management.
SIDDHARTH DESWALSenior Marketer, Visual Website Optimizer
@SiddharthDeswal
@NEILPATEL #KissWebinar @Kissmetrics
#KissWebinar
@SiddharthDeswal
1 Find your top optimization opportunities
2 Ask some questions
3 Easy usability testing
TABLE OF CONTENTS
4 Visual analysis
5 Build your hypotheses
5 Putting it all together
WATCH WEBINAR RECORDING NOW
Step 1 - Find Your Top Optimization Opportunities
LANDING PAGES WITH HIGH BOUNCE
FUNNEL PAGES WITH HIGH EXIT
Image source
Start with your most valuable traffic sources
ANALYTICS - TOP TRAFFIC SOURCE
Landing pages with high bounce and high entries are optimization gold
ANALYTICS - MOST VALUABLE ENTRY POINTS
Landing pages with high bounce and high entries are optimization gold
ANALYTICS - MOST VALUABLE ENTRY POINTS
Funnel pages with high exit are the holes in the leaking bucket
ANALYTICS - FUNNEL DROP OFF
ANALYTICS - FUNNEL DROP OFF
LOG IN WITH GOOGLE
Start Your Free Kissmetrics Trial
Segment Your Traffic For Accurate Analysis
USEFUL SEGMENTS - NEW VISITORS WHO DIDN’T BOUNCE
USEFUL SEGMENTS - BEHAVIOR SIMILAR TO CONVERTERS
Create a segment about your goal converters (in our case, Free Trial signups)
USEFUL SEGMENTS - BEHAVIOR SIMILAR TO CONVERTERS
Map their key metrics
USEFUL SEGMENTS - BEHAVIOR SIMILAR TO CONVERTERS
Create a segment to see those who behaved like goal converters but did not convert
USEFUL SEGMENTS - BEHAVIOR SIMILAR TO CONVERTERS
You can use this segment in multiple ways1. Retarget them with a discount or promotional offer2. Give them higher lead scores in your marketing automation system3. Show them a right hand side pop-up4. If you have their email, trigger a specific message if they don’t convert
and leave the website5. If they’ve opted in to browser push-notifications, trigger a message
Step 2 - Ask Some Questions
ON-SITE SURVEYS
ON-SITE SURVEYS - SAAS
Page Question Insights drawn
Pricing / Features / Trial-Signup
What’s stopping you from signing up with us?
Common concerns, objections, anxieties
Top exit pages Is there something you were looking for that you couldn’t find on the website?
Reasons that users leave, Information that they needed to make a decision but couldn’t find
Top landing pages / Homepage
How familiar are you with [business offering]?
Understand how much convincing/explaining do your visitors need
Top landing pages / Homepage
Where did you come to know of us?
Top marketing channels
ON-SITE SURVEYS - SAAS
ON-SITE SURVEYS - ECOMMERCE
Page Question Insights drawn
Product page What’s stopping you from adding this product to cart?
Common concerns, objections, anxieties or ‘just looking’
Top exit pages (product or category)
Is there something you were looking for that you couldn’t find on the website?
Reasons that users leave, Information that they needed to make a decision but couldn’t find
Top landing pages / Homepage
Where did you come to know of us?
Top marketing channels
Checkout / Cart What’s stopping you from buying this product?
Why are visitors not completing purchases in spite of adding to cart
Step 3 - Easy Usability Testing
EASY USABILITY TESTING
Putting 5 to 10 users in front of your website or landing pages and observing them (or hearing them vocalize their thoughts) can teach you far more than most other forms of research.
Caveat: these users have to be representative of your target customers.
USABILITY TESTING - ASKING THE RIGHT QUESTIONS
“You received a damaged product, try and return it”
vs.
“You received a damaged product, what do you do now?”
USABILITY TESTING - ASKING THE RIGHT QUESTIONS
“You received a damaged product, try and return it”
vs.
“You received a damaged product, what do you do now?”
1) User searches for ‘Returns Policy’ page and then proceeds to follow the steps listed
USABILITY TESTING - ASKING THE RIGHT QUESTIONS
“You received a damaged product, try and return it”
vs.
“You received a damaged product, what do you do now?”
1) User searches for ‘Returns Policy’ page and then proceeds to follow the steps listed
1) User opens ‘Live Chat’ and asks rep what to do, or2) Takes to Twitter to complain to the brand, or3) Looks for the ‘Returns’ page, or4) Something that none of us were able to anticipate
EASY USABILITY TESTING
Choose your usability tasks/problems based on these parameters:1. Do you want to understand problems in a specific, goal driven process,
or2. Do you want to understand how users approach a problem they are
looking to solve?
Step 4 - Visual Analysis
HEATMAPS
Heatmaps can reveal some surprising insights
HEATMAPS
More people clicked on the top-right “Free Trial” CTA instead of the primary, middle-of-page CTA
HEATMAPS
What is it that we want our visitors to do, versus what are they doing instead?
The goal should be to align our interests and theirs’ to a common outcome.
Heatmaps quickly show you the difference.
By now, problems on your pages should have ‘bubbled’ out.
Step 5 - Hypothesis
HOW TO CREATE GREAT HYPOTHESIS
Research
Hypothesis
HOW TO CREATE GREAT HYPOTHESIS
Image source: MarketingExperiments.com
HOW TO CREATE GREAT HYPOTHESIS
The Change: Effect model depends on
1. Presumed problem
2. Proposed solution
3. Anticipated result
Image source: MarketingExperiments.com
Putting It Together
PUTTING IT ALL TOGETHER
1. Start by analyzing your top traffic source2. Look at the landing pages that have high traffic and relatively higher
bounce3. Look at pages that have the highest exit rates in a funnel4. Segment your traffic to get more accurate insights (goal converters
vs. non converters, new visitors who didn’t bounce)5. Run on-site surveys on your important landing or funnel pages6. Ask live users to complete tasks or solve problems that involve these
pages7. Run visual analysis on your most important pages8. At the end of this process, issues with the pages should bubble
out, i.e. they will become obvious9. Create hypotheses and start testing!
SIDDHARTH DESWALSenior Marketer, Visual Website Optimizer
THUE MADSENMarketing Operations Manager, Kissmetrics
Questions?