Download - How to Run AB Tests that Get REAL Results
How to Run A/B Tests that Get REAL Results
Lars LofgrenMarketing Analyst - July 2012 [email protected] - Confidential - Do not distribute
@larslofgren #KISSwebinar
We’ll cover...
The wrong way to A/B test1
5 places to get ideas for testing2
Get the most from your A/B testing tools3
HOW MOST PEOPLEA/B Test
A/B testing is totally rad! We should do it!
Everyone loves button colors
VS
Next up? Layouts!
What about photos?
VS
And conversions went up by 4%! Party time!
There’s a better way...
Testing random elements is very inefficient. You will not find the big
wins this way.
You’re also not learning
Red vs green buttons doesn’t help you understand your customer.
GETQualitative
Qualitative data definition
Data that isn’t a number.
Start with qualitative insights
Use other sources of data to get a basic understanding of what your
customers want.
Understand first, then do the A/B test
You’ll focus on the right tests and increase your conversions a LOT
faster.
The A/B Testing Process
Get qualitative insights1
Predict how to improve2
Confirm prediction with an A/B test3
Why is this process better?
Helps you find the 40% increase in conversions instead of only the 4%.
WHERE IS THE QUALITATIVEData?
Surveys
Location #1
Use one-question surveys on your site
Also try long surveys
Survey best practices
Keep it short1
Only ask for answers that you’ll use2
Start with open-ended questions3
Feedback
Location #2
Do you have a feedback email?
They know exactly what problems keep coming up.
Talk to your support team
They talk to customers every day.
Also talk to your sales team
Reach Out
Location #3
Get them on the phone and deeply understand how they use your
product.
Nothing beats talking to customers personally
If local, take them out to lunch or coffee.
Focus on asking good questions and listening.
This is not a pitch
User Activity
Location #4
You’ll need customer analytics for this.
Look at activity of individual people
KISSmetrics People report
Confirm by looking at other people.
When you have an insight...
Usability Tests
Location #5
Have someone use your site or product while you watch.
What’s a usability test?
Grab someone off the street
Amazon’s Mechanical Turk - mturk.com
UserTesting.com
Places to Find Qualitative Insights
Surveys1
Feedback2
Reach Out3
User Activity4
Usability Tests5
The A/B Testing Process
Get qualitative insights1
Predict how to improve2
Confirm prediction with an A/B test3
After you’ve found the big wins and you have a full-time growth team.
When should you test the small stuff?
A/B TESTINGTools
Great for testing copy, photos, and layouts. Terrible for bigger tests like
pricing.
A/B testing tools only measure the next step
Critical for measuring how your changes impact customers over the
long term.
Tie A/B tests back to the customer life-cycle
How do we actually do this?
First, make sure you’re using customer analytics
Then use Visual Website Optimizer or Optimizely
OR
Now all your A/B testing data is in KISSmetrics
For a demo of KISSmetrics or to learn how KISSmetircs can help grow your
business, contact Ben Sardella at: