how to run ab tests that get real results

How to Run A/B Tests that Get REAL Results

Lars LofgrenMarketing Analyst - July 2012 info@kissmetrics.com - Confidential - Do not distribute

@larslofgren #KISSwebinar

We’ll cover...

The wrong way to A/B test1

5 places to get ideas for testing2

Get the most from your A/B testing tools3

HOW MOST PEOPLEA/B Test

A/B testing is totally rad! We should do it!

Everyone loves button colors

VS

Next up? Layouts!

What about photos?

VS

And conversions went up by 4%! Party time!

There’s a better way...

Testing random elements is very inefficient. You will not find the big

wins this way.

You’re also not learning

Red vs green buttons doesn’t help you understand your customer.

GETQualitative

Qualitative data definition

Data that isn’t a number.

Start with qualitative insights

Use other sources of data to get a basic understanding of what your

customers want.

Understand first, then do the A/B test

You’ll focus on the right tests and increase your conversions a LOT

faster.

The A/B Testing Process

Get qualitative insights1

Predict how to improve2

Confirm prediction with an A/B test3

Why is this process better?

Helps you find the 40% increase in conversions instead of only the 4%.

WHERE IS THE QUALITATIVEData?

Surveys

Location #1

Use one-question surveys on your site

Also try long surveys

Survey best practices

Keep it short1

Only ask for answers that you’ll use2

Start with open-ended questions3

Feedback

Location #2

Do you have a feedback email?

They know exactly what problems keep coming up.

Talk to your support team

They talk to customers every day.

Also talk to your sales team

Reach Out

Location #3

Get them on the phone and deeply understand how they use your

product.

Nothing beats talking to customers personally

If local, take them out to lunch or coffee.

Focus on asking good questions and listening.

This is not a pitch

User Activity

Location #4

You’ll need customer analytics for this.

Look at activity of individual people

KISSmetrics People report

Confirm by looking at other people.

When you have an insight...

Usability Tests

Location #5

Have someone use your site or product while you watch.

What’s a usability test?

Grab someone off the street

Amazon’s Mechanical Turk - mturk.com

UserTesting.com

Places to Find Qualitative Insights

Surveys1

Feedback2

Reach Out3

User Activity4

Usability Tests5

The A/B Testing Process

Get qualitative insights1

Predict how to improve2

Confirm prediction with an A/B test3

After you’ve found the big wins and you have a full-time growth team.

When should you test the small stuff?

A/B TESTINGTools

Great for testing copy, photos, and layouts. Terrible for bigger tests like

pricing.

A/B testing tools only measure the next step

Critical for measuring how your changes impact customers over the

long term.

Tie A/B tests back to the customer life-cycle

How do we actually do this?

First, make sure you’re using customer analytics

Then use Visual Website Optimizer or Optimizely

OR

Now all your A/B testing data is in KISSmetrics

For a demo of KISSmetrics or to learn how KISSmetircs can help grow your

business, contact Ben Sardella at:

bsardella@kissmetrics.com