Fail Faster with Multivariate Testing

Tom Wentworth| VP, Web Solutions | @twentworth12

In this session, you’ll learn:

• The Importance of website testing

• The difference between A|B and multivariate testing

• Testing best practices

• Creating A|B and multivariate Tests in Ektron

Optimize Landing Pages with Website Testing

• Test copy, images, pricing, call to action, layout

• Track Revenue, Leads, Subscriptions, Registrations, Donations

• A|B vs. Multivariate

When to use A|B vs. Multivariate

• A|B Testing– Test Broader Page Concepts– “Low Hanging Fruit”– Lower Traffic Requirements– Difficult to Control External Factors

• Multivariate Testing–More Learning–More Complexity, Planning, Traffic

Starting with A|B Testing

A B(Winner: A – 44% more Registrations)

Evolve with Multivariate Testing

B

A

D

E

C

“There are two possible outcomes: if the result confirms the hypothesis, then you've made a measurement. If the result is contrary to the hypothesis, then you've made a discovery.”

- Enrico Fermi

A|B Testing at Take Charge America

Challenger Champion

38% - 4% over lift

Landing Page Optimizationat Lloyds TSB

• Full range of banking and financial services

• 5th largest bank in Europe

• Results:– 90% increase in visitors– 150% increase in leads– 30% increase in visitor

to registered lead conversion rate

Testing the Three C’sContent, Copy, Call-to-Action

Testing the Three C’s

• Content– Test Images, layout, forms, rich media,

social widgets

• Copy– Strong, Clear, Direct Headline

• Call To Action– Be Specific, Obvious, and Singular

Select the Right Pages to Test

• A single, specific and “easy to measure” goal

• A page with a poor conversion rate• Enough conversions to test in a

reasonable amount of time

Know When to Hold Them…

Resist the Temptation to End Early

Don’t stop running a test too early

or too late

• Too early risks promoting the wrong version

• Too late wastes time that could be spent on new tests

Rinse. Lather. Repeat

Drive a culture of continuous Improvement

Design for Continuous Improvement

Building your FirstMultivariate Experiment in Ektron

(Demo)

\“Continuous Improvement is Better than Delayed Perfection”

- Mark Twain

Additional Resources

• Ektron Whitepaper – Fail Faster with Multivariate Testing– http://www.ektron.com/literature/whitepapers/fail

_faster_with_multivariate_testing.pdf• Sample Size Estimation Tool

– http://www.websharedesign.com/tools/sample-size-estimation-tool/

• Tim Ash “Seven Deadly Sins of Landing Page Design” Webinar– http://www.youtube.com/watch?v=erdEZvOq6wo

• Occam’s Razor Blog by Avinash Kaushik– http://www.kaushik.net/avinash/