december 2009, multivariate testing

MULTIVARIATE TESTING

2012 Planning

Confidential – Do Not Distribute

TESTING INTRODUCTION

• Marketers use testing to optimize the performance of their chosen tactical vehicle– Media– Creative– Messaging– Offer

• Testing allows us to quantify the effects of design and user interface iterations– Does the “hard sell” copy really drive more sales?– Is the Flash navigation causing users to abandon the homepage?– Does the search box in the header drive more product views?

• Online channels present an efficient testing opportunity due to the production efficiencies of the medium– Relatively short lead time– Accountable medium

2


THREE TESTING METHODOLOGIES

• A/B testing (Split run testing)– Tests a single design variation per test– Typically all users in the control group receive the baseline page and the users in the test group

receive the design variation– Pros: Quick to deploy, simple test matrix, minimal resources need to be created– Cons: Doesn’t scale well

• Multivariate testing– This test involves testing multiple items as one time– Pros: Can uncover success with multiple design elements combinations not tested with A/B, will

almost always demonstrate improvement over control group– Cons: Doesn’t scale well. A small number of variables will cause the test matrix to quickly grow very

large. Also many resources must be produced to fill the iterations in the matrix.

• Multivariate testing using the Taguchi method– Similar to multivariate testing but it only uses a subset of the possible test iterations – Pros: Theoretically provides the optimal test strategy– Cons: Can be expensive as external vendors are recommended for optimal knowledge transfer

3


ENSURE ACTIONABLE OBJECTIVES

1. Clearly identify success metrics before the test– “Increase customer service” is too broadly defined

2. Stakeholders must agree in advance to the test objectives– Ensure that all business owners understand the test and have

given input3. Ensure the test results can be acted upon

– Don’t test variables that may be outside the control of the business unit

4


ALIGN SCENARIOS WITH TEST OBJECTIVES

• Scenario Design and use case development should inform testing objectives

• Utilize existing scenario’s or use cases to guide the test– Testing scenarios are developed to test the effectiveness of a

campaign to increase applications for “Student Visa Platinum” from the Student Banking Center landing page. BAC is specifically interested in growing this business on the west coast and in particular California.

– If the campaign is geared specifically towards to drive users in the California market don’t test the effectiveness of the campaign for the East Coast.

5


MINIMIZE VARIABLES TO ENSURE DATA INTEGRITY

• Based on the objectives of the test determine if the users who will see the test are broadly representative of the overall audience segment

• Ideally you can control the source of the traffic to the test via a media vehicle

• Ensure a statistically reliable sample. This depends on the test setup but is typically at least 200 responses

6


THE VENDOR LANDSCAPE

Depending on the complexity of the test we recommend working with a partner to deal with the logistics of multivariate testing

1. Dedicated outsourced ASP based solution – Several vendors dedicated to multivariate testing exist. They can provide the infrastructure necessary for dealing with the complexities of running a multivariate test.

2. Existing transactional systems – Many application servers have testing modules that can be utilized. For example: ATG’s Campaign Optimizer

3. Custom solution – This is the most flexible but also the least efficient due to time and cost constraints

7


PRIVACY ISSUES

• Ensure that your privacy policy reflects the fact that you’re performing testing:– Additional cookies will likely be issued– Multiple users will get different versions of the site for purposes of

the test. You want to avoid the scenario that Amazon ran into a few years ago where they offered different prices to different users as this was very unpopular with the general populous

8


TESTING IS AN ONGOING PROCESS

• Testing is not a one off project – It’s part of an ongoing optimization cycle

• Learnings should be applied to future iterations of the test• Ideally you can test variables that can be applied to a larger

body of work

9


PREPARE THE TEAM FOR ORGANIZATIONAL CHALLENGES

• People may be resistant to testing as the results often invalidate their assumptions

• Create a case study to illustrate the value to the team• Strive to create a culture of testing• Start with simple A/B testing to develop internal processes for

change management

10


TESTING BEST PRACTISES

• Start small– Start off testing manageable elements – Don’t try to solve all your

problems on the first round of testing• Test actionable data

– Testing will generate “best possible” success criteria – ensure that you can enact the changes the test has proven most successful

• Get the creative teams involved early– Waiting until the results are complete will alienate the create teams

• Test messaging, offers and user interface elements separately – Depending on your test methodology be sure to test offers and

messaging in a separate test• Apply your test learnings to future projects

– React to the information that the test has generated– Use this information in inform future testing

11

Confidential – Do Not Distribute 12

DISCUSSION

december 2009, multivariate testing

Business