americaneagle.com’s guide to a successful testing · the power of conversion rate optimization...

Americaneagle.com’s Guide to a

Successful Testing& Optimization Program

Corte SwearingenDirector of Testing & Optimization

Corte Swearingen

Americaneagle.com’s Guide to aSuccessful Testing & Optimization Program

Table of Contents

I. Introductiona. What is split testing?

b. Different types of tests

c. The power of conversion rate optimization

d. The Optimizely platform

e. Expectations when testing

II. Creating the Proper Foundation for Testinga. Developing website objectives, goals and key performance indicators

b. Defining all site micro-conversion goals

c. Technical analysis

III. Gathering the Right Kinds of Visitor Dataa. Web analytics data

b. Exit & page-level surveys

c. Form and checkout funnels

d. Heat and scroll maps

e. User testing

IV. The Americaneagle.com Testing Methodologya. The Optimization Road map

b. The Testing Cycle

c. Testing velocity

V. Developing the Optimization Road map Documenta. The advantage of Google Sheets

b. The ‘KPI’ tab

c. The ‘Brainstorming’ tab

d. The ‘Optimization Road map’ tab

e. The ‘Testing Road map’ tab

2

6

10

14

18

Different Types of Tests

At Americaneagle.com, we run plenty of split tests, but there are other types of tests

we may run depending on the traffic your site gets as well as the section of your site

being targeted for a test. Here is a quick summary of these additional tests.

1. Multivariate Test – If your site gets enough traffic, we may conduct what is

called a multivariate test. A multivariate test will test multiple changes within

a page and keep track of which combinations of those elements produce the

most business value. The advantage of multivariate testing is quicker insights

as to how different page element changes work together to make the most

improvement.

2. Multi-Page Test – A multi-page test (also known as a funnel test) lets you

test changes along a series of sequential pages. A common example of a

multi-page test would be testing a completely different checkout process

against the original checkout in order to determine which one produces

more orders. Another example might be a multi-step form or registration

process.

3. Radical Redesign Test – A radical redesign test pits the original page against

a version that is completely different. This is the type of test we would run if

we wanted to compare two totally different designs on a form or product

detail page. The advantage of a radical redesign test is they can bring

dramatic results. The disadvantage is that you lose granularity in

understanding which combinations of element changes really drove overall

improvement.

Unless your website gets a very large amount of traffic, our focus will be on using split

testing and multi-page tests to optimize the performance of the site. Split testing will

allow us to be much more granular in our testing and to gain better insights on how

each specific change affects performance.

If your site or page has a lower volume of traffic, we may suggest focusing more on

radical redesign tests. In this strategy, we lose the granularity and precision of split

testing, but improve the chances of seeing a major improvement in a shorter period of

time.

The Power of Conversion Rate Optimization

Many companies spend thousands of dollars in PPC and other traffic acquisition

strategies, but very little when it comes to ` optimization. As long as your site is getting

a reasonable amount of monthly traffic, allocating a portion of your digital marketing

budget to conversion rate optimization is absolutely critical.

Figure 3 shows how small increases in conversion rate can affect revenue, even if your

traffic and average order values stay constant.

The Optimizeley Platform

While the Americaneagle.com optimization team can use any split testing platform, we

prefer to work within Optimizely because of its power and flexibility.

Optimizely will allow us to produce any of the test types discussed in the previous

section. Our in-house testing engineers are Optimizely Certified and can quickly build

out simple tests within the Optimizely editor or engage our team of designers,

developer and programmers for more complicated tests.

If you are interested in building an internal testing team within your organization, we

can help you by running a series of training sessions to get your employees up to

speed. We’ll start with very simple tests for them to build out, and then slowly work up

to more complex tests. Your overall optimization program will produce better results if

you have one or more employees engaged with our team on a weekly basis.

Expectations When Testing

Creating and running a testing optimization program is a long-term endeavor. It takes

time to assemble all the correct pieces and build the proper foundation to ensure

success. This includes defining all the proper on-site goals and key performance

indicators, as well as using both quantitative and qualitative data sets to help form

stronger test hypothesis.

While every site and industry is different, our experience has been that only 1 in 5-6

tests will show positive results. The rest will be statistical ties, tests that are

inconclusive, or tests where the variation actually underperforms the original. Because

of this, it is important to keep up a certain weekly test velocity so that new tests are

constantly entering the queue as existing tests are running and completing

Table of Contents (continued)

VI. The Seven Steps to Every Testa. Creating a Strong Hypothesis Ties to a Specific Goal

b. Setting up the experiment in the Optimizely Editor

c. Set URL Targeting Conditions

d. Define the Test Audience

e. Set the Percentage of Visitors to be Included in the Test

f. Setting the Test Goals

VII. Common Questions & Answers

21

25

What is Split Testing?

INTRODUCTION

FIGURE 1: Simple A/B Split Test

2

I. Introduction

Split testing is a scientific approach to continuously evolving and improving your site

over time. This is accomplished by showing different page variations to different

visitors and measuring which page produces more overall business value.

The most common type of split test is commonly referred to an A/B test as two pages

are being tested against each other (the original page and a variation page) to see

which one produces more results.

Split testing brings a more controlled scientific approach to continuously optimizing

your site over time.










improvement.






process.







improvement.








time.































Traffic50%50%

Version A20% Conversion Rate

Version B25% Conversion Rate

Winner!

INTRODUCTION 3










improvement.






process.



FIGURE 2: Difference between Split & Multivariate Testing





improvement.








time.































VS

Test different versions of a web page, slitting the traffic evenly between both

pages.

Test different elements within a single web page. Multivariate testing is ideal

for high-traffic websites.

MultivariateTestingA/B Testing

Defining Site Micro-Conversion Goals

Most companies have one single most important action they would like a visitor to

take when visiting their site. This typically takes the form of a purchase or filling out

some type of lead form.

Considering the vast majority of visitors will not take your primary action (the average

conversion rate for a retailer is only 2-3%), what other smaller conversion goals can be

measured that still provide business value.

Here are some examples of viable micro-conversion goals for different types of sites:

Content-Based Website

• # visitors registering

• # visitors signing up for premium content

• # visitors signing up for newsletters/announcements

• Content consumption/Length of Visit/pages per Visit

• Forwarding content to a friend

• # downloads

• Social engagement (social sharing, likes, comments, etc.)

Technical Support Website

• # tickets opened (compared to those opened over the phone)

• Net Promoter Score (the % of people who would recommend the company products after an experience on the tech support site)

• % of solution guide downloads

• Number of completed videos watched

Ecommerce Site

• Create an account

• Download trial versions of product/service

• % of visits that viewed a product detail page

• # of free samples ordered

• Placing an item in a wishlist

In summary, start measuring the total business value of your site, not just the primary

conversion action.

Total Business Value = Primary Goal Conversions + Sum of all Micro-conversions

Technical Analysis

Before we start your testing program, it is important to ensure there aren’t any

technical site issues that might interfere with conversion rate. As part of our initial

audit, we will look at the following areas:

• Look at the browser report in your web analytics tool and monitor

conversion rates per browser segmented by desktop, tablet and mobile and

fix all discovered issues

• Look at page timings in your web analytics tool and identify all URLs that load

with sub-optimal speed

• Use Google PageSpeed Insights to enter all URLs from the previous step and

collect list of all fixable issues

• Check your Google Webmaster Tools account to make sure there aren’t any

html or page indexing issues

INTRODUCTION 4










improvement.






process.







improvement.








time.































FIGURE 3: Conversion Rate vs Revenue















• # downloads







Ecommerce Site







conversion action.


Technical Analysis













INTRODUCTION 5










improvement.






process.







improvement.








time.































II. Creating the Proper Foundation for Testing

Developing website objectives, goals and key performance indicators

The very first step for a successful testing program is to develop your overall website

objectives goals and key performance indicators. A key performance indicator (KPI) is a

metric that allows you to understand how you are doing against your business

objectives.

Before defining and establishing your KPI’s, think about your broad website objectives.

Pick 3-4 objectives and then work to develop one or more goals for each of those

objectives. The next step is to define the best key performance indicator for each goal.

Try to tie your KPIs directly to either revenue generation (or lead generation), cost

savings or improved customer satisfaction.

Use an Excel document to publish your web objectives, goals and KPIs for your entire

team. This will help keep everyone focused on the most important business metrics as

your testing & optimization program takes shape.

The following two figures are great examples of how to structure your web objectives

document.

CREATING THE PROPER FOUNDATION FOR TESTING 6















• # downloads







Ecommerce Site







conversion action.


Technical Analysis






















improvement.






process.







improvement.








time.
































FIGURE 4: Example of Website Objectives, Goals & KPIs

FIGURE 5: Example of Website Objectives, Goals & KPIs















• # downloads







Ecommerce Site







conversion action.


Technical Analysis













Increase Traffic

Company X Website Objectives, Goals & KPI’s

Increase Site Revenue Grow the EmailProgram

ImproveSatisfaction

Website Goal:More Visitors

Segments:Traffic Sources

Free vs Paid


Free vs PaidVisitor Type


Free vs Paid


Free vs Paid

KPI:Unique Visitors

Target:50K Visitors/Month

KPI:# Transactions

Target:5000 Transactions

per Month

KPI:AOV

Target:$200

KPI:Email Revenue

Target:$250K per Month

KPI:Task Completion

Rate

Target:80%

Website Goal:More Purchases

Website Goal:Higher Average

Order Value

Website Goal:Increase $ tied toEmail Campaigns

Website Goal:Improved Task

Completion










improvement.






process.







improvement.








time.













































• # downloads







Ecommerce Site







conversion action.


Technical Analysis




























• # downloads







Ecommerce Site







conversion action.


Technical Analysis













FIGURE 6: Example of page speed report with recommendations for improvement


GATHERING THE RIGHT KINDS OF VISITOR DATA 10

III. Gathering the Right Kinds of Visitor Data

Web Analytics Data

Your web analytics data will be a rich source of information for helping locate

high-traffic pages that have a higher than average bounce rate. In addition, for

e-commerce sites, we’ll be able to visually look at your multi-step checkout process to

monitor the exit rates of each step. If we find one or more steps that have a high

degree of abandonment, we can concentrate tests in those areas in an effort to reduce

checkout abandonment.

Exit and Page-Level Surveys

Website surveys offer a great way to quickly collect more qualitative information that

complements the more numbers-based data coming from your web analytics

FIGURE 7: Example of Using Web Analytics to Uncover Problem Pages

program. Exit surveys use technology to pop up a small survey invite just as it looks

like the visitor is about to abandon the site. The survey asks if they were able to

complete the goal of their visit, and if not, why?

A page-level survey can be used to collect visitor information on a specific page of your

site. Below are visual examples of both an exit survey and a page-level survey.















• # downloads







Ecommerce Site







conversion action.


Technical Analysis













Your web analytics data will be a rich source of information for helping locate

high-traffic pages that have a higher than average bounce rate. In addition, for

e-commerce sites, we’ll be able to visually look at your multi-step checkout process to

monitor the exit rates of each step. If we find one or more steps that have a high

degree of abandonment, we can concentrate tests in those areas in an effort to reduce

checkout abandonment.

Exit and Page-Level Surveys

Website surveys offer a great way to quickly collect more qualitative information that

complements the more numbers-based data coming from your web analytics

program. Exit surveys use technology to pop up a small survey invite just as it looks

like the visitor is about to abandon the site. The survey asks if they were able to

complete the goal of their visit, and if not, why?

A page-level survey can be used to collect visitor information on a specific page of your

site. Below are visual examples of both an exit survey and a page-level survey.

Form and Checkout Funnels

For e-commerce sites, understanding the abandonment rates for each step of your

checkout process can help uncover problem areas. We’ll set up checkout funnels as

well as funnels for all major forms on the site. If we see a high abandonment rate on a

particular form field or checkout step, we’ll concentrate testing efforts on the area in

an effort to make improvements.

Heat and Scroll Maps

Heat-maps and scroll-maps will allow us to look at any page on your site and

understand the entire pattern of click-engagement; which links are getting clicked and

which ones are being ignored. The scroll-maps will allow us to measure how far down

the page, on average, visitors scroll. These two toolsets will be vital in helping us

formulate stronger test hypotheses.


FIGURE 8: Page-level and Exit Survey Examples















• # downloads







Ecommerce Site







conversion action.


Technical Analysis













Stage 1: Analytics Analysis – Using a combination of web analytics, user testing, heat

maps, scroll maps and exit surveys to better understand the friction points within the

site.

Stage 2: Generating Test Ideas – Using the data in Stage 1 to formulate a brainstorm

list of possible test ideas

Stage 3: Tying Test Ideas to Key Performance Indicators – In this stage, we start

breaking out test ideas by the key performance area it will affect the most. Examples

might include increasing average order value, or improving lead capture rate.

Stage 4: Building the Testing Road map Document – This stage involves building out

the optimization road map document in Google Sheets so that it can be shared with all

team members. The road map document will show details of all tests currently

running, next tests in the queue and final results of all completed tests.

Stage 5: Building & Launching Tests: In this stage, tests are prioritized in the road

map document and work begins on building them out in the Optimizely editor. Each

week, testing overview calls are held where we discuss current tests running, any

completed tests and the test schedule for the following week.

The Testing Cycle

Once a page or section of the site has been identified for a test, the five-step testing

cycle begins.

Step 1: Hypothesis Creation – This is where we define our hypothesis of what the test

will achieve. This will be a short direct statement that ties the test idea to one or more

specific outcomes. An example would be “If we add a cross-sell item within the

shopping cart and make it easy for the visitor to add that item to their order, we’ll

increase the overall average order value for the site.”

Step 2: Graphic Design & Copy – Depending on the type of test running, there may be

the need to build out a graphic preview for approval before the test is assembled. This

will be especially true for radical redesign tests and tests were the copy has been

changed significantly. For simpler where only small elements are being changes, this

step may be skipped.

Step 3: Technical Install – Once a test has been approved for launch, the next step is

to actually build out the test in the Optimizely editor and program in the test goals as

well as any test restrictions like running on desktop only or including only certain

traffic segments. Once the test is set up, you’ll receive an email stating the name of the

test, the test hypothesis, all defined goals and audience segments, and links to view

both the original page and any test variations that were created.

Step 4: QC and Launch – Before a test is set live, and depending on the complexity, we

may turn the test on only for internal Americaneagle.com employees. This allows us to

see the test as a visitor would so that we can be confident that everything is working

properly. Once our QC check is completed, we’ll then turn the test on for everyone.

Step 5: Results Analysis and Rollout – We’ll make sure to filter out both your internal

employees and Americaneagle.com employees from tests results. When a test reaches

statistical significance, assuming the variation is the winner, we will immediately direct

100% of the traffic to the winning variation while we work to install the new page in

your content management system. A short result summary with recommendations will

be logged in the Testing Road map document in Google Sheets.

FIGURE 9: Example of a Form Funnel with Drop-Off Points by Field

FIGURE 10: Example of a Heat-Map




site.














The Testing Cycle


cycle begins.



























FIGURE 11: Conducting User Testing to Uncover Insights

User Testing

Another valuable tool in our testing arsenal will be periodic user testing. This involves

formulating a half-dozen or so specific tasks we want to watch people perform on your

site. We’ll create a handful of testers focused on desktop and another handful for your

mobile test. You’ll receive videos and audio of each tester working their way through

the defined task list. Discovered friction points will then be discussed and developed

into testing hypotheses.


IV. The Americaneagle.comTesting Methodology

Americaneagle.com follows a strict testing methodology that includes the stages

depicted in the following graphic.



site.














The Testing Cycle


cycle begins.



FIGURE 12: The Optimization Road map Cycle

THE AMERICANEAGLE.COM TESTING METHODOLOGY 14

























1.AnalyticsAnalysis

2.GeneratingTest Ideas

3.Trying test

ideas to key performaceindicators

4.Building out

the road mapdocument

5.Building &Launching

Tests



site.














The Testing Cycle


cycle begins.



FIGURE 13: The Americaneagle.com Testing Cycle


























1.Hypothesis

Creation

2.Graphic

Design &Copy

3.Technical

Install

4.QC &

Launch

5.Results

Analysis &Rollout

Optimization Road map Tab

This tab organizes and maps our test ideas to specific key performance indicators

(KPIs) and website objectives. This helps us organize our tests better and to ensure

there is proper test focus on each defined KPI.

The Brainstorming Tab

The brainstorming tab is an area to capture test ideas. As ideas come up weekly, they

can be logged in this tab so that we all have visibility to them. We’ll highlight test ideas

in green if we feel they should have priority.



site.














The Testing Cycle


cycle begins.
































Testing Velocity

On average, only about 1 in 4-5 tests will show a positive result. The rest will be either

statistical ties or the test variation will actually underperform against the original.

Because of this, it is important to maintain a schedule whereby we are launching new

tests each week. There are a few considerations when launching new tests on a regular

basis.

1. Test Complexity – Because some test may be simple to set up while others

may require hours of graphical design or programming, we will work to

balance out easy, medium and hard tests so that we aren’t trying to launch 3

very difficult tests all at one time.

2. Test Location – As we build the breadth and depth of the program, we will

make sure to minimize cross-pollination of tests as much as possible. This

means NOT having multiple checkout tests all running at the same time or 5

different product detail tests running concurrently. Typically, we will work to

have at least one test running in all major sections of the site. For a retail site,

this would include the homepage, the category page, the product detail page

and the checkout funnel. We may also run tests on the global navigation as

well as areas within the header/footer as well as key form pages.







site.














The Testing Cycle


cycle begins.




























V. Developing theTesting Road map Document

DEVELOPING THE TESTING ROADMAP DOCUMENT 18

The Advantage of Google Sheets

We will develop your customized testing & optimization road map using Google

Sheets. This will allow us to share the document and build it out collaboratively. At the

bottom of the sheet will be various tabs containing specific testing information. The

rest of this section will go through each of these tabs.

The rest of this section will describe the various tabs of the Google Sheet Optimization

Road map.

The KPI Tab

The KPI tab lists out overall website objectives, goals and key performance indicators

and will drive the foundation of our testing efforts.

FIGURE 14: Test Ideas Listed in the Brainstorm Tab

Website Objectives

Increase Traffic Aquisition

Website Goals Key Performance Indicators

Improve Overall Visits Total Sessions

Increase Traffic Aquisition

Improve Overall Revenue

CR for Email Captures

Improve Open Rates

Improve Click-Though Rates

Increase Email Revenue

Email Capture Conversion Rate

Open Rate %

Click-Through Rate %

Total Revenue Form Emails

Improve Conversion Rate

Improve Average Order Value

Lower Checkout Abondonment

Conversion Rate %

AOV

Checkout Abandonment Rate











site.














The Testing Cycle


cycle begins.




























FIGURE 15: Test Ideas Listed in the Brainstorm Tab

FIGURE 16: Mapping Key Performance Indicators to Test Ideas





Product Detail Page

Product Detail Page

Product Detail Page

Shopping Cart

Shopping Cart

Shopping Cart

Shopping Cart

Shopping Cart

“Add Now” button on recommended items

Move/remove recommended items module to bottom/horizontal.Stretch/enlarge remaining UI components

Test working on “Item Added” window buttons. “Checkout Now” vs “Secure Checkout.”

Test CTA on checkout button

Change treatment on promo code box

Remove promo code link to see if it improves transactions

CHange “Est. Shipping” to “Estimate Your Shipping”

Test removal of SSL badge under checkout button

Add text that tells people how far they are away from getting freeshipping; need to decide where to place this. Could test different locations





Website Objectives

Optimization Road map

KPI’s Experiment Type

Improve OverallRevenue

Improve the Email Program

Conversion Rate %

Average Order Value

Click-Through Rate %

Total Revenue From Emails

Checkout Abandonment Rate

Open Rate %

Email Capture Conversion Rate



site.














The Testing Cycle


cycle begins.



























Step 5: Set the Percentage of Visitors to Be Included in the Test

In most cases, we’ll want to run as many visitors through our experiments as possible.

This will allow us to achieve statistical significance faster and move on to a new test.

However, for certain types of tests, we may decide to dial down the number of visitors

entering a test in order to view early results and ensure the test variation is performing

as expected. A good example of this might be a major test we are conducting on a

checkout funnel. Complex checkout tests can make people nervous, so dialing down

the number of visitors may make sense.

Step 6: Setting the Test Goals

1. Clicking on an element like a submit button or a banner

2. Landing on a specific page (like a ‘Thank-You’ page after completing a

form)

3. Revenue


FIGURE 17: Part of the testing road map showing tests currently running





Test Status

Test Description, Hypothesis & Goals

Experiment Name Test Description Test Hypothesis

Test in Progress

Test in Progress

Test in Progress

Test in Progress

Test in Progress

E-dition Product Detail Page Radical redesign test

By offering up a radical redesign of thisproduct detail page, we will improve clicks

to the “Purchase Licenses” button.

By offering up a radical redesign of thisproduct detail page, we will improve clicks

to the “Renew” or “Purchase” buttons.

By redesigning the entire lead gen page,adding photos and additional benfit

statements, we will see an increase in lead submissions.

By redesigning the entire lead gen page,adding photos and additional benfit

statements, we will see an increase in lead submissions.

By making the demo button stand outmore and using a stronger CTA, we will

improve clicks on this element.

Radical redesign test

Testing a radical redesign forlead gen improvement

Testing a radical redesign forlead gen improvement

Test to determine if a buttoncolor/text change to the

“Demo” button increases clicks

Perspectives ProductPages

Software Landing PageDemo Buttons

Consulting Lead Gen

Software Lead Gen





THE SEVEN STEPS TO EVERY TEST 21

VI. The Seven Steps toEvery Test

Step 1: Creating a Strong Hypothesis Tied to a Specific Goal

As we begin adding detail to our test ideas, the first step is create a strong hypothesis

that is tied to a goal outcome. A weak or fuzzy test hypothesis usually correlates to

weak and fuzzy outcomes. Here’s an example of a strong hypothesis and test goal.

Hypthesis – By allowing shoppers to add an inexpensive accessory item to

their order from within the shopping cart, we will improve overall average

order value.

Goal – Improvement in Average Order Value

Step 2: Setting up the Experiment in the Optimizely Editor

In this step, we build out the actual test variation page within the Optimizely editor.

Optimizely will create a complete copy of the original page and allow us to make

changes to it on the fly from within the editor. Below is a screenshot of a page being

modified in Optimizely.

Step 3: Set URL Targeting Conditions

In this step, we need to tell Optimizely where the test is going to be run. In the case of

a test running on a single page, we simply need to enter the page URL into the URL

targeting box. If the test involves more than one page, we can enter all the URLs

associated with the test.

We can also run site-wide tests by using what is known as a “substring” match. A

substring match would allow us, for example, to target all product detail pages or all

category pages. Optimizely offers great flexibility in carving out which exact section or

sections of your site you want to test.

Finally, we can also exclude specific pages from a test.

Step 4: Define the Test Audience

Now that we have built out the test variation in the editor and defined exactly where

the test will run, the next step is to define what types of conditions need to be met

before a visitor is included in a particular test. Here are some examples of audience

segments we can build out and allocate to a specific test.

• Visitors using a desktop/laptop computer

• Visitors using a mobile phone

• Visitors coming in from a specific email or PPC campaign

• Visitors living in California

• New visitors

• Returning visitors

• Visitors coming to the site on Monday afternoon

• Visitors coming in from the Bing search engine

• Visitors coming in from a specific 3rd-party website

• Visitors who logged in

The above represents just a small fraction of audience segments that can be defined

for a test. Below is an example of setting an audience condition for people using

iPhones. This means that the test will only include visitors that are viewing the site

through an iPhone. Any other type of mobile phone user would be excluded from the

experiment.












form)

3. Revenue















order value.

























FIGURE 18: Editing a page from within the Optimizely editor


• New visitors










experiment.













form)

3. Revenue















order value.


























• New visitors










experiment.

FIGURE 19: Creating a test audience of only iPhone users













form)

3. Revenue














FIGURE 20: Changing test traffic is as easy a moving a slider bar

FIGURE 21: Setting up a goal to monitor Clicks on a Button







form)

3. Revenue


Website Objectives

Specify the percentage of traffic that should be included in this experiment and be tracked for conversions. Visitors that are not included will not count towards your monthly quota.

56%Included in experiment.

44%Excluded from experiment.

Slide bar makes it easy to changetraffic allocation

COMMON QUESTIONS & ANSWERS 25

VII. Common Questions& Answers

How do you QC tests before they are set live?

Before any test is launched, we will QC all areas of the experiment, including

verification that all experiment goals are firing properly.

Depending on the complexity of the test, we may also turn the test live within the

Americaneagle.com building. This allows our testing teams to move through an

experiment just as an actual visitor would and to double-check that page variations are

showing up as they should.

Once a test finished the QC process, preview links will be sent to you via email for

approval. Once approved, the test will be set live and closely monitored for the first

few days.

We’d like to eventually build our own internal testing team. Can you help with that?

Yes, we can hold online training sessions to get your internal team up to speed with

using the Optimizely platform and setting up your own experiments. We’ll start with

simple experiments, things like button color or call-to-action text changes, and then

move to more complicated tests as your team gains experience.

How long will tests take to complete?

Test length depends on two factors; how much traffic the page gets, and the minimum

conversion improvement you wish to detect. Tests in which the variation is only

producing a minor improvement in the conversion goal will take much longer to

achieve statistical significance. Here are a few examples of test length depending on

traffic and minimum detectable conversion lift.












form)

3. Revenue

How do you QC tests before they are set live?

Before any test is launched, we will QC all areas of the experiment, including

verification that all experiment goals are firing properly.

Depending on the complexity of the test, we may also turn the test live within the

Americaneagle.com building. This allows our testing teams to move through an

experiment just as an actual visitor would and to double-check that page variations are

showing up as they should.

Once a test finished the QC process, preview links will be sent to you via email for

approval. Once approved, the test will be set live and closely monitored for the first

few days.

We’d like to eventually build our own internal testing team. Can you help with that?

Yes, we can hold online training sessions to get your internal team up to speed with

using the Optimizely platform and setting up your own experiments. We’ll start with

simple experiments, things like button color or call-to-action text changes, and then

move to more complicated tests as your team gains experience.

How long will tests take to complete?

Test length depends on two factors; how much traffic the page gets, and the minimum

conversion improvement you wish to detect. Tests in which the variation is only

producing a minor improvement in the conversion goal will take much longer to

achieve statistical significance. Here are a few examples of test length depending on

traffic and minimum detectable conversion lift.

TABLE 1: Estimated time to complete a split test

What is the process for evaluating overall program performance?

The testing/optimization road map that we develop will contain a tab that projects out

incremental revenue/lead gain over a 12 month period and will be updated the first

week of every month. You will be able to access this information at any time and see

the projected incremental lift of all winning tests that have completed. This will make it

easy for you to calculate the benefit/ROI of the testing program at any time.

Will split testing negatively affect my SEO?

Optimizely will not affect your SEO negatively. In fact, Google not only permits split

testing, but actively promotes and encourages it as a way to learn more about visitor

behavior and conversions.

Why is my test not reaching statistical significance?

In some cases, a test variation may show only a very small increase in the test goal.

When this happens, achieving statistical significance may take many months. In these

cases, we will usually be able to call a tie or pronounce the variation to be

“non-negative” so we can officially stop the test and decide whether to implement the

variation.

COMMON QUESTIONS & ANSWERS 26












form)

3. Revenue

Copyright © 2015 Americaneagle.com | All rights reserved

Contact us today at

[email protected]

for a free A/B testing demo!

Contact Us

americaneagle.com’s guide to a successful testing · the power of conversion rate optimization...

Documents