a how-to session featuring a compliance software company...

A how-to session featuring a compliance software company that achieved a 52% lift by testing across 45 pages

Former Website Optimization Manager,MasterControl

Senior Optimization Analyst,MECLABS Institute

Former Website Optimization Manager, MasterControl

Senior Optimization Analyst, MECLABS Institute

• Compliance software company

• Provides documentation development and tracking software to FDA and ISO regulated organizations

• Products span across multiple industries

• ROI attribution

• Process is methodical

• Provides a quantifiable level of accuracy

• Allows you to ask and get answers to specific questions

• Wanted to increase conversion rates

• Increase lead generation for the high intent users of the website (A-Leads)

21 2 3

Testing in a way you will actually learn about the customer

Treatment 1 Treatment 2Control

Treatments Conversion Rate

Control 0%

Treatment 1 9.2%

Treatment 2 7.6%

Treatment 1 increased conversion rate by 9.2%.

Co

ntr

ol

Tre

atm

en

t

TreatmentsLeads

Captured Rate

Relative Difference Level of Confidence

Control 3.12% -

Treatment 4.75% 52.3% 99.7%

The treatment increased leads captured rate by 52.3%.

Rule-based Science-basedvs.

• Emphasize hero shots

• Keep it above the fold

• Avoid too much copy

• Emphasize the benefits

• Don’t use reverse text

• Don’t rely too much on flash

• Implement basket recovery emails

• Avoid auto-on audio

Practice

Methodology

Meta-theory

C = 4m + 3v + 2(i-f) - 2a©

Goal: To increase total leads by increasing both resource downloads and video resource downloads.

Problem Statement: The current page lacks product, process and prospect-level value to effectively guide the users to the resources that best suit their needs.

Test Hypothesis: By providing additional, specific value about the resources available, how they will be obtained, and for which prospects they will beneficial, we will direct users to the correct resources.

Primary Research Question: Which deployment of value will improve clickthrough, resource downloads and leads?

Con

trol

Tre

atm

ent

C = 4m + 3v + 2(i-f) - 2a©

Con

trol

Tre

atm

ent

C = 4m + 3v + 2(i-f) - 2a©

21 2 3

Avoiding validity threats to make sure what you think you learned about the customer is really true

• Validity is the most important criterion for a quality test

• A valid test measures what it claims to measure

• Valid tests are well-founded and correspond accurately to the real world

• Results of a valid test can be trusted to be accurate and representative of real-world conditions

Treatments Conversion Rate

Control 0%

Treatment 1 9.2%

Treatment 2 7.6%

Treatment 1 increased conversion rate by 9.2%.

• A lot of analytics platforms don’t have statistical significance built in, so they don’t give you a reliable statistical analysis of the data

Treatments Conversion Rate Level of Confidence

Control 0%

Treatment 1 9.2% N/A

Treatment 2 7.6%

Treatment 1 increased conversion rate by 9.2%.

• There are hundreds of product pages primarily reached via SEO

• A single product page does not have nearly enough traffic to run any kind of statically valid test

45 product pages

• To aggregate sufficient traffic the top traffic, pages were identified to test on

• To reach statistical significance within approximately 1 month (10% RL @ 95 LoC), 45 product pages needed to be aggregated

• This required significant effort to plan, design, develop and quality check

TreatmentsLeads

Captured Rate

Relative Difference Level of Confidence

Control 3.12% -

Treatment 4.75% 52.3% 99.7%

The treatment increased leads captured rate by 52.3%.

1 2 23

Executive buy-in and budget authority to keep learning about the customer

• Starting small and gradually increasing degree and prominence of changes to the page to gain credibility and trust

• Treatment designs backed by a scientific approach and research rather than just an idea or best practice

• Less financial risk in testing — if the data doesn’t prove out the treatment, it’s not going to be implemented

• Less subjective and risky — prove ROI

Treatments Clickthrough Rate Relative Difference

Control 6.9% -

Treatment 15.8% 131%

The treatment increased clickthrough rate by 131%.

Treatments Rate Relative Difference

Control 6.9% -

Treatment 15.8% 131%

The treatment increased clickthrough rate by 131%.

• Tools and technology• Support from graphic design

and development teams• Access to someone with

statistical analysis skills (data scientist)

• Optimization manager• Executive champion

• Start.

• Measured improvement outweighs a perfect test

• Testing should be a company-wide effort — get everyone onboard

• Have someone with testing experience you can turn to when problems arise — because they will

• Ideas are about the company; a hypothesis forces us to ask a question about the consumer

• Customer theorist

Roundtable

Former Web Optimization Manager,MasterControl

Bryce Miller, MasterControl

Quin McGlaughlin, MECLABS Institute