how do you know....designing multi-channel campaign tests that optimize your budget
TRANSCRIPT
How Do You Know…Designing Multi-Channel
Campaign Tests That Optimize Your Budget
Jim DeLashNovember 18, 2014
One version had a Conversion Rate 24.6% higher than the other one
Page layout
Moving registration to the right side form increased the site’s Conversion Rate by
24.6%
Page layout
How do you know?
The right frequency
The best subject line
The optimal channel mix
Designing successful multi-channel tests
requires….
Establishing hypotheses
Developing test strategies by segment
Creating a measurement plan
Establishing hypotheses
Statements to be proven
All tests must support a hypothesis
Extrapolating the results to other brands
Low unaided recall scores for advertising creative will require
additional frequency….
Establishing hypotheses
Low unaided recall scores for advertising creative will require
additional frequency….
Establishing hypotheses
Low unaided recall scores for advertising creative will require
additional frequency….
BUT HOW MUCH MORE?
Establishing hypotheses
TEST…
More frequency in one channel
More frequency in multiple channels
Different levels of frequency
Establishing hypotheses
TEST…
More direct mail or e-mail
More frequency in multiple channels
Different levels of frequency
Establishing hypotheses
TEST…
More direct mail or e-mail
More direct mail and e-mail
Different levels of frequency
Establishing hypotheses
TEST…
More direct mail or e-mail
More direct mail and e-mail
10 15 20
Establishing hypotheses
TEST…
More frequency in one channel
More frequency in multiple channels
Different levels of frequency
Establishing hypotheses
How many times must th
e test
be run to
ensure enterprise
application
The best design is not always the best approach
Do not let design conflate the objective of an email or message
Every unnecessary piece of content is waste and reduces your chances of getting a click
Designing successful multi-channel tests
requires….
Establishing hypotheses
Developing test strategies by segment
Creating a measurement plan
Developing Test Strategies by Segment
Specialist or Generalist
Current User or Prospect
Decile
Awareness & Trial Offer
Developing Test Strategies by Segment
Multiple Brands Cadence
What will we learn from this test….
1. Best frequency by Specialty and Quintile2. One channel vs. Two channels (Segments 5,7,9)3. Total frequency: 10 vs. 12 vs. 14
Version B, the one with the ‘Save Now’ button, increased opt-ins for logged-in users by 333% at a 99.9% confidence level.
Designing successful multi-channel tests
requires….
Establishing hypotheses
Developing test strategies by segment
Creating a measurement plan
Control Groups
Measurement Milestones
Test Analysis
Creating a Measurement Plan
There are only two ways to get fired from Harrah’s:
stealing from the company,
or
failing to include a proper control group in your business experiment
Control Groups…
Not a ‘holdout’ group
Must include all relevant variables
Large enough to be statistically
valid
Creating a Measurement Plan
Measurement Milestones…
Track progress against goal
Forecast test impact early
Connect KPIs with Rx impact
Creating a Measurement Plan
Test Analysis…
Evaluate overall campaign impact
Both TRx and NRx
Creating a Measurement Plan
How can an e-mail test be successful with a low open
rate?
MICRO TESTS
Headline
copy> 30%
Call to action
> 80%
Image
s>
100%
Video> 80%
Use of
Color
20-50%
Page layout> 25%
MICRO TESTS
Subject Line
Subject Line
Version B, the subject line broken into three consumable sections, increased opens by 73.7%.
Call to Action
Version B, the ‘Get My Free Paper’ Call to Action, increased downloads by 25% at a 95% confidence
level.
Call to Action
Subject Line
Version B, the ambiguous subject line, increased opens by 112% at a 95%
confidence rate.
Subject Line
Designing successful multi-channel tests
requires….
Establishing hypotheses
Developing test strategies by segment
Creating a measurement plan
What is the optimal number of samples to offer in a multi-
channel campaign?
So, how will you know?