direct mail testing & key metrics nedma 2016
TRANSCRIPT
Direct Mail Testing& Key Metrics
Presented by: Brett JonesMay 17, 2016 2:30 – 3:20
Today We’ll Discuss
§ WHY its important to test
§ HOW to design a valid test
§ WHAT to test & how to measure results
1
WHY IT’S IMPORTANT TO TEST
2
3 Ways to Make More Money
More Donors
• Acquisition Lists / Select criteria for Lapsed
• Package/Creative
• Lower initial ask
• Upfront / back-end premium
More Gifts per Donor
• RFM segmentation to identify those donors most likely to give an additional gift
• Follow up or additional appeals
• Less aggressive ask string
Higher Average Gift
• RFM segmentation to identify those with higher previous gifts
• More aggressive ask string
3
#4: Cutting Costs
The Testing Process
Determine your
objectiveIdentify an opportunity
Test your hypothesis
Take action based on
results
4
Possible Actions:1. Retest2. Rollout3. Move on to
another hypothesis
HOW TO DESIGN A VALID TEST
5
HOW to design a valid test
What change do you want to create?
Develop a hypothesis
Isolate a variable & construct a
valid testExecute Test
Determine key metric
Read results
6
WHAT TO TEST
7
Focus on the thingsthat will make a difference
List, 60%Creative, 20%
Offer, 20%
8
Two Key Metrics
Response Rate
Average Gift
9
Costs
CREATIVE TESTS
10
Package Test in Acquisition:Action v. Memo
QTY # Gifts % Resp.
Avg. Gift
Gross $ Net $ Net Cost to Acquire
“Action”Control
29,905 384 1.28% $35.99 $13,819 ($4,723) ($12.29)
“Memo”Test
29,901 208 0.70% $39.19 $8,152 ($9,999) ($48.07)
99% Confidence LevelWhat change did we want to create?What was the hypothesis?How would we determine success?How did we construct a valid test?What won? What might we do next?
Picture Window
vs. Standard
OE
Which format won?
Segment QTY # Gifts % Resp. Avg. Gift
Gross $ GrossRev/M
Control OE
13,033 229 2.29% $48.09 $14,380 $1,103
Picture Window
13,033 373 2.86% $44.36 $16,547 $1,270
99% Confidence Level
What change did we want to create?Our hypothesis. What variable did we isolate?How would we determine success?What we will do next?
Why / When to use this
metric
Component Test in Acquisition:RE v. BRESegment QTY # Gifts % Resp. Avg. Gift Gross $BRE 9,931 97 0.98% $36.81 $3,571RE 9,931 69 0.69% $41.09 $2,835
95% Confidence Level
What change did we want to create?What was the hypothesis?What won? What we will do next?
Format Test:Attached vs.
Separate Reply
Format Test:Attached vs. Separate ReplySegment QTY # Gifts % Resp. Avg. Gift Gross $ Cost/PieceSeparate Reply
63,239 1,034 1.60% $66.87 $69,141 $0.36
Attached Reply
63,773 1,176 1.84% $75.64 $88,962 $0.30
99% Confidence Level
What change did we want to create?What was the hypothesis? What about the increase in the average gift?What won? What we will do next?
OFFER TESTS
19
Longwood GardensCan an offer be too rich?
20
Test Offer:2 Guest Passes
Control Offer: 2 Additional Guest Passes
Segment QTY # Gifts % Resp. Avg. Gift
Gross $ GrossRev/M
Control:MOREpasses
56,763 484 0.85% $75.63 $36,605 $645
Test : fewer passes
56,761 558 0.98% $74.98 $41,840 $737
What change did we want to create? What was the hypothesis?What would we measure to determine success?What variable did we isolate?What won? Why?
95% Confidence Level
Longwood GardensOffer Test
Any Amount Makes You a Member
22
Sometimes you need to evaluate results over a l-o-n-g period of time
Open Ask Test Results§ Response Rate increased by 122%§ Average Gift fell from $55 to $28§ Net Cost to Acquire a member cut in half§ Minimal reduction in retention rate Year 2
23
LIST TESTS
24
Segment TestSegment QTY #
Gifts% Resp.
Avg. Gift
Gross $ Net Rev.
Recently Lapsed
9,682 139 1.44% $48.10 $6,686 $1,861
Longer Lapsed
21,681 197 0.91% $52.56 $10,354 ($451)
Would you mail this group again?
How would you decide?
Segment TestSegment QTY #
Gifts% Resp.
Avg. Gift
Gross $ Net Rev. Net Cost to Acquire
Acquisition 63,521 381 0.60% $46.49 $17,713 ($13,984) ($36.70)Recently Lapsed
9,682 139 1.44% $48.10 $6,686 $1,861 +
Longer Lapsed
21,681 197 0.91% $52.56 $10,354 ($451) ($2.29)
Would you mail this group again?
Goal: More New Donors
Key Metric: Net Cost to Acquire Longer Lapsed donor vs. Acquisition
Every acquisition mailing is an opportunity to tests new lists
Performance Risk
A Balanced List PlanTest Lists10% max
RetestsApprox. 10%
Core Lists80% of names
§ List testing opportunity #1:Identify lists that are mediocre performers whose response may be increased with a tighter select
List Segment Historical Resp. % Avg. Gift
Habitat For Humanity/24 Mos. .87% $48.09
List Segment % Resp. Avg. Gift
Habitat For Humanity/12 Mos. 1.01% $48.07
Current Performance
Sample Test Results
If the available universe doesn’t shrink too much, you may be better served by tightening the select criteria
§ List testing opportunity #2: The other side of the coinIdentify lists that are very strong performers that could generate additional members with an expanded select (larger universe) and comparable (or acceptable) response
List Segment Mailing Universe
Historical Resp. %
Members
World Wildlife Fund/12 Mos. 4,000 1.03% 41
World Wildlife Fund/12 Mos.
Mailing Universe
% Resp. Members
World Wildlife Fund/24 Mos.
7,000 .91% 63
Current Performance
Sample Test Results
Sometimes the greater good is served by a lower response rate on a larger universe of prospects – ultimately delivering you more new donors.
Is an UNsuccessful test one that doesn’t give you the
answer you expected?
What if a test loses?
What if I’m “too small” to test?
33
Remember…
