2013 dma post conference certification
DESCRIPTION
I presented this presentation at the close of the 2013 DMA show.TRANSCRIPT
![Page 1: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/1.jpg)
The Answer is Always, “It Depends” – So Test It!
Ryan PhelanVice President, Strategy
Acxiom
![Page 2: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/2.jpg)
• Prior Industry Experience– Vice President, Strategy at BlueHornet
– Director, Email Marketing & Acquisition at Sears Holdings
– Responsible for East Coast Operations at Responsys
• Thought Leadership– DM News : Email Gets Personal (Cover Story)
– Keynote address – March 2012, EEC12
– Ranked as one of the top 40 Digital Marketing Strategists in the country by OMI
– Co-Chair of the EEC
– Member of:Ryan PhelanVice President, Global Strategic Services
![Page 3: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/3.jpg)
Join the Email Evolution Council for education, events, networking and
resources!
![Page 4: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/4.jpg)
“OMG Honey, look at that compelling subject line ….totally made me open it”
![Page 5: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/5.jpg)
This is more like it…
![Page 6: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/6.jpg)
Tier 3:Advanced
Tier 2:Medium Complex
Tier 1: Foundation programs
Existingprograms
Persona development, cluster analysis, next logical product, behavioral \ attitudinal segmentation, dynamic messaging, shopping cart abandonment,
preference center phase 2, social messaging
Video in email, creative testing, promotional optimization, triggers, win-back, preference center, social media
Testing & Reporting
Welcome, transactional messaging, opt-down, acquisition, promotional, attrition
Optimize programs based on easy changes (Low Hanging Fruit)
Testing & Reporting
Testing & reporting
Email Program Development
![Page 7: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/7.jpg)
1,000 Consumers from across the United States
![Page 8: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/8.jpg)
Nearly 49% of respondents have an email account for emails they rarely intend to open
Lesson: Ensure that when you ask for an email address, you make the reason very compelling
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
![Page 9: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/9.jpg)
9
When an email is saved to be read
later, 60% never read it
Lesson: Make your CTA immediate, urgent, laced with benefit and time sensitive in email
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
![Page 10: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/10.jpg)
Lesson: 21% to get updates and 13% because they love the brand10
40% of consumers sign up for email to receive discounts
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
![Page 11: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/11.jpg)
11
36% of respondents check email, social media and texts before doing anything else after they wake up
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
Lesson: Think about your message and test the optimal time of the day to send your message.
![Page 12: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/12.jpg)
Lesson: Test when you send and be wary of complex CTA in the morning.12
21% check their email before breakfast
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
![Page 13: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/13.jpg)
Consumers have shifted their consumption and are active at the very start of the day
Lesson: Try testing in the evening or even Friday evening for retail based business.2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
![Page 14: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/14.jpg)
14
Things are tiny in the morning…
Lesson: If you have to pinch it, you’re doing it wrong. Track mobile opens and design a template.
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
![Page 15: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/15.jpg)
15
Phone calls, texts, browsing the internet and email are the top uses for
smart phones
Lesson: Lesson: Track statistics for your consumers that are consuming email on a mobile device.
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
![Page 16: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/16.jpg)
Lesson: Track statistics for your consumers that are consuming email on a mobile device. 16
The shift in consumption is increasing based on consumer need for and ease of access
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
![Page 17: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/17.jpg)
17
Consumers don’t see silos, they see devices
Lesson: Consumers have adopted more devices and marketers have to be truly Omni-Channel
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
![Page 18: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/18.jpg)
Lesson: Capture and track the mobile opens of your subscribers and implement a template18
91% of consumers check email on their mobile phones
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
![Page 19: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/19.jpg)
19
…25% of those that use iPhone Passbook used it to access coupons
Of those that use iPhone Passbook (33%), 22% use it for Movie Tickets…
Lesson: Try linking a coupon for a sale or event to the Passbook functionality and then track use
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
![Page 20: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/20.jpg)
20
72% of consumers read email when they are bored…
…29% read email while in the bathroom
Lesson: Don’t make your email creative boring – inspire, delight and amaze
2013 Acxiom Digital Impact Consumer Digital Behavior Study n=1,006
![Page 21: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/21.jpg)
Nothing anyone says in marketing is right(except me of course)
![Page 22: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/22.jpg)
It depends…(the strategists mantra)
![Page 23: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/23.jpg)
QUIZ: WHICH TEST WON?DMA2013 | Email Testing in the Digital Age
![Page 24: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/24.jpg)
Which Test Won?
Up or Down Arrows
![Page 25: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/25.jpg)
Which Test Won?
Up or Down Arrows
![Page 26: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/26.jpg)
Which Test Won
![Page 27: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/27.jpg)
Which Test Won
![Page 28: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/28.jpg)
Which series won?Images Courtesy of:
![Page 29: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/29.jpg)
Which series won?Images Courtesy of:
![Page 30: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/30.jpg)
CHALLENGES OF THE EMAIL MARKETER
DMA2013 | Email Testing in the Digital Age
![Page 31: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/31.jpg)
The email marketer’s challenge• More subscribers• More data• More email
• More targeting• More expectations• Poor attribution
![Page 32: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/32.jpg)
What do we struggle with
You’re not alone…everyone struggles with email
![Page 33: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/33.jpg)
Chart 4.21 Email campaign element testing and optimizationWhich of the following email campaign elements do you routinely test to optimize performance? Please select all that apply.
Subject line
Call-to-action
Message (eg greeting, body, closing)
Days of the week sent
Layout and images
Time of day sent
Landing page
Target audience
Personalization
From line
Layout and images specifically for mobile viewing
Other
None of the above
86%
62%
58%
48%
47%
46%
44%
44%
42%
32%
26%
Source: ©2013 MarketingSherpa Email Marketing Benchmark Survey Methodology: Fielded December 2012, N=264
2%
2%
Message (e.g. greeting, body, closing)
![Page 34: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/34.jpg)
Why do we care?
• Campaign on 10/17– 166
individual email segments
– 84 were tests
Images Courtesy of:
![Page 35: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/35.jpg)
Does this look bad???
The highlighting worked…for a while
Images Courtesy of:
![Page 36: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/36.jpg)
Testing Challenges
We don’t (as an industry) know how to test
![Page 37: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/37.jpg)
Types of TestingA|B Testing• Testing one element against a control
Multi-Variant• Testing multiple elements against a control
Local Control Groups• Isolation of a population on a campaign level to see % of lift
Universal Control Groups• Isolation of a permanent population to see percentage of lift over time
![Page 38: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/38.jpg)
A|B Testing
Cons• Populations must be equal• Time periods must be
significant to judge results• Limited to one variable so
extended testing can be long
Definition• 2 email creative that are
identical but with one element changed in one version
Pros• Simple testing that is built into
most email platforms• Systems usually handle
division of population• Results are easy to
understand and act upon
![Page 39: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/39.jpg)
Multivariate Testing
Cons• Populations in each group
must be equal• Most email populations do
not have enough equal parts to be statistically relevant
DefinitionUsing one region in multiple email creative with changes in each sent to equal populations to determine a better performing email
Pros• Multiple elements can be
tested at the same time
![Page 40: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/40.jpg)
Local Control Groups
Cons• Sometimes hard to manage
from a population• Some people that are
active/buyers will not get a message
• Must involve pre-planning
DefinitionIsolation of a small but significant population from an email campaign to see what happens with their behavior against those that received an email
Pros• Can show the influence that
and individual email has on a campaign level
• Population must be reflective of the entire list
![Page 41: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/41.jpg)
Universal Control Group
Cons• Isolation of a population
means that some customers don’t get an email
• Hard sell internally• Must educate various groups
internally
DefinitionIsolating a population over a longer period of time to see what their behavior is against those that receive email
Pros• Gives email a true ROI
number
![Page 42: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/42.jpg)
CONFIDENCE INTERVALSDMA2013 | Email Testing in the Digital Age
![Page 43: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/43.jpg)
Sample Size DeterminationThere is no perfect answer in determining sample size. It is a trade-off between sample size and the difference between A and B it is important for us to detect.
Acxiom presents a chart (as seen below) to help clients find their optimal balance.
43
The smaller the difference we want to be able to detect, the greater the required
sample size
![Page 44: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/44.jpg)
Example Confidence Interval
• We have some results from an A/B Test:
oHow much confidence do we have in these estimates?
oDo we feel comfortable enough in the observed gain of .3% to switch to
B?
–It’s all about Sample Size when considering confidence in Response Rates.
44
![Page 45: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/45.jpg)
Example Confidence Interval
• The 95% Confidence Interval indicates we are 95% certain the RANGE of the interval captures the True response rate
• We observed B as 0.3% greater than A.
• We can now use our confidence interval for the difference (B – A) to establish how tight that 0.3% difference is based on our sample sizes
• The range includes A being .8% greater than B all the way to B being 1.4% greater than A. With this wide range of possibilities, taking action based on our estimates becomes very dangerous
45
![Page 46: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/46.jpg)
Example Confidence Interval
• Let’s say we have the same results, but instead of results based on samples of 1,000 each, they are now based samples of 50,000 each.
• We now have more evidence, due to our greater sample size. This results in greater belief in our results (estimates), and hence tighter intervals
• We still observe B as 0.3% greater than A, however now we can now conclude B is greater than A by between 0.1% and 0.5%.
46
![Page 47: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/47.jpg)
THE RULES OF TESTINGDMA2013 | Email Testing in the Digital Age
![Page 48: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/48.jpg)
buck·shot mar·ket·ing n. Marketing without a plan, clue, intelligent design or path toward success
![Page 49: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/49.jpg)
Rules for Testing
1. What needs to be tested2. Get a plan3. Execute the test4. Report on the results
![Page 50: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/50.jpg)
What needs to be tested
• To develop a plan, take time to define what’s broken– Mobile creative– Subjects– CTA– Lifestyle images vs action images– personalization– Audience/Segment– Landing page– Discount type
![Page 51: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/51.jpg)
Get a Plan
1. Develop a plan that lasts from 1-3 months1. Set rules for populations
1. What determines a valid population
2. What is success in each test1. What is the KPI that will be judged2. Define from prior test/campaigns what the range of KPI success
exists
3. Determine the right amount of time to build a significance to achieve the desired KPI
4. What are your exclusions5. Get extra pairs of eyes – make it a team effort
![Page 52: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/52.jpg)
One point to remember about the plan
• Does the result really prove the point?– Always work to validate
your testing– Sometimes thing “win”
because they’re different
• Recognize the “shiny objects”
Images Courtesy of:
![Page 53: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/53.jpg)
Execute the Test
• Define equal populations• Define the optimal timeline for response and
adhere to it• Monitor results• Verify that the test has been carried out
![Page 54: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/54.jpg)
Report on the Results
• Each test should have a post-mortem report– Reason– Goal– Creative– Population– Results
• Should be one page per report and stored• How does the result inform the next test
1. Track results1. You must be able to track testing results2. Determine who’s on the strike team to examine results
![Page 55: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/55.jpg)
EXERCISESDMA2013 | Email Testing in the Digital Age
![Page 56: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/56.jpg)
YOU NOW ALL WORK FOR
…AND GET PAID $1,000,000 A YEAR(SO DON’T SUCK)
![Page 57: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/57.jpg)
• Come up with a testing plan for this creative
Images Courtesy of:
![Page 58: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/58.jpg)
Small Groups• Define what you could test
(10 things over 3 months)– Cannot be subject line– Why are you testing it
• Define how you would test it• Define the audience• Define the exclusions
Assumptions• Customer is a male• 42 years old• Lives in Half Moon
Bay, CA• 20 miles south
of SF• Single
Teams will present their testing plan to the group. Best one, wins something gooooooood.
![Page 59: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/59.jpg)
Go…now…be smart
![Page 60: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/60.jpg)
Conclusion
• Think about your testing plan• Find equal populations• Test things that move the needle• Stop testing only subject lines
![Page 61: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/61.jpg)
Questions?
![Page 62: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/62.jpg)
© 2013 Acxiom Corporation. All Rights Reserved.
Ryan PhelanVice President, Strategic [email protected]
Thank you!
Keep the learning going. Follow me on twitter @ryanpphelan @acxiom
![Page 63: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/63.jpg)
APPENDIX
![Page 64: 2013 DMA Post Conference Certification](https://reader038.vdocuments.mx/reader038/viewer/2022103110/548b52a2b4795994488b46ac/html5/thumbnails/64.jpg)
Example Confidence Interval
64
How did we get the confidence intervals?
Confidence Interval for one proportion:
Test A = 0.015 +/- 1.96*SqRt ( (0.015 x 0.985) / 1,000 ) = 0.015 +/- 0.008 = (0.7%, 2.3%) Test B = 0.018 +/- 1.96*SqRt ( (0.018 x 0.982) / 1,000 ) = 0.018 +/- 0.008 = (1.0%, 2.6%)
Confidence Interval for difference between two proportions:
Test (B – A) = 0.003 +/- 1.96*SqRt ( ((0.015 x 0.985) / 1,000 ) + ((0.018 x 0.982) / 1,000) )Test (B – A) = 0.003 +/- 0.011 = Test (B – A) = (-0.8%, 1.4%)