namp conference - a/b testing your way to success
TRANSCRIPT
Na#onal Arts Marke#ng Project | #nampc November 11, 2017
A/B Testing Your Way to Success
Devon Smith Eva RosenbergMeasure Creative Harvard Ed Portal
First, a demonstration…
Version A
Version B
What is A/B testing? 1. Goal 2. Hypothesis 3. Segment 4. Split 5. Show 6. Measure 7. Change
Version A
Version B Winner
✓Each shown to Limited Group Shown to Full Group
Why do we A/B test?
Because we don't know the right answer, but our audiences do.
The best way to learn is to start testing.
Social Media Ads
Week 1: Tes?ng Image Week 2: Tes?ng Headline Week 3: Release Winner
Headline Winner
Image Winner
Headline A
Image A
✓
Image B
Headline B
Each Shown to Limited GroupEach shown to Limited Group Shown to Full Group
✓
Goal: Click on the ad Segment: Audience interested in “contemporary classical” on Facebook Hypothesis: Focus on the ar#st vs. focus on the arMorm
Week 1: Tes?ng Image
Image A Image B
Clickthrough rate: 1.0%
Week 1: Tes?ng Image
Image A Image B
Clickthrough rate: 2.9%
✓
Goal: Buy a #cket Segment: Audience interested in “contemporary classical” on Facebook Hypothesis: audience knows the ar#sts vs. audience wants background about the ar#sts
Week 2: Tes?ng Headline
Headline A Headline B
Week 2: Tes?ng Headline
Headline A Headline B
Cost per #cket sold: $3.06Cost per #cket sold: $4.70
✓
Tips for A/B testing social ads: • Make a campaign plan with a list of tests you want to run • Let ads run for at least a week before upda=ng • Capture what you learned each week • Different audience segments will respond differently to tests • Strive for cost per conversion (but be wary of bidding on it)
From: Name A
Tes?ng “From” Name Send Winner
From: Name A
From: Name B
✓
Each sent to Limited Group Sent to Full Group
Variation
Segment
Goal
Tes?ng “Subject line”
Goal: Open the email Hypothesis: Be specific about email content vs. general invita#on to an unspecified event.
A St. Patrick’s Day Celtic Sojourn A St. Patrick’s Day Celtic Sojourn Tickets Now On Sale!
2/10/17
Version B
A St. Patrick’s Day Celtic Sojourn Join us this March!
2/10/17
Version A
Tes?ng “Subject line”
A St. Patrick’s Day Celtic Sojourn A St. Patrick’s Day Celtic Sojourn Tickets Now On Sale!
2/10/17
Version B
A St. Patrick’s Day Celtic Sojourn Join us this March!
2/10/17
Version A
Open rate: 13.2%
Open rate: 12.6%
✓
Tes?ng Email Image
Goal: Click on the “nominate” buVon Hypothesis: Many faces of diversity vs. personalizing the impact on one recipient
Image A Image B
Tes?ng Email Image
Image A Image B
Clickthrough rate: 15.1% Clickthrough rate: 41.5%
✓
Image A: 1.2% began nomina#on process
Image B: 5.9% began nomina#on process
Tes?ng Email Image
Conversions ultimately matter more than opens or clicks.
Tips for A/B testing emails: • Your total list size needs to be at
least 4,000 people • Don’t expect same results to apply
to every campaign • Test “bad ideas” too • Don’t get overwhelmed by all the
tesAng opAons • Strive to measure on-site
conversions, but use the data you can get quickly
Open Rate
Clickthrough Rate
On-site Conversion
Date: ___________ To: ___________ From: ___________ Subject: ___________ Pre-Hear Text: ___________
Headline
Website
Website
DONATE
✓
DONATE
Website A/B Tes?ng Implement Winner
DONATE
Each Shown to Half of Users Shown to All Users
Website A/B Tes?ng
Original (A) Variant (B)
Goal: Donate Hypothesis: Users pay more aVen#on to the le\ side of the page
Website A/B Tes?ng
Original (A) Variant (B)
Conversion: 7.4% Conversion: 8.7%
✓
Tips for A/B testing on your website: • Start with your most important conversion goals and
areas of your site • Try experimenAng with audience targeAng opAons • Elements on the page you can test: text content, image
content, text size/styling in an element, size of element, locaAon of an element, existence of an element
• TesAng tool will tell you when you’ve reached staAsAcal significance (usually 2+ weeks)
• Don’t forget you have to roll out winning soluAon across your site permanently
Your turn…
Instructions: Part 1
1. Head to your group’s corner 2. Reflect on the exisAng Version A 3. Develop a hypothesis for what to change 4. Sketch your own version B on your card
Version B
Sketch your own version B.
Facebook Ad Email Website
Page
My hypothesis…
Facebook Ad
Audience: Women, 45+, interest in psychedelic rock
Audience: Boston area prior #cket buyers
A Christmas Celtic Sojourn Join us for the 15th season of A Christmas Celtic Sojourn!
12/11/17
Website Page
Choose a page from your
own website
Audience: You choose
Example Index Card
“My hypothesis: an email from a named person will have a higher open rate than an email from an
organiza?on”
Robert Lue Join us this March!
12/10/17
Need help? slideshare.net/devonvsmith
Facebook Ad
Website Page
Instructions: Part 2 1. Turn to someone in your color group 2. Say your hypothesis to each other 3. Play Rock, Paper, Scissors to idenAfy
the “winning version” of the test 4. Winner finds a new person to play 5. Loser becomes the cheering secAon
behind their partner 6. Repeat
Let’s hear from the winners…
Check out the NAMP blog for the results of these tests…hUps://namp.americansforthearts.org/get-smarter/arts-marke?ng-blog
Eva Rosenberg | [email protected] | edportal.harvard.edu Devon Smith | @devonvsmith | measurecrea#ve.com
Q&A
Resources
Calculators