search marketing thursday - januari 2014 - martijn geerlings

Friday, January 31, 14

Best Practices and Lessons Learned from

100,000 A/B TestsA/B TestsMartijn Geerlings

100,000


9:1Friday, January 31, 14

Lessons Learned


Lesson #1:

Define success .


Lesson #1:

Define quantifiable success metrics.


Visitvisitors per day



Emailsign up

rate



Emailsign up

rate$

$ per recipient


Visit Email $sign up

rate $ per recipientvisitors per day


Media

Button


Button:

1.Sign Up

2.Learn More

3. Join Us Now

4.Sign Up Now

Media:

1.Get Involved Image

2.Family Image

3.Change Image

4.Barack’s Video

5.Springfield Video

6.Sam’s Video

Splash page experimentVariations:


Button: “Sign Up”Friday, January 31, 14

Button: “Learn More”Friday, January 31, 14

Button: “Join Us Now”Friday, January 31, 14

Button: “Sign Up Now”Friday, January 31, 14

Button:

1.Sign Up

2.Learn More

3. Join Us Now

4.Sign Up Now

Media:


2.Family Image

3.Change Image

4.Barack’s Video

5.Springfield Video

6.Sam’s Video

Variations:Splash page experiment


Media: “Get Involved”Friday, January 31, 14

Media: “Family”Friday, January 31, 14

Media: “Change”Friday, January 31, 14

Media: “Barack’s Video”Friday, January 31, 14

Media: “Springfield Video”Friday, January 31, 14

Media: “Sam’s Video”Friday, January 31, 14

Button:

1.Sign Up

2.Learn More

3. Join Us Now

4.Sign Up Now

Media:


2.Family Image

3.Change Image

4.Barack’s Video

5.Springfield Video

6.Sam’s Video

Variations:Splash page experiment


Splash page experiment


Email Subscriptions Volunteers Amount Raised

Original: 7,120,000 712,000 $143,000,000

+40.6% +2,880,000 +288,000 +$57,000,000

New: 10,000,000 1,000,000 $200,000,000

Splash page experiment


Lesson #1:

Define quantifiable success metrics.


Lesson #2:

Less is more. Reduce choices.


+43.8% Purchases


vs.


vs.

+14.8% conversion rate


Lesson #2:

Less is more. Reduce choices.


Lesson #3:

Words matter. Focus on your call to action.


vs.


vs.

+14.6% clicks


Donation button experiment

Variations


Donation button experiment

VariationsNever

Signed UpSigned Up,

Never DonatedPreviously Donated

0.0% 0.0% 0.0%+2.3% +27.8% +16.3%-27.8% N/A N/A+15.2% -24.6% +11.9%+8.5% +2.9% +18.4%


Lesson #3:

Words matter. Focus on your call to action.


Lesson #4:

Fail fast.


vs.


vs.

-10% conversions


Lesson #4:

Fail fast.


Lesson #5:

Seek the Global Maximum.


+0.5% Retention +1.4% Engagement


Lesson #5:

Seek the Global Maximum.


1. Define quantifiable success metrics.2. Less is more. Reduce choices.3. Words matter. Focus on your call to action.4. Fail fast.5. Seek the Global Maximum.

Lessons Learned


Thank You


vs.


vs.

+8% engagement


2009! 2010! 2012!2011! 2013!


search marketing thursday - januari 2014 - martijn geerlings

Marketing

baracks video

springeld video

splash page experimentfriday

sign up1

donation button

media buttonfriday

sams videofriday

involved image2