optimizely: a/b testing
DESCRIPTION
TRANSCRIPT
AB Testing
+17.1% $ per visitor
AB Testing?
2008
Lesson #1
Define Quantifiable Success Metrics
Optimizely
Media
Button
Button1. Sign up
2. Learn More
3. Join Us Now
4. Sign Up Now
Media1. Get Involved Image
2. Family Image
3. Change Image
4. Barack’s Video
5. Springfield Video
6. Sam’s Video
Button1. Sign up
2. Learn More
3. Join Us Now
4. Sign Up Now
Media1. Get Involved Image
2. Family Image
3. Change Image
4. Barack’s Video
5. Springfield Video
6. Sam’s Video
Media: “Barack’s Video”
Media: “Springfield Video”
Media: “Sam’s Video”
Button1. Sign up
2. Learn More
3. Join Us Now
4. Sign Up Now
Media1. Get Involved Image
2. Family Image
3. Change Image
4. Barack’s Video
5. Springfield Video
6. Sam’s Video
And the Winner Was…
???????????
???????????
Button1. Sign up
2. Learn More
3. Join Us Now
4. Sign Up Now
Media1. Get Involved Image
2. Family Image
3. Change Image
4. Barack’s Video
5. Springfield Video
6. Sam’s Video
And the Winner Was…
And the Winner Was…
What Did That Mean?
Email Subscriptio
nsVolunteers
Amount Raised
Original 7,120,000 712,000 $143,000,000
+40.6% +2,880,000 288,000 +$57,000,000
New 10,000,000 10,000 $200,000,000
Lesson #1
Define Quantifiable Success Metrics
Lesson #2
Less Is More
+8% Engagement
Vs
+15% Conversion Rate+10% $ per visitor
+16.8% $ per visitor
Lesson #2
Less Is More
Lesson #3
Words MatterFocus on Your Call to Action
Vs+15.75% $ per visitor
Never Signed Up
Signed Up, Never
Donated
Previously Donated
0.0% 0.0% 0.0%
+2.3% +27.8% +16.3%
-27.8% N/A N/A
+15.2% -24.6% +11.9%
+8.5% +2.9% +18.4%
Vs
+14.6% Clicks
Lesson #3
Words MatterFocus on Your Call to Action
Lesson #4
Fail Fast
Vs
-92.3%
VideoClicks
-10% Conversion Value
Vs
Lesson #4
Fail Fast
Lesson #5
Seek the Global Maximum
+600% Engagement
“This new interface is just awful” +0.5% Retention
+1.4% Engagement
+49% Conversion Rate from 240 A/B Tests
Vs
Lesson #5
Seek the Global Maximum
Lesson #6
Start Today
Optimizely for Magento
1. Define Quantifiable Success Metrics2. Less is More3. Words Matter4. Fail Fast5. Seek The Global Maximum6. Start Today
Lessons Learned
Happy Testing!