optimization summer games - test ideation
TRANSCRIPT
![Page 1: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/1.jpg)
![Page 2: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/2.jpg)
Generating High-Impact ExperimentsHazjier Pourkhalkhali
Senior Strategy Consultant
@Hazjier
![Page 3: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/3.jpg)
Optimizely is the world’s largest digital laboratory
![Page 4: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/4.jpg)
![Page 5: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/5.jpg)
We researched the world’s top companies
Interviewed experts
§ Spoke with thought leaders in the industry
§ Met with top customers
Built maturity model
§ Developed comprehensive maturity model
§ Assessed 100 customers
§ Surveyed 50+ customers
Held 100+ Business Reviews
§ Met with executives at over 100 Optimizely clients
Analyzed Top Revenue Experiments
§ Analyzed the 100 best experiments run on Revenue Per Visitor
§ Asked customers to describe test and process
![Page 6: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/6.jpg)
User acquisition:
1.5m
User retention
33%
Add-to-Cart:4.00%
Cart Checkout:
50.0%
Average quantity:
2.00
Average per unit price:€25.00
Average Order Value:€50.00
Conversion Rate:2.00%
Revenue Per Visitor:€1.00
Visitors:2.0m
Revenue:€2.0m
Understand the revenue drivers
![Page 7: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/7.jpg)
Goals
Strategies
Tactics
Cart Checkout
Rate
Wording / messaging
Location / size
Design / color /
iconography
Establish a visual
hierarchy
Remove non-
critical content
Limit number of
choices
Select/target the right USP’s
Refine the messaging
Crystallize visually
Clear and consistent
UI
Structure, naming,
and order of sections
Describe visually
Test ideal product visuals
Use the right
review systems
DescriptionsReviewsImages
Emphasize the Primary
Call to ActionMinimize
DistractionsCommunicate
Unique Selling Points
Design Intuitive
Navigation
Market Products
EffectivelyBuild a Sense
of Urgency
Show quantity
left
Push temporary
offers
Promote buying
before set times
![Page 8: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/8.jpg)
Testing requires freedom
LOW FREEDOM HIGH FREEDOM
“Every change we make to ourwebsite has to be approvedby our senior staff”
-Financial Services company
![Page 9: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/9.jpg)
Testing requires freedom
LOW FREEDOM HIGH FREEDOM
“Our Brand Styleguide is key.We cannot make changes that do not fit our design book”
-Financial Services company
![Page 10: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/10.jpg)
Testing requires freedom
LOW FREEDOM HIGH FREEDOM
“We would change our company logo if itconverted better”
-Travel Company
![Page 11: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/11.jpg)
![Page 12: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/12.jpg)
NO BEARD CAVEMAN
.30% .29% .28% .32% .25% .74%
.11% .28% .24% .32% .09% .59%
.74%
.59%
![Page 13: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/13.jpg)
Minimize Distractions:Remove Non-Critical Content +14%
Revenue PerVisitor
![Page 14: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/14.jpg)
BASED ON KEYWORDCURATED
+24%Click to Sale
+30%Add-to-Cart
+210%Conversions
![Page 15: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/15.jpg)
![Page 16: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/16.jpg)
![Page 17: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/17.jpg)
§ The user is reminded repeatedly of price
§ The user must remember the product benefits
§ The user must scroll down to the CTA when their basket is 3 or more
Problem Solution Result
§ Reduce size of price
§ Highlight discounts
§ Add product visuals
§ Duplicate CTA at top of page
§ Increase Conversion Rate
§ Increase Conversion Rate
§ Increase CTR to checkout
Build powerful hypotheses
![Page 18: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/18.jpg)
§ Analyze one section of your application / website
§ Document 10 different problems for your users
§ Group them by common root causes
§ Prioritize by urgency
Problem Solution Result
Optimizely Hypothesis Workshop
![Page 19: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/19.jpg)
§ Analyze one section of your application / website
§ Document 10 different problems for your users
§ Group them by common root causes
§ Prioritize by urgency
Problem Solution Result
Optimizely Hypothesis Workshop
§ Select 1 concrete problem
§ Design 10 solutions per theme
§ Choose 4 preferred solutions
![Page 20: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/20.jpg)
The problem with “A/B Testing”
2
A B
4
A B C D
![Page 21: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/21.jpg)
Most people test only 2 variations
0
10,000
20,000
30,000
40,000
50,000
60,000
2 3 4 5 6 7 >8
Tests 65% 17% 17%
# of Variations
![Page 22: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/22.jpg)
Most people test only 2 variations
0
10,000
20,000
30,000
40,000
50,000
60,000
2 3 4 5 6 7 >8
Tests
# of Variations
![Page 23: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/23.jpg)
Test 5 variations and nearly double your win rate!Win Rate
# of Variations
14% 18% 20% 27% 20% 22% 23%0%
10%20%30%40%50%60%70%80%90%
100%
2 3 4 5 6 7 >8
+90%
![Page 24: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/24.jpg)
Running more variations will significantly improve your testing!
![Page 25: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/25.jpg)
§ Analyze one section of your application / website
§ Document 10 different problems for your users
§ Group them by common root causes
§ Prioritize by urgency
Problem Solution Result
Optimizely Hypothesis Workshop
§ Select 1 concrete problem
§ Design 10 solutions per theme
§ Choose 4 preferred solutions
§ Primary Goals:What do you seek to impact immediately
§ Secondary Goals:What would you like to impact, but do not have to
§ Monitoring Goals:What are metrics you want to track
![Page 26: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/26.jpg)
Optimizely funnel analysis
HomepageFlight
ResultsBookFlight
PassengerDetails
FlightOptions
Payment Page
Step
45%30%
65% 65%
90%
70%
40%
40%
15% 20%
5%
25%
15%30%
20% 15%5% 5%
Exit Rate
Divergence
Completion
![Page 27: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/27.jpg)
Optimizely funnel analysis
HomepageFlight
ResultsBookFlight
PassengerDetails
FlightOptions
Payment Page
Step
Primary /SecondaryGoals
§ Searches Submitted
§ Category tabs§ Deals§ Frequent Flyer
§ Flight Selection§ Filter Usage§ Refine Search§ Sort Order
§ Continue button
§ Edit search§ Upsells
§ Field completion rate
§ Continue button
§ Continue button
§ Upsells§ Seat
selection
§ Conversion rate
§ Form field completion
§ Payment Type
MonitoringGoals
§ Checkboxes§ Flight tracker
§ Home page clicks
§ Menu interactions
§ Site search
§ Home page clicks
§ Menu interactions
§ Site search
§ Field error rate
§ Home page clicks
§ Menu interactions
§ Site search
§ Shopping cart pageviews
§ Home page clicks
§ Return to previous page
§ Edit address, information
§ Error rate
![Page 28: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/28.jpg)
§ Analyze one section of your application / website
§ Document 10 different problems for your users
§ Group them by common root causes
§ Prioritize by urgency
Problem Solution Result
Optimizely Hypothesis Workshop
§ Select 1 concrete problem
§ Design 10 solutions per theme
§ Choose 4 preferred solutions
§ Primary Goals:What do you seek to impact immediately
§ Secondary Goals:What would you like to impact, but do not have to
§ Monitoring Goals:What are metrics you want to track
![Page 29: Optimization Summer Games - Test Ideation](https://reader031.vdocuments.mx/reader031/viewer/2022021922/5875da021a28ab8f438b725b/html5/thumbnails/29.jpg)