Download - Optimizely Experience - LA - Disney-ABC
Bringing A/B Testing to Entertainment Disney-ABC Digital Media
The Optimizely LA Experience 8.19.14
Proprietary. Do not distribute without permission (Khai Tran).
Khai Tran Sr. Mgr, Analytics & Customer Insights Disney-ABC Digital Media
Building A/B Testing within Disney-ABC Digital Media
Background on our business
A case study
Final thoughts
2Proprietary. Do not distribute without permission (Khai Tran).
Why A/B test?
3
Disney-ABC Television – WATCH Products & Platforms
Proprietary. Do not distribute without permission (Khai Tran).
Own, operate, and stream full-episodes, short-form, live video content and games. Serve as a marketing channel for our shows and brands.
What’s our Business?
Supported Platforms
Supported Brands
4Proprietary. Do not distribute without permission (Khai Tran).
5Proprietary. Do not distribute without permission (Khai Tran).
6Proprietary. Do not distribute without permission (Khai Tran).
7Proprietary. Do not distribute without permission (Khai Tran).
8
Where A/B Testing Lives in our Org
WATCH Products & Platforms
Product Engineering Video Programming MarketingAnalytics &
Customer Insights PMO
Data "Data Design
Requirements Tagging
Data Integrity
Analytics "A/B Testing
Path Analysis Case Studies KPI Reporting
Customer Insights "Support Agents
Customer Feedback
Escalation Mgmt Voice of Customer
Proprietary. Do not distribute without permission (Khai Tran).
Why A/B Testing Makes Sense in Entertainment
9
1. Controls for external factors"• Traffic is randomly split across test variations during
time period of test"• Based on real users, not focus groups "
2. Launch and iterate faster"• Eliminates guesswork in evaluating different options"• Reduces time and resources "
3. Drive innovation and growth"• Whether the test wins or loses, it enables to learn more
about your users"• Speeds up innovation by minimizing risk and fear in
trying out new ideas
Controls for External Factors
"Originals vs. Repeats
Show Popularity Seasonality Windowing
Competitive Offerings Macro Factors
Proprietary. Do not distribute without permission (Khai Tran).
10
Driving Actionable Insights – Testing in the Larger Picture Track Key Success Drivers
Analyze User
Interactions
Segment for Deeper Insight
Test & Optimize
Drive Actionable
Insights
1. Track key success drivers • Keep pulse of the business • See if we’ve made a difference
"2. Analyze user interactions • Heat mapping, path analysis,
etc. • Identify areas for improvement "
3. Segment for deeper insight • Sub-groups behave differently • Solve for key segments "
4. Test and Optimize • Form hypotheses and tests based
on insights • Validate different variations
based on cause-and-effect
Proprietary. Do not distribute without permission (Khai Tran).
11
Full-Episodes Main Page""• Top page of our user path"• High traffic page"• High potential impact"• Bounce rate: 20-35%
* Omniture: Jan – Jun 2011, FEP visits only (excluded iPad)
40%
35%
0.2% 1% 1%
0.5%1%
2% 4%
7%
4%3%
What’s Interesting "35% of users clicked “All Shows” in bottom nav. Why? "Insight: Users want to get directly to the show that they’re interested in.
ABC.com Case Study – What and How We Tested
12Proprietary. Do not distribute without permission (Khai Tran).
Variation A: "Control Version""Full-Episode Home Page "www.abc.com/watch
13
Variation: Display all shows with logos "Hypothesis: "Will increase video starts because it: " • Reduces click to all shows • Easier to access non-recent content "
Potential Concerns: • Requires lots of scrolling. • Choice overload may lead users to leave. • Increases page load time.
Proprietary. Do not distribute without permission (Khai Tran).
Variation B: "All Shows with Show Logos
14
Variation: Display all shows text "Hypothesis: "Will increase video starts because it: " • Is easier to scan and find shows • Loads faster, requires less scrolling "
Potential Concerns: • Choice overload • Too much text. Unattractive.
Proprietary. Do not distribute without permission (Khai Tran).
Variation C: "All Shows with Text
15
Variation: Display top 8 noncurrent shows with link to “All Shows” "Hypothesis: "Will increase video starts because it: " • Highlights most popular Fall shows • Extends long tail for non-recently aired content • Distinct from current shows displayed in the
slideshow
Proprietary. Do not distribute without permission (Khai Tran).
Variation D: "Top Shows Fall Season "(Noncurrent Shows)
16
Variation: Display top 8 shows that are in season at time of test "Hypothesis: "Will increase video starts because: " • These shows account for 80% of current video
views • Easy to find and access these shows
Proprietary. Do not distribute without permission (Khai Tran).
Variation D: "Top Shows Summer Season "(Current shows)
17
4.24% 3.74% 3.93% Video Starts 4.94%
Chance to beat 99.3% 98.4% 97.8% 97.9%
So Which One Won vs. the Control?
A Control
B All Shows Logos
C All Shows Text
D Top Shows Noncurrent
E Top Shows Current
18
0.5 Days "• Define goals,
success metrics, scope, etc. ""
2 Days "• Assess dev needs
• Build test versions
0.5 Day "• Configure
campaign in Optimizely
Planning Design Development
Campaign Configura
tion0.5 Day "• QA
• Launch test
1 Day "• Hypothesize test
variations and design comps
Setup time – 4.5 days
7 Days "• Advised by
T&T to run at least 1 week ""
Run
A/B Test
Analysis
2 Days "• Analyzed real-
time results throughout
Run + Analysis time – 9 Days
A/B Test Details ""• Ran for one week (June 28 – July 5) "• 660k+ unique visitors"• 20% traffic allocation "• Test groups maintained across sessions
A/B Testing Steps and Actual Time Spent
19
"Even with a slight page tweak, which appears below the fold, we can produce a sizable revenue lift for the business.
Video Starts Driven from Homepage Revenue
146 MM $13.7 MM
+ 7.2 MM + $677 KLift from
Test + 4.94%
Projected Contribution
Connect A/B Testing Results to Revenue Impact
Actual numbers have been masked, but results are in the ballpark. This test ran in 2011.
x $0.0938 =
x $0.0938 =
Avg. Ad Revenue per Start
Before Test
20Image from Entrepreneur magazine (Dec 2012). http://www.entrepreneur.com/article/224967
Final Thoughts