optimizely experience - la - disney-abc

Download Optimizely Experience - LA - Disney-ABC

Post on 10-May-2015

315 views

Category:

Documents

1 download

Embed Size (px)

DESCRIPTION

Learn how Disney-ABC uses Optimizely and experience optimization to drive content consumption of hit shows like Modern Family and The Bachelorette

TRANSCRIPT

  • 1.Bringing A/B Testing to Entertainment Disney-ABC Digital Media The Optimizely LA Experience 8.19.14 Proprietary. Do not distribute without permission (Khai Tran). Khai Tran Sr. Mgr, Analytics & Customer Insights Disney-ABC Digital Media

2. Building A/B Testing within Disney-ABC Digital Media Background on our business A case study Final thoughts 2Proprietary. Do not distribute without permission (Khai Tran). Why A/B test? 3. 3 Disney-ABC Television WATCH Products & Platforms Proprietary. Do not distribute without permission (Khai Tran). Own, operate, and stream full-episodes, short-form, live video content and games. Serve as a marketing channel for our shows and brands. Whats our Business? Supported Platforms Supported Brands 4. 4Proprietary. Do not distribute without permission (Khai Tran). 5. 5Proprietary. Do not distribute without permission (Khai Tran). 6. 6Proprietary. Do not distribute without permission (Khai Tran). 7. 7Proprietary. Do not distribute without permission (Khai Tran). 8. 8 Where A/B Testing Lives in our Org WATCH Products & Platforms Product Engineering Video Programming MarketingAnalytics & Customer Insights PMO Data " Data Design Requirements Tagging Data Integrity Analytics " A/B Testing Path Analysis Case Studies KPI Reporting Customer Insights " Support Agents Customer Feedback Escalation Mgmt Voice of Customer Proprietary. Do not distribute without permission (Khai Tran). 9. Why A/B Testing Makes Sense in Entertainment 9 1. Controls for external factors" Traffic is randomly split across test variations during time period of test" Based on real users, not focus groups " 2. Launch and iterate faster" Eliminates guesswork in evaluating different options" Reduces time and resources " 3. Drive innovation and growth" Whether the test wins or loses, it enables to learn more about your users" Speeds up innovation by minimizing risk and fear in trying out new ideas Controls for External Factors " Originals vs. Repeats Show Popularity Seasonality Windowing Competitive Offerings Macro Factors Proprietary. Do not distribute without permission (Khai Tran). 10. 10 Driving Actionable Insights Testing in the Larger Picture Track Key Success Drivers Analyze User Interactions Segment for Deeper Insight Test & Optimize Drive Actionable Insights 1. Track key success drivers Keep pulse of the business See if weve made a difference " 2. Analyze user interactions Heat mapping, path analysis, etc. Identify areas for improvement " 3. Segment for deeper insight Sub-groups behave differently Solve for key segments " 4. Test and Optimize Form hypotheses and tests based on insights Validate different variations based on cause-and-effect Proprietary. Do not distribute without permission (Khai Tran). 11. 11 Full-Episodes Main Page" " Top page of our user path" High traffic page" High potential impact" Bounce rate: 20-35% * Omniture: Jan Jun 2011, FEP visits only (excluded iPad) 40% 35% 0.2% 1% 1% 0.5% 1% 2% 4% 7% 4% 3% Whats Interesting "35% of users clicked All Shows in bottom nav. Why? " Insight: Users want to get directly to the show that theyre interested in. ABC.com Case Study What and How We Tested 12. 12Proprietary. Do not distribute without permission (Khai Tran). Variation A: " Control Version" " Full-Episode Home Page " www.abc.com/watch 13. 13 Variation: Display all shows with logos " Hypothesis: "Will increase video starts because it: " Reduces click to all shows Easier to access non-recent content " Potential Concerns: Requires lots of scrolling. Choice overload may lead users to leave. Increases page load time. Proprietary. Do not distribute without permission (Khai Tran). Variation B: " All Shows with Show Logos 14. 14 Variation: Display all shows text " Hypothesis: "Will increase video starts because it: " Is easier to scan and find shows Loads faster, requires less scrolling " Potential Concerns: Choice overload Too much text. Unattractive. Proprietary. Do not distribute without permission (Khai Tran). Variation C: " All Shows with Text 15. 15 Variation: Display top 8 noncurrent shows with link to All Shows " Hypothesis: "Will increase video starts because it: " Highlights most popular Fall shows Extends long tail for non-recently aired content Distinct from current shows displayed in the slideshow Proprietary. Do not distribute without permission (Khai Tran). Variation D: " Top Shows Fall Season " (Noncurrent Shows) 16. 16 Variation: Display top 8 shows that are in season at time of test " Hypothesis: "Will increase video starts because: " These shows account for 80% of current video views Easy to find and access these shows Proprietary. Do not distribute without permission (Khai Tran). Variation D: " Top Shows Summer Season " (Current shows) 17. 17 4.24% 3.74% 3.93%Video Starts 4.94% Chance to beat 99.3% 98.4% 97.8% 97.9% So Which One Won vs. the Control? A Control B All Shows Logos C All Shows Text D Top Shows Noncurrent E Top Shows Current 18. 18 0.5 Days " Define goals, success metrics, scope, etc. " " 2 Days " Assess dev needs Build test versions 0.5 Day " Configure campaign in Optimizely Planning Design Developm ent Campaign Configura tion 0.5 Day " QA Launch test 1 Day " Hypothesize test variations and design comps Setup time 4.5 days 7 Days " Advised by T&T to run at least 1 week " " Ru n A/B Tes t An aly sis 2 Days " Analyzed real- time results throughout Run + Analysis time 9 Days A/B Test Details " " Ran for one week (June 28 July 5) " 660k+ unique visitors" 20% traffic allocation " Test groups maintained across sessions A/B Testing Steps and Actual Time Spent 19. 19 " Even with a slight page tweak, which appears below the fold, we can produce a sizable revenue lift for the business. Video Starts Driven from Homepage Revenue 146 MM $13.7 MM + 7.2 MM + $677 K Lift from Test + 4.94% Projected Contribution Connect A/B Testing Results to Revenue Impact Actual numbers have been masked, but results are in the ballpark. This test ran in 2011. x $0.0938 = x $0.0938 = Avg. Ad Revenue per Start Before Test 20. 20 Image from Entrepreneur magazine (Dec 2012). http://www.entrepreneur.com/article/224967 Final Thoughts