optimizely experience - la - disney-abc

20
Bringing A/B Testing to Entertainment Disney-ABC Digital Media The Optimizely LA Experience 8.19.14 Proprietary. Do not distribute without permission (Khai Tran). Khai Tran Sr. Mgr, Analytics & Customer Insights Disney-ABC Digital Media

Upload: optimizely

Post on 10-May-2015

323 views

Category:

Documents


1 download

DESCRIPTION

Learn how Disney-ABC uses Optimizely and experience optimization to drive content consumption of hit shows like Modern Family and The Bachelorette

TRANSCRIPT

Page 1: Optimizely Experience - LA - Disney-ABC

Bringing A/B Testing to Entertainment Disney-ABC Digital Media

The Optimizely LA Experience 8.19.14

Proprietary. Do not distribute without permission (Khai Tran).

Khai Tran Sr. Mgr, Analytics & Customer Insights Disney-ABC Digital Media

Page 2: Optimizely Experience - LA - Disney-ABC

Building A/B Testing within Disney-ABC Digital Media

Background on our business

A case study

Final thoughts

2Proprietary. Do not distribute without permission (Khai Tran).

Why A/B test?

Page 3: Optimizely Experience - LA - Disney-ABC

3

Disney-ABC Television – WATCH Products & Platforms

Proprietary. Do not distribute without permission (Khai Tran).

Own, operate, and stream full-episodes, short-form, live video content and games. Serve as a marketing channel for our shows and brands.

What’s our Business?

Supported Platforms

Supported Brands

Page 4: Optimizely Experience - LA - Disney-ABC

4Proprietary. Do not distribute without permission (Khai Tran).

Page 5: Optimizely Experience - LA - Disney-ABC

5Proprietary. Do not distribute without permission (Khai Tran).

Page 6: Optimizely Experience - LA - Disney-ABC

6Proprietary. Do not distribute without permission (Khai Tran).

Page 7: Optimizely Experience - LA - Disney-ABC

7Proprietary. Do not distribute without permission (Khai Tran).

Page 8: Optimizely Experience - LA - Disney-ABC

8

Where A/B Testing Lives in our Org

WATCH Products & Platforms

Product Engineering Video Programming MarketingAnalytics &

Customer Insights PMO

Data "Data Design

Requirements Tagging

Data Integrity

Analytics "A/B Testing

Path Analysis Case Studies KPI Reporting

Customer Insights "Support Agents

Customer Feedback

Escalation Mgmt Voice of Customer

Proprietary. Do not distribute without permission (Khai Tran).

Page 9: Optimizely Experience - LA - Disney-ABC

Why A/B Testing Makes Sense in Entertainment

9

1. Controls for external factors"• Traffic is randomly split across test variations during

time period of test"• Based on real users, not focus groups "

2. Launch and iterate faster"• Eliminates guesswork in evaluating different options"• Reduces time and resources "

3. Drive innovation and growth"• Whether the test wins or loses, it enables to learn more

about your users"• Speeds up innovation by minimizing risk and fear in

trying out new ideas

Controls for External Factors

"Originals vs. Repeats

Show Popularity Seasonality Windowing

Competitive Offerings Macro Factors

Proprietary. Do not distribute without permission (Khai Tran).

Page 10: Optimizely Experience - LA - Disney-ABC

10

Driving Actionable Insights – Testing in the Larger Picture Track Key Success Drivers

Analyze User

Interactions

Segment for Deeper Insight

Test & Optimize

Drive Actionable

Insights

1. Track key success drivers • Keep pulse of the business • See if we’ve made a difference

"2. Analyze user interactions • Heat mapping, path analysis,

etc. • Identify areas for improvement "

3. Segment for deeper insight • Sub-groups behave differently • Solve for key segments "

4. Test and Optimize • Form hypotheses and tests based

on insights • Validate different variations

based on cause-and-effect

Proprietary. Do not distribute without permission (Khai Tran).

Page 11: Optimizely Experience - LA - Disney-ABC

11

Full-Episodes Main Page""• Top page of our user path"• High traffic page"• High potential impact"• Bounce rate: 20-35%

* Omniture: Jan – Jun 2011, FEP visits only (excluded iPad)

40%

35%

0.2% 1% 1%

0.5%1%

2% 4%

7%

4%3%

What’s Interesting "35% of users clicked “All Shows” in bottom nav. Why? "Insight: Users want to get directly to the show that they’re interested in.

ABC.com Case Study – What and How We Tested

Page 12: Optimizely Experience - LA - Disney-ABC

12Proprietary. Do not distribute without permission (Khai Tran).

Variation A: "Control Version""Full-Episode Home Page "www.abc.com/watch

Page 13: Optimizely Experience - LA - Disney-ABC

13

Variation: Display all shows with logos "Hypothesis: "Will increase video starts because it: " • Reduces click to all shows • Easier to access non-recent content "

Potential Concerns: • Requires lots of scrolling. • Choice overload may lead users to leave. • Increases page load time.

Proprietary. Do not distribute without permission (Khai Tran).

Variation B: "All Shows with Show Logos

Page 14: Optimizely Experience - LA - Disney-ABC

14

Variation: Display all shows text "Hypothesis: "Will increase video starts because it: " • Is easier to scan and find shows • Loads faster, requires less scrolling "

Potential Concerns: • Choice overload • Too much text. Unattractive.

Proprietary. Do not distribute without permission (Khai Tran).

Variation C: "All Shows with Text

Page 15: Optimizely Experience - LA - Disney-ABC

15

Variation: Display top 8 noncurrent shows with link to “All Shows” "Hypothesis: "Will increase video starts because it: " • Highlights most popular Fall shows • Extends long tail for non-recently aired content • Distinct from current shows displayed in the

slideshow

Proprietary. Do not distribute without permission (Khai Tran).

Variation D: "Top Shows Fall Season "(Noncurrent Shows)

Page 16: Optimizely Experience - LA - Disney-ABC

16

Variation: Display top 8 shows that are in season at time of test "Hypothesis: "Will increase video starts because: " • These shows account for 80% of current video

views • Easy to find and access these shows

Proprietary. Do not distribute without permission (Khai Tran).

Variation D: "Top Shows Summer Season "(Current shows)

Page 17: Optimizely Experience - LA - Disney-ABC

17

4.24% 3.74% 3.93% Video Starts 4.94%

Chance to beat 99.3% 98.4% 97.8% 97.9%

So Which One Won vs. the Control?

A Control

B All Shows Logos

C All Shows Text

D Top Shows Noncurrent

E Top Shows Current

Page 18: Optimizely Experience - LA - Disney-ABC

18

0.5 Days "• Define goals,

success metrics, scope, etc. ""

2 Days "• Assess dev needs

• Build test versions

0.5 Day "• Configure

campaign in Optimizely

Planning Design Development

Campaign Configura

tion0.5 Day "• QA

• Launch test

1 Day "• Hypothesize test

variations and design comps

Setup time – 4.5 days

7 Days "• Advised by

T&T to run at least 1 week ""

Run

A/B Test

Analysis

2 Days "• Analyzed real-

time results throughout

Run + Analysis time – 9 Days

A/B Test Details ""• Ran for one week (June 28 – July 5) "• 660k+ unique visitors"• 20% traffic allocation "• Test groups maintained across sessions

A/B Testing Steps and Actual Time Spent

Page 19: Optimizely Experience - LA - Disney-ABC

19

"Even with a slight page tweak, which appears below the fold, we can produce a sizable revenue lift for the business.

Video Starts Driven from Homepage Revenue

146 MM $13.7 MM

+ 7.2 MM + $677 KLift from

Test + 4.94%

Projected Contribution

Connect A/B Testing Results to Revenue Impact

Actual numbers have been masked, but results are in the ballpark. This test ran in 2011.

x $0.0938 =

x $0.0938 =

Avg. Ad Revenue per Start

Before Test

Page 20: Optimizely Experience - LA - Disney-ABC

20Image from Entrepreneur magazine (Dec 2012). http://www.entrepreneur.com/article/224967

Final Thoughts