Quantifying the Value ofBetter Recommendations

The Value of Better Recommendations

Stakeholders:■ Business value■ Consumer value■ Producer value■ Cultural and Societal value

Recommenders have power for great benefit,but also for harm

Use your power wisely!

Neil Hunt

1999: Started work on recommendations at NetflixGoal was to improve satisfaction, while solving the %New problem

2006-2009: Netflix PrizePublic recognition of the importance of recommenders

2007-2011: Transition to streamingComplete catalog, short supply → curated catalog, unlimited supply

2014: 300 people working on “content discovery”$150M investment

Context

Technology EliminatesConstraints on Personal ChoiceConstraints on Personal Choice Falling Away

■ Geographic “trade radius”■ Production / manufacturing■ Shelf-space■ Channel (TV, radio)

Recommenders allow access to the long-tail of choices:■ Discovery■ Evaluation

Recommenders Enable Long Tail Media

■ There are no bad shows,just shows with small audiences

■ It’s our job to find and motivateexactly the right audience

Linear TV Channel - one choice available■ Only watch what’s being broadcast■ 21 hours/week of prime time - nothing else matters

On-demand - unlimited catalog accessible instantly■ Paradox of choice:

1000s of possibilities, most not interesting

Need a custom channel for each user (50M channels):■ 20-50 personalized choices

Netflix - TV of the Future - 50M Channels

No More Commercials

… or only relevant and interesting ads(chosen by recommenders…)

Richer Storytelling

Freed of the constraints of linear TV,not all shows must be 42 minutes with a cliffhanger end

Discovery from outside a channel grid liberates the format

The same was true for novels in Dicken’s time:Pickwick Papers was published in 20 weekly magazineswith 32 pages of text (a 4-fold broadsheet)and 16 pages of advertising support

They too, were liberated by advances in technology - making books possible

Business Objectives

Why Do Businesses Invest in Recommenders?

Better Economics…■ Makes a traditional business better, or

(Netflix, Amazon, Spotify, Pandora, ...)

■ Enables new businesses not possible before(LinkedIn, GoogleNews, Instagram, Waze, Pinterest, any free service with ads, …)

Why Do Businesses Invest in Recommenders?

The Tension:■ Enhancing customer satisfaction

■ Better choices■ Shorter time to choose

■ Suggesting more profitable products■ Choices with better margins■ Advertising

■ Long-term vs. Short-term tradeoff?

Netflix Choices

All our content is licensed to a fixed fee:Each possible choice has same cost impact

We don’t sell advertising on our service. Never will.

We don’t sell our recs or data to third parties in any form.

For Netflix, it’s all about customer satisfaction

Quantifying Netflix Benefits

7B hours per quarter50M subscribers worldwide90 minutes/day average150M choices/day

Quantifying Netflix Benefits

A good choice leads to a complete viewingA poor choice leads to abandonment, and risk of cancel

10% “better” choices → +500M/month good outcomesIf 1% of those avoids a cancellation → $500M/year

Our measurement thresholds:0.1% retention improvement ($5..50M/year)0.1% more viewing per time period

Our Business Metrics

Business Value

New Trials Retention

Hours of ViewingWord of Mouth

Retention is a Blunt Measuring Instrument

Retention is a Blunt Measuring Instrument

Measuring Users-at-Threshold

Hours of Viewing

Freq

uenc

y

Medians AveragesBaseline

Measuring Users-at-Threshold

Hours of Viewing

Freq

uenc

y

Medians AveragesBaseline Higher Avg

Measuring Users-at-Threshold

Hours of Viewing

Freq

uenc

y

Medians AveragesBaselineHigher Median Higher Avg

But We’re Still Measuring the Wrong Thing...

We optimize hours of viewing…But all hours are not created equal

Implication:■ We machine-learn addictive over compelling■ Partly innoculated by also measuring retention

What signal can we find for valued hours?

What if the Retention Driver is Something Else?Avoiding Failed Sessions (user found nothing to watch)Reducing Time-to-PlayMaximizing fraction-of-content-viewedMaximizing velocity of episode consumption

Consumer Objectives

What Consumers Say...

“I don’t need suggestions, just show me the good stuff”

“Don’t hide anything - I want to evaluate it all”

Winning The Moment of Truth

■ Moment of truth■ 1-2 minutes to find something■ 20-50 chances to connect■ Or the user has moved on...

The Intrinsic/Social Spectrum

Oracle vs. Advisor

✗ ✓

Content Producer Implications

The Cliff of Conventional Media

Producers must aim for broad audience or be irrelevant

Target Audience

Con

sum

ptio

n

Too smallNo-one knowsNo-one cares

Consu

mption

matc

hes

targe

t aud

ience

Recommender Systems Level The Cliff

Economics of high-end producers is less exponential

Producers can target the audience of their choice

New producers with niche product can emerge

Target Audience

Con

sum

ptio

n

Even small audiencescan be engaged

Consu

mption

matc

hes

targe

t aud

ience

Recommender Systems Level The Cliff

Long-tail producers aren’t excluded

Much greater cultural diversity is enabled

Does Data Drive the Product?

■

■

■

■

■

■

Netflix Use of Data for Content

✓ Predict reach & hours for a project given what we know

? Give insight to choice of cast, location, etc. if requested

✗ DO NOT dictate “she has to die at the end of S2-E1”

The director’s choices matter!

Cultural and Societal Implications

Democratization of Media

The cultural implication of the media cliff is lack of access to less prominent voices, channels, products

Recommendations systems can provide the market.

Producers are stepping in to fill that niche

The Cultural Exception

■ Marketing economics drives large commercial cultureto displace local, regional, niche culture

■ France: l’exception culturelle under GATT■ Canada: cultural exemption under NAFTA

■ Recommendation systems can reduce the swamping effect of large commercial culture

■ More to gain by exporting French culture to the world than by limiting import of global culture to France

Protectionism can yield to multiculturalism

Filter Bubbles and Echo Chambers

Proposition:■ Recommendation systems reinforce existing taste,

don’t expose users to the new, unexpected or different

If this keeps users happy, it’s likely to be true

Our experience is that diversity and serendipity play a large role in delivering recommendations that win

?

Final Thoughts

We are just scratching the surface of what’s possible

We are just scratching the surface of what’s possible

We depend upon our users trusting us with their data-- they might lose that trust

We are just scratching the surface of what’s possible

We depend upon our users trusting us with their data-- they might lose that trust

We have the ability to do amazing things for cultureor distort it horribly by following a false north-star

We are just scratching the surface of what’s possible

We depend upon our users trusting us with their data-- they might lose that trust

We have the ability to do amazing things for cultureor distort it horribly by following a false north-star

Be creative, but humble, and amaze the world!