1

The Netflix Data Platform – A Recipe for High Business ImpactKurt Brown, Director, Data Platform

Agenda

• Architecture

• Cloud

• OSS

• Philosophy

Architecture

S3

~5 PB

S3

vs.

Sting

S3

Sting

Data Platform

Tez

/ Shark

?

S3

Data Platform

TezYarn

???

Sting

Cloud

Yes

OSS

Cost

Cost

Flexibility

• Growth

• Spikes

• Self-service

Services

• SQS, EMR, S3,…

• Available

• Focus!

Redundancy / Global Presence

• Multi-DC

• International

Cloud Native

• Build for failure

• Swing systems

• Excess capacity

Right for everyone?

• Are you Google or Facebook?

• Netflix

• Undifferentiated

• Opportunity Cost

Cost?

OSS

General OSS

• Pooled goodness

• Manage own destiny

• Vendor relationships

• Conflicting agendas

Netflix OSS

• Meet Netflix needs

• Shared direction

• Recruiting & Retention

• Code quality

• Contributions

Data Platform OSS

• Genie

• Lipstick

• Aegisthus

• S3mper

• Suro

• Future: Inviso

Should you open source?

• We’re all using it

• You get value

• Others would get value

• Will invest some time

Philosophy

Netflix Expense Policy

“Act in Netflix’s Best Interest”

Development & Deployment Flow

BestPractices Components

ConsultationTools

Automation

Clean-up

DBAs?

QA Team?

Testing

Accept things WILL break

Safety Nets

Give folks what they need

Eliminate Process

• Status Reports

• Tickets

• Documentation

Tell vs. Ask

Does it really matter?

Questions?

http://jobs.netflix.com kurtbrown@netflix.com