Building 200 Gbit/svideo platformVyacheslavOlhovchenkov

Load profile• Peak bandwidth 210Gbit/s• Peak bandwidth per edge

40Gbit/s• Daily traffic 1PTB• Views per day 6M • RPS to ruby app: 1000• RPD to ruby app: 60M

Core components• Balancers• Edge servers• Video manager• Encoders• Storage servers• Stats server• Player w/ adv.support: IMA; VPAID,

VAST 2,3• API

Core components (2)

Request flowchart

Balancer (architecture)

Balancer (features)• Geo targeting• Stats• Edge selection• Serving player iframe• Secured manifest transfer• URL signing• Advertising module

Balancer (configuration)• CPU: E3-1270 V2 @ 3.50GHz• RAM: 8GB• SSD• Ubuntu 14.04• RoR, unicorn• PostgreSQL, redis

Balancer• 1000 RPS at peak• 75% CPU utilization

Balancer (tips & tricks)• PgSQL => Redis for hot data• ORM optimization

joins, indexes, pure sql• Linux/nginx optimization

net.core.somaxconn=65535net.ipv4.tcp_fin_timeout = 25…

• RoR optimizationworker_processes = 24timeout = 60

Edge — history. V1.0. 5Gbit/s• Dual E5-2620@ 2.00GHz• RAM: 32GB• 1x10Gbit Intel NIC• ZMIRROR:

– 12 SATA 3TB– L2ARC (6SSD 256GB)

• ZFS default settings

Edge (tuning 1)Reason for multiple writes?

zfs set atime=off tank8Gbit/s

Edge (tuning 2)• HDS, HLS: segment size 300-

500K• Too many files• Prefetch isn’t working

Concatenate segmentsSecond NIC12 Gbit/s

HDD performancetotal = seek + rotate + transfer

Seek: 9ms (datasheet)Rotate: 7200rpm, 4.17 ms half turnTransfer: BlockSize/160MB/s

Total: 13ms + transfer time128KB: 0.8 ms => 71.4 IOPS, 9.1MB/s1MB: 6.4ms => 53.8 IOPS, 55MB/s

Edge (tuning 3)• zfs set recordsize=1M

ZFS larger block supportMatthew Ahrens

• HEAD: r274673• STABLE: r27608

• vfs.zfs.prefetch_disable=1• Limited by CPU: 20Gbit/s

Edge — configuration (CPU)

• Xenon E3 vs E5E3: 32GB RAM Limit

• 40Gbit => 50K clients• 1MB/client => 50GB buffers

• UP E5 vs DP E5High end => high priceLess PCIe slots

Edge — configuration (disk)SATA average transfer:• 160MB/s linear:

40/(.160*8) = 31 SATA HDD• 50MB/s random:

40/(.050*8) = 100 SATA HDDSSD cost:• x8 expensive $/TB• x4 less size

Edge — configuration (final)• CPU: 2x E5-2640v3

8 core @ 2.60GHz)• RAM: 128GB• NIC: Chelsio 40Gbit• HBA: LSI2308 + onboard• HDD: SATA x18 (storage)• SSD: 477GB x 8 (cache)

Edge — performance• 40 Gbit• 40-50K clients• 5Mpps

Edge — tips & tricks• No HT• Dedicated package for

apps• Dedicated package for

NIC• No NUMA

Warm content

Warm content (2)

Warm content (3)

200Gbit

P2P (WebRTC)• Streamroot (Fr)• Hot content: wins 67%

13:00

14:00

15:00

16:00

17:00

18:00

19:00

20:00

21:00

22:00

23:00

23:59

0

2.25

4.5

6.75

9

P2PCDN

P2P Architecture

• Viewer authenticates and initializes the connection with Integros load balancer

• At the same time, viewer connects to Streamroot tracker and requests peers

• Tracker sends back intelligently-selected list of peers• A P2P webRTC data channel is created among viewers to exchange

segments

INTEGROS LOAD BALANCER

STREAMROOTTRACKER

PEER POOL

% SERVERWebRTC

WebRTC

WebRTC

P2P Issues and Current Status• Issue Aug 2015: Significant traffic

offload followed by partial playback issues

• Reason: Flash-JavaScript interface:– Javascript based WebRTC module – Callback is not error resilient– Fun fact: not documented anywhere

in Flash• Solution: fixed by tweaking Flash-JS

interface• Next step: HTML5 first (hls.js / Dash.js)

to avoid Flash associated issues

P2P Future• 80% Offload• More clients

TODAY 2016Chrome YES YESFirefox YES YESOpera YES YESEdge BETA YESSafari NO ?iOS SDK NO YESAndroid SDK NO YESSTBs BETA YES

Linkshttps://www.youtube.com/watch?v=tujArcDozq0 (railsclub Ruby Meetup №2) https://events.yandex.ru/lib/talks/2694/ (ruBSD2104)slawa@integros.com