2016 streaming media west - how to improve video quality of service with an effective ab testing...

How to improve quality of service with player-side iterations and an effective AB testing workflow

Streaming Media West – Track DTuesday, November 1, 2016 2:45 to 3:30 pm

● Creators of peer-accelerated video delivery for large-scale OTT

● Pioneers in hybrid video delivery systems to accompany exponential growth

● Experts in HTML5 video through integrations into open-source & proprietary players

Infinite scale, limitless delivery.

Streamroot: Who are we?

I. How to build an AB Testing workflowWhen and what to test, analytics pipeline, AB testing setup

II. Dailymotion use case: ABR iterations for quick, dramatic QoS improvements

ABR iterations to hls.js to cut rebuffering

III. Streamroot use case: using effective AB testing to improve our productStreamroot workflow and example

Presentation Outline



AB Testing is Data-Driven

Development.…here applied to Media engines & QoS.

I. Building an effective AB testing workflow



AB testing makes sense when:

You can’t test locally: distributed and unmanaged networks (ABR, P2P).

Test subjective hypothesis, magic numbers.

You deploy a completely new feature that breaks your old logic.



Before starting:

Define your metrics

Determine an appropriate sample size and testing timeframe

Define your variables

Define your populations


For testing, you need:

The right data

A DB to store it

A BI/analytics tool



I. How to build: Data pipelineGeneral workflow


Streamroot workflow

I. How to build: Data PipelineGeneral workflow

Axel Delmas

You can also put a map/reduce framework between the message broker and the analytics database.(or any kind of data processing consumer. Could be a hyperloglog consumer, or whatever)

Many options based on your requirements: open-source, proprietary, Cloud...


I. How to build: Data PipelineWhat to chose from

Message broker / stream processor

Storage + possible query/visualization

BI/viz tools Advanced analytics

Kafka (Kinesis) Druid Caravel R

RabbitMQ InfluxDB Grafana Matlab

ActiveMQ Cassandra (DynamoDB) Tableau

PostGresSQL QlikView

ElasticSearch Kibana


I. How to build: AB Testing workflowGeneral setup

Client AB testing methods:

1) Dynamic Config Injection2) Different builds with a reverse proxy


I. How to build: AB Testing workflowGeneral setup


I. How to build: AB Testing workflowConfig injection


I. How to build: AB Testing workflowDynamic config injection

PROs

Simple & easy splitsSeparate - fast to deploy & rollbackScalable

CONs

Limited scope Complications in codeNeed to handle scaling & high availability

WARNING

Async implementation!


I. How to build: AB Testing workflowDifferent builds with reverse proxy


PROs

Serve very different (major) versions of playerPerfect for rolling out new features smoothly (10% then 30% then...)

CONs

Good CI/CD necessary to roll out new versions quicklyHeavy stack to maintain and scale

WARNING

Make sure the file request is not slower!Single point of failure!

I. How to build: AB Testing workflowDifferent builds with reverse proxy

1. AB test when it makes sense.

2. Choose the right metrics.

3. Build a pipeline with tools that are adapted to your use case.

4. Use the right method depending on what you’re testing.


I. How to build: AB Testing workflowTips & sum up


II. DailymotionSome numbers: world’s 2nd largest video hosting platform

100 million video views each day 63%

Desktop

31%Mobile

8%Tablet

1%SmartTV

69%HTML5

31%Flash


II. DailymotionABR iterations for quick, dramatic QoS improvements

I should have rehearsed

this more…When’s lunch?

Adaptive bitrate algorithms can have a huge impact on QoS.

They are easy to tweak, but improvements depend on your specific use case.

One size does not fit all.


II. DailymotionABR iterations for quick, dramatic QoS improvements


II. DailymotionImproving hls.js’s ABR logic



Introduce history parameter to bandwidth estimation.



Average out the values.

https://github.com/dailymotion/hls.js/blob/master/src/controller/abr-controller.js

EWMA:



Shaka Player:

https://github.com/dailymotion/hls.js/blob/master/src/controller/abr-controller.js

ABR now based on two bandwidth moving averages:

• Fast → adapt down quickly

• Slow → adapt up more slowly

Bandwidth estimation = min(fast, slow)



Control over media engine

Control over events

QoS events

AB testing & customizationABR algos, config, etc.


CDNDirector

Analytics aggregation and visualization

II. DailymotionAB testing workflow: total control over their stack

67.4% native

70.3%hls.js

78.6%flashls


II. DailymotionComparing the results - live

Rebuffering, percentile rank

native flashls hls.js,s=9,f=3 hls.js,s=0,f=0 hls.js,s=6,f=1


II. DailymotionComparing the results - live

Rebuffering, percentile rank

hls.js,s=0, f=0

hls.js, s=9, f=3


II. DailymotionNumber of level switches - live

With the right workflow, big improvement in less than 2 weeks of testing.

Numbers are use-case specific (segment length, network, etc.).

Lots of other options to test → a continuous effort to improve QoS!


II. DailymotionConclusions


III. StreamrootPeer-accelerated delivery

...We have a very distributed network!

AB testing on:

- matching devices to get the best connections- Transfer algorithms and protocol improvements- QoS impact and improvements


III. StreamrootWhy are we using AB testing


III. StreamrootThe pipeline

GET build-a.js

Pipeline

AB testing File Distribution

AB Test on AskSegment


III. StreamrootThe test case


III. StreamrootResults

AskSegment Timeout Samples P2P %

Longer 15,772,542 67.41%

Shorter 16,152,624 67.43%

No effect?


III. StreamrootResults

AskSegment Timeout Type Samples P2P %

Longer Live 9,004,140 66.43%

Shorter Live 9,340,504 68.02%

Longer VOD 6,768,403 68.72%

Shorter VOD 6,812,120 66.61%

Live: +1.59% for shorter timeoutVOD: +2.11% for longer timeout

- Understand and know your data

- Ab testing is sometimes the only way to prove your hypothesis

- You will be wrong sometimes: Make sure you can rollback quickly!!

- The most difficult is to start! Once it is set up: be relentless!

Continuous AB testing FTW


III. Streamroot Takeaways


Questions?

Nikolay Rodionov, Co-Founder and CPO, [email protected]

Erica Beavers, Head of Partnerships, [email protected]


Contact us!

mailto:[email protected]

mailto:[email protected]

Axel Delmas

You have to make a choice for the talk: you wrote COO at the begining of the script

2016 streaming media west - how to improve video quality of service with an effective ab testing...

Engineering