Live HTTP Streaming of Video and

Subtitles within a Browser

Cyril Concolato

Jean Le Feuvre

{concolato, lefeuvre} @ telecom-paristech.fr

MMSys 2013

Context

1

■ Video Streaming is popular on the Web• With subtitle support in VOD

• With live streaming solutions (DASH, HLS, …)

■ What about live subtitles ?• Several specifications being developed

− ISOBMFF Amendment on Timed Text and Timed Overlays, for use with DASH

− WebVTT draft (used in HLS v10)

• No deployment yet within browsers

■ Can we today stream and play live video+subtitleswithin a browser?• Yes !

• Development of a Proof-of-Concept

Live HTTP Streaming of Video and Subtitles within a Browser

- MMSys Demo 2013

Architecture

3Live HTTP Streaming of Video and Subtitles within a Browser

- MMSys Demo 2013

Demonstration Media Formats

■ The ISO Base Media File Format

• To carry video stream segments

• Conformant to MPEG DASH and Google Chrome restrictions

■ The WebVTT format

• To carry subtitle data

− as separate text files (not multiplexed in the video container)

− Similar to HLS v10, with some differences (no embedded video timing)

• Generation of non-overlapping, constant duration segments (compatible with MPEG DASH)

■ MPEG DASH MPD

• To represent a dynamic live/long-running streaming service

• To enable synchronization of the video and subtitle streams.

4Live HTTP Streaming of Video and Subtitles within a Browser

- MMSys Demo 2013

Demonstration software components

■ GPAC MP4Box

• To produce video segments on-the-fly and delete old ones

− From a pre-recorded file (looped indefinitely)

■ VTTLiveGenerator

• Ad-hoc tool

• To produce subtitle segments on-the-fly, synchronized with the video and delete old ones

■ Google Chrome

• Canary version

• To display the content using HTML5 and JavaScript

5Live HTTP Streaming of Video and Subtitles within a Browser

- MMSys Demo 2013

Usage of MP4Box

■ Fragmenting a file repeatedly using -dash-ctx option

MP4Box.exe -dash-live 2000 -rap -dash-ctx dash-live.txt -segment-name counter-live counter.mp4

■ Example of MP4Box’s DASH context[DASH]

InitializationSegment=counter_liveinit.mp4

BitstreamSwitching=yes

InitializationSegmentSize=860

MaxSegmentDuration=1.000000

NextSegmentIndex=11

NextFragmentIndex=11

PeriodDuration=9.960000

[TrackID_1]

NextDecodingTime=249000

6Live HTTP Streaming of Video and Subtitles within a Browser

- MMSys Demo 2013

Example of Video Content

7Live HTTP Streaming of Video and Subtitles within a Browser

- MMSys Demo 2013

Example of WebVTT Segment

■ Each segment is a complete WebVTT file with multiple cues, • changing header, cue indices, and timing values

WEBVTT Segment 6

61

00:00:12.200 --> 00:00:12.400

This is cue 61 (start: 00:00:12.200 -- end: 00:00:12.400)

...

70

00:00:14.000 --> 00:00:14.200

This is cue 70 (start: 00:00:14.000 -- end: 00:00:14.200)

8Live HTTP Streaming of Video and Subtitles within a Browser

- MMSys Demo 2013

Example of DASH Description

■ Use of a “dynamic” MPD

• No minUpdate attribute

• Without updates (assuming known/static timing)

■ A single period with 2 adaptation sets

• 1 video representation

• 1 subtitle representation

• Same video and subtitle segments duration (easy)

■ live-vtt-mp4.mpd

9Live HTTP Streaming of Video and Subtitles within a Browser

- MMSys Demo 2013

Rendering using Google Chrome Canary

■ HTML page

• Use of HTML5 <video> element with an HTML <track> element for playback synchronization

• Use of JavaScript code to control the download logic

■ JavaScript algorithm

• Fetch and parse the MPD,

• Create the <video> and <track> elements,

• Determine and fetch the video and subtitles segments using the XMLHttpRequest API,

• Feed the video segments to the media decoder engine using the Media Source extension API,

• Load each subtitle segment using an HTML5 work-around

10Live HTTP Streaming of Video and Subtitles within a Browser

- MMSys Demo 2013

Demo

11Live HTTP Streaming of Video and Subtitles within a Browser

- MMSys Demo 2013

Discussion/Limitations

■ Working demo• Supports time shifting or not

• Needs better integration of video & subtitle MPDs at server side

• Needs at client side:

− Better MPD parsing

− Better simple segment index algorithm

■ Issues• Rendering performance problem if more than 4 cues per second

• User Interface problem

− Unusable built-in UI for long-running live events

− Needs custom UI to disable seeking before the first received data

• HTML 5 limitations to load cues

− No MSE equivalent for cues (outside of VTT in MP4)

− No easy method to parse cues

12Live HTTP Streaming of Video and Subtitles within a Browser

- MMSys Demo 2013

Questions ?

Come and see me