www.kurento.org

Developing rich multimedia applications with Kurento

Developing rich multimedia applications with Kurento: a tutorial for JavaScript developers

lulop@kurento.orghttp://www.kurento.org

http://www.nubomedia.eu

http://www.fi-ware.org

http://ec.europa.eu

www.kurento.org 2

WebRTC technologies from the perspective of a developer

Developing the client side

Developing the infrastructure

side

Before WebRTCFirst wave of

WebRTC technologiesSecond wave of

WebRTC technologies

Begin End Begin End

• APIs• Standards• FOSS

• Coherent APIs• Standards• FOSS• Multiplatform

Begin End• APIs• Standards• FOSS• Platforms• Out of the box

capabilities

2012 2014

www.kurento.org 3

What’s a WebRTC infrastructurePeer-to-Peer WebRTC Application (without media infrastructure)

WebRTC video stream

WebRTC Application with media infrastructure

Media infrastructure

www.kurento.org 4

Some WebRTC media infrastructure providers

www.kurento.org

Media infrastructure common capabilities

5

Transcoding media infrastructure

VP8 H.264

MCU media infrastructures

Recording media infrastructure

www.kurento.org

Can’t we go beyond?Media is

here Media gotthere

Media gotthere

AnalyzeTransform

Store

Transcoding

MCURecording

EnrichAugment

Adapt

SensorsContext

EventsMedia is

here

What WebRTC infrastructures are• Transcoding• MCU• Recording

What WebRTC infrastructures should be• Flexible processing• Augmented reality• Blending• Mixing• Analyzing• Etc.

6

www.kurento.org

• Interoperable media exchange (multiplatform/multiprotocol)• WebRTC, RTP, HTTP (video tag), etc.

• Standard capabilities Capabilities• Transcoding, MCU, recording

• Advanced capabilities• Computer vision, augmented reality, mixing, blending, etc.

Multimedia infrastructure

• REST API• JavaScript API• Java API

APIs

• LGPL 2.1

Is distributed through a flexible FOSS license

What’s Kurento

7

www.kurento.org

Future Internet

Multimedia Infrastructure

Simple Development

APIs

Kurento: the equation

8

www.kurento.org

Kurento Media Server (KMS): the nucleus of Kurento

• KMS is a middleware for media streams– Receives the stream– Process the stream– Issues the stream

SendReceiveAnalyze

AugmentEnrich

TransformTranscode

RecordProcess

Replicate

MediaSource

MediaSink

KMS

9

www.kurento.org

The Media API: The API for accessing KMS capabilities

SendReceiveAnalyze

AugmentEnrich

TransformTranscode

RecordProcess

Replicate

MediaSource

MediaSink

KMS

JavaMedia API

JavaScriptMedia API

RESTAPI

Applications define the processingof streams getting through KMS

10

www.kurento.org

Media API: Media Elements and Media Pipelines

Media Element

• Provides a specific media functionality

› Send/receive media

› Process media

› Transform media

• Exchange media through

› Sources

› Sinks

Media pipeline

• Chain of media elements implementing the desired media logic.

• The Media API provides the capability of creating media pipelines by joining media elements of the toolbox

Media Element

Sink

SRC

11

www.kurento.org

The Lego Game: Developers create applications connecting media elements

Application 2

Application 3

Application 1

Toolbox of media elements

12

www.kurento.org

Media API: trivial example (JavaScript)kwsMedia.create("MediaPipeline", function(error, pipeline){

pipeline.create("HttpGetEndpoint", function(error, httpGetEndpoint){pipeline.create("PlayerEndpoint", {uri : file_uri}, function(error, playerEndpoint){

playerEndpoint.connect(httpGetEndpoint, function(error){httpGetEndpoint.getUrl(function(error, url){videoInput.src = url;});playerEndpoint.play(function(error){});

});});

});});

Media Pipeline (inside KMS)

HttpGetEndpoint

Media fromfile or URI

HTTP mediastreaming Si

nk

SRC

PlayerEndpoint

13

www.kurento.org

Kurento Architecture: putting it all together

Kurento Media Server (KMS)

ReceiveVideo

AugmentedReality

SendVideo

Computer Vision

Video Playingand Recording

Kurento WS JSON-RPC API

Med

ia

Med

ia

Kurento Media Connector (KMC)

14

SIP Servlet

SOAP

HTTPServlet REST

Browser

Developer code with specific application

logic (pipelines)

Media capabilities of the browser

Kurento Media API

Specificlogic

www.kurento.org

Application execution flowClientCode

Media Connector(KMC)

MediaServer (KMS)

Crate pipeline/element…(JSON-RPC)

Commands requestingthe creation of a pipeline

Reference to media object …(JSON-RPC)

Pipelinecereation

phase

Media exchange

phase

1

2

Specific logic(e.g. AAA)

Mediapipelinecreation

Media exchange between client and server

15

www.kurento.org 16

What about Node.js?

Kurento Media Server (KMS)

ReceiveVideo

AugmentedReality

SendVideo

Computer Vision

Video Playingand Recording

Kurento WS JSON-RPC API

Med

ia

Med

ia

Kurento Media Connector (KMC)

SIP Servlet

SOAP

HTTPServlet REST

Node.js Pipeline creation +

AAA

Kurento Media API

BrowserBrowser-side code

www.kurento.org

Let’s develop with Kurento• What you need

– A Kurento instance• You can install your own Kurento instance

– http://www.kurento.org/docs/current/

• You can launch a Kurento instance at the FI-LAB (FI-WARE project)– http://lab.fi-ware.org – FI-LAB is a cloud infrastructure making possible to execute instances from images or blueprints

• kws-media-api – https://github.com/kurento/kws-media-api– http://github.com/kurento/kws-utils

» Node.js• npm install kws-media-api• npm install kws-utils

» Browser• bower install kws-media-api• bower install kws-utils

• Getting help– Kurento web site

• http://www.kurento.org

– Kurento mailing list• https://groups.google.com/forum/#!forum/kurento

– Twitter• @kurentoms

17

www.kurento.org

Kurento Hello World: Playing a file with an HTML5 client

Media Pipeline

HttpGetEndpoint

Media fromfile or URI

HTTP mediastreaming Si

nk

SRC

PlayerEndpoint

Kurento WS JSON-RPC API

Create / excec …

Ref / answer …

18

Kurento Media Connector (KMC)

www.kurento.org

Playing a file with an HTML5 client: JavaScript code

const ws_uri = “ws://your.kurento.server:8888/thrift/ws/websocket”; //change thisconst file_uri = “http://uri.to.video.file/”; //change this

var videoInput = document.getElementById("videoInput"); //video tag element

KwsMedia(ws_uri, function(kwsMedia){ //instantiate kws-media-api managerkwsMedia.create("MediaPipeline", function(error, pipeline){ //instantiate pipeline

pipeline.create("HttpGetEndpoint", function(error, httpGetEndpoint){ pipeline.create("PlayerEndpoint", {uri : file_uri}, function(error, playerEndpoint){

playerEndpoint.connect(httpGetEndpoint, function(error){httpGetEndpoint.getUrl(function(error, url){

videoInput.src = url;});

playerEndpoint.play();

});

});});

});});

Source: https://github.com/Kurento/kws-tutorial/tree/develop/VideoTagPlayer

19

www.kurento.org

Playing a file with an HTML5 client: HTML code

<!DOCTYPE html><html><head><meta charset="US-ASCII"><title>Insert title here</title>

<script type="text/javascript" src="../api/kws-media-api.js"></script><script type="text/javascript" src="demo.js"></script>

</head><body>

<button id="playButton">Click to start playing</button><br><video id="videoInput" autoplay></video>

</body></html>

Source: https://github.com/Kurento/kws-tutorial/tree/develop/VideoTagPlayer

20

www.kurento.org

Playing a file with an HTML5 client: See the example working

https://www.youtube.com/watch?v=Mp6UEL9U-v4

21

www.kurento.org

WebRTC loopback

Kurento Media Server

WebRTCStreaming

Sink

SRC

22

WebRtcEndpoint

www.kurento.org

WebRTC loopback: JavaScript code

23

Source: https://github.com/Kurento/kws-tutorial/tree/develop/WebRtcLoopback

const ws_uri = “ws://your.kurento.server:8888/thrift/ws/websocket”; //change this

var videoInput = document.getElementById("videoInput");var videoOutput = document.getElementById("videoOutput");

var webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput, onOffer, onError);

function onOffer(offer){KwsMedia(ws_uri, function(kwsMedia){

kwsMedia.create("MediaPipeline", function(error, pipeline){

pipeline.create("WebRtcEndpoint", function(error, webRtc){

webRtc.processOffer(offer, function(error, answer){

webRtcPeer.processSdpAnswer(answer);

});

webRtc.connect(webRtc);

});});

});};

23

www.kurento.org

WebRTC loopback: HTML code

24

<!DOCTYPE html><html><head><meta charset="US-ASCII"><title>Insert title here</title>

<script type="text/javascript" src="../api/adapter.js"></script><script type="text/javascript" src="../api/kws-media-api.js"></script><script type="text/javascript" src="../api/kws-utils.js"></script><script type="text/javascript" src="demo.js"></script>

</head><body>

<button id="launchButton">Click me ...</button><br><div style="float: left; border: 2px solid black; margin: 5px;">

<h1>Local stream</h1><video id="videoInput" autoplay></video>

</div><div style="float:left; border: 2px solid black; margin: 5px;">

<h1>Remote stream</h1><video id="videoOutput" autoplay></video>

</div></body></html>

Source: https://github.com/Kurento/kws-tutorial/tree/develop/WebRtcLoopback

24

www.kurento.org

WebRTC loopback: see the example working

25

https://www.youtube.com/watch?v=hUChVutQ2og

25

www.kurento.org

WebRTC with FaceOverlay

Media Pipeline

WebRTCStreaming

Sink

SRC

26

Sink

SRC

WebRtcEndpoint

FaceOverlayFilter

www.kurento.org

WebRTC with FaceOverlay: JavaScript code

27

Source: https://github.com/Kurento/kws-tutorial/tree/develop/FaceOverlay

var webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput , onOffer, onError);

function onOffer(offer) { KwsMedia(ws_uri, function(kwsMedia) { kwsMedia.create("MediaPipeline", function(error, pipeline) { pipeline.create("WebRtcEndpoint", function(error, webRtc) { pipeline.create("FaceOverlayFilter”, function(error, filter) {

//hat_uri is the uri pointing to the overlay image to draw on top of detected faces //These percent variables need to be defined to adjust overlay image to face box

filter.setOverlayedImage(hat_uri, offsetXPercent, offsetYPercent, widthPercent, heightPercent); webRtc.connect(filter, function(error) { filter.connect(webRtc, function(error) { webRtc.processOffer(offer, function(error, answer) {

webRtcPeer.processSdpAnswer(answer);});

…//many }); omited here for simplicity});

27

www.kurento.org

WebRTC recorder

2828

Media Pipeline

WebRTCStreaming

Sink

SRC

Sink

Mediato

file or URI

Media Pipeline

WebRTCStreaming

Sink

SRC

Mediafrom

file or URI

SRC

WebRtcEndpoint

WebRtcEndpoint

RecorderEndpoint

PlayerEndpoint

www.kurento.org

WebRTC Recorder: JavaScript codewebRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput, onOffer, onError);

function onOffer(offer) { KwsMedia(ws_uri, function(kwsMedia) {

kwsMedia.create('MediaPipeline', function(error, pipeline) { pipeline.create('RecorderEndpoint', {uri : file_uri}, function(error, recorder) {pipeline.create('WebRtcEndpoint', function(error, webRtv) {

webRtc.connect(recorder, function(error) {recorder.record();webRtc.connect(webRtc, function(error) { webRtc.processOffer(offer, function(error, answer) { webRtcPeer.processSdpAnswer(answer); });

... }); //many }); omited here for simplicity}

29 29

Source: https://github.com/Kurento/kws-tutorial/tree/develop/Recorder

www.kurento.org

WebRTC with Recorder: see the example working

30

https://www.youtube.com/watch?v=wp5A4NZ-kd4

30

www.kurento.org

But … is this code beautiful?

3131

Developers love spaghetti for eating, but not for code

From that code structureto this code structure

www.kurento.org

Avoiding the callback hell• The hidden secret of the content API

– Every call to a kws-media-api returns a promise• var p1= kwsMedia.create(“MediaPipeline”);• p1.then(function(pipeline){…})

– Promises• A proxy to a value not knows when the proxy is created• Allows associating handlers to

– Value successfully obtained– Error

• https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise

• http://www.html5rocks.com/en/tutorials/es6/promises/

– Example

32 32

www.kurento.org

Are promises really useful for our objectives?

• Developing with promises– WebRTC loopack with FaceOverlay

• Source– https://github.com/Kurento/kws-media-api/tree/develop/example/WebRtcEndpoint-

FaceOverlayFilter_promises_2

• Does structure improve?– Not much– Problems

• Nesting avoiding, but chaining isn’t much beautiful• Nesting eliminated => closures eliminated

– Scope of variables (pipelines, elements, etc.) lost.– Need to pass variables along on each call

• What’s the point of using promises then?– Continue reading if you are the kind of guy of like living on the edge …

33 33

www.kurento.org 34

Generators+co: the callback hell killer

• Generators– Cooperative (non preemtive) simulation of synchronous

behavior on iterator functions by maintaining execution state• https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/

Iterators_and_Generators

– Allow multiple entry points for suspending and resuming execution at certain locations.

– Available on EC6 (Harmony)• Co

– Combines promises and generators– Uses promises to “connect” generator exit and reentrance– https://github.com/visionmedia/co

www.kurento.org 35

Revisiting the WebRTC with loopback example

var webRtcPeer = kwsUtils.WebRtcPeer.startSendRecv(videoInput, videoOutput, onOffer, onError);

function onOffer(offer){co(function*(){

var kwsMedia = yield KwsMedia(ws_uri);var pipeline = yield kwsMedia.create("MediaPipeline");

var webRtc = yield pipeline.create("WebRtcEndpoint");var filter = yield pipeline.create("FaceOverlayFilter");

//you need to define the hat_uri and percents variablesyield filter.setOverlayedImage(hat_uri, offsetXPercent, offsetYPercent, widthPercent, heightPercent);

var answer = yield webRtc.processOffer(offer);webRtcPeer.processSdpAnswer(answer);

yield webRtc.connect(filter);yield filter.connect(webRtc);

})();}

Source code of previous examples using generators:https://github.com/Kurento/kws-tutorial/tree/develop/WebRtcLoopback https://github.com/Kurento/kws-tutorial/tree/develop/FaceOverlayGeneratorhttps://github.com/Kurento/kws-tutorial/tree/develop/RecorderGenerator

www.kurento.org 36

Isn’t it beautiful ?

www.kurento.org

Let’s do some more complex stuff: a WebRTC game

Media Pipeline

Sink

SRC

Sink

SRC

Sink

SRC

37

Sink

SRC

Sink

WebRtcEndpoint

MirrorFilter PointerDetectorFilter

FaceOverlayFilter

RecorderEndpoint

www.kurento.org

WebRTC game: code

38

Source: https://github.com/Kurento/kws-tutorial/tree/develop/RecorderFilterGenerator

38

www.kurento.org

WebRTC game: see the example working

39

https://www.youtube.com/watch?v=5eJRnwKxgbY

39

www.kurento.org 40

Summary and conclusions• Kurento

– Open Source Softare• LGPL v2.1

– Multimedia infrastructure for HTML5• WebRTC• HTML5

– Advanced media processing capabilities• Recording, MCUs, Transcoding, Computer Vision, Augmented Reality, Mixing, Blending.

• kws-media-api– Access Kurento capabilities from JavaScript

• Media elements and media pipelines• Like a “Lego game”

– Beautiful code• Promises• Generators• CO

www.kurento.org

Thank you

41

Suggestions, comments and complains:lulop@kurento.org

http://www.kurento.orghttp://www.github.com/kurentoTwitter: @Kurentoms

http://www.nubomedia.eu

http://www.fi-ware.org

http://ec.europa.eu