Open APIs for Open Minds

The Stream Oriented Generic Enabler – SO GE

Developing rich multimedia applications with FI-WARE.

Speakers

2

Luis López

• Leader of the Kurento project

• As. Prof. at URJC

• Coordinator of the Stream Oriented GE at FI-WARE

Jose Antonio Santos

• Chief Architect of Kurento project

• Senior Developer at Naevatec

• Researcher at FI-WARE project

The Stream Oriented Generic Enabler – SO GE

3

• Send and receive media (Multidevice/multiprotocol)• Process media (Computer vision, augmented reality, media indexing, etc.)• Transform and adapt media (H.264, H.263, VP8, Ogg, and others)

Provides multimedia capabilities to the FI-WARE infrastructure

• No need of protocol/codec low level understanding

Exposes those capabilities through a simple to use API

• LGPL 2.1

Is distributed through a flexible FOSS licence

• Sorry for the instabilities…

Is still work in progress

How media APIs (usually) work

4

Client side- WWW browser- Native API

Server side- Media repositories- Media capabilities

Hey, I want to send/receive media in this way

Here you have the media you requested

API API

The SO-GE API is a WWW inspired API

5

Dynamic WWW page development

• Lots of capabilities (ex. Java EE)

Multimedia development with the SO-GE

• Don’t want to loose any capability

Create HTML- DDBB access- Transactions- Security tools- Etc.

HTTP request: I want this resource

HTTP response:The resource

Create Media- Media API- DDBB access- Transactions- Security tools- Etc.

HTTP request: I want this media

HTTP response:The media is

here

The SO-GE Media API: media elements and pipelines

6

Media Element

• Provides a specific media functionality

› Building block

› Send/receive media

› Process media

› Transform media

• The Media API provides a toolbox of media elements ready to be used.

• New media elements can be added

Media pipeline

• Chain of media elements implementing the desired media logic.

• The Media API provides the capability of creating media pipelines by joining media elements of the toolbox

Media Element

Sink

SRC

The global vision of a SO-GE application

7

SO-GE Media Server Infrastructure

DecodeVideo

AugmentedReality

Encode Video

Computer Vision

Video Playingand Recording

Examples of media elements: HttpEndPoint

8

HttpEndPoint

Function

• Sends/receives media basing on HTTP transports (HTML5 video tag)

• Associates media to a temporary-one-time-use URL

Formats

• WebM

• H.264 support under development

Input parameters

• None

HttpEndPoint

Media Source

Media Sink

HTTP POST requests

HTTP answers

Example of media elements: RtpEndPoint

9

RtpEndPoint

Function

• Sends/receives media basing on RTP transports (Real Time Communications).

• Negotiates media parameterization through SDPs

Formats

• H.263, MPEG4, H.264, VP8

Input parameters

• Input SDP

RtpEndPoint

Media Source

Media Sink

RTP inputmedia

RTP outputmedia

Example of media elements: PlayerEndPoint

10

PlayerEndPoint

Function

• Plays media from file or URI.

Formats

• WebM

• MP4 (H.264)

• 3GPP

• AVI

• Etc.

Input parameters

• File URI

PlayerEndPoint

Media Source

Media fromfile or URI

Example of media elements: RecorderEndPoint

11

RecorderEndPoint

Function

• Records media into a file.

Formats

• WebM

• H.264 formats under development

Input parameters

• File path

RecorderEndPoint

Media Source

Media Sink

Media tofile

Example of media elements: JackVaderFilter

12

JackVaderFilter

Function

• Example of filter

• Recognizes faces

• Adds a Jack Sparrow or Darth Vader hat onto faces

Formats

• Raw

Input parameters

• None

JackVaderFilter

Sink

SRCMedia

StreamAugmented

Media stream

Example of media elements: ZBarFilter

13

ZBarFilter

Function

• Example of filter

• Recognizes Bar and QR codes

• Generates events

Formats

• Raw

Input parameters

• None

ZBarFilter

Sink

SRCMedia

StreamMediastream

Event

Developing media application with the SO-GE

Runtime environment

• Install Kurento Media Server (soon available at FI-LAB)

• Install JBoss Java EE container (soon available at FI-LAB)

Development environment

• Maven (repository: http://repository.kurento.com/archiva/repository/snapshots)

• Soon available at maven central

› kmf-content-api

› Provides handers

› kmf-media-api

› Provides media elements and pipelines

• Eclipse

• Javascript libraries (available at the same repository)

› kws-content-api

14

My first application with the SO-GE: play a video

15

HttpEndPoint

Media Source

Media Sink

PlayerEndPoint

Media Source

Media fromfile or URI

HTTP mediastreaming

Client side Server side

My first application with the SO-GE: play a video

Source code …

16

My second application: including a simple filter

17

HttpEndPoint

Media Source

Media Sink

PlayerEndPoint

Media Source

HTTP mediastreaming

JackVaderFilter

Sink

SRC

Client side Server side

My second application: including a simple filter

Source code …

18

Making it more fun: introducing real time media

19

Media Source

Media Sink

Real timemedia

HTTP mediastreaming

JackVaderFilter

Sink

SRC

RtpEndPoint

Media Source

Media Sink

Client side Client sideServer Side

Making it more fun: introducing real time media

Source code …

20

Programming with media events: the ZBarFilter

21

Real timemedia

ZBarFilter

Sink

SRC

RtpEndPoint

Media Source

Media Sink

HttpEndPoint

Media Source

Media Sink

Media events to browser

Client side Client sideServer Side

Programming with media events: the ZBarFilter

Source code …

22

Digging into the media API: implementing your own filters: requirements Requirements

• GStreamer

› Based on GObject and GLib

› C programing skills needed

• OpenCV or your preferred image processing software

• C++ basic knowledge

• Thrift basic knowledge

23

Digging into the media API: implementing your own filters: GStreamer media element implementation Download media server source code

Look for sub-module gst-kurento-plugins

Enter into src/filters folder

Create your filter as couple of filtername.c filtername.h files

• Your filter must inherit from GstVideoFilter

• You have to implement virtual method transform_frame_ip

› This method gives you each frame to be processed

• Be aware of init and finalize functions if you use attributes that need to be freed

24

Digging into the media API: implementing your own filters: Integration in media server Modify kms-interface/mediaServer.thrift file adding a new value to enum FilterType

Create a C++ file to manage your GStreamer filter (by convention we use the same name in camel case)

• This class inherits from kurento::Filter class

• Its constructor creates a “filterelement” gstreamer element and add your filter name as a parameter (it will be created automatically)

• Destructor should free this element

Modify kurento::MediaPipeline::createFilter method in order to add your filter type in the switch statement

25

Facing the FI-WARE challenge using the SO-GE

Getting started• Use the FI-WARE Catalogue (ready since 15 minutes ago hopefully!)

› http://catalogue.fi-ware.eu/enablers/stream-oriented-kurento/• Ask the SO-GE team around the FI-WARE stand

Augmenting videos in novel ways• Sensor information on live stream videos

› Pollution/noise/traffic Red-filtering level• Combine audio-visual & sensor information through augmented reality

› Thermometers, charts, gauges on top of a media flow.

Make a camera to become an advanced sensor using computer vision• Traffic status

› Car counter• Weather

› Sky/ground color

Conceive novel WWW and Smartphone interaction GUIs• Up to your imagination

26

http://fi-ppp.eu

http://fi-ware.eu

Follow @Fiware on Twitter !

Thanks !

27

Technical details: Javascript KwsContent API

Creating a media connection

• conn = new KwsContent(uri, options);

• Options

› remoteVideoTag: id of the video tag rendering the remote stream

› Others for WebRtc (currently not available)

• Semantics

› Connects to URI and requests media

› Upon media reception, renders it on the specified video tag

Adding event handlers

• conn.onstart( function …

• conn.onremotestream (function …

• conn.onterminate( function …

• conn.onmediaevent( function …

28

The SO-GE Content API: handlers

Handler

• Is implemented by the application developer (equivalent to a Servlet)

• Defines the sequence of instructions executed when a request for media is received

Four types

• PlayerHandler

› Handles requests for playing media with HTTP transport

› Compatible with the HTML5 video tag

• RecorderHandler

› Handlers request for recording media with HTML transport

› Compatible with HTTP file upload mechanism

• RtpMediaHandler

› Handlers request for establishing RTP media sessions

• WebRtcMediaHandler (under development)

29

Technical details: the SO-GE Architecture

30

Pla

yerH

anld

er i

nte

rnal

det

ails

: se

qu

ence

dia

gra

m

31