video streaming on e-lab

R. B. Henriques | Lisbon, 23 March 2012 | Video Streaming

e-labInstituto Superior TécnicoLisboa, Portugalhttp://elab.ist.utl.pt

Video Streaming

Rafael Bagagem [email protected]

2 R. B. Henriques | Lisbon, 23 March 2012 | Video Streaming

Streaming multimediaStreaming multimedia

● Multimedia that is constantly received by, and normally displayed to, the end-user

● A media stream can be on demand or live● Usually applied to media that are distributed

over telecommunications networks— Radio and TV are inherently streaming

● Real world examples— Youtube, on-line TV and Radios, etc.


OSI ModelOSI Model

● A set of seven layers that define the different stages that data must go through to travel from one device to another over a network


OSI Model – layers and OSI Model – layers and instancesinstances

● Layer – a collection of conceptually similar functions that provide services to the layer above and receives service from the layer below.

● Instance – on each layer an instance provides services to the instances at the layer above and requests service from the layer below; conceptually two instances at one layer are connected by a horizontal protocol connection on that layer.


OSI Model – layers OSI Model – layers descriptiondescription

● Physical Layer – electrical and physical specifications (layout of pins, voltages, cable specifications, hubs, network adapters …

● Data Link Layer – functional and procedural means to transfer data; detect/correct errors on Physical.

● Network Layer – functional and procedural means of transferring variable length data via one or more networks; performs network routing functions; perform fragmentation/reassembly, and report delivery errors; maintain the QoS requested by the Transport.

● Transport Layer – provides transparent transfer of data between end users, providing reliable data transfer services to upper layers; controls reliability of a given link through flow control, segmentation and de-segmentation, and error control; keep track of the segments and retransmit those that fail.


OSI Model – layers OSI Model – layers descriptiondescription

● Session Layer – controls the dialogues between computers; establishes, manages and terminates the connections between the local and remote application; provides for full-duplex, half-duplex, or simplex operation, and establishes check-pointing, adjournment, termination, and restart procedures.

● Presentation Layer – independence from differences in data representation (e.g., encryption) by translating from application to network format, and vice versa; works to transform data into the form that the application layer can accept; formats and encrypts data to be sent across a network, providing freedom from compatibility problems.

● Application layer – interacts with software applications that implement a communicating component; functions include identifying communications partners and resource availability, and communications synchronization.


Transport Layer (I)Transport Layer (I)

● Most common protocols for media broadcast are UDP and TCP

● User Datagram Protocol (UDP)

— Unreliable: no concept of acknowledgement, retransmission and time-out

— Unordered - the order in which they arrive cannot be predicted.

— Lightweight - Small transport layer designed on top of IP.

— Datagrams - Packets are sent individually and are guaranteed to be whole if they arrive.


Transport Layer (II) Transport Layer (II)

● Transmission Control Protocol (TCP)

— Reliable: manages message acknowledgement, retransmission and time-out

— Ordered - the order is well known.

— Heavyweight - Large transport protocol designed on top of IP

● TCP requires three packets just to set up a socket, before any actual data can be sent....

— Streaming - nothing distinguishing where one packet ends and another begins. Packets may be split or merged into bigger or smaller data streams arbitrarily.


Streaming Media Streaming Media ProtocolsProtocols

● MMS – Microsoft Media Services

— Deprecated in favour of RTSP

— UDP or TCP● RTP/RTSP – Real Time Streaming Protocol

— Open, widely used● PNM/PNA

— Real Audio/Networks proprietary● RTMP – Real Time Messaging Protocol

— Proprietary protocol developed by Adobe Systems


Broadcasting raw dataBroadcasting raw data

● The number of bytes produced in one second is given by:— Image width x Image height x Colours (typically

RGB=3 at 8 bits) x number of frames per second (FPS)

— 10FPS with 320 x 240 RGB image:

— 2304000 bytes/s ~ 2.3MB/s ~ 18432000 bits/s ~ 18.4Mbits/s


Problems with this Problems with this approachapproach

● Typically a user has at home a bandwidth of something between 2 and 24 Mbits/s

● If 10 users are connected the bandwidth on the server grows to 180Mbits/s!!!

● Not so good solutions— lower the image size

— Use only one color (gray scale)


SolutionSolution

● Compress the image● Two approaches

— Compress each image

— Compress over time● This is clearly the most clever solution since most of the

time the images don't change that much!

● CODECs are specialized algorithms to perform this work


Frame compressionFrame compression

● Compress each frame:

● Compress over time:— The difference between two the successive frames


Video compressionVideo compression

● Lossless— Ideal but not feasible for video broadcast. The

bandwidth is still to high

— Examples are: PNG, JPEG 2000, ZIP

● Lossy— Define the best relation between quality loss and

data reduction

— Find a format which allows to maximize the previous point


Lossy compressionLossy compression

● The big trick:— Remove information from video that is not important

for human perception in order to achieve very high compression rates while still keeping very good visual quality.

● Some video formats allow ratios of 200:1 !


Common Video Coding Common Video Coding FormatsFormats

● H.261● H.263● H.264

— x264

● MPEG-4 AVC (Advanced Video Coding)

— Xvid, FFmpeg, DivX, x264...

● WMV● RealVideo


Why x264?Why x264?

● Open source● H.264/MPEG-4 AVC video CODEC. Lots of

advanced encoding features● 200:1 ratios and more...● Available for free in every operating system● Supported by most popular video players:

— Quicktime, VLC, Windows Media Player 12, mplayer


Video sizesVideo sizes

● VGA (Video Graphics Array)● QQVGA 160 × 120 quarter of a quarter of VGA

● HQVGA 240 × 160 half of a quarter VGA

● QVGA 320 × 240 e-lab videos (quarter of VGA)

● VGA 640 × 480 VGA

● XGA 1024 × 768 Extended Graphics Array


Best bit ratesBest bit rates

● As discussed before, the output of the CODEC must maximize the quality of the image at the lowest bandwidth

● 8 kbit/s — telephone quality (using speech codecs)

● 120kbit/s — elab videos● 5 Mbit/s — DVD quality● 15 Mbit/s — HDTV quality


Multicast vs UnicastMulticast vs Unicast

● Multicast— Streams the same media to all the receivers

— It is the perfect solution... but NO Internet Provider will allow multicast on its networks

— It only works in private networks

● Unicast— Multiple connections for the same stream

— The only solution


Streaming serversStreaming servers

● Microsoft Media Services

— Not free

— In the past only support the MMS protocol (now uses RTSP)● Adobe Flash Media Streaming Server 3

— Not free● Darwin Streaming Server (DSS)

— Open source

— Shares the same code base as QuickTime Streaming Server

— Streams MPEG-4


e-lab structure diagram e-lab structure diagram Video Streaming ServerVideo Streaming Server


DSS – ReflectorDSS – Reflector

● Allows to deliver live broadcast to clients● Client connects to DSS which then reflects the

connection to the desired broadcast● Separate program sends RTP packets to DSS

with the video● UDP packets are sent from the source● DSS makes the bridge between the

broadcaster and the client


Session Description Session Description Protocol (SDP)Protocol (SDP)

● DSS stores files, describing each broadcaster, saved with the SDP format

● Most important information:

— IP of the source

— Port of the source

— Media type (payload)

— Bit rate

— Title● There are programs which automatically generate these files


SDP ExampleSDP Example

● v=0

● o=- 1204476969191332 1204476969191334 IN IP4 192.168.0.123

● s=GAMMA

● b=RR:0

● t=0 0

● a=tool:vlc 1.1.13

● a=type:broadcast

● m=video 2026 RTP/AVP 96

● c=IN IP4 192.168.0.2

● b=AS:80

(red lines can be optional)


Connection flowConnection flow

● Client connects to DSS with an URL like:

— rtsp://elabmc.ist.utl.pt/gamma.sdp● DSS parses the SDP file and connects it to the desired

source (typically an UDP port)

● On the previous example the video is being sent to the port 2026 of the 192.168.0.2 from 192.168.0.123

● Client also uses the SDP file to check the media type (in the previous case 96=MPEG4)


Video broadcastVideo broadcast

● In order to send the video to the streaming server one needs— Cameras

— Drivers

— Generic API to access the images from the cameras

— Encoder

— Broadcaster


CamerasCameras

● In the case of remote laboratories, commercially available webcams are the perfect solution

● Problem is that a lot of webcams only work under Windows

— Imply encoding under Windows

— Costs money

— Highly unreliable● Some webcams are supported quite well under Linux


Video 4 Linux (V4L)Video 4 Linux (V4L)

● Video capture API for Linux;● Unifies the way programs access the cameras● Closely integrated with the Linux kernel● V4L is in its second version (V4L2)

— Include a compatibility mode for V4L1

— Better use V4L2 native devices/drivers


Linux webcamsLinux webcams

● Look for cameras which already have a driver in the kernel tree

● Camera drivers must support native V4L2● All the cameras supported by the PWC (Phillips

USB Webcam Driver for Linux) project are a good choice— Philips

— Logitech Quickcams


Video encodingVideo encoding

● The ideal program grabs the video using V4L2● Encodes using MPEG-4● Sends the data using RTP to the streaming

server● Automatically writes an SDP file which can be

placed on the DSS● All these features are present on the VLC

project


VLCVLC

● Open-source● VLC live streaming

— audio/video capture utility that can capture and encode audio and video in real-time

— results can be written to either an .mp4 file, transmitted onto the network via either unicast or multicast, or both simultaneously


VLC internalsVLC internals

● Download source from

— http://www.videolan.org/vlc/

— Configure, make and install

● Gentoo Linux OS:

— emerge vlc

— USE=”X ffmpeg libv4l2 live rtsp stream v4l2 x264”

● Each broadcaster uses a specific configuration command

● The program is launched with the following syntax:

— vlc + “configurations” (with GUI)

— cvlc + “configurations” (without GUI)


VLC configurations VLC configurations

● Entire commandsu elab --command "cvlc -vvv v4l2:///dev/video1 :input-slave=alsa:// :v4l2-standard=0 :file-caching=300 \--sout '#transcode{vcodec=h264,vb=80,fps=10,scale=1,width=320,height=240,acodec=none}'\':rtp{proto=udp,dst=192.168.0.2,port-video=20002,ttl=127,name=SCUBA,sdp=file:///home/elab/webcam_streaming/sdps/scuba.sdp}' \ --no-sout-audio" &

● Parts of the command— Open the webcam:

● cvlc -vvv v4l2:///dev/video1 :input-slave=alsa:// :v4l2-standard=0 :file-caching=300

— Stream output (transcoding):

● '#transcode{vcodec=h264,vb=80,fps=10,scale=1,width=320,height=240,acodec=none}'

— RTP and SDP configurations:

● ':rtp{proto=udp,dst=192.168.0.2,port-video=20002,ttl=127,name=SCUBA, sdp=file:///home/elab/webcam_streaming/sdps/scuba.sdp}'

— No audio:

● --no-sout-audio


Launching VLC atLaunching VLC ate-lab computer clustere-lab computer cluster● Each broadcaster have a script file with VLC

commands corresponding to the respective webcams— /etc/local.d/vlc.start

● Information about the streaming port number is located in a web-page:— elab1.ist.utl.pt

● All broadcaster execute the script file located in their /etc/local.d directory by executing local daemon:— /etc/local.d/vlc.start which is executed by:

— /etc/init.d/local start


RemarksRemarks

● This set-up allows to have a complete, open-source and free video broadcasting solution

● It can be easily adapted to broadcast other kind of sources (stored files, movies, music, etc.)

● Required software— Linux installation

— Darwin Streaming Server

— VLC


The final pictureThe final picture

video streaming on e-lab

Education

application layer

henriques lisbon

session layer

streaming youtube

network layer functional

physical layer electrical

data representation

raw data