Concepts of Multimedia Concepts of Multimedia Processing and TransmissionProcessing and Transmission

IT 481, Lecture #8Dennis McCaughey, Ph.D.

30 October, 2006

08/28/2006IT 481, Fall 20062

Broadcast EnvironmentBroadcast Environment

08/28/2006IT 481, Fall 20063

Video Transmission System ExampleVideo Transmission System Example

Network Adapter

EncoderRate

Control

D/AError

ControlNetwork Adapter

Decoder

Monitor

Camera A/D

Frame Sequence

To Network

From Network

08/28/2006IT 481, Fall 20064

Rate ControlRate Control

Video applications involve real-time display of the decoded sequence.

Transmission across a Constant Bit Rate (CBR) channel requires a constant end-to-end delay between the time that the encoder processes a frame and the time at which that same frame is available to the decoder

A buffer is require to match the Variable Bit Rate (VBR) encoder with the CBR channel

08/28/2006IT 481, Fall 20065

Admission ControlAdmission Control

Decision if a new connection with a given set of QoS parameters can be allowed into the network

Criteria: the new connection will not degrade the QoS of other ongoing connections

Need models for predicting the expected bit rates of video sources especially packet switched networks

Simpler for circuit switched networks where the transmission resources are constant over the duration of the connection

08/28/2006IT 481, Fall 20066

Usage Parameter ControlUsage Parameter Control

Prevent sources from exceeding the traffic parameters negotiated at call setup– Maliciously or unintentionally

Example: tracking cell rate in an ATM network and verifies that the source remains within that rate

Example: “Leaky Bucket” – a cell counter incremented with each cell arrival– Decremented at fixed intervals– If counter exceeds a threshold cells are dropped

or marked for deletion

08/28/2006IT 481, Fall 20067

Multi-Resolution EncodingMulti-Resolution Encoding

Separation of the information into two or more layers or resolutions

Coarse Resolution– Reduce the spatial or temporal resolution of the sequence

or by having images of lower quality– Transmitted by high priority packets

Detailed Resolution– Transmitted by lower priority packets than can be

discarded first Multi-Resolution Encoding enables efficient error

concealment techniques– e.g. interpolate lost information from packets that were not

lost

08/28/2006IT 481, Fall 20068

Layered Video CodingLayered Video Coding

Provide multiple image quality levels simultaneously across multiple network channels– e.g. Standard Resolution and High Definition TV

Each receiver individually tunes its reception rate by adjusting the number of layers that it receives

Two problems to solve– Layered Compression

Multiple image quality levels

– Layered Transmission Problem Selective delivery of layer subsets to individual

receivers

08/28/2006IT 481, Fall 20069

Structure of a Simulcast CoderStructure of a Simulcast Coder

08/28/2006IT 481, Fall 200610

Simulcast CoderSimulcast Coder

Produces a multirate set of signals that are independent of one another

Each layer provides improved quality, independent of sublayers

Single layer (nonscalable) decoder can decode any layer

Total available bandwidth is partitioned dependent on the desired quality for each independent sublayer

08/28/2006IT 481, Fall 200611

Structure of a Layered CoderStructure of a Layered Coder

08/28/2006IT 481, Fall 200612

Layered CompressionLayered Compression

Input signal is compressed into a number of discrete layers, arranged in a hierarchy that provides progressive refinement– If first layer is received, decoder will produce the

lowest quality version of the signal– If the decoder receives two layers, it will combine

the second layer with the first layer to produce improved quality

– Overall, the quality improves with the number of layers that are received and decoded

Layered compression and layered transmission must be combined to be effective

Slide courtesy of Hung Nguyuen

08/28/2006IT 481, Fall 200613

Simulcast vs. Layered ApproachSimulcast vs. Layered Approach

Distortion measures quality degradation between reconstructed and original signals– DI(R): an ideal coder

– DR(R): a real coder

– DL(R): a layered coder

– DS(R): a simulcast coder

08/28/2006IT 481, Fall 200614

Layered TransmissionLayered Transmission

Each layer is transmitted on a different network channel.

Network only forwards the number of layers that each physical link can support.

Each user receives the best quality signal that the network can deliver.

The network must be able to drop layers selectively at each bottleneck link

08/28/2006IT 481, Fall 200615

Video Communication SystemVideo Communication System

Video Source Encoding

Multiplexing, Packetizing &

Channel Encoding

Network

Demultiplexing, Depacketizing

& Channel Decoding

Video Source Decoding

Other Data Other Data

Original VideoReconstructed

Video

08/28/2006IT 481, Fall 200616

Error ResilienceError Resilience

Redundancy is added to the compressed bitstream to allow the detection and correction of errors– Can be added in either the source or channel encoder

Shannon Information Theory:– Separately design the source and channel coders to

achieve error-free transmission so long as the source is represented by a rate below the channel capacity

Source should compress the source as much as possible

Channel coder, via Forward Error Correction (FEC) adds redundancy bits to enable error detection and correction

08/28/2006IT 481, Fall 200617

Binary Symmetric ChannelBinary Symmetric Channel

1-p

pp

1-p

Capacity C = 1 + plog2(p) + (1-p)log2(1-p)

p C1.00E-06 1.00001.00E-05 0.99981.00E-04 0.99851.00E-03 0.98861.00E-02 0.91921.00E-01 0.53105.00E-01 0.0000

08/28/2006IT 481, Fall 200618

Shannon’s Capacity TheoremShannon’s Capacity Theorem

If the Rate (R) of a code R = log2(m)/L is less than channel capacity C, there exists a combination of source and channel encoders such that the source can be communicated over the channel with fidelity arbitrarily close to perfect – m = Number of message words– L = Number of code word bits

08/28/2006IT 481, Fall 200619

Types of CodesTypes of Codes

Block Codes– Hamming Codes– Bose-Chaudhuri-Hocquenhem BCH Codes– Reed-Solomon Codes

Convolutional Codes

08/28/2006IT 481, Fall 200620

Example: (Example: (n,kn,k) Hamming Code) Hamming Code

Block Coder

k-bit Shift Register

n-bit Shift Register

Source Bitsmk

m0

m3

c0

c6 Coded Bitscn

(n,k) = (2m-1,2m-1-m), for any m

08/28/2006IT 481, Fall 200621

(7,4) Hamming Code(7,4) Hamming Code

0 1

0 1

, , , Message Word

, , , Code Word

For a (7,4) Hamming Code:

1 0 1 1 0 0 0

1 1 1 0 1 0 0 Generator Matrix

1 1 0 0 0 1 0

0 1 1 0 0 0 1

All arithmetic done modulo 2

k

n

k

m m m m

c c c c

c mG

G P I

G

Corrects all single errors

08/28/2006IT 481, Fall 200622

Parity Check MatrixParity Check Matrix

0 1, , , Message Word

e Received Word

Where:

For a (7,4) Hamming Code:

1 0 0 1 1 1 0

0 1 0 0 1 1 1 Parity Check Matrix

0 0 1 1 1 0 1

n

T

Tn k

e e e e

r c

s rH

H I P

H

08/28/2006IT 481, Fall 200623

DecodingDecoding

errors no are thereif zero be willand syndrome thecalled is

WordReceived e

Error Word ,,, 10

s

HeHeHcHrs

cr

eeee

TTTT

n

08/28/2006IT 481, Fall 200624

(7,4) Messages and Codes(7,4) Messages and Codes(Systematic Code)(Systematic Code)

0 0 0 0

0 0 0 1

0 0 1 0

0 0 1 1

0 1 0 0

0 1 0 1

0 1 1 0

0 1 1 1

1 0 0 0

1 0 0 1

1 0 1 0

1 0 1 1

1 1 0 0

1 1 0 1

1 1 1 0

1 1 1 1

0 0 0 0 0 0 0

1 0 1 0 0 0 1

1 1 1 0 0 1 0

0 1 0 0 0 1 1

0 1 1 0 1 0 0

1 1 0 0 1 0 1

1 0 0 0 1 1 0

0 0 1 0 1 1 1

1 1 0 1 0 0 0

0 1 1 1 0 0 1

0 0 1 1 0 1 0

1 0 0 1 0 1 1

1 0 1 1 1 0 0

0 0 0 1 1 0 1

0 1 0 1 1 1 0

1 1 1 1 1 1 1

Messages Code Words

08/28/2006IT 481, Fall 200625

Received Code WordsReceived Code Words

1 0 0 0 0 0 0

0 1 0 0 0 0 0

0 0 1 0 0 0 0

0 0 0 1 0 0 0

0 0 0 0 1 0 0

0 0 0 0 0 1 0

0 0 0 0 0 0 1

1 0 0 0 0 0 0

0 1 0 0 0 0 0

0 0 1 0 0 0 0

0 0 0 1 0 0 0

0 0 0 0 1 0 0

0 0 0 0 0 1 0

0 0 0 0 0 0 1

0 0 0 0 0 0 0

0 0 0 0 0 0 0

Received Words

1 0 0 0 0 0 0

1 1 1 0 0 0 1

1 1 0 0 0 1 0

0 1 0 1 0 1 1

0 1 1 0 0 0 0

1 1 0 0 1 1 1

1 0 0 0 1 1 1

1 0 1 0 1 1 1

1 0 0 1 0 0 0

0 1 0 1 0 0 1

0 0 1 0 0 1 0

1 0 0 1 1 1 1

1 0 1 1 1 1 0

0 0 0 1 1 0 0

1 1 0 1 1 1 0

1 0 1 1 1 1 1

errors

0 0 0 0 0 0 0

1 0 1 0 0 0 1

1 1 1 0 0 1 0

0 1 0 0 0 1 1

0 1 1 0 1 0 0

1 1 0 0 1 0 1

1 0 0 0 1 1 0

0 0 1 0 1 1 1

1 1 0 1 0 0 0

0 1 1 1 0 0 1

0 0 1 1 0 1 0

1 0 0 1 0 1 1

1 0 1 1 1 0 0

0 0 0 1 1 0 1

0 1 0 1 1 1 0

1 1 1 1 1 1 1

Transmitted Words

08/28/2006IT 481, Fall 200626

SyndromesSyndromes

1 0 0

0 1 0

0 0 1

1 1 0

0 1 1

1 1 1

1 0 1

1 0 0

0 1 0

0 0 1

1 1 0

0 1 1

1 1 1

1 0 1

1 0 0

0 1 0

4

2

1

6

3

7

5

4

2

1

6

3

7

5

4

2

Binary Decimal

0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 1 0

Syndrome Table

•Rows in Syndrome Table indicate error locations

•Syndromes point to rows

08/28/2006IT 481, Fall 200627

PerformancePerformance

Bit Error Probability p=10-6

M = 100 Bits Probability that message is received correctly, Pc=(1-10-2)100 = .367 Use a hamming (7,4) code

– 25 7-Bit code words with 4 info bits per word– Probability that a word is correctly decoded

Pw = probability that one or fewer bits are decoded incorrectly =(1-10-2) + 7 (10-2) =.998

– Probability that message is received correctly =(.998)25 = .95 A substantial improvement

08/28/2006IT 481, Fall 200628

Video Streaming ArchitectureVideo Streaming Architecture

08/28/2006IT 481, Fall 200629

Scalable and Non-Scalable Scalable and Non-Scalable Coder/DecoderCoder/Decoder

Scalable Encoder/Decoder

Scalable Encoder/Decoder

•A nonscalable video encoder generates one compressed bit-stream. •Scalable video encoder compresses a raw video sequence into multiple sub-streams

08/28/2006IT 481, Fall 200630

Video Stream Architecture Building Video Stream Architecture Building BlocksBlocks

Video Compression Application Layer QoS Control Continuous Media Distribution Services Streaming Servers Media Synchronization Protocols for Multimedia

08/28/2006IT 481, Fall 200631

Video CompressionVideo Compression

Video compression schemes can be classified into two categories: scalable and nonscalable

Scalable video is capable of gracefully coping with the bandwidth fluc tuations in the Internet.

Primarily concerned with scalable video coding techniques. We will also discuss the requirements imposed by streaming applications on the video encoder and decoder.

08/28/2006IT 481, Fall 200632

Application Layer QoS ControlApplication Layer QoS Control

Congestion control takes the form of rate control. Three kinds of rate control: 1) Source-based,

– The source-based rate control is suitable for unicast;– Most recent studies on source-based rate control have been

focused on TCP-friendly adaptation – A number of TCP-friendly adaptation schemes have been

proposed and demonstrated to achieve certain degree of fairness among competing connections, including TCP connections.

– However, strictly TCP-like rate control may result in sharp reductions in the transmission rate, and possibly unpleasant visual quality [66].

– Needs further investigation on how to trade off responsiveness in detecting and reacting to congestion with smooth fluctuation in visual quality.

2) Receiver-based, and 3)Hybrid rate-control. – Both are suitable for multicast since both can achieve good

tradeoff between bandwidth efficiency and service flexibility for multicast video.

08/28/2006IT 481, Fall 200633

Source Based Rate ControlSource Based Rate Control

08/28/2006IT 481, Fall 200634

Continuous Media Distribution ServicesContinuous Media Distribution Services

Continuous media distribution services are built on top of the best-effort Internet with the aim of achieving QoS and efficiency for streaming video.

A major topic of active research is how to build a scalable, efficient, cost-effective and incremental deployable infrastructure for continuous media distribution.

08/28/2006IT 481, Fall 200635

Streaming ServersStreaming Servers

Essential in providing streaming services. Current research efforts include:

– 1) Efficient support VCR-like interac tive control; – 2) The design of efficient and reliable storage and retrieval

of multimedia objects on disk arrays; – 3) The design of highly scalable multimedia servers in a

variety of environments ranging from video-on-demand servers to integrated multimedia file system

– 4) The design of fault-tolerant storage systems Desirable features of both parity and mirroring

– Trade off the parity group size with the number of disks across which original data of a single disk is replicated for mirroring.

08/28/2006IT 481, Fall 200636

Media SynchronizationMedia Synchronization

Intra-stream synchronization: The lowest layer of con tinuous media or time-dependent data (such as video and audio) is the media layer.

– Maintains the continuity of logical data units. – Prevents pauses and gaps.

Inter-stream synchronization: The second layer of time-dependent data is the stream layer.

– Maintains temporal relationships among different continuous media. – Prevents skew between the streams may become intolerable. – eg, users could be annoyed if they notice that the movements of the lips

of a speaker do not correspond to the presented audio. Inter-object synchronization: The highest layer of a mul timedia

document is the object layer, which integrates streams and time-independent data such as text and still images.

– Synchronization at this layer is referred to as inter-object synchronization.

– The objective of inter-ob ject synchronization is to start and stop the presentation of the time-independent data within a tolerable time interval,

– e.g., the audience of a slide show could be annoyed if the audio is commenting one slide while another slide is being presented

08/28/2006IT 481, Fall 200637

Protocols for MultimediaProtocols for Multimedia

Network Layer Protocol Transport Protocol Session Control Protocol

– UDP– TCP– RTP– RTCP

08/28/2006IT 481, Fall 200638

Standardized ProtocolsStandardized Protocols

Several protocols have been standardized for communication between clients and streaming servers.

Future research topics on design of protocols include: – 1) How to take caches into account (e.g., how to

communicate with continuous media caches and how to control continuous media caches);

– 2) How to efficiently support pause/resume operations in caches (since the pause/resume operations interfere with the sharing of a multimedia stream among different viewers); and

– 3) How to provide security in the protocols

08/28/2006IT 481, Fall 200639

Unicast and MulticastUnicast and Multicast

Unicast video distribution using multiple point-to-point connections.

Multicast video distribution using point-to-multipoint transmission.

08/28/2006IT 481, Fall 200640

Streaming Server ComponentsStreaming Server Components

Communicator: – A communicator involves the application layer and

transport protocols implemented on the server. – Through a communicator, the clients can communicate

with a server and retrieve multimedia contents in a continuous and synchronous manner.

Operating system: – Different from traditional operating systems, – An operating system for streaming services needs to

satisfy real-time requirements for streaming applica tions. Storage system: A storage system for streaming services

08/28/2006IT 481, Fall 200641

Protocol Stack for MultimediaProtocol Stack for Multimedia

08/28/2006IT 481, Fall 200642

RTPRTP

RTP does not guarantee QoS or reliable delivery, but rather, provides the following functions in support of media streaming:

Time-stamping: RTP provides time-stamping to synchronize different media streams.

Sequence numbering: RTP employs sequence numbering to place the incoming RTP packets in the correct order. Since packets arriving at the receiver may be out of sequence (UDP does not deliver packets in sequence),.

Payload type identification: The type of the payload contained in an RTP packet is indicated by an RTP-header field called payload type identifier.

– The receiver interprets the content of the packet based on the payload type iden tifier.

– Certain common payload types such as MPEG-audio and video have been assigned payload type numbers

– For other payloads, this assignment can be done with session control protocols.

Source identification: The source of each RTP packet is identified by an RTP-header field called Synchronization Source identifier (SSRC), which provides a means for the receiver to distinguish different sources.

08/28/2006IT 481, Fall 200643

RTCPRTCP

QoS feedback: This is the primary function of RTCP. – RTCP provides feedback to an application regarding the quality of

data distribution. – The feedback is in the form of sender reports (sent by the source)

and receiver re-ports (sent by the receiver). – The reports can contain in-formation on the quality of reception

such as: – 1) Fraction of the lost RTP packets, since the last report; – 2) Cumulative number of lost packets, since the beginning of re

ception; – 3) Packet interarrival jitter; and – 4) Delay since receiving the last sender’s report.

Participant identification: A source can be identified by the SSRC field in the RTP header.

– RTCP provides a human-friendly mechanism for source identification.

– RTCP SDES (source description) packets contain textual information called canonical names as globally unique identifiers of the session participants.

– It may include a user’s name, telephone number, email address, and other information.

08/28/2006IT 481, Fall 200644

RTCPRTCP

Control packets scaling: To scale the RTCP control packet transmission with the number of participants, a control mechanism is designed as follows.– The control mechanism keeps the total control packets to 5% of

the total session bandwidth. – Among the control packets, 25% are allocated to the sender

reports and 75% to the receiver reports. – To prevent control packet starvation, at least one control packet

is sent within 5 s at the sender or receiver. Inter-media synchronization: RTCP sender reports contain an

indication of real time and the corresponding RTP timestamp. This can be used in inter-media synchroniza tion like lip synchronization in video.

Minimal session control information. This optional functionality can be used for transporting session information such as names of the participants.

08/28/2006IT 481, Fall 200645

Session Control ProtocolsSession Control Protocols

RTSP functions – Support VCR-like control operations such as stop, pause/re sume,

fast forward, and fast backward. – Provides a means for choosing delivery channels (e.g., UDP, mul

ticast UDP, or TCP), and delivery mechanisms based upon RTP. – RTSP works for multicast as well as unicast. – Also establishes control streams of continuous audio and video

media between the media servers and the clients. Specifically, RTSP provides the following operations.

– Media retrieval: The client can request a presentation description, and ask the server to setup a session to send the requested media data;

– Adding media to an existing session: The server or the client can notify each other about any additional media becoming available to the established session

08/28/2006IT 481, Fall 200646

Session Control ProtocolsSession Control Protocols

Similar to RTSP, SIP can also create and terminate sessions with one or more par ticipants.

Unlike RTSP, SIP supports user mobility by proxying and redirecting requests to the user’s current location

08/28/2006IT 481, Fall 200647

ReferencesReferences

D. Wu et. Al. “Streaming Video over the Internet: Approaches and Directions”, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 11, No.3, March 2001