Multimedia ApplicationsMultimedia Applications

Ali Saman TosunComputer Science

Department

2

Compression - NecessityCompression - Necessity

E.g., video sequence 25 images/sec.

PAL standard 3 byte/pixel

YUV (luminance + 2 chrominance values) RGB (red-green-blue values)

Image resolution 640 * 480 pixel Data rate = 640 * 480 * 3 Byte * 25/s = 23040000 byte/s

~ 22 MByte/s Approx. 1/16 stream over Ethernet Approx. 1/2 stream over Fast Ethernet

Compression is necessary

3

Multimedia Networking Multimedia Networking Applications Applications

Fundamental characteristics:

Typically delay sensitive

end-to-end delay delay jitter

But loss tolerant: infrequent losses cause minor glitches

Antithesis of data, which are loss intolerant but delay tolerant.

Classes of MM applications1) Streaming stored audio and

video2) Streaming live audio and

video3) Real-time interactive audio

and video

Jitter is the variability of packet delays within the same packet stream

4

From Broadcast to True From Broadcast to True Media-on-DemandMedia-on-Demand

Broadcast (No-VoD) Traditional, no control

Pay-per-view (PPV) Paid specialized service

Quasi Video On Demand (Q-VoD) Distinction into interest groups Temporal control by group change

Near Video On Demand (N-VoD) Same media distributed in regular time intervals Simulated forward / backward

True Video On Demand (T-VoD) Full control for the presentation, VCR capabilities Bi-directional connection

5

Streaming Stored Video Streaming Stored Video

Streaming media stored at source transmitted to client streaming: client playout begins

before all data has arrived

timing constraint for still-to-be transmitted data: in time for playout

6

constant bit rate videotransmission

Cum

ula

tive

data

time

variablenetwork

delay

client videoreception

constant bit rate video playout at client

client playoutdelay

bu

ffere

dvid

eo

Streaming MultimediaStreaming Multimedia

Client-side buffering, playout delay compensate for network-added delay, delay jitter

7

Proxy-based Video Proxy-based Video DistributionDistribution

Server

Proxy

Client

Client

Proxy adapts videoProxy caches video

8

Proxy OperationsProxy Operations Drop frames

Drop B,P frames if not enough bandwidth Quality Adaptation

Transcoding Change quantization value Most of current systems don’t support

Video staging, caching, patching Staging: store partial frames in proxy Prefix caching: store first few minutes of movie Patching: multiple users use same video

9

Basic Encoding StepsBasic Encoding Steps

10

H.261 (px64)H.261 (px64) International Standard

Video codec for video conferences at p x 64kbit/s (ISDN): Real-time encoding/decoding, max. signal delay of 150ms Constant data rate

Intraframe coding DCT as in JPEG baseline mode

Interframe coding, motion estimation

Search of similar macroblock in previous image and compare Position of this macroblock defines motion vector Difference between similar macroblocks

11

MPEG (Moving Pictures Expert MPEG (Moving Pictures Expert Group)Group)

International Standard: Compression of audio and video for playback (1.5 Mbit/s): Real-time decoding

Sequence of I-, P-, and B-Frames:

Random access at I-frames at P-frames: i.e. decode previous I-frame first at B-frame: i.e. decode I and P-frames first

12

MPEG-2MPEG-2

Beyond MPEG-1: Higher quality encoding Higher data rates Interleaved modes

Higher data rates MPEG-1: about 1.5 MBit/s MPEG-2: 2-100 MBit/s

Scaling: Signal to Noise Ration (SNR) scalin

progressive compression Spatial scaling

several pixel resolutions Temporal scaling

frame dropping

13

Scalable codingScalable coding

Typically used asLayered coding

A base layer Provides basic quality Must always be

transferred One or more

enhancement layers Improve quality Transferred if possible Sending rate

Qu

alit

yBest p

ossible quality

at possi

ble sending ra

te

Base layer

Enhancement layer

14

Temporal ScalabilityTemporal Scalability Frames can be dropped

In a controlled manner Frame dropping does not violate dependancies Low gain example: B-frame dropping in MPEG-1

15

Spatial ScalabilitySpatial Scalability Base layer

Downsample the original image Send like a lower resolution version

Enhancement layer Subtract base layer pixels from all pixels Send like a normal resolution version

If enhancement layer arrives at client Decode both layers Add layers

72

75 83

6173

-1 -12

2 10

Base layer

Enhancement layerBetter compression due to low values

Less data to code

16

SNR ScalabilitySNR Scalability SNR – signal-to-noise ratio Idea

Base layer Is regularly DCT encoded A lot of data is removed using quantization

Enhancement layer is regularly DCT encoded Run Inverse DCT on quantized base layer Subtract from original DCT encode the result

If enhancement layer arrives at client Add base and enhancement layer before running Inverse

DCT

17

Multiple Description Multiple Description CodingCoding Idea

Encode data in two streams Each stream has acceptable quality Both streams combined have good quality The redundancy between both streams is low

Problem The same relevant information must exist in both

streams Old problem: started for audio coding in telephony Currently a hot topic

18

Delivery Systems Delivery Systems DevelopmentsDevelopments

Network

Saving network resources:

Stream scheduling

Several Programs or

Timelines

19

PatchingPatching

Server resource optimization is possible

multicast

Unicast patch stream

Central server

1st client 2nd client

Join !

cyclicbuffer

20

Proxy Prefix CachingProxy Prefix Caching

Split movie Prefix Suffix

Operation Store prefix in prefix cache

Coordination necessary! On demand

Deliver prefix immediately Prefetch suffix from central

server

Goal Reduce startup latency Hide bandwidth limitations,

delay and/or jitter in backbone Reduce load in backbone

Client

Unicast

Unicast

Central server

Prefix cache

21

Interval Caching (IC)Interval Caching (IC) caches data between requests following requests are thus served from the cache sort intervals on length

Video clip 1

S11

Video clip 1

S11S12

Video clip 1

S12 S11S13

Video clip 2

S22 S21

Video clip 3

S33 S31S32S34

I11I12

I21

I31I32I33

I32 I33 I21I11I31I12

22

Receiver-driven Layered Multicast Receiver-driven Layered Multicast (RLM)(RLM)

Requires IP multicast layered video codec

(preferably exponential thickness) Operation

Each video layer is one IP multicast group Receivers join the base layer and extension layers If they experience loss, they drop layers (leave IP multicast

groups) To add layers, they perform "join experiments“

Advantages Receiver-only decision Congestion affects only sub-tree quality Multicast trees are pruned, sub-trees have only necessary traffic

23

Receiver-driven Layered Multicast Receiver-driven Layered Multicast (RLM)(RLM)

24

Multimedia Disk Multimedia Disk SchedulingScheduling Suitability of classical algorithms

minimal disk arm movement (short seek times) no provision of time or deadlines generally not suitable

Continuous media server requirements serve both periodic and aperiodic requests never miss deadline due to aperiodic requests aperiodic requests must not starve support multiple streams balance buffer space and efficiency tradeoff

25

Group Sweep Scheduling Group Sweep Scheduling (GSS)(GSS)GSS combines Round-Robin (RR) and SCAN requests are serviced in rounds (cycles) principle:

divide S active streams into G groups service the G groups in RR order service each stream in a group in C-SCAN order playout can start at the end of the group

special cases: G = S: RR scheduling G = 1: SCAN scheduling

tradeoff between buffer space and disk arm movement try different values for G giving minimum buffer requirement – select

minimum a large G smaller groups, more arm movements, smaller buffers

(reuse) a small G larger groups, less arm movements, larger buffers

26

Power ManagementPower Management Based on functionality

Server Proxy Client

Based on component Disk Processor