a graduate course on multimedia technology 3. multimedia communication © wolfgang effelsberg 3.5 -...

A Graduate Course on

Multimedia Technology3. Multimedia Communication© Wolfgang Effelsberg 3.5 - 1

3.5 Media Scaling and Media Filtering

Definition of Media ScalingAdaptation of the data rate of a media stream to the capacity of the network

Requirements for scaling algorithms• Quick and precise adaptation to the capacity of the network• Minimum possible visual/audible effect• Robustness in case of packet loss



Resource Reservation vs. Stream Adaptation

Reservation of Resources• The application requests the necessary QoS (and thus network

resources) in advance. Those resources will be guaranteed for the duration of the stream.

Adaptation• The application adapts dynamically to the resources/capacity currently

available from the network.• This works for existing „best effort“ networks, in particular for the Inter-

net with IP Version 4.• Adaptive applications must implement audio and/or video codecs that

allow scaling at run-time.

Reservation and adaptation can be combined, in particular for probabi-listic QoS guarantees: A reasonable amount of resources is reserved in advance, and in short phases of exceedingly high bit rates, the appli-cation scales down the stream.



Messages Exchanged for Scaling

• The source starts to send at some initial data rate.• The network or the destination sends feedback messages indicating their

current load.• The source adapts the data stream by dynamically changing the para-

meters of the codec.

source sink

data

more/less data

feedback



Scaling Example

While an MPEG video is being played from a video server to a user the network signals congestion. As a consequence, the video server changes the quantization table of the MPEG codec (increases the quantization step size) and thus generates VBR video at a lower bit rate (and a lower image quality).

Note the difference to the reaction you would observe on a network without adaptive (scaling) applications: the congestion control algorithm of the net-work (e.g., the one contained in TCP) would reduce the packet rate, leading to a lower video frame rate at the receiver. The network has no alternative; only the source (the application) can handle the overload situation in a more intelligent way!



Media Layering and Filtering

Layering and filtering are especially relevant in multicast trees:• Different receivers require different bit rates and stream qualities.• Only those parts of the stream are transported to a receiver that it can

really use.• Irrelevant parts of the stream are removed by the inner nodes of the

network.

A

B C

E

D

layer 0 (base layer)layer 1 (enhancement layer for medium quality)layer 2 (enhancement layer for high quality)



Layered Video Encoding

The two most important layering techniques for video are temporal scaling and spatial scaling.

Temporal Scaling

Assign frames to layers in a round-robin fashion. The more layers the re-ceiver gets, the higher is the frame rate.

Spatial ScalingAssign data values to layers such that for each frame, a low-quality version can be reproduced from layer 0; adding higher layers will add “visible qua-lity“ to the frame.

There are other scaling/layering schemes, e.g., spatio-temporal techniques, which we will not discuss here.



Temporal Scaling for Video

The base layer (i.e., layer 0) transmits every third frame. So do the first and the second enhancement layers. Together, the full frame rate is transmitted.

0 1 2 3 4 5 6 7 8 9 10 11

01

2

34

5

67

8

9

0 3 6 9

1 4 7 10

2 5 8 11

10

disassembling

reassembling

layer 0layer 1layer 2

layer 0

layer 1

layer 2

11



Spatial Scaling for Video (1)

Pyramid EncodingThe encoder first down-samples the image, compresses it according to the chosen encoding technique, and then transmit the result in the base layer. At the same time, it decompresses the image, up-samples it and gets a coarse version of the original. To compensate for the difference, the encoder subtracts the resulting copy from the original and sends the encoded diffe-rence in the enhancement layer. The MPEG-2 standard supports pyramid encoding.




Layered FrequenciesEach 8x8 block is DCT-transformed and quantized. After quantization, the coefficients are stored in different layers, e.g., the first three coefficients in the base layer, the next three in the first enhancement layer, the next four in the second enhancement layer, etc.




Layered QuantizationEach 8x8 block is DCT-transformed. The bits of the DCT coefficients are distributed over several layers. For example, the base layer may contain the first two bits of all coefficients (corresponding to a very coarse quantization), the first enhancement layer the next two bits of all coefficients, etc. Recei-ving more layers is then equivalent to choosing a finer quantization.




Layered Encoding of Wavelet-Transformed Video

Original frame Wavelet-transformed, five iterations

Wavelet-transformed, one iteration




Layered Wavelet EncodingThe approximation (i.e., the low-pass filtered part) of a wavelet-transformed frame is stored in the base layer. The coefficients of the details (i.e., the high-pass filtered part) are stored in enhancement layers in decreasing order of their values.

base layer l0 enhancement layer l1 enhancement layer l2 enhancement layer l3



Examples of Spatial Layering

Sample frames of a video sequence (a cartoon). Note that the images are shown at different quality levels in order to illustrate the characteristic artifacts.

Pyramid encoding Layered DCT frequencies

Layered wavelet encodingLayered DCT quantization



Conclusions on Media Scaling and Layering

• Scaling techniques for digital video were developed in recent years and work well.

• Layering (i.e., the decomposition of the video into separate layers) is not yet available in most commercial video codecs.

• The integration of layer filters into internal network nodes of multicast trees is still a research issue.

a graduate course on multimedia technology 3. multimedia communication © wolfgang effelsberg 3.5 -...

Documents

base layer layer

multimedia technology

graduate course

scaling example

medium quality layer

mpeg video

data stream

lower video frame rate