LOSS CONCEALMENTS FOR REAL-TIMEMULTIMEDIA OVER THE INTERNET

Benjamin W. Wah

Department of Electrical and Computer Engineeringand the Coordinated Science Laboratory

University of Illinois at Urbana-ChampaignUrbana, IL 61801, USA

January 14, 2005

Loss Concealments for Real-Time Multimedia over the Internet Outline

Motivations

Real-time multimedia Voice-over-IP

Image downloads from Web servers

Real-time audio over IP networks

Real-time video transmissions

Characteristics of IP networks Loss rate

Burstiness

End-to-end delay

Packet rate

Loss concealments without a channel-loss model Trade-offs among delay, bandwidth, and (perceptual) quality

Modified codec design (standard compliant versus quality)

Reconstruction algorithm

Benjamin W. Wah 1

Loss Concealments for Real-Time Multimedia over the Internet Outline

VoIP Speech Quality

Interactive real-time communications: End-to-end delay due to codec, network, and jitter buffer

ITU G.114: one way delay, < 150 ms acceptable, < 400 ms noticeable

Mobile loop one-way delay about 100 ms; Mobile-VoIP-Mobile: about 300ms

Acoustic echo: due to PSTN wiring or PC setup Noticeable for delays more than 30ms

Loss: some degradations on voice samples tolerable Low bandwidth/congestion: last-mile problem

Long-burst or frequent short-burst intolerable

Codec Codec in tandem: code conversions at hosts or gateway, causing degradedquality and increased delay

Using PC as phone: Speaker and microphone not optimal for phone conversation

Standard low bit-rate speech codecs: Error propagation

Benjamin W. Wah 2

Loss Concealments for Real-Time Multimedia over the Internet Outline

Delay-Quality Trade-offs in Image Downloads

Lena at 0.125bpp 2 out of 8 packets lost in UDP, 470 ms end-to-end delay

TCP UDP UDPJPEG2000 SDC JPEG2000 SDC Frequency-based MDC30.97 dB 20.51 dB 25.21 dB4.01 sec. 0.71 sec. 0.71 sec.

Some quality degradations can be tolerated in real-time downloads

Benjamin W. Wah 3

Loss Concealments for Real-Time Multimedia over the Internet Outline

Network Environments: Packet Network

IPv4: best-effort, no real-time support

Packet size: less than MTU to avoid fragmentation

Packet rate: 20 - 30 packets per second

IPv6: best-effort, may support real-time traffic

Wireless: future IP-based

Loss unavoidable in packet networks

Benjamin W. Wah 4

Loss Concealments for Real-Time Multimedia over the Internet Outline

Network Environments: Transport-Layer Protocol

TCP- Reliable but not suitable for real-time transmissions

- Connection oriented, more secure

- Allowed through firewalls

UDP- Lossy and unreliable

- No congestion control mechanism to slow the flow

- Not permitted through firewalls

TCP in real-time mode- Provide connection-oriented transmission without congestion avoidance

- Suitable for real-time multimedia for firewall compatibility

Loss of real-time multimedia not handled at the transport layer

Benjamin W. Wah 5

Loss Concealments for Real-Time Multimedia over the Internet Outline

Network Environments: Application-Layer Protocol

H.323: umbrella standard for interoperability

RTP: no loss recovery scheme for real-time multimedia

Packet losses in real-time multimedia left for end-point applications

Benjamin W. Wah 6

Loss Concealments for Real-Time Multimedia over the Internet IP Traffic Loss Characteristics

IP Traffic Loss Characteristics

Example connectionsConnection Loss rate

UIUC-Berkeley low-medium

UIUC-W. China medium-high

UIUC-Central Europe high

Loss behavior

00.050.1

0.150.2

0.250.3

0.350.4

0.450.5

0 5 10 15 20

Los

s R

ate

Time of day, hour

UIUC-BerkeleyUIUC-W. China

UIUC-C. Europe

0.4

0.5

0.6

0.7

0.8

0.9

1

1 2 3 4 5 6 7

Dis

trib

utio

n

Burst length

UIUC-BerkeleyUIUC-W. China

UIUC-C. Europe

Loss rate can go up to 50% Most losses have short burst lengths

Benjamin W. Wah 7

Loss Concealments for Real-Time Multimedia over the Internet IP Traffic Loss Characteristics

Reducing Unrecoverable Loss by Interleaving

Bursty losses are difficult to handle Interleaving: disperse bursty losses to isolated losses P (fail | i): prob. of losses that cannot be recovered under interleaving factor i

0

0.05

0.1

0.15

0.2

0.25

0.3

0 5 10 15 20

P(f

ail|i

)

Time of day (hour)

UIUC-Berkeley (i=1)UIUC-Berkeley (i=2)UIUC-Berkeley (i=3)UIUC-Berkeley (i=4)

00.050.1

0.150.2

0.250.3

0.350.4

0.450.5

0 5 10 15 20

P(f

ail|i

)

Time of day (hour)

UIUC-W. China (i=1)UIUC-W. China (i=2)UIUC-W. China (i=3)UIUC-W. China (i=4)

00.050.1

0.150.2

0.250.3

0.350.4

0.450.5

0 5 10 15 20

P(f

ail|i

)

Time of day (hour)

UIUC-C. Europe (i=1)UIUC-C. Europe (i=2)UIUC-C. Europe (i=3)UIUC-C. Europe (i=4)

Small interleaving factor 2 4 is enough Multiple-description coding is promising

Benjamin W. Wah 8

Loss Concealments for Real-Time Multimedia over the Internet Problem Addressed

Loss Concealment Problem

Design, analyze and evaluate robust end-to-end loss-concealment schemes

Allow reliable and real-time multimedia transmissions

Unreliable IP networks, like the Internet and wireless wide area networks

Benjamin W. Wah 9

Loss Concealments for Real-Time Multimedia over the Internet Loss concealments for low bit-rate coded speech

Coder-Independent Sample-Based MDC of Low Bit-Rate Speech

Coded UDPPacket

Description 1

PacketCoded UDP

Description 0

Lost

Dep

acke

tiza

tion

and

Dec

odin

g

Reconstructedframe withodd samples

Dei

nter

leav

ing

Pla

yed

Spee

chSe

quen

ce

Frame witheven samples

Pac

keti

zati

onC

odin

g an

d

Frame witheven samples

Frame withodd samples

Inte

rlea

ving

Sam

ple

base

d

Ori

gina

l spe

ech

sequ

ence

Drawbacks:

Aliasing: caused by down sampling Coding-frame time span lengthened

Benjamin W. Wah 10

Tests on FS-CELP 1016Original voice streamSDC (no loss)Sample-based MDC (no loss)2-way LSP-based MDC

Both descriptions recd

Only 1 description recd

2-way LSP-based MDC (PW)Both descriptions recd

Only 1 description recd

SDC (UIUC-Slovak, 46% loss)Adaptive MDCAdaptive MDC (PW)

Synt

hetic

Tes

tsIn

tern

etTe

sts

Synthetic Tests (With Concealments) Original voice stream

Loss % FS-CELP G.723-ACELP

G.732-MP/MLQ

G.729

0%

15%

30%

45%

Loss Concealments for Real-Time Multimedia over the Internet Loss concealments for low bit-rate coded speech

Coder-Dependent Parameter-Based MDC for Low Bit-Rate Speech

Description 0

Description 1

Pla

yed

Spee

chSe

quen

ce

LostCoded UDPpacket with

param. setpacket withCoded UDP

param. set

Ori

gina

l spe

ech

sequ

ence

Mod

ifie

d co

ding

Cod

ed f

ram

e

Par

amet

erb

ased

Inte

rlea

ving

Dep

acke

tiza

tion

Frame withparam. set

Reconstructedframe with

param. set

Dei

nter

leav

ing

Dec

oded

fra

me

Mod

ifie

d de

codi

ng

S2

S1

S2

S1

MDC design by correlation analysis Interleave correlated parameters to multiple descriptions and reconstruct lostparameters from description(s) received

Replicate uncorrelated parameters to multiple descriptions after reducing theamount of parameters generated

Analyze speech sequence to determine critical parameters to replicate

Benjamin W. Wah 11

Loss Concealments for Real-Time Multimedia over the Internet Loss concealments for JPEG2000 images

Coder-Dependent MDC for JPEG2000 Images

OriginalImage

Packet sequence

Description 0

Description 1

......

02subdescriptionsubdescription subdescription

00 01......

12subdescriptionsubdescription subdescription

10 11......

......subdescription00 10

subdescription subdescription01 11

subdescription subdescription02 12

subdescription

intradescription redudancyinterdescription redudancy

interleaving setUDP packet

Generate subdescriptions that minimize intra-description redundancies, withoutsacrificing inter-description redundancies

Probability of losing both subdescriptions in an interleaving set is less than 5%

Presence of intra-description redundancies reduces coding efficiency

Benjamin W. Wah 12

Loss Concealments for Real-Time Multimedia over the Internet Loss concealments for JPEG2000 images

Frequency-Based MDC System for JPEG2000 Images

hdr 1 2 3

hdr 1 2 3

hdr 1 2 3

hdr 1

Sender sideReceiver side

description 1lost data subbands

reconstructed

0

1

description 2

0

1

description

subdescriptions

Lost

originalimage

decodedimage

Packet4

Packet1 Packet3

Packet2 Packet4

Packet3Packet1

Packet2

Benjamin W. Wah 13

Loss Concealments for Real-Time Multimedia over the Internet Loss concealments for JPEG2000 images

An Illustration on Smoke

Sending 512 512 smoke compressed at 0.25 bpp by JPEG2000 Between UIUC and Thailand1 (www.iced.moe.go.th) Five out of the sixteen packets were lost in UDP

SDC and TCP

PSNR: 30.96 dB,

Delay: 13.03 s.

SDC and UDP

PSNR: 22.03 dB,

Delay: 0.46 sec.

Frequency-Based MDC+UDP

PSNR: 28.72 dB,

Delay: 0.46 sec.

Benjamin W. Wah 14

Loss Concealments for Real-Time Multimedia over the Internet Future Work

Future Work

Low bit-rate coded speech

Bandwidth-quality trade-offs

Exploiting voice properties in generating protection and concealments

MP3 coded audio

Exploiting redundancies in stereo audio for reconstruction

JPEG2000 coded images

Real-time TCP protocol with retransmissions to improve image quality

Benjamin W. Wah 15

0412demo.pdfTests on FS-CELP 1016Synthetic Tests (With Concealments)