1

Concealment of Whole-Picture Loss in Hierarchical B-Picture Scalable Video Coding

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 11, NO. 1, JANUARY 2009

Xiangyang Ji, Debin Zhao, and Wen Gao

2

Outline

ه Introductionه Proposed Error Concealment Strategyه Experimental Resultsه Conclusion

3

Introduction

ه Error concealment(EC) algorithmه a post-processing tool at decoderه conceal the erroneous region due to transmission errors according

to the correctly received information.

ه Temporal error concealment is one of the most important approaches to combat transmission errors.ه temporal replacement (TR) [3]: each damaged macroblock is

directly replaced by the co-located one in the temporally previous picture with zero motion.

[3] J. W. Suh and Y. S. Hu, “Error concealment based on directional interpolation,” IEEE Trans. Consumer Electron., vol. 43, no. 3, pp. 295–302, Aug. 1997.

4

Introduction

ه TR can be improved by a boundary matching algorithm (BMA) [4].ه a suitable motion vector for a damage MB can be selected from

the candidates based on side match distortion measure.

ه This paper not apply BMA due to the high computational estimation.

[4] W. M. Lam, A. R. Reibman, and B. Liu, “Recovery of lost or erroneously received motion vectors,” in Proc. IEEE Int. Conf. Acoustics, Speech, and Signal Process., Apr. 1993, vol. 5, pp. 417–420.

5

Introduction

ه To conceal the damaged MB, its neighboring MBs need to be correctly received.ه split the current MB and its neighboring MBs into different slices.ه one slice data typically fits one packet.ه Flexible Macroblock Ordering (FMO)

ه But, FMO is only supported for Baseline and Extended Profiles NOT for Main Profile and High Profile in H.264/AVC and SVC.

6

Purpose

ه Tackle the whole-picture loss problem.

ه The advantage of the temporal relationship can be taken when hierarchical B-picture coding is used to support temporal scalability.

7

Proposed Strategy

ه Take the temporal relationship among the adjacent video pictures into account.

ه The motion information of the lost picture is derived simply and efficiently based on the principle of temporal direct mode (TDM).

8

Proposed Strategy

ه The MV of the damaged MB can be derived from the MVs of the co-located MBs.ه temporally neighboring left and/or right B-pictures at next higher

temporal level.

ه This method could also be used to conceal lost key picture as a P-picture.

9

Outline

ه Introductionه Proposed Error Concealment Strategy

ه Hierarchical B-Picture Codingه Motion Parameters Recovery Based on the Enhanced TDM

(ETDM_EC)ه Further Improvement on Motion Parameters Recovery

(WTDM_EC)ه Experimental Resultsه Conclusion

10

Hierarchical B-Picture Coding

11

B-Picture Prediction (1/2)

ه A B-picture allows two prediction blocks from List 0 and List 1 reference buffers which contain an arbitrary set of reference pictures in forward and/or backward directions [16].

ه At encoder, its motion-compensated prediction signal can be obtained by

[16] A. M. Tourapis, F.Wu, and S. Li, “Direct mode coding for bipredictive slices in the H.264 standard,” IEEE Trans. Circuits Syst. Video Technol., vol. 15, no. 1, pp. 119–126, Jan. 2005.

12

B-Picture Prediction (2/2)

ه At decoder, for a block S in the lost picture , which is concealed as a B-picture, its prediction signal can be achieved by

13

Motion Parameters Recovery Based on the Enhanced TDM (ETDM_EC)

ه In the case of the whole-picture loss, the motion vectors of each block have to be recovered based on the temporal motion correlation.

ه Assume motion among the adjacent pictures is translational, the motion vector of the block in the lost B-picture can be estimated based on Temporal Direct Mode (TDM), as proposed in [17].

[17] Y. Chen, J. Boyce, and K. Xie, “Frame loss error concealment for SVC,” Joint Video Team(JVT), Doc. JVT-Q046, Oct. 2005.

14

Temporal Direct Mode (TDM) (1/2)

ه of the co-located block in point to the temporally most recent List 0 picture.

(4)

15

Temporal Direct Mode (TDM) (2/2)

ه of the co-located block in does not point to the temporally most recent List 0 picture.

(5)

16

TDM- Cases

ه If the picture pointed by or of the co-located block in the corresponding or picture, can not be accessed.

(5)

(4)

17

Enhanced TDM Error Concealment

18

Further Improvement on Motion Parameters Recovery (WTDM_EC)

ه For a lost B-picture which is not at the highest temporal level.ه the temporal motion relationship tends to considerably weaken as

the temporal distances become longer.

ه If the key picture is lost and recovered as a P-picture, the motion vectors of its block usually can not be derived based on TDM.

19

Further Improvement on Motion Parameters Recovery (WTDM_EC)

20

Further Improvement on Motion Parameters Recovery (WTDM_EC)

(6)

(7)

21

Further Improvement on Motion Parameters Recovery (WTDM_EC)

22

Outline

ه Introductionه Proposed Error Concealment Strategyه Experimental Resultsه Conclusion

23

Experimental Parameters

ه H.264/AVC SVC reference software: JSVM_8_6.ه Test sequences: Mobile, Stefan, Foreman, Bus, Tempete

and Coastguard.ه Group of Pictures: 16.ه Frame rate: 30 Hz.ه Frame size: 352 X 288 CIF.ه I-picture is inserted for every 32 pictures. ه Quantization parameters for highest temporal level are

composed of 28, 32, 36 and 40.ه Four packet loss patterns with average packet loss rates of

3%, 5%, 10%, and 20%

24

Rate Distortion Curves (1/2)

25

Rate Distortion Curves (2/2)

26

Quality Gain of Proposed Strategy

27

Mobile (1/5)

ه 195th frame, temporal level 4

Anchor_EC

Proposed Strategy

Correct

28

Mobile (2/5)

ه 230th frame , temporal level 3

Anchor_EC

WTDM_EC Correct

ETDM_EC

29

Mobile (3/5)

ه 156th frame , temporal level 2

Anchor_EC

WTDM_EC Correct

ETDM_EC

30

Mobile (4/5)

ه 104th frame , temporal level 1

WTDM_EC Correct

Anchor_EC&ETDM_EC

31

Mobile (5/5)

ه 256th key frame

WTDM_EC Correct

Anchor_EC&ETDM_EC

32

Outline

ه Introductionه Proposed Error Concealment Strategyه Experimental Resultsه Conclusion

33

Conclusion

ه The proposed algorithm derive the motion vector of the damaged block in the lost picture by utilizing the motion information in the temporally neighboring.

ه The proposed method also can be easily extended to the slice loss case with the error resilient tool like FMO.

ه The further improvement on motion parameters recovery perhaps introduce an extra delay for video decoding within one GOP.