J. Vis. Commun. Image R. 22 (2011) 522–528

Contents lists available at ScienceDirect

J. Vis. Commun. Image R.

journal homepage: www.elsevier .com/ locate / jvc i

An integrated temporal error concealment for H.264/AVC based on spatialevaluation criteria

Chih-Cheng Wang, Chih-Yao Chuang, Kuan-Ru Fu, Shinfeng D. Lin ⇑Department of Computer Science and Information Engineering, National Dong Hwa University, Hualien, Taiwan, ROC

a r t i c l e i n f o

Article history:Received 12 November 2010Accepted 16 June 2011Available online 30 June 2011

Keywords:H.264/AVCError concealmentError resilienceFlexible macro-block ordering (FMO)Boundary matching algorithm (BMA)Mean absolute difference (MAD)Boundary distortionTexture intensity

1047-3203/$ - see front matter � 2011 Elsevier Inc. Adoi:10.1016/j.jvcir.2011.06.002

⇑ Corresponding author. Fax: +886 3 8634010.E-mail address: david@mail.ndhu.edu.tw (S.D. Lin)

a b s t r a c t

Owing to error-prone transmission networks, the compressed video bit stream is prone to packet loss in thetransmission channel. This loss causes serious distortion and the distortion will propagate to successiveframes, especially in highly compressed video coding standard. Therefore, it is very important to efficientlyenhance the restored result. In this paper, an integrated temporal error concealment technique for H.264/AVC is proposed. The technique could effectively restore the corrupted data by adaptively integrating errorconcealment approaches with the adaptive weight-based switching algorithm. The integrated mechanismis based on spatial evaluation criteria, judged by boundary distortion estimation and texture intensity. Exper-imental results show that the technique could effectively enhance the performance of error concealment.

� 2011 Elsevier Inc. All rights reserved.

1. Introduction

H.264/AVC (advanced video coding) is the state-of-the-art videocoding standard of the ITU-T Video Coding Experts Group and theISO/IEC Moving Pictures Experts Group [1]. It provides an excellentcompression efficiency and visual quality than previous standards,because it adopts some unique techniques to reduce the redundantinformation [2], such as 4 � 4 integer transform, multiple referenceframes, variable block size, quarter–sample–accurate motion com-pensation, etc. Therefore, this CODEC can efficiently reduce theamount of bits for video representation. In error-prone transmis-sion environments, the packet loss of the highly compressed videobit stream will cause the serious distortion. The transmission errornot only results in distorting current frame but also propagates toits successive frames (error propagation, as shown in Fig. 1). Thus,how to recover the lost video data in decoder is very essential.

For solving above-mentioned problems, the error resilience andthe error concealment techniques have been adopted. The errorresilience is a mechanism in the encoder for preventing packet loss.This preventative mechanism is designed to achieve robustness ofbit streams in error-prone networks [3]. On the other hand, thespatial error concealment is the other mechanism in the decoder.It applies to concealing corrupted regions by referencing previousdecoded data. Each processing unit is macro-block (MB) and itsreferencing data is located in reference frames (temporal error con-

ll rights reserved.

.

cealment) or the current frame (spatial error concealment). Gener-ally, the recovery performance of TECs is better than SECs.

Temporal error concealment (TEC) employs the existing similar-ities between continuous frames to estimating the motion vectors(MVs). Further corrupted MBs can be replaced with the correspond-ing MBs in reference frames. Corresponding MBs are judged by esti-mated MVs, obtained by boundary matching algorithms (BMA) [4]or extrapolation algorithms [5]. Deng et al. [6] proposed an edge-direction-based boundary matching algorithm. The method utilizesedge directions to identify whether the lost MB contains multipleobjects, and then determine the restored mode of the four 8 � 8blocks in the lost MB. However, it could not recover very well with16 � 16 MBs. Qian et al. [7] used the mean absolute difference(MAD) to represent the boundary distortion. Then it recovers errorregions orderly based on the texture intensity of the correctly re-ceived adjacent MBs. This method could efficiently enhance therecovery performance. Youjun et al. [8] presented a method whichcombines three boundary matching algorithms by adopting threefixed thresholds. Cui et al. [9] adopted the boundary residual estima-tion for appending to the motion compensated MB. This methodcould enhance the recovery performance especially for smooth se-quence due to strong correlation of pixels. However, this methodis not efficient for non-smooth sequences.

Spatial error concealment (SEC) restores the lost data accordingto spatial dependency. In general, SEC is based on interpolation ap-proaches, such as bilinear interpolation (BI), multi-directional(MDI) [10] and selective directional interpolation (SDI) [11]. It uti-lizes the neighboring boundary pixels to interpolate the lost pixels.

Fig. 1. Error propagation with dispersed FMO error pattern.

C.-C. Wang et al. / J. Vis. Commun. Image R. 22 (2011) 522–528 523

Thus, as long as more neighboring data surrounding the corruptedMB exist, the recovery performance will be better. Yi et al. [12] pre-sented a method switching between two interpolation approacheswith a fixed threshold according to the directional entropy. Zhanand Zhu [13] proposed a spatial integrated approach to enhancethe recovery performance. This method switches between an inter-polation approach and the integrated approach, integrating twointerpolation approaches with the fixed weight value.

This paper proposes an integrated temporal error concealmenttechnique suited for dispersed FMO (flexible macro-block ordering)error pattern. In order to find out the best restored data, the techniqueadaptively switches between a conventional temporal error conceal-ment mode and the proposed integrated mode. The integrated modeis obtained by integrating two temporal error concealment ap-proaches with an adaptive weight. This adaptive weight is deter-mined by two spatial evaluation criteria. One of the criteria is astandard deviation, corresponding to the texture intensity, of regionsurrounding the corrupted MB. The other criterion is based on bound-ary distortion estimation. Experimental results show the proposedtechnique could effectively enhance the recovery performance.

This paper is organized as follows. The proposed temporal errorconcealment technique in H.264/AVC is presented in Section 2. InSection 3, experimental results are demonstrated. Finally, Section 4draws the conclusion.

2. The proposed temporal error concealment technique forH.264/AVC

The proposed temporal error concealment technique could effi-ciently find out the optimal restored data with dispersed FMO errorpattern. Its flowchart is shown as Fig. 2. The similar macro-block(MB) is estimated by a conventional temporal method to replacethe corrupted MB initially. Secondly, the enhanced MB is calculated

Fig. 2. The flowchart of the proposed temporal error concealment technique.

by appending enhanced residuals to the replaced MB. Then theproposed adaptive weight-based switching (AWS) algorithmadaptively integrates above two estimated data to be the inte-grated MB. Finally, the proposed texture-based selective calibra-tion technique will find out the most appropriate restored datafor corrupted MBs based on spatial evaluation criteria. The exhaus-tive steps of the proposed temporal error concealment are listed asfollows, and exhaustively described in the following sub-sections:

Step 1: Searching for the most similar MB by mean absolute dif-ference (MAD).Step 2: Calculation of boundary residuals for generatingenhanced MBs.Step 3: Calculation of the standard deviation and proposed eval-uation criterion.Step 4: Applying the proposed adaptive weight-based switchingalgorithm.Step 5: Applying the proposed texture-based selective calibra-tion technique.

2.1. Searching for the most similar MB by mean absolute difference(MAD)

The most similar MB for corrupted MB is estimated in the firststep. Like one component of AECOD [7], the mean absolute differ-ence (MAD) of external boundary pixels is adopted to search forthe most similar MB. Fig. 3 shows that the 4-pixel wide externalboundary pixels are utilized in MAD procedure. Then, the cor-rupted MB is replaced with the most similar MB. We abbreviateit as replaced MB, MBrep.

The equations of MAD are expressed as follows:

MAD mv i;mv j� �

¼wT �MADT þwB �MADBþwL �MADLþwR �MADR

M �N � wT þwBþwLþwRð Þ ;

ð1Þ

MADT mv i;mv j� �

¼X15

x¼0

X�4

y¼�1

f iþ x; jþ y;nð Þj

�f iþmv i þ x; jþmv j þ y;n� 1� ���; ð2Þ

Fig. 3. Four neighboring boundaries (4-pixel width) of the corrupted macro-block.

524 C.-C. Wang et al. / J. Vis. Commun. Image R. 22 (2011) 522–528

MADB mv i;mv j� �

¼X15

x¼0

X3

y¼0

f iþ x; jþ N þ y;nð Þj

�f iþmv i þ x; jþ N þmv j þ y; n� 1� ���; ð3Þ

MADL mv i;mv j� �

¼X�4

x¼�1

X15

y¼0

f iþ x; jþ y;nð Þj

�f iþmv i þ x; jþmv j þ y;n� 1� ���; ð4Þ

MADR mv i;mv j� �

¼X3

x¼0

X15

y¼0

f iþ N þ x; jþ y;nð Þj

�f iþ N þmv i þ x; jþmv j þ y; n� 1� ���; ð5Þ

M and N denote the size of a corrupted MB, where M and N are equalto 16. Let n and n � 1 denote the current frame and the referenceframe, respectively. (i, j) is the coordinate of one pixel in the currentframe. (mvi, mvj) means the estimated motion vector for the cor-rupted MB. The top, bottom, left, and right MBs are marked as T,B, L, and R. The coefficients, wT, wB, wL and wR, are equal to 0.25, ifthe corresponding boundaries around the corrupted MB are avail-able. If certain boundary of the corrupted MB is not available, thecorresponding coefficient is set to 0. In addition, there are at leasttwo available boundaries around each corrupted MB due to the dis-persed FMO error pattern.

2.2. Calculation of boundary residuals for generating enhancedmacro-blocks

This step utilizes boundary residuals to perform the proposed 5-tap interpolation filter in order to enhance the recovery perfor-mance. Firstly, boundary residuals are obtained by subtractingthe boundary pixels around corrupted MB from the boundary pix-els around the most similar MB in the previous frame. These arecalculated by Eq. (6):

MBBR x; y;nð Þ ¼MBCBP x; y; nð Þ �MBRBP x; y; n� 1ð Þ; ð6Þ

BR, CBP, and RBP denote the boundary residual, the boundary pixelsof the current frame and the boundary pixels of the reference frame,respectively.

Inspired by Zhan and Zhu [13], enhanced residuals for the re-placed MB are estimated secondly. The improved 5-tap filter isdeveloped to interpolate the enhanced residuals. Its equationsare expressed as Eqs. (7) and (8), and its corresponding figure is

Fig. 4. The proposed 5-tap interpolation filter: (a) the filter in the outer loop o

shown as Fig. 4. Then, the estimated residuals will be appendedto the replaced MB. At the beginning, the proposed interpolationfilter estimates the enhanced residual of the outermost loop byEq. (7), except for the four corner points. Next, the enhanced resid-ual of corner point is calculated by adopting Eq. (8) to average thefour neighboring values, where BRc3, BRc4, erc1 and erc3 are shown inFig. 4(a). Similarly, enhanced residuals of the second-outer loop areinterpolated by adopting enhanced residuals of the first-outer loop.The filter can estimate enhanced residuals for the replaced MBsequentially from the outermost loop to inner loops. Then, en-hanced residuals of the innermost loop are calculated by averagingthe partially five surrounding values, such as er1, er2, er3, er4 and er5

in Fig. 4(b). Therefore, 256 enhanced residuals can be estimated byabove-mentioned procedure. It is worth pointing out that the pro-posed 5-tap filter utilizes different interpolation policies to im-prove the weakness of the previous method [11]. This is becausefour diagonal neighboring MBs of the corrupted MB are usually lostwith dispersed FMO error pattern (e.g., f1 in Fig. 1):

eri ¼P4

n¼0xnynP4n¼0xn

; where

for general casesxn; ynð Þ ¼ ½1;3;1�; ½BRc1;BRc2; BRc3�ð Þ;

for boundary conditionsxn; ynð Þ ¼ ½1;3;6;3;1�; ½BR1;BR2;BR3;BR4; BR5�ð Þ;

8>>><>>>:

ð7Þ

erc2 ¼BRc3 þ BRc4 þ erc1 þ erc3½ �

4; four corner points only: ð8Þ

Finally, the estimated residuals are added to the replaced MB, MBrep,to form the enhanced MB, MBen.

2.3. Calculation of the standard deviation and proposed evaluationcriterion

This step calculates the standard deviation and temporal evalu-ation criterion for the following steps. The standard deviation, r, ofcorrectly received 4-pixels wide neighboring boundary pixels iscalculated to represent the texture intensity of corrupted MB. Itsequations are as follows:

r ¼

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi1

N �M � 1

XN

i¼1

XM

j¼1

ðPði; jÞ � lÞ2vuut ; ð9Þ

l ¼ 1N �M

XN

i¼1

XM

j¼1

Pði; jÞ; ð10Þ

f the replaced MB, (b) the filter in the innermost loop of the replaced MB.

Fig. 5. The proposed spatial criterion based on boundary distortion estimation.

C.-C. Wang et al. / J. Vis. Commun. Image R. 22 (2011) 522–528 525

In Eqs. (9) and (10), P(i, j) and l denote the pixel value of correctlyreceived neighboring boundary region and the mean value ofboundary pixels, respectively. N �M denotes the amount of allboundary pixels.

The proposed temporal evaluation criterion based on theboundary distortion estimation is illustrated in the following steps.This criterion is estimated from two values of the replaced MB andenhanced MB. It is based on the boundary distortion estimationshown in Fig. 5 and its equation is expressed as:

BDði; jÞ ¼XjMBEBði; jÞ �MBIBði; jÞj: ð11Þ

In Eq. (11), BD(i, j), MBEB(i, j) and MBIB(i, j), respectively, denote theboundary distortion value of the replaced/enhanced MB, the exter-nal boundary of current MB and the internal boundary of replaced/enhanced MB.

2.4. Applying the proposed adaptive weight-based switchingalgorithm

In this step, the proposed adaptive weight-based switchingalgorithm (AWS) is applies to determining a weight, x, and calcu-lating the optimal integrated MB, MBopt_aw, for the ultimate step.The optimal integrated MB is calculated by adaptively integratingthe replaced MB, MBrep, and enhanced MB, MBen, with the adaptiveweight. This algorithm is described as follows:

x = 1For (i = 1; i < 10; i++){

MBaw(i) = x �MBrep + (1 �x) �MBen

x = x � 0.125}MBopt_aw = MinBD(MBaw(i))

In above algorithm, MBaw is the integrated MB with the adaptiveweight-based switching algorithm. And MinBD() is the functionutilized to find the optimal integrated MB, MBopt_aw, with minimalboundary distortion. This algorithm is effectively utilized to opti-mize the recovery performance.

2.5. Applying the proposed texture-based selective calibrationtechnique

In the ultimate step of the proposed temporal error conceal-ment technique, the texture-based selective calibration technique(TSCT) is proposed to determine the optimal restored MB, MBopt.The determination is based on many criteria, such as boundary dis-tortions and standard deviation, r. It is expressed by the followingalgorithm:

#01

Set initial thresholds (SDup = 100; SDlow = 75; BDth = 1;) #02 For (i = 0; i < 4; i++) #03 { #04 if (i = 0) #05 if (r > SDlow) #06 if (BDrep > BDen & |BDrep � BDen| > BDth) #07 MBopt = 0.5 �MBrep + 0.5 �MBen

#08

else #09 MBopt = MBrep

#10

else #11 if (r > SDlow & r < SDup) #12 if (BDrep > BDen & |BDrep � BDen| > BDth) #13 MBopt = MBopt_aw

#14

else #15 MBopt = MBrep

#16

SDup = SDup � 25 #17 SDlow = SDlow � 25 #18 BDth = BDth + 2 #19 }

In above algorithm, all standard deviations are normalized form0 to 100 firstly. SDup and SDlow are the dynamic upper-bound anddynamic lower-bound of the standard deviation, respectively. Theycould determine four intervals of standard deviation to representdifferent texture intensity: high, medium-high, medium-low andlow texture. The threshold, BDth, is utilized to determine the mag-nitude of boundary distortion. It is calculated by BDrep and BDen, theboundary distortion of the reconstructed MB, MBrep, and the en-hanced MB, MBen, respectively. Lines 16 to 18 of the algorithmmean that SDup and SDlow decrease 25 and BDth increases 2 afterthe iteration. In other words, the interval with higher texture cor-responds to smaller threshold for boundary distortion, BDth. In ourobservations, the optimal restored MB is generally the replacedMB, MBrep (obtained by step1) in smooth sequences. As to theinterval with higher texture, it will be obtained by averaging MBrep

and MBen (line 7). Therefore, this proposed TSCT could optimize therecovery performance for damaged MBs by many above-men-tioned criteria.

3. Experimental results

The proposed technique is implemented in the H.264/AVC ref-erence software JM16.2 [14]. Several benchmark sequences(QCIF, 144 � 176; CIF, 288 � 352), such as Akiyo, foreman and flow-er, etc, are employed in this experiment. Different random packeterror ratio (PER), 5%, 10% and 15%, are utilized in order to evaluatethe performance of the proposed error concealments. Main param-eter settings are listed as follows:

Sequence type

IPPP (QP: I 28, P 28) Group of pictures (GOP) 15 Search setting Fast full search (range = ±16) Number of reference frames 1 Slice setting Dispersed FMO (2 slice groups)

The proposed integrated temporal error concealment tech-nique, texture-based selective calibration technique (TSCT), isexperimented with initial thresholds: SDup = 25, SDlow = 75, andBDth = 1. Comparisons with other schemes at different error ratesare listed in Tables 1 and 2. The tables show that the proposed TSCTis better than BMA [4], EDBMA [6], BRR [13] and JM16.2. In Table 1,it is worth pointing out that BRR [13] restores smooth sequences,such as Akiyo, better than high texture sequences, such as flower.Conversely, EDBMA [6] restores high texture sequence better than

Table 1Comparisons with other schemes at different error rates (QCIF sequences).

Sequence PER (%) BMA [4] (dB) EDBMA [6] (dB) BRR [13] (dB) JM 16.2 (dB) Proposed TSCT (dB)

Akiyo 5 32.04 32.35 37.53 37.61 37.7710 29.72 30.14 36.13 37.05 37.3815 28.36 28.55 35.61 36.50 36.93

Foreman 5 31.11 31.72 31.24 33.01 33.7010 28.15 28.94 27.92 30.89 31.6315 26.80 27.55 25.75 29.27 30.29

Flower 5 28.74 29.26 28.46 29.16 31.0010 25.75 26.35 26.07 26.18 28.3715 24.36 24.97 24.05 24.47 26.91

Table 2Comparisons with other schemes at different error rates (CIF sequences).

Sequence PER (%) BMA [4] (dB) EDBMA [6] (dB) BRR [13] (dB) JM 16.2 (dB) Proposed TSCT (dB)

Carphone 5 33.36 33.57 35.09 34.40 35.3710 30.37 30.63 32.45 32.12 32.9315 28.59 28.78 30.46 30.20 31.10

Foreman 5 32.09 32.44 33.57 32.21 33.7510 28.59 29.05 30.54 29.41 30.7715 26.16 26.61 27.89 26.49 28.15

Stefan 5 30.28 30.54 30.94 30.29 31.2210 26.37 26.73 26.91 27.14 27.3615 24.11 24.35 24.35 24.64 24.92

Fig. 6. Comparisons of PSNR curves with other schemes.

526 C.-C. Wang et al. / J. Vis. Commun. Image R. 22 (2011) 522–528

Fig. 7. Recovery performance for the 16th frame of foreman (QCIF): (a) the error-free frame, (b) the corrupted frame, (c) JM (25.5213 dB), (d) BMA (25.2553 dB), (e) EDBMA(26.1184 dB), (f) TSCT (28.5749 dB).

Fig. 8. Recovery performance for the 4th frame of carphone (CIF): (a) the error-free frame, (b) the corrupted frame, (c) JM (32.1745 dB), (d) BMA (29.6117 dB), (e) EDBMA(30.2700 dB), (f) TSCT (33.3733 dB).

C.-C. Wang et al. / J. Vis. Commun. Image R. 22 (2011) 522–528 527

smooth sequence. But the proposed technique could outperformtwo above methods in each sequence due to the adaptive integra-tion of two error concealment modes. Moreover, it also outper-forms JM in each sequence, especially in the high texture sequence.

The PSNR curves of Tables 1 and 2 are shown in Fig. 6. InFig. 6(c), the PSNR curve of the proposed technique is significantlybetter than other schemes due to the high texture sequence, flower.Then, the comparisons of subjective quality are shown in Figs. 7and 8. In Fig. 8, the recovery performance of the proposed tech-nique, Fig. 8(f), is better than other schemes, Fig. 8(d) and (e), espe-cially in the left background. It is noted that the texture of curtainis straighter in Fig. 8(f).

4. Conclusion

An integrated temporal error concealment technique for H.264/AVC has been proposed. It could effectively restore the corrupteddata by adaptively integrating two error concealment approacheswith the proposed adaptive weight-based switching algorithm.The integrated mechanism is based on spatial evaluation criteria,

judged by boundary distortion estimation and texture intensity.Therefore, the proposed technique could obtain the optimal recov-ery data for corrupted macro-blocks. Experimental results demon-strate the technique could effectively enhance the recoveryperformance and outperform other schemes.

References

[1] ITU-T Rec. H.264/ISO/IEC 11496-10, Advanced Video Coding, Final CommitteeDraft, Document JVTG050, 2003.

[2] T. Wiegand, G.J. Sullivan, G. Bjontegaard, A. Luthra, Overview of the H.264 AVCvideo coding standard, IEEE Transactions on Circuits and Systems for VideoTechnology 13 (7) (2003) 560–576.

[3] Y. Wang, S. Wengers, J. Wen, A.K. Katsaggelos, Error resilient video codingtechniques, IEEE Signal Processing Magazine 17 (4) (2000) 61–82.

[4] W.-M. Lam, A.R. Reibman, B. Liu, Recovery of lost or erroneously receivedmotion vectors, in: IEEE International Conference on Acoustics, Speech, andSignal Processing, vol. 5, April 1993, pp. 417–420.

[5] B. Yan, H. Gharavi, A hybrid frame concealment algorithm for H.264/AVC, IEEETransactions on Image Processing 19 (2010) 98–107.

[6] X. Deng, Y. Liu, C. Hong, J. Bu, C. Chen, A temporal error concealment algorithmfor H.264/AVC based on edge directions, in: IEEE International Conference onImage Processing, November 2009, pp. 941–944.

528 C.-C. Wang et al. / J. Vis. Commun. Image R. 22 (2011) 522–528

[7] X. Qian, G. Liu, H. Wang, Recovering connected error region based on adaptiveerror concealment order determination, IEEE Transactions on Multimedia 11(4) (2009) 683–695.

[8] X. Youjun, L. Na, F. Liangmou, X. Shengli, Video error concealment using spatio-temporal boundary matching, in: International Congress on Image and SignalProcessing, October 2009, pp. 1–5.

[9] Y. Cui, Z. Deng, W. Ren, Novel temporal error concealment algorithm based onresidue restoration, in: International Conference on Wireless Communications,Networking and Mobile Computing, September 2009, pp. 1–4.

[10] W. Kwok, H. Sun, Multi-directional interpolation for spatial error concealment,IEEE Transactions on Consumer Electronics 39 (1993) 455–460.

[11] W.-Y. Kung, C.-S. Kim, C.-C. Jay Kuo, Spatial and Temporal error concealmenttechniques for video transmission over noisy channels, IEEE Transactions onCircuits and Systems for Video Technology 16 (7) (2006) 789–803.

[12] J.-W. Yi, E. Cheng, F. Yuan, An improved spatial error concealment algorithmbased on H.264, in: International Symposium on Intelligent InformationTechnology Application, vol. 3, November 2009, pp. 455–458.

[13] X. Zhan and X. Zhu, Refined spatial error concealment with directionalentropy, in: International Conference on Wireless Communications,Networking and Mobile Computing, September 2009, pp. 1–4.

[14] H.264/AVC Reference Software Joint Model 16.2 (JM 16.2). Available at:<http://iphome.hhi.de/suehring/tml/download/>, February 2010.