By: Hitesh YadavSupervising Professor: Dr. K. R. Rao

Department of Electrical EngineeringThe University of Texas at ArlingtonThe University of Texas at Arlington

Optimization of the Optimization of the Deblocking Filter Deblocking Filter Algorithm in H.264 Algorithm in H.264 Codec for Real Time Codec for Real Time ImplementationImplementation

Brief Information about Brief Information about H.264H.264 H.264 is the latest video coding standard It addresses practical applications such as

internet multimedia, wireless video , video conferencing etc.

In terms of compression efficiency it is up by a factor of two over MPEG-2.

The increase in compression efficiency comes at the expense of complexity.

The resulting complexity does depend upon the profile of a standard implemented which is application dependent.

H.264 EncoderH.264 Encoder

H.264 H.264 DecoderDecoder

Why Deblocking Filter?Why Deblocking Filter? At very low bit rates, coding visual

artifacts are noticed in decoded frames.

Prominent among them are blocking effects and ringing effects.

Deblocking filter is used in the H.264 encoder and decoder to remove the blocking effects from decoded frames.

Causes of Blocking EffectsCauses of Blocking Effects

Transform coding causes discontinuity between adjacent blocks.

Severity of blocking effects is subject to the coarseness of quantization of the transform coefficients.

Motion-compensated prediction also contributes to the blocking effects but mostly in mildly textured areas.

Regions Susceptible to Regions Susceptible to Blocking effectsBlocking effects For intra coded blocks the effect is

hidden in either the more spatially active areas or smooth areas.

For predictive coded blocks the effect mainly occurs in mildly textured areas.

For predictive coded blocks the artifact known as false edge typically occurs for macro blocks with smoothly texture content.

Loop filtering vs. post Loop filtering vs. post filteringfiltering Post filters offer maximum freedom

for decoder implementation as they are not normative part of the standard.

Empirical tests have shown that loop filtering improves both objective and subjective quality of video streams with significant reduction in decoder complexity compared to post filter.

Desired Deblocking filterDesired Deblocking filter Smoothing artificial discontinuities

between blocks. Differentiating between image

edges and artificial edges. Image edges should not be

smoothed as it degrades image quality.

If needed, filters can be applied specific to image edges.

Desired deblocking filterDesired deblocking filter

It should remove the blocking effects without blurring the image.

Its computational complexity should be low.

It can be implemented in real time systems.

Summary on Relative Summary on Relative ComplexityComplexity

POCS-based algorithm

Weighted sum based algorithm

Adaptive algorithms

Algorithm Flow

Iteratively projecting back and forth

between two sets on entire picture

Grading of blocks with grading

matrix iterative on every pixel

Iteratively classify and applying filter on

every block edge.

Major Operations

Low-pass filtering,DCT

Weighted sum of 4 pixels for each

pixel

3-tap or 5-tap filter on pixels across

edges

Relative Computation Complexity

High Medium Low

Relative Implementation Complexity

High Low Medium

Visual Quality Best Good Good

Algorithm used for Algorithm used for deblocking filter in H.264 deblocking filter in H.264 StandardStandard

As the relative computation complexity of adaptive algorithm is low as shown in the table, they are the first choice in real time implementation.

Deblocking filter uses adaptive algorithm in H.264 standard to remove the blocking effects.

Deblocking filter Deblocking filter operationoperation The deblocking filter is applied to all

the edges of a 4x4 pixels block in each macroblock except to the edges on the boundary of a frame or a slice.

For each block,vertical edges are filtered from left to right first, and then horizontal edges are filtered from top to bottom.

Main characteristics of Main characteristics of deblocking filterdeblocking filter On slice level, the filtering strength

can be adjusted to the individual characteristics of the video sequence.

On edge level, the filtering strength is dependent on inter/intra, motion and coded residuals.

On pixel level, quantizer dependent threshold can turn off filtering for every individual pixel.

Principle of deblocking Principle of deblocking filterfilter

The decision tap for each pixel is based on the following factors.

1. Boundary strength2. Thresholds α and β.3. The content of sample pixels

Decision flow of bS where Decision flow of bS where P and Q denote adjacent P and Q denote adjacent blocksblocks

Decision flow ofDecision flow offilter tap selectionfilter tap selection

bS!=0 AND |A0-B0| <α AND |A1-A0| <β AND|B1-B0|<β

Problems with the Problems with the Deblocking FilterDeblocking Filter

Analysis of run-time profiles of decoder sub-functions reported that deblocking filter process in H.264 standard is the most computationally intensive part.

Deblocking filter took as much as one-third of the computational resources of the decoder.

Reasons for the Reasons for the Complexity Complexity High adaptivity of the filter which

requires conditional block edge and pixel levels.

As a result, conditional branches almost inevitably appear in the innermost loops of the algorithm.

Small block size employed for residual coding also contributes to high complexity.

Also the code exposes little parallelism.

How can complexity be How can complexity be reduced?reduced? Some of the branches are inherited to

the algorithm itself.So it is hard to eliminate them at programming level.

Nothing can be done about the small block size employed for residual coding.

If the conditional branches in the innermost loop of the algorithm and access to memory are reduced ,complexity can be reduced.

Why loops add to Why loops add to complexity in real time?complexity in real time? Program code includes extensive

conditional branching which makes it unsuitable for deeply pipelined processors. Also the little parallelism exhibited by code makes it unsuitable for VLIW processors.

VLIW processors otherwise are well suited for video encoding/decoding applications.

Proposed AlgorithmProposed Algorithm

Intra Frame Results for Main Profile

Test clip(QCIF)

QP

PSNR (dB)

Reconstruction with

ProposedMethod

Reconstruction without

Loop Filter JM 9.2 (H.264 reference

software)

Reconstruction with Loop

Filter JM 9.2 (H.264 reference software)

Foreman 37 31.615 31.363 31.637

Car phone 37 31.616 31.318 31.620

Car phone 45 26.671 26.291 26.612

News 35 32.204 32.040 32.043

News 39 29.515 29.313 29.55

Silent 39 29.489 29.269 29.330

Container 39 29.539 29.383 29.493

Container 45 25.718 25.553 25.710

Bridge-close 37 29.886 29.738 29.828

Bridge-close 45 25.816 25.635 25.805

Blocking Artifacts

Reconstructed I frame without using a loop filter with QP=37

Reconstructed I frame with proposed method with QP=37

Reconstructed I frame without using a loop filter with QP=45

Reconstructed I frame with proposed method with QP=45

Reconstructed I frame without using a loop filter with QP=37

Reconstructed I frame with proposed method with QP=37

Reconstructed I frame without using a loop filter with QP=45

Reconstructed I frame with proposed method with QP=45

Test clip(QCIF)-Type of frames

QP

PSNR (dB) Total number of bits used

Reconstruction with ProposedMethod

Reconstruction without Loop filter JM 9.2 (H.264 software)

Reconstruction with Loop filter JM 9.2 (H.264 software)

Reconstruction with ProposedMethod

Reconstruction withoutLoop filterJM 9.2 (H.264 software)

Reconstruction with Loop filter JM 9.2 (H.264 software)

Foreman-P 39 29.883 29.834 29.692 2085 2131 2107

News-P 39 27.846 27.528 27.771 4119 4074 4235

Car phone-P 39 30.271 30.024 30.171 817 897 720

Bridge close-P 39 28.484 28.495 28.480 105 52 126

Foreman-B 39 28.925 28.879 29.054 489 515 499

News-B 39 28.085 28.307 27.982 1424 1802 1687

Car phone-B 39 29.637 29.438 29.491 197 203 193

Bridge close-B 39 28.532 28.558 28.480 53 53 53

P Frame and B-frame P Frame and B-frame results for Main profileresults for Main profile

Reconstructed P frame without using a loop filter with QP=39

Reconstructed P frame with proposed method with QP=39

Reconstructed B frame without using a loop filter with QP=39

Reconstructed B frame with proposed method with QP=39

Results for a GOP of size 10

Frame Type- frame number(Foreman _qcif)

QP

PSNR (dB)

Reconstruction with ProposedMethod

Reconstruction withoutLoop FilterJM 9.2 (H.264 software)

Reconstruction with LoopFilter JM 9.2 (H.264 software)

Intra -0 45 26.058 25.851 26.096

B -3 39 28.1341 28.066 28.044

P -8 39 28.668 28.556 28.411

B -9 39 28.269 28.439 28.157

P-10 39 28.731 28.727 28.534

AdvantagesAdvantages Memory is saved as the proposed loop filter

code size is 11kb compared to JM 9.2(H.264 software) loop filter code size of 21kb.

JM 9.2 uses 2 tables of size 52 bytes and one table of size 260 bytes to check the pixel filtering condition. No such tables are used in the proposed method to check the filtering condition. Hence time of the processor is saved.

Conditional loops in the innermost loops of the algorithm are reduced compared to JM 9.2.

Further ResearchFurther Research The proposed deblocking filter can be

implemented in a DSP or VLIW processor. The deringing filter can also be incorporated

to see its effects on the reconstructed video. Computationally realizable image recovery

techniques can be explored in H.264. Transforms which do not produce blocking

artifacts but at the same time provides the benefits of integer DCT can be explored.

1. Draft ITU-T Recommendation and final draft international standard of joint video specification (ITU-T Rec. H.264/ ISO/ IEC 14 496-10 AVC), Mar. 2003.

2. I. E. G. Richardson, “H.264 and MPEG-4 Video Compression: Video coding for next-generation multimedia”, Hoboken, NJ, Wiley, 2003.

3. H.R.Wu and K.R.Rao, “Digital Video Image Quality and Perceptual Coding”, Taylor and Francis, Dec. 2005.

4. Y. –L. Lee and H. W. Park, “Loop filtering and post-filtering for low- bit-rates moving picture coding”, Signal processing: Image Communication, vol. 1, pp. 94-98, Dec. 1999.

5. J. Lainema and M. Karczewicz, “TML 8.4 loop filter analysis”, ITU-T SG16 Doc. VCEG-N29, 2001.

6. V. Lappalainen, A. Hallapuro, and T. D. Hamalainen, “Complexity of Optimized H.26L Video Decoder Implementation”, IEEE Trans. CSVT, vol. 13, pp. 717-725, July 2003.

7. M.-Y. Shen and C. - C Jay Kuo, “Review of postprocessing techniques for compression artifact removal”, Journal of Visual Communication and Image Representation, vol. 9, pp. 2-14, Mar. 1998.

8. J. J. Zou and H. Yan, “A POCS-based method for reducing artifacts in BDCT compressed images,” Proc. 16th Int’l Conf. on Pattern Recognition, vol. 2, pp. 11-15, Aug. 2002.

ReferencesReferences

9. S. A. Karunasekra and N. K. Kingsbury, “A distortion measure for blocking artifacts on images based on human visual sensitivity”, IEEE Trans. Image Processing, vol. 4, pp. 713-724, June 1995.

10. G. Fan and W. K. Chan, “Model-based edge reconstruction for low-bit rate wavelet compressed images”, IEEE Trans. CSVT, vol. 10, pp. 120-132, Jan.2000.

11. N. Jayant, J. Johnson, and R. Safranek, “Signal compression based on models of human perception”, Proc. IEEE, vol. 81, pp. 1385-1422, Oct. 1993.

12. H. S. Malvar, Signal Processing with Lapped Transforms. Boston, MA: Artech House, 1992.

13. Y. Zhang, R. Pickholtz, and M. Loew, “A new approach to reduce the blocking effect of transform coding”, IEEE Trans. Commun., vol. 41, pp. 299-302, Feb. 1993.

14. B. Ramamurthi and A. Gersho, “ Nonlinear space-variant post processing of block coded images”, IEEE Trans. ASSP, vol. ASSP-34, pp. 1258-1267, Oct. 1986.

15. K. R. Rao and P. Yip, “Discrete cosine Transform: Algorithms, Advantages, Applications”, Boston, MA: Academic press, 1990.

16. J. J. Zou and H. Yan, “A POCS-based method for reducing artifacts in BDCT compressed images,” Proc. 16th Int’l Conf. on Pattern Recognition, vol. 2, pp. 11-15, Aug. 2002.

17. P. List, A. Joch, J. Lainema, G. Bjntegaard, and M. Karczewicz, “Adaptive deblocking filter,” IEEE Trans. CSVT, vol.13, pp. 614-619, Jul. 2003.

18. C. Wang, W-J Zhang, and X-Z Fang, “Adaptive reduction of blocking Artifacts in DCT domain for highly compressed images,” IEEE Trans. Consumer Electronics, vol.50, pp. 647-654, May 2004.

19. Y. Zhao, G. Cheng and S. Yu, “Post processing technique for blocking artifacts reduction in DCT domain,” Electronics Letters, vol. 40, issue 19, pp. 1175-1176, Sept. 2004.

20. S. D. Kim, et al, “A deblocking filter with two separate modes in block-based video coding”, IEEE Trans. CSVT, vol. 9, pp. 156-160, Feb. 1999.

21. S.-K. Kwon, A. Tamhankar, K. R. Rao, “Overview of H.264 / MPEG-4 Part 10”, Journal of Visual Communication and Image Representation, vol. 17, pp. 186-216, April 2006.

22. S. C. Chang, et al, “A platform based bus-interleaved architecture for deblocking filter in H.264/ MPEG-4 AVC”, IEEE Int’l Conf. on Consumer Electronics, vol. 51, pp. 249-255, 2005. 23. H.264 software (JM9.8/ FRExt) from

http:/ / iphome.hhi.de/ suehring/ tml/ download/ jm98.zip. 24.A. Puri, H. Chen and A. Luthra, "Video Coding using the

H.264/ MPEG-4 AVC compression standard", Signal Processing: Image Communication, vol. 19, pp.793-849, Oct. 2004.

25 Test sequences are obtained from: http:/ / trace.eas.asu.edu/ yuv/ qcif.html

Thank Thank You !!You !!