Unified Loop Filter for High-performance Video Coding

Yu Liu and Yan Huo

ICME2010, July 19-23, Singapore

Outline

2

• Introduction• Proposed Unified Loop Filter• Experimental Results• Conclusion

Introduction

• Conventional Video Coding Standard– Block-based DPCM coding

• Transform, quantization, ME/MC, in-loop deblocking filter, entropy coding

• In-loop Deblocking Filter– A bank of fixed low-pass filters to alleviate blocking artifacts

• Assume smooth image model– thus singularities, such as edges and textures, are not handled correctly– an analysis on the gradients across the boundary is performed to check whether the

filtering should be skipped to preserve image sharpness.

• Pre-defined filter coefficients– do not retain the frequency-selective properties or have the ability to suppress

quantization noise optimally

3

Introduction

• Adaptive Wiener Filter [4-7]– Well-known optimal linear filter: adaptive post/loop filter

• Improve the quality of reconstructed picture degraded by compression

– Pros:• Guarantee the optimized objective quality restoration

– Cons:• Can’t efficiently improve the subjective quality if used alone; thus• Have to be utilized on top of in-loop deblocking filtered picture to achieve

both improved objective and subjective quality

4

Introduction

• Proposed Unified Loop Filter– Is there a way to combine the advantages of deblocking

filter and Wiener filter into a unified filtering framework?– Motivation

• The fact: multiple sources of information loss in current video coding standards, such as H.264/AVC

• Quantization: the only source of information loss, prior to H.264/AVC• Deblocking loop filter in H.264/AVC: another source of information loss• Adaptive loop filter in KTA:

– may not be able to reach the capability upper-bound of picture restoration– due to additional information loss brought by the deblocking filtering

– Unified Loop Filter• Reduces the number of the sources causing information loss• Thus further improves the capability of picture restoration

5

Introduction

• Block Diagram of Conventional Video Codec

6

T Q

Intra-Pred

Inter-Pred

ME

Reference Buffer

Adaptive Loop Filter

DeblockingLoop Filter

Q-1+T-1

Entropy Coding

+

+

Adaptive Post Filter

Video

-

Display

Introduction

• Block Diagram of Video Codec with Unified Loop Filter

7

T Q

Intra-Pred

Inter-Pred

ME

Reference Buffer

Unified Loop Filter

Q-1+T-1

Entropy Coding

+

+Video

-

Display

Proposed Unified Loop Filter

• Order Statistics Filter– Filters utilizing order statistics information [9], improved on median filters, can

effectively remove the blocking artifacts and ringing artifacts, while retaining the sharpness of edges.

– Although order statistics filter is a nonlinear filter, it can be optimized to minimize the mean square error by using linear combination of ordered statistics.

– Here, the order statistics filter is used to combine nonlinear enhancement filter and linear restoration filter into one unified filtering framework.

• Unified Loop Filter– Suppose that X=(x1, x2, …,xc,… , xN)T is a support vector

containing N pixels of the reconstructed picture arranged by the spatial order surrounding the central pixel xc.

– The unified loop filter is constructed as follows:

8

...

... xc

...

x1

x2

...

...

xN

...

...

...

...

Proposed Unified Loop Filter

• Step 1: Vector of Similarity Statistics– The support vector X is converted to form a vector of similarity statistics X’=(x’1,

x’2, … , x’N)T by using the following equation:

where f(xc,xi) is the similarity function. Real-valued similarity functions have to satisfy the following constrains:

– In this paper, the following similarity function is adopted:

where σ is the spread parameter controlling the strength of similarity function.

9

iiccici xxxfxxxfx ),()],(1['

. if),,(),(.3

,0),(lim.2

,1),(lim.1

22112211

||

0||

bababafbaf

baf

baf

ba

ba

)/)(10(1),( 22 ba,,Clipbaf

Proposed Unified Loop Filter

• Step 2: Similarity-Ordered Statistics Filter– The vector of similarity statistics X’ is further ordered to form a vector of

similarity-ordered statistics Xn=(x’(1),x’(2), …, x’(N))T by using the following rule:

– Then the output of non-linear similarity-ordered statistics filter becomes

where Wn is the vector of N optimized filter coefficients.

10

cici xxxx )1()( ''

nTnn XWy

• Step 3: Unified Loop Filter– In order to improve the coding efficiency, linear Wiener filter should also be

incorporated into the unified loop filter. Suppose that Xl=(x1, x2, …,xc,… , xM)T is the support vector of Wiener filter, the output of Wiener filter becomes

where Wl is a vector of M optimized filter coefficients. – Generally speaking, Wiener filter is also a kind of order statistics filter, called as

spatially ordered statistics filter, because its support vector Xl is constructed by arranging the M pixels in a spatial order. Therefore, nonlinear similarity-ordered statistics filter is concatenated with linear spatially ordered statistics filter, aka Wiener filter, to form the unified loop filter:

where Xu =(x’(1), x’(2), … , x’(N) , x1, x2, … , xM )T and Wu is a vector of M+N optimized filter coefficients.

Proposed Unified Loop Filter

11

uTuu XWy

Nonlinear Part Linear Part

lTll XWy

• Optimization of Unified Loop Filter– The optimization of the unified loop filter falls into the classical optimization

framework of least mean square error (LMSE). The solution can be obtained by solving the Wiener-Hopf equations:

where xo is the original video frame. The minimization problem can be solved by the Wiener-Hopf equation, which is given by

where Ru,u(k,l) is the auto-correlation function of xu, which is defined as

and Ro,u(l) is the cross-correlation function between xo and xu , which is defined as

Proposed Unified Loop Filter

12

2

1~

2

~))()((minarg)(minarg

~

NM

kuuo

Wuo

Wu kxkwxEyxEW

uu

NMlforlRlkRkwNM

kuouuu

1),(),()(1

,,

)]()([),(, lxkxElkR uuuu

)]([)(, lxxElR uouo

Proposed Unified Loop Filter

• Filter Design– Two considerations in the filter design of unified loop filter:

• not only the subjective enhancement (for removing the blocking and ringing artifacts)• but also the objective restoration (for improving the coding efficiency)

– For Luma Component (Y)• In unified loop filter, nonlinear part consists of one 12-tap diamond filter, and linear part consists

of four kinds of different taps (1-tap, 13-tap, 25-tap, and 41-tap) diamond filters with central point symmetry

13

2

5 6

9

0

1

4

8

11

10

7

3

x

0

or or

or

Nonlinear part Linear part

2

5 6

2

0

1

4

3

0

1

4

3

5

2

5 6

2

0

1

4

3

0

1

8

3

7

11 1210 10119 9

7 68 45

2

5 6

2

0

1

4

3

0

1

8

3

7

11 1210 14139 15

7 68 45

19 2018 181917 1716 16

13 1214 101115 9

Proposed Unified Loop Filter

• Filter Design – For Chroma Components (Cr/Cb)

• In unified loop filter, nonlinear part consists of one 4-tap diamond filter, and linear part consists of two kinds of different taps (1-tap and 13-tap) diamond filters with central point symmetry

• Selection of Filter Tap Type– The tap type of linear part in unified loop filter is decided by rate-distortion optimization

selection within the whole frame:

– The filter side information includes the filter tap type and filter coefficient quantization bits, which are encoded and transmitted to the decoder side.

14

0

or

Nonlinear part Linear part

2

0

1

3

x

2

5 6

2

0

1

4

3

0

1

4

3

5

RDJ filter

Experimental Results

• Test Conditions– The proposed unified loop filter has been implemented within JM11.0KTA2.4r1 reference

software. The test conditions are listed as follows:

Table 1. Test conditions

15

Test Sequence 1280x720 progressiveCoding Structure IBBPBBP…

Intra Frame Period Only the first frameEntropy Coding CABAC

R-D Optimization ON

QP I(22, 27, 32, 37) P(23, 28, 33, 38) B(24, 29, 34, 39)

Transform Size 4x4, 8x8Reference Frame 4

Search Range ± 64Frame Number 61

• Objective Performance Comparison

Experimental Results

Table 2. Coding gain comparison of different coding schemes, compared with H.264/AVC High Profile, in BD bitrate reduction (BR) and BD-PSNR gain (for Luma)

16

SequenceALF w/o DLF ALF + DLF QALF + DLF ULF

Δ BR (%) Δ PSNR (dB) Δ BR (%) Δ PSNR (dB) Δ BR (%) Δ PSNR (dB) Δ BR (%) Δ PSNR (dB)

Bigships -4.82 0.13 -6.93 0.19 -8.38 0.23 -9.95 0.28City -15.49 0.51 -15.43 0.52 -16.32 0.55 -17.27 0.58

Crew -7.39 0.2 -10.74 0.28 -14.43 0.39 -12.12 0.33Harbour -12.64 0.49 -14.26 0.55 -14.21 0.55 -14.37 0.56

Jet -5.51 0.12 -6.46 0.15 -10.64 0.28 -11.27 0.30Night -1.46 0.05 -5.52 0.21 -6.32 0.24 -6.54 0.25Optis -5.75 0.15 -5.84 0.14 -7.87 0.20 -12.28 0.32

Sailormen -7.63 0.20 -10.24 0.27 -11.14 0.30 -11.77 0.31Sheriff -5.89 0.16 -7.72 0.22 -8.90 0.25 -10.64 0.30

ShuttleStart -7.73 0.23 -7.27 0.21 -9.36 0.27 -9.54 0.28SpinCalendar -4.11 0.09 -7.38 0.19 -14.31 0.36 -11.03 0.29

Average -7.13 0.21 -8.89 0.27 -11.08 0.33 -11.53 0.34

Experimental Results

• Subjective Quality Comparison

17

(a) Anchor (b) ALF w/o DLF

(c) ALF + DLF (e) ULF Figure. Part (128x128) of the reconstructed SpinCalendar sequence at the 54th frame with QP=38.

• Complexity Comparison– Compared with “ALF+DLF”, extra complexity introduced by ULF includes:

• Similarity computation: lookup-table (LUT) technique • Sorting process: counting sort algorithm with linear complexity O(n).

– With the help of the two fast algorithms• ULF only increases the execution time 13.70% and 26.52% on average for the encoder and the

decoder, respectively, compared with “ALF+DLF”

Experimental Results

Avg. Exec. Time (s)

Anchor ALF w/o DLF ALF+DLF QALF+DLF ULF

Enc Dec Enc Dec Enc Dec Enc Dec Enc Dec

Bigships 1019.695 12.332 1116.576 16.809 1116.710 19.074 1507.433 16.715 1368.247 23.549

City 1102.105 12.789 1198.029 16.813 1197.776 19.137 1611.106 17.707 1467.945 23.794

Crew 1010.664 11.746 1109.336 15.347 1109.758 18.145 1501.561 15.902 1173.826 23.790

Harbour 1158.656 14.234 1251.545 18.152 1253.102 21.360 1654.652 21.055 1396.649 26.969

Jet 914.743 11.840 1011.421 13.864 1012.875 15.496 1300.529 13.352 1104.618 19.751

Night 1101.743 12.445 1199.932 16.985 1197.756 19.219 1580.087 16.715 1292.450 26.666

Optis 995.519 12.461 1095.312 15.683 1093.754 17.532 1450.333 16.035 1259.534 21.738

Sailormen 1102.997 14.032 1197.463 18.000 1199.272 20.309 1612.117 18.324 1352.911 25.752

Sheriff 1006.427 12.945 1103.943 16.508 1105.292 18.793 1468.728 17.258 1258.969 23.923

ShuttleStart 874.494 10.856 973.000 12.438 973.945 14.828 1246.393 13.008 1085.115 17.301

SpinCalendar 1117.885 14.329 1214.541 16.945 1217.469 20.535 1614.543 17.219 1426.570 25.406

Total Average 1036.812 12.728 1133.736 16.140 1134.337 18.584 1504.316 16.663 1289.712 23.513

18

Table 3. Average execution time comparison of different coding schemes

Conclusion

• Unified Loop Filter (ULF)– Combine the advantages of linear filter and nonlinear filter to achieve both

objective and subjective quality optimization– The deblocking loop filter in conventional video codec can be removed, and thus

replaced by the proposed unified loop filter– The proposed unified loop filter can be used in any hybrid video coding system

• Classification-based Unified Loop Filter (CULF)– Global v.s. Local: different regions have different quantization error characteristics– It is better to classify pixels into different groups:

• one group for boundary pixels with blocking artifacts• one group for non-boundary and boundary pixels without blocking artifacts• other group for non-filtering pixels

– Different unified loop filter with different characteristics is applied to each group– Additional 1.57% bitrate reduction is achieved, compared with ULF

19

Thank You!

Q&A

20