Class-count Reduction Techniques for ContentAdaptive Filtering

Hao HuEindhoven University of Technology

Eindhoven, the NetherlandsEmail: h.hu@tue.nl

Gerard de HaanPhilips Research Europe

Eindhoven, the Netherlands

Abstract—In the field of image/video enhancement, contentadaptive filtering has shown superior performance over fixedlinear filtering. The content adaptive filtering, first classifies thelocal image content based on different image features, such asstructures and contrast. Then in every class, a least mean square(LMS) optimal filter is applied. A disadvantage of the concept isthat many classes may be redundant, which leads to an inefficientimplementation. In this paper, we propose various class-countreduction techniques based on class-occurrence frequency, coef-ficient similarity and error advantage, which can greatly simplifythe implementation without sacrificing much performance.

I. INTRODUCTION

Image/video enhancement often involves filtering. Contentadaptive filtering has received considerable attention recentlydue to its superior performance over fixed filters [1]. A contentadaptive filter first classifies local image content based ondifferent image features, such as structures and contrast. Thenin every class, a least mean square (LMS) optimal filter isemployed. The structure adaptive filter was first proposed forimage interpolation in Kondo’s method [2], where only theimage structure such as edge direction or luminance pattern isused for the classification. In applications to coding artifactreduction, additional features like block grid position [3],local contrast[4] and local variance[5], have been utilized.Incorporating more features in the classification improves theperformance of the content adaptive filters, but also leads to anexplosion of the class-count, many of which may be redundant.

For hardware implementation, some class-count reductiontechniques which allow a graceful degradation of the perfor-mance would be desirable. In this paper, we will investigatethree such techniques, which use class-occurrence frequency,coefficient similarity and error advantage, to reduce the num-ber of classes. The results show that with the techniques thenumber of classes can be greatly reduced without seriousperformance loss.

The rest of the paper is organized as follows. Section 2gives a brief introduction to the content adaptive filtering. InSection 3, three different class-count reduction techniques arepresented. The evaluation of the techniques in the applicationsof image interpolation and coding artifact reduction are shownin Section 4. Finally, in Section 5, we draw our conclusions.

II. CONTENT ADAPTIVE FILTERING

In this section, we will introduce the framework of contentadaptive filtering. The block diagram in Fig. 1 shows thatthe input pixel vector x from local image content within afilter aperture is first classified by image features such aslocal structure and contrast. A LMS-optimal linear filter isused to calculate the output pixel y with filter coefficientsfrom a look-up-table (LUT). The filter coefficients are obtainedfrom an off-line supervised training using simulated input andreference output images. A typical structure classification isAdaptive Dynamic Ranging Coding (ADRC) [2]. The 1-bitADRC code of every pixel is defined as:

ADRC(xi) ={

0, if xi <xmax+xmin

21, otherwise

(1)

where xi is the value of pixels in the filter aperture and xmax,xmin are the maximum and minimum pixel value in the filteraperture. For applications like image interpolation, structureclassification seems to be enough. But for other applicationlike coding artifacts reduction, the combination with otherclassification such as local contrast is advantageous [4][5].

Fig. 1. The block diagram of the content adaptive filtering: the local imagestructure is classified using content classification and the filter coefficients areobtained from an offline training and stored in the LUT.

III. CLASS REDUCTION TECHNIQUES

Although the framework of content adaptive filtering leadsto a simple hardware implementation, the size of the look-up-table may be significant. For example, the number of ADRCclasses increases exponentially with the pixel count in thefilter aperture. With a large number of classes, the methodmay not be efficient, i.e., there may be some redundancyin the classes. In this section, we explore three clustering

TABLE IPERCENTAGE OF MOST DOMINANTLY OCCURRING CLASSES IN A IMAGE

DATASET

Number of classes 64 128 256 512 1024 2048Percentage 78.52 87.96 93.06 96.32 98.46 99.59

techniques all capable to reduce the total number of classes,namely, class-occurrence frequency, coefficient similarity anderror advantage. These techniques all use a similar scheme asfollows. First one or more content classes will be clustered ina class-cluster. In every class-cluster an optimal linear filter isused. Every content class is assigned with a class-cluster labelto indicate to which class-cluster it belongs. We use f(·) todenote the labeling function that maps the content class k tothe class-cluster number j.

The filtering process is shown in Fig 2. First the localimage content will be classified into content classes usingdifferent features, then the label look-up-table (LUT) is usedto find out which cluster the content class belongs to. Thenthe corresponding filter is chosen for computing the output.

Fig. 2. The block diagram of the class-reduced algorithm: the input vector isfirst pre-classified using content classification, then the content class is usedto get the cluster number from the label LUT. Finally, the filter coefficientsfor the cluster is used for the filtering.

A. Class-occurrence Frequency (CF)

One way to reduce the number of classes, is to merge theclasses which are less important for the perceived image qual-ity. The importance of a class is likely reflected to how often itoccurs in an image. Therefore, we could count the occurrencefrequency of every content class and hope that it can tell howimportant the content class is for the perceived image quality.Table I shows the percentage of most dominantly occurringclasses in a set of sequences.

We would expect that if we merge the rarely occurringclasses, the overall performance will suffer little. Suppose Xk

denotes all the input vectors in content class k. We sort theinput vectors X1, X2, ..., Xk to X[1], X[2], ..., X[k] from highto low by the occurrence frequency of its content classes.The M least frequent content classes will be merged intoa cluster. The most popular classes, each will remain as aseparate cluster. The labels will be:

f(i) ={

[i], if [i] < MM, otherwise

(2)

In this reduction technique, only one cluster includes morethan one content classes, therefore, in the hardware imple-mentation, a number of comparators can be used insteadof the more expensive label LUT. Fig. 3 shows a blockdiagram of using such comparators. M−1 comparators containthe class codes C[1], C[1], ..., C[M−1] which are sorted by itsoccurrence frequency. When the input pixels are classified bythe ADRC, the class code will be compared with the mostfrequent occurring class code C[1]. If the result is yes, thenthe output pixel is calculated using the filter coefficients forC[1]. Otherwise, it will be compared with the following classcodes. The advantage is that for most of the input pixels, itwill only need the first several comparators since these classesare dominant. Therefore, the overall computation load is low.

Fig. 3. The block diagram of using comparators for class-count reduction:M − 1 comparators which contain the class codes sorted by their occurrencefrequencies are used.

B. Coefficient Similarity (CS)

Another option to reduce the class-count is to examine thesimilarity between the filter coefficients obtained from thetraining in every content class. The filter coefficients directlyshow the filtering behavior and classes with similar coefficientscan be merged. The similarity, here, is indicated by the theEuclidian distance between coefficient vectors. We propose touse the K-means algorithm to cluster the classes.

The clustering consists of the following steps:1. Specify the number of clusters M according to the

requirement and initialize the labels randomly.2. Apply iterative steps to update the mean vector µi

j inevery cluster and the labels where i is the iteration number.Calculate the mean vector:

µij =

∑f(k)i−1=j

Wk/Ni−1j (3)

where Wk is the coefficient vector from the content class kand Nj is the number of content classes that belongs to thecluster j.Update the labels with respect to the minimal distance fromthe mean vector:

f(k)i = arg minjD(Wk, µj). (4)

where D(Wk, µj) is the Euclidian distance between Wk andµj .

3. Repeat step 2 until the clustering converges. Convergence,here, means the labels do not change compared to the labelsfrom the previous iteration.

C. Error Advantage (EA)

The previous two approaches offer means to reduce thenumber of classes, however, they do not guarantee the mini-mization of total intra-cluster LMS estimation error. Therefore,a third technique is proposed to cluster the content classes withthe respect to the error advantage of cluster LMS filters. Thenthe minimal total error can be achieved given a fixed numberof clusters.

The clustering consists of the following steps:1. Build the training set. As described in the Section 2,

the vector pairs are collected from the simulated input andreference images, respectively. Sk denotes the collection of allthe vector pairs whose input vectors belong to content class k.Specify the number of clusters M according to the requirementand initialize the labels randomly.

2. Apply the EM iterations [7] to the content classes toupdate the LMS filter coefficients CW i

j of all the clusters andcluster labels f(·)i, where i is the number of iterations. Thetotal intra-cluster mean square error (MSE) will decrease afterevery iteration until the clustering converges.M-step: Obtain the cluster LMS filter coefficients CW i

j byLMS algorithm using the labels f(·)i−1.

CW ij = (

∑f(k)i−1=j

Sk,xSk,x)−1∑

f(k)i−1=j

Sk,xSk,y (5)

where Sk,x and Sk,y denote all input vectors and referencevectors from the content class k, respectively.E-step: Evaluate the regression coefficients CW i

j on everycontent class and update the labels of sub-clusters with respectto minimal MSE.

f(k)i = arg minjE[(Sk,y − CW iT

j Sk,x)2] (6)

3. Repeat step 2 until the clustering converges.Comparing to the coefficient similarity approach, the itera-

tion here involves much more calculations to evaluate all thecluster coefficients on the whole training set. Therefore, highercomputation load and more training time is expected for theerror advantage approach.

IV. RESULTS

In this section, we will evaluate the three class reductiontechniques in the application to coding artifact reduction andimage interpolation. In the experiment, we use a training setincluding about 2000 high resolution (1920 by 1080) imagesof various contents. For the evaluation, we use some testsequences shown in Fig. 4, which are not included in thetraining set. The test sequences are downgraded, as in thetraining, to generate the simulated input sequences. Then theinput sequences are processed by different methods and themean square error (MSE) between the original sequences andthe processed ones are calculated and used for the subjectiveevaluation.

(A) Bicycle (B) Lena

(C) Birds (D) Boat (E) Motor

(F) Football (G) Siena (H) TokyoFig. 4. The testing material used for the evaluation.

A. Combined coding artifact reduction and sharpness en-hancement

In the application of combined coding artifact reductionand sharpness enhancement, it is concluded [4] that ADRCclassification alone is not enough to distinguish between thecoding artifacts and the real image structures. Therefore, oneextra classification describing the contrast information in thefilter aperture is added. Here we use the same filter setting asin [4]. The filter aperture is a diamond shape consisting of 13pixels as shown in Fig. 1. An extra classification bit is used forthe local contrast classification. The total number of classes is8192.

In the experiment, the test sequences are first blurred, thencompressed using JPEG compression, as in the training in [4],to generate the simulated input. Then the content adaptivefilters with all the three class-count reduction techniquesare evaluated using these sequences. For a fair comparison,we use the same number of clusters for the three class-count reduction techniques. An attractive number for hardwareimplementation, M = 32, is chosen for the experiment. Forreference, we also include a fixed LMS filter which uses noclassification.

Table II shows the MSE comparison of the evaluated meth-ods. In terms of MSE score, one can see that three reductiontechniques can reduce the number of classes by a factor of256 with a modest increase of the MSE, compared to the MSEscore of the fixed LMS filter. Among the three techniques, EAachieves the lowest MSE score, which it is expected, as it aimsat minimizing the MSE.

TABLE IIMSE SCORES OF EVALUATED METHODS IN CODING ARTIFACT REDUCTION

AND SHARPNESS ENHANCEMENT

Mean Square ErrorSequence No reduction CF CS EA FixedClass No. 8192 32 32 32 1Bicycle 76.9 88.0 87.0 85.1 97.7Birds 15.7 17.0 16.5 16.9 19.8Boat 87.5 91.1 90.7 90.2 96.5Lena 40.4 43.0 42.6 42.3 46.0

Motor 141.6 150.2 149.1 148.0 157.5Average 72.4 77.9 77.2 76.5 83.5

Fig. 5 shows image fragments from the original sequenceBicycle, the simulated one, the processed ones by the originalmethod without class reduction and with these reductiontechniques. The three reduction techniques degrade the per-formance of the method without class reduction only little,while CS and EA show a better performance at suppressingthe ringing artifacts than CF.

B. Image interpolation

For image interpolation, we apply the three class-countreduction techniques to Kondo’s method. As a comparison, wealso choose Atkins’ method [6]. Atkins’ method is a contentadaptive filtering method which applies soft probability-basedclassification and allows a flexible number of classes. For theevaluation, we use some test sequences shown in Fig. 4, whichare not included in the training set. The test sequences first aredown-scaled two times to generate the down-scaled versionas the simulated input. Then Atkins’ method, the proposedclass-count reduction methods and original Kondo’s methodare evaluated using these sequences. For a fair comparison,we use the same number of clusters for the proposed methodand Atkins’ method and the same aperture for the proposedmethods and Kondo’s method. Since Atkins’ method performsbest at the cluster number M = 100 [6], we use the same num-ber here. Similar to the coding artifact reduction application,we also include a fixed filter for reference.

Table III shows the MSE comparison of the evaluatedmethods. In terms of MSE score, one can see that theproposed method outperforms Atkins’ method, while it is farless computationally expensive for both the classification andobtaining the filter coefficients. Comparing to the originalKondo’s method, the proposed methods only show a modestincrease at MSE score.

Fig. 6 shows image fragments from the original highresolution sequence Bicycle and processed ones by all themethods. All the three proposed methods render the lines indifferent directions correctly where Atkins’ method producessome staircase artifacts. Among them, EA and CS produceslightly smoother results at reconstructing the lines than CF.They show more or less the same interpolation quality asthe original Kondo’s method, though the LUT size had beenreduced nearly by a factor of 40 and only one extra fetchoperation is needed. The quality difference between CF and

TABLE IIIMSE SCORES OF EVALUATED METHODS IN IMAGE INTERPOLATION

Mean Square ErrorSequence No reduction CF CS EA Fixed Atkin’sClass No. 4096 100 100 100 1 100Bicycle 41.4 47.0 45.6 45.2 65.4 49.8Birds 20.3 20.4 20.3 20.2 22.4 22.1Boat 55.1 57.7 56.9 56.9 63.2 59.3Lena 78.1 82.6 81.5 81.4 90.2 83.1

Motor 52.4 53.1 52.8 52.7 53.1 52.7Average 49.4 52.2 51.4 51.2 58.9 53.4

EA, CS is rather small. Therefore CF is more suitable forhardware implementation where the cost is more important.

V. CONCLUSION

In this paper, we have proposed three class-count reductiontechniques, class-occurrence frequency, coefficient similarityand error advantage for the content adaptive filtering frame-work. In the applications of coding artifact reduction andimage interpolation, it has been shown that these techniquescan greatly reduce the number of content classes withoutsacrificing much performance and are promising for contentadaptive filter with a large number of features. Among them,the coefficient similarity and error advantage approach producethe best result. Taking the cost into consideration, the class-occurrence frequency approach seems to be the best choicefor implementation.

REFERENCES

[1] M. Zhao, M. Bosma, and G. de Haan,“Making the best of legacy videoon modern displays”, Journal of the Society for Information Display,Vol. 15, Issue 1, pp. 49-60, Jan. 2007.

[2] T. Kondo and Y. Fujimori, S. Ghosal, J. J. Carrig, “Method and apparatusfor adaptive filter tap selection according to a class”, US-Patent: US6,192,161 B1, Feb. 20, 2001.

[3] M. Zhao, R. E. J. Kneepkens, P. M. Hofman and G. de Haan, “Contentadaptive image de-blocking”, IEEE International Symposium on Con-sumer Electronics, pp. 299-304, Sept. 2004.

[4] H. Hu and G. de Haan, “Simultaneous coding artifacts reduction andsharpness enhancement”, Digest of IEEE International Conference onConsumer Electronics, Las Vegas, USA, Jan. 2007, pp. 213-214.

[5] L. Shao, H. Hu and G. de Haan, “Coding Artifacts Robust ResolutionUp-conversion”, IEEE International Conference on Image Processing2007, Vol. 5, pp. 409-412, Sept. 2007

[6] C. B. Atkins, C. A. Bouman, and J. P. Allebach, “Optimal imagescaling using pixel classification”, International Conference on ImageProcessing, Vol. 3, pp. 864-867, Oct. 2001.

[7] A. Dempster, N. Laird, and D. Rubin.“Maximum likelihood fromincomplete data via the EM algorithm”. Journal of the Royal StatisticalSociety, Series B, Vol. 39, pp. 1-38, 1977.

[8] J. A. Hartigan and M. A. Wong, “A K-means clustering algorithm”,Applied Statistics, Vol. 28, No. 1, pp.100-108, 1979.

(A) Original (B) Simulated input

(C) Without reduction (D) Using CF

(E) Using CS (F) Using EA

(G) Fixed filterFig. 5. Image fragments the coding artifact reduction results on the Bicyclesequence: (A) Original, (B) Simulated input, (C) Without reduction, (D)Using occurrence frequency, (E) Using coefficient similarity, (F) Using erroradvantage, (G) Fixed filter

(A) Original (B) Simulated input

(C) Without reduction (D) Using CF

(E) Using CS (F) Using EA

(G) Atkins’ method (H) Fixed filterFig. 6. Image fragments from the image interpolation results on the Bicyclesequence: (A) Original, (B) Simulated input, (C)Without reduction - Kondo’smethod, (D) Using occurrence frequency, (E) Using coefficient similarity, (F)Using error advantage, (G) Atkins’ method, (H) Fixed filter.