HEVC-based Adaptive Quantization for ScreenContent by Detecting Low Contrast Edge Regions

Hong Zhang, Oscar C. Au, Yongfang Shi, Xingyu Zhang, Ketan Tang, Yuanfang GuoDepartment of Electronic and Computer EngineeringHong Kong University of Science and Technology

Clear Water Bay, Kowloon, Hong KongEmail: {hzhangad, eeau, yshiab, eexyzhang, tkt, eeandyguo}@ust.hk

Abstract— High-Efficiency Video Coding (HEVC) is the newestvideo coding standard which can significantly reduce the bit rateby 50% compared with existing standards. The key features andnew tools in HEVC are designed for natural video sequencescaptured by a real camera. Different from natural videos, screencontent contain much more edges in text and icon regions. Thecurrent video coding standards may blur or even remove low con-trast edges, which are very important in screen content for humaneyes to recognize the character and the icon. Therefore, this paperproposes an effective modification on HEVC to preserve the lowcontrast edges in screen content. First, discrete laplacian filter isadopted for edge detection, and then we adaptively adjust QPs forlow contrast edge regions, which can be detected based on ourdesigned measurement for edge contrast. Experimental resultsshow that nearly all the regions containing low contrast edgescan be detected, and the adjustment of QPs for these regions cangreatly protect the edges with no RD performance reduction.

I. INTRODUCTION

Recently, the next generation video coding standard HEVCis being established by ITU-T VCEG and ISO/ICE MPEGorganizations. The main goal of HEVC [1] is to significantlyimprove compression performance, until now it has beendesigned to achieve equal perceptual video quality using onlyon average 50% bit rate relative to the performance of previousstandards. Based on conventional hybrid coding framework,nearly all the advanced technology in HEVC are developed fornatural video content and do not treat artificial screen contentspecially, which will lead to some undesired performance forscreen content.

Screen content refer to images or videos artificially gen-erated by computers, such as word files, PPT slides, andcomputer games, which are widely used in various applicationsincluding remote desktop, game video, etc. Different fromnatural content, screen content usually contain much more textand icon regions with lots of edges. For human being, theseedges are very necessary for recognizing and identifying thetext character and icon in screen content. In contrast, naturalvideo content are smooth, and edges in these content will notgreatly influence visual perception. Conventional lossy imageor video coding standards will reduce high frequencies, andthese edges will be blurred. This performance is acceptablefor natural content but not for screen content. Therefore, itbecomes important to evaluate novel techniques aiming atscreen content coding.

Some schemes have been presented for coding compoundimage which is the combination of text and natural imageregions. Considering different texture properties of text andnatural regions, researchers propose to do segment on com-pound image and then treat these two kinds regions separately.In [2] [3], a layer based approach, named mixed raster content(MRC) image model is adopted. They proposed to decomposecompound image into different layers. This model requiresaccurate segmentation for text and natural image parts, andproper filling algorithms for fore- and back- ground layers.People also consider to categorize blocks in compound imageinto different types depending on their spatial features [4][5]. In [6] [7], they analyze the properties of different blocksfurther and develop another two modes RSQ and BICM forH.264/AVC. RSQ mode utilizes the property that signals inspatial domain is sparser for text region than in transformdomain, and directly applies quantization to residues withouttransform. In BCIM mode, blocks are represented by basecolors and index map, since colors in text blocks are muchfewer than in natural image blocks. The methods mentionedabove may be effective for screen content, however, if theyare inserted into HEVC much more extra complexity forencoder will be introduced. Besides, the encoded bit stream isincompatible with existing video standards.

In this paper, we take advantage of the state-of-the-artvideo standard HEVC and propose a non-normative adaptivequantization method for screen content. Since edges are veryimportant in screen content, but treated with less attention inHEVC, we attempt to preserve these edges, especially for lowcontrast edges as they will be removed by a great chance. Ourproposed method is, first, the laplacian filer is applied to screencontent frame to detect edges. Second, we propose a simplemeasurement for edge contrast. Based on this measurement,low contrast edges blocks for certain QP are detected, andthen adaptive QPs are computed for these blocks to improvethe reconstructed visual performance.

The rest of the paper is organized as follows: In SectionII, some key techniques related to our method in HEVC areintroduced. Section III presents the proposed adaptive quanti-zation method, including measure of edge contrast, detectionfor low contrast edge blocks and adaptive QPs selection forlow contrast edge blocks. The Experimental results and furtherdiscussions are provided in Section IV. At last, a conclusion

978-1-4673-5762-3/13/$31.00 ©2013 IEEE 49

����������� ���

�

��������� ������������ �����������

�

�����

�����������

�����

���������

��

��������

�������� �!"!#$%���������

$���

&������

'�������

����

'�������

������

'�������

Fig. 1. Hybrid Video Encoder

is draw in Section V.

II. HEVC TECHNIQUES

Our proposed method is applied to I slices, so advancedtechniques for I slices are introduced. HEVC standard adoptsthe well-known block based hybrid coding scheme, as depict-ed in Fig.1. Similar to conventional video coding schemes,the coding performance relies on intra and inter prediction,transform, quantization and effective entropy coder. Besides,block structure in HEVC is more flexible. It defines threenew kinds of units, coding unit (CU), prediction unit (PU)and transform unit (TU), and employs quad-tree structures fordifferent unit partitions. CU is the basic intra/inter coding unit,with size from 8×8 to 64×64 for luma component, and canbe recursively split into smaller CU size. PU is the basic unitused for prediction mode selection. For intra prediction, PUis the same size as CU except for the smallest CU size wherePU can be split further. The root of PU quad-tree is in CU.TU is basic unit for transform and quantization. The size ofTU can be from 4×4 to 32×32. Fig. 2 illustrates a possiblepartition of CU and TU.

In addition, HEVC employs more modes for intra predic-tion. Compared to 8 directional intra prediction modes ofH.264/AVC, there are 35 modes defined for PU, including DCprediction, planar prediction and 33 directional orientations, asillustrated in Fig. 3. For chroma prediction block, mode DC,planar, vertical, horizontal and mode of luma are candidatemodes. The new added modes provide lots of directionsfor prediction which can improve the accuracy of predictiondirection and generate rather good prediction result for thecurrent block.

Combined with the use of flexible unit partition structureand various block sizes, HEVC can deal with most unsmoothregions of screen content by trying small block size andmany directions to do prediction. However, the edges stillcan be blurred to some extent for lossy coding. From ourexperiments, we observe that the reconstructed performanceof strong contrast edges in screen content is acceptable, whilewith the same configuration, the results of the low contrastedges were severely blurred. Therefore, a modification ofHEVC becomes necessary to preserve the low contrast edgesregions for screen content coding.

III. PROPOSED METHOD

In this paper, we propose to adaptively adjust QP for CUcontaining low contrast edges. Edge regions should be firstly

Fig. 2. Partition of CU and TUblock. Dash line: CU boundary.Dotted line: TU boundary

�

����������

���������

��������

�������

��������

���������

���������

�������

������������������

�������������������

������������������

��

������������������������������������������������������������������������������������������������������������������������������������

���� �����������

����� �������

Fig. 3. Modes and directionsfor intra-prediction

detected from original picture, and then a measurement ofedge contrast is defined. Based on this measurement, regionscontain low contrast edges are detected. At last, adaptivequantization is applied to these regions.

A. Edge Detection

There are a lot of ways to perform edge detection, andmost of them can be categorized into gradient based methodand zero-crossing based method. Gradient methods usuallycompute the first derivatives Gx of image in x-directionand Gy in y-direction, and search the position of maximumgradient G =

√(G2

x +G2y). The most commonly used kernels

are Sobel, Robert and Prewitt operators. Using operator withsecond derivative of signals for edge detection is called zero-crossing method, and the well-known operator is laplacianfilter. The laplacian has the advantage that it is an isotropicmeasure, as the magnitude of the 2nd derivative is computed inall orientations. In addition, zero-crossing method can detectany positions where intensity gradient tents to increase ordecrease, including very low contrast gradient changes. Sincethere is little noise in computer generated contents, we cantake use of laplacian filter to detect all edges for screen

contents. We apply the laplacian operator H=

1 1 11 -8 11 1 1

to

the original picture, and obtain the edge mask MH , indicatingthe positions of edges.

B. Edge Contrast Measure

For conventional hybrid video coding scheme, quantizationis applied to the transformed residues and is the main causeof distortion. We hope the contrast measure is directly relatedto the quantization step, which can help build the relationbetween the quality of reconstructed block and edge contrast.So, edge contrast measurement C is defined in the frequencydomain.

Let F denotes the original picture and XN×N denotes anN×N block in the detected edge picture E, where E is equalto F.∗MH . Based on HEVC, here DCT is adopted to transformXN×N to frequency domain, and the resulting DCT coeffi-cients are di,j , (i, j = 0, 1, ...., N − 1). Each coefficient di,jdescribes the contribution of the corresponding basis function[8], which is arranged in an increasing order of frequency from

50

��

��

����

�����

�

Fig. 4. Classified bands for DCT transform coefficients

left-up to right-bottom. d0,0 is DC coefficient and other di,jare AC coefficients. The spatial intensity changes are properlydescribed by these coefficients in frequency domain, whichprovides us a natural way to define contrast measurement Cin DCT domain.

Transformed coefficients di,j , (i, j = 0, 1, ...., N − 1) arefirst classified into 2N -1 frequency bands in a diagonal scan[9]. Fig. 4 illustrates the structure of frequency band, eachfrequency band Bt(t = 0, 1, 2, ....2N−2) contains coefficientsdi,j , where i+j = t. This is a multiscale structure for differentfrequency levels, as each band contains the coefficients of basisfunctions under the same frequency level. Our edge contrastmeasurement C is designed based on Bt, (t = 1, 2, ....2N−2),except B0 band.

C =

2N−2∑t=1

wtct (1)

where wt is the weight for each band Bt and ct is

ct =∑

i+j=t

|di,j |, t = 1, 2, ...2N − 2 (2)

which is the sum of coefficients in the t-th band. Edge contrastmeasurement C is the weighted combination of every ACband contrast measurement ct, and weight wt can be used toaddress interested frequency band. In screen content, we treatall AC bands equally as most text and icon edges are complex,so wt = 1/(2N − 2) .

C. Low Contrast Edge Region Detection

Our goal is to improve the performance for low contrastedge regions. It is because that edges are very importantfor screen content, and HEVC encoder will blur them at thequantization step. Usually, the transform coefficients of pictureare smaller for high frequency than for low frequency, afterquantization is applied to the transformed coefficients, someAC coefficients will be quantized to zero, which cannot berestored and will cause distortion. Therefore, we expect tosearch these regions that the edge contrast C drops greatlyafter quantization, which means most AC coefficients havelarge distortion and the reconstructed frame is of low quality.Since the quad-tree partitioning structure is complex and timeconsuming, the detection for low contrast edge regions will notbe conducted in every CU partition. Instead, the detection isimplemented as a preprocess using a certain block size beforecurrent frame is compressed.

First, two-dimensional DCT is applied to block XN×N , andedge contrast measurement C is computed using equation(1),(2). Based on the input Qstep, a simple rounding quantizationis applied to transformed coefficients

qi,j = round(di,j

Qstep), i, j = 0, 1, 2....N − 1 (3)

where qi,j is the quantized coefficients. We would like toknow the performance of reconstructed result, so inversequantization is applied,

d′i,j = qi,j ·Qstep, i, j = 0, 1, 2....N − 1 (4)

d′i,j is the reconstructed frequency coefficient. Now, we canget edge contrast measurement C ′ using d′i,j for reconstructedblock. If the edge block XN×N is seriously blurred afterquantization, the contrast measurement C ′ of the reconstructedblock will be much smaller than C. It is because most ofAC coefficients are small relative to current Qstep, and arequantized to zeros which result the reconstructed d′i,j arealso zeros. We want to detect this kind block, so a contrastreduction measurement Re is defined

Re = 1− C ′

C(5)

when Re is larger than a threshold T (T is set 1/3 in ourexperiments), we regard this block LN×N as low contrastregion under current QP.

D. Adaptive Quantization

This is a pre-process step, we adjust QP for the detected lowcontrast edge block LN×N adaptively. For simplicity, we williteratively reduce QP by 6, as Qstep will reduce by half whenQP decreases by 6. In each iteration, the contrast measure C ′

and contrast reduction measure Re will be recomputed. Theiteration terminates when the measure Re is larger than thethreshold T . After all the low contrast blocks are assignedproper QPs, a QP map is build to indicate the QP value foreach block in current picture. This map will be used in theencoder for adaptive QP selection and CU partition. It meansif one CU contains different QPs, this CU will be split furtherwithout compression until the spitted CU covers same QP.With the selected QP for the CU, the following steps are sameas HEVC, continue divide current CU if possible, and choosethe best partition based on R-D cost.

IV. EXPERIMENTAL RESULTS

We evaluate the performance of proposed method on thereference software HM7.0 [10] of HEVC. Recommendedintra-main test conditions by JCT-VC are used, and deblockingfiler is disabled as it will blur the decoded screen content.As shown in Fig. 5, two screen content pictures from Fclass test sequences in HEVC are used as test frames. In ourexperiments, the block size for detecting low contrast edgeregions is set 16 × 16 on Y component, and four base QPs(22, 27, 32, 37) are selected to test our method. We computeadaptive QP for each low contrast edge block according to thebase QP.

51

The subjective results are shown in Fig. 6, limited by thelength of paper, only part of the result from SlideEditing ispresented. Non-black regions in (c) represent detected lowcontrast edge regions in Y component. Compared with theoriginal picture in (a), we can see that most low contrastedge regions are detected. (b) is the decoded result usingHM7.0 when QP is 37. Look at the regions in red rectangles,edges of icons are seriously blurred that makes it difficult torecognize these icons, while other decoded regions are accept-able. Implement the proposed method in HM7.0, the decodedperformance of low contrast edge regions are improved greatlydue to the adaptive quantization, and result is shown in (d).Edges of icons in red rectangles become more clear, andpeople can identify these icons easily. The visual quality isimproved significantly.

One concern of this method is that R-D performance maydecrease. The objective results, bit rate versus PSNR curvesof test frames, are depicted in Fig.7. The curves show thatthe RD performance of our proposed method is almost thesame as HEVC (HM7.0), no reduction. Therefore, we can usesame bits to encode screen content but obtain better visualperformance than HEVC.

(a) (b)

Fig. 5. Test pictures. (a) SlideEditing 1280x720, (b) SlideShow 1280x720

(a)

(b)

(c)

(d)

Fig. 6. Example Performance of SlideEditing 1280x720, QP is 37 (a)Original, (b) Reconstructed result of HM7.0, (c) Lowcontrast Regions forY component, (d) Reconstructed result of proposed method

0.4 0.5 0.6 0.7 0.8 0.9 130

32

34

36

38

40

42

44

46

48

bpp

dB

SlideEditing

Proposed

HM7.0

(a)

0.2 0.25 0.3 0.35 0.4 0.45 0.534

36

38

40

42

44

46

48

bpp

dB

SlideShow

Proposed

HM7.0

(b)

Fig. 7. Experimental results of rate distortion cost

V. CONCLUSION

In this paper, we propose an adaptive quantization methodto improve the compression performance of low contrast edgeregions in screen content. Before compression, a preprocess isconducted to detect low contrast edge regions and build a QPmap for adaptive quantization. From the experimental results,we can see that nearly all the low contrast edge regions can bedetected under certain QP. After the adaptive quantization isapplied to these regions, the decoded subjective performanceis good enough for people to recognize these low contrasttext and the icons. Compared with the results of HEVC,our proposed method achieves almost same objective RDresults as HEVC, however, can greatly improve the subjectivecompression performance for low contrast regions, which arevery important for screen content.

ACKNOWLEDGEMENT

This work has been supported in part by the HKUST(HKUST Project no. FSGRF12EG01).

REFERENCES

[1] G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview ofthe high efficiency video coding (HEVC) standard,” to apperar in IEEETrans. on Circuuts and Systems for Video Technology, Dec 2012, pre-publication draft.

[2] L. Bottou, P. Haffner, P. G. Howard, P. Simard, Y. Bengio, and Y. LeCun,“High quality document image compression with ”djvu”,” Journal ofElectronic Imaging, vol. 7, no. 3, pp. 410–425, 1998.

[3] R. de Queiroz, R. Buckley, and M. Xu, “Mixed raster content (MRC)model for compound image compression,” Proceedings of SPIE - TheInternational Society for Optical Engineering, vol. 3653, no. II, pp.1106–1117, 1999.

[4] A. Said and A. Drukarev, “Simplified segmentation for compound imagecompression,” in Image Processing, 1999. ICIP 99. Proceedings. 1999International Conference on, vol. 1, 1999, pp. 229 –233 vol.1.

[5] T. Lin and P. Hao, “Compound image compression for real-time com-puter screen image transmission,” Image Processing, IEEE Transactionson, vol. 14, no. 8, pp. 993 –1005, aug. 2005.

[6] C. Lan, F. Wu, and G. Shi, “Compress compound images inH.264/MPEG-4 AVC by fully exploiting spatial correlation,” in Circuitsand Systems, 2009. ISCAS 2009. IEEE International Symposium on, may2009, pp. 2818 –2821.

[7] C. Lan, G. Shi, and F. Wu, “Compress compound images inH.264/MPGE-4 AVC by exploiting spatial correlation,” Image Process-ing, IEEE Transactions on, vol. 19, no. 4, pp. 946 –957, april 2010.

[8] G. Strang, “The discrete cosine transform,” SIAM Review, vol. 41, no. 1,pp. 135–147, 1999.

[9] J. Tang, E. Peli, and S. Acton, “Image enhancement using a contrastmeasure in the compressed domain,” Signal Processing Letters, IEEE,vol. 10, no. 10, pp. 289 –292, oct. 2003.

[10] https://hevc.hhi.fraunhofer.de/svn/svn HEVCSoftware/tags/HM-7.0rc2/,Refrence Software HM7.0 of HEVC 2012 [Online].

52