Visual Processing Driven by Perceptual Quality Gauge:

A Perspective

Weisi Lin, Zhongkang Lu, Susanto Rahardja, Weisi Lin, Zhongkang Lu, Susanto Rahardja, EePingEePing Ong and Susu YaoOng and Susu Yao

Media Processing Department Institute for Infocomm Research, Singapore

Outline of PresentationOutline of Presentation

• Review on – perceptual visual quality gauges– perceptual image/video processing

• Some of our recent research attempts– visual quality evaluation– perceptual signal maniputions

• Concluding remarks

Facts about Visual Quality Evaluation

as a standalone metric:•image evaluation•algorithm benchmarking

as an embedded module:shaping algorithms/systems

The HVS: ultimate appreciator of most images

PSNR/MSE/MAE:not matching the HVS perception

Perceptual metrics so far:•much research interest

(VQEG, IEEE G-2.1.6, many others)•a difficult odyssey•existence of general solution?

Different factors for perceptual metric building

sensory perceptual emotional domain-specific

PSNR/SNR/MSE/MAE

perceptual metrics

application-specific perceptual metrics

performance, difficulty, complexity

application scopes

A Glimpse on

Different Metrics

medical

mobile comm

HDTV

domain knowledge to be used; piece-wise formulation for different quality ranges; PSNR largely irrelevant for mobile comm.;

SDTVapplication

H.264

H.261/3

MPEG 4

MPEG 1,2

JPEG 2000

knowledge of artifacts to be incorporated JPEGcodec

PSNR not applicable; wider scopesno-reference

feature selectionreduced-reference

more info availablefull-referencereference

relatively efficientbottom-up

general, complextop-downmethodology

third-party

second-party

most interested: third-party onesfirst-partyviewer

computer graphics

Majority of work on natural picturesnatural scenesource

overall qualityquality

for single, multiple or overall distortiondistortionoutput

new area3-D views

simple temporal effect; pooling over frames; further modeling needed

video

relatively well explored; color difference to be probed further

imageinput signal

RemarksMetric typeCriterion

A closer look…Top-down Metrics

single channel approach —CSF filtering

Mannos & Sakrison’74Fauger’79Lukas & Buddrikis’82HeegerLambrecht’99

multi-channel decompositionDaly’93, Lubin 95Lambrecht’96Winkler’99Watson’01

Bottom-up Metricslumonance/color difference

Miyahara’98Zhang & Wandell’98

sharpnessCaviedes & Gurbus’02Winkler’01Dijk, et al.’03

common coding artifactsWu & Yuen’97Yu, et al.’02Marziliano, et al.’02Tan & Ghanbari’00Mylene’03Caviedes & Oberti’03

other featuresSuresh & Jayant’05Lu, et al.’05

Hybrid (top-down & bottom-up) metrics

Yu, et al.’02Ong, et al.’04Tan & Ghanbari’00

No-reference MetricsWu & Yuen’97Caviedes & Gurbus’02 Marziliano, et al.’02Caviedes & Oberti’03Dijk, et al.’03

Full-reference MetricsDaly’93Lubin 95Lambrecht’96Miyahara, et al.’98Zhang & Wandell’98Wang, et al.’99, 04Tan & Ghanbari’00Winkler’99Watson’01Yu, et al.’02Lin, et al.’05

Reduced-reference MetricsWolf’97Horita, et al.’03

Perception-driven Visual Processing

watermarkingWolfgang, et al.‘99

self-embedment for error correctionunequal error protection

Jiang, et al.‘99joint source-channel coding

visual communication

demoaicingLongere, et al.‘02

synthesisRamasubramanian, et al.‘99

super-resolution formationpost-processing

Yao, et al.’05edge-enhancement

Lin, et al.’05

enhancement/reconstruction

quantizer and rate controlWatson’93Hontsch & Karam’00,02Yang, et al.‘05

foveation-based codingWang & Bovik’01Wang, et al.’03Itti’04

motion searchMalo, et al.’01Yang, et al.‘03

inter-frame replenishmentChiu & Berger’99

filtering of residues/coefficients

Safranek’94Yang, et al.‘05

scalabilityWang, et al.’03Lu, et al.‘05

closed-lopp controlTan, et al.’04

image/video compression

Just-noticeable Difference (JND)

• JND: the visibility threshold below which any change cannot be detected by the HVS (Jayant, et al.’93)

• differentiation in quality evaluation– near-JND– supra-JND

• 2 JNDs, 3 JNDs, …can be also determined

DCT subbandsAhumada & Peterson’92, Watson’93, Hontsch & Karam’00,02, Zhang, et al.’05

wavelet subbandsWatson, et al.’93

pyramid subbandsRamasubramanian, et al.’99

pixel domainChou & Li’95, Chiu &Berger’99, Yang, et al.’03

contrast maskingTong&Venetsanopoulos’98, Zhang, et al.’05

temporal effecteye motion: Daly’98

frame difference: Chou & Chen’96

temporal CSF:for subbands-- Daly’98, Watson, et al.’01for pixel-- Zhang’04

Visual-attention modulationLu, et al.‘05

Visual Quality Gauge• new ideas:

noticeable edge contrast increase--enhancementnoticeable edge contrast decrease--the worst degradation noticeable non-edge contrast decrease--degradationnoticeable non-edge contrast increase—degradation

where

• D reduces to the mean absolute error (MAE) measure, if– JND is not considered– different contrast changes are not differentiated

3α > ),max( 21 αα > 4α >0

eenene ccccD +−+− −++= 4321 αααα

Recent research attempts

“…to tell a good picture from a good one…”

Better quality than the original image

(our method)(Longere, et al.’02)

Recent research attempts

Pearson & Spearman correlations

Tests with VQEG-I Data

(P0,1,3,5,8: the five best VQEG-I proponents)95% CI

std for all 9 test groups

the new metric:• outperforms the relevant existing metrics with both databases:

VQEG-I (compressed video)Longere, et al.’02 (demosaiced images)

• has small variation in performance under different test conditions

Recent research attempts

1-D illustration…

MC Residue (x10-1)

Pixel

original signal

modification of signal: for better compression

MC Residue (x10-1)

Pixel

Reasonable modification: the mean in the neighborhood, B

Simplest butmeaningless modification

Problem: noticeable distortion introduced

Recent research attempts

Perceptual Signal Modification

making the distortion unnoticeable

MC Residue (x10-1)

Pixel

JND range

Noticeable distortion

MC Residue (x10-1)

Recent research attempts

Perceptual Quality Significance Map (PQSM)

The HVS:•not with a ideal sensor •with limited source

-processing power-internal memory

•as a result of the evolution =>visual attention

hierarchical PQSM (full to rough)

PQSM generation

integration of multiple stimuli:

Recent research attempts

in line with eye tracking results

Applications of Perceptual Significance Map• JND models• quality metrics• ROI-based compression• scalable coding• other visual processing, for

resource savings/allocationbandwidth, computing power, memory space, display/printing resolution

and/orperformance enhancement

picture quality

YCb Cr

JND Modulated JND

Recent research attempts

Concluding Remarks• interesting areas for further work

– modeling more temporal effectsmotion, jerkiness, mean time between errors, etc.

– more effective accounting for chrominance effectsesp. for non-coding distortion

– joint modeling with other mediaaudio, text, and so on

– no-reference situationsPSNR not applicable; wider scope of application

– mobile comm applicationsPSNR largely irrelevant

– codec dependent metricse.g. targeting H.264 artifacts

– ROI-based scalable coding• ROI coding• scalability • SVC standardization

– adaptive watermarking• authentication• error resilience

• significant progress– perceptual quality gauges

• various types of metrics– perceptual image/video

processing• compression• other related areas

References

[1] S. Daly, “The visible differences predictor: an algorithm for the assessment of image fidelity”, Digital Images and Human Vision (A.B. Watson, ed.), pp.179-206, The MIT Press, 1993.

[2] J. Lubin, “A visual discrimination model for imaging system design and evaluation”, Vision Models for Target Detection and Recognition (E. Peli, ed.), pp.245-283, World Scientific, 1995.

[3] S. Winkler, “A perceptual distortion metric for digital color video”, Proc. SPIE Human Vision and Electronic Imaging IV, Vol. 3644, B.E. Rogowitz and T.N. Pappas eds., pp. 175 – 184, Bellingham, WA, 1999.

[4] VQEG (Video Quality Expert Group), “Final Report from the Video Quality Expert Group on the validation of Objective Models of Video Quality Assessment”, www.vqeg.org, 2000.

[5] VQEG (Video Quality Expert Group), “Final Report from the Video Quality Expert Group on the validation of Objective Models of Video Quality Assessment, Phase II”, www.vqeg.org, 2003.

[6] A.M. Eskicioglu and P.S. Fisher, “Image quality measures and their performance”, IEEE Trans. Communications, Vol. 43(12), pp.2959-2965, Dec. 1995.

[7] H.R. Wu and M. Yuen, “A generalize block-edge impairment metric for video coding,” IEEE Sig. Proc. Lett., Vol. 4(11), pp.317-320, 1997.

[8] P. Marziliano, F. Dufaux, S. Winkler and T. Ebrahimi, “a no-reference perceptual blur metric”, Proc. IEEE Int’l Conf. Ima. Proc .(ICIP), 2002.

[9] S. Wolf, “Measuring the end-to-end performance of digital video systems”, IEEE Transactionson Broadcasting, vol.43(3), pp. 320-328, 1997.

[10] M. Miyahara, K. Kotani, K., and V.R. Algazi, “Objective picture quality scale (PQS) for image coding”, IEEE Trans. Communications, Vol. 46(9), pp.1215-1225, 1998.

[11] X. Zhang and B.A. Wandell, “Color image fidelity metrics evaluated using image distortion maps”, Signal Processing, Vol. 70 (3), pp.201-214, 1998.

[12] K.T. Tan and M. Ghanbari, “A multi-metric objective picture-quality measurement model for MPEG video”, IEEE Trans. Circuits Syst. Video Technol., Vol. 10, No. 7, Oct. 2000, pp. 1208-1213.

[13] S. Wolf and M. Pinson, “Video quality measurement techniques”, NTIA Report 02-392, June 2002.[14] E. Ong, W. Lin, Z. Lu, S. Yao and M. Etoh, “Visual Distortion Assessment with Emphasis on Spatially Transitional

Regions”, IEEE Trans. Circuits and Systems for Video Technology, Vol. 14(4), PP.559 – 566, April 2004.[15] Z. Yu, H.R. Wu, S. Winkler, and T. Chen, “Vision-model-based impairment metric to evaluate blocking artifacts in

digital video”, Proc. IEEE, Vol. 90(1), pp. 154-169, 2002.[16] Z. Wang, L. Lu and A.C. Bovik, “Foveation scalable video coding with automatic fixation selection”, IEEE

Transactions on Image Processing, Vol. 12(2), pp.243 - 254, Feb. 2003.[17] Z. Lu, W. Lin, X. Yang, E. Ong and S. Yao, “Modeling Visual Attention's Modulatory Aftereffects on Visual

Sensitivity and Quality Evaluation”, IEEE Trans. Image Processing, Vol.14(11), pp.1928 – 1942, Nov. 2005.[18] A. B. Watson, ``Proposal: Measurement of a JND Scale for Video Quality", prepared for the IEEE G-2.1.6

Subcommittee on Video Compression Measurements meeting, August 7th, 2000.

[19] ITU-R Recommendation 500-10, ``Methodology for the Subjective Assessment of the Quality of Television Pictures," ITU, Geneva, Switzerland, 2000.

[20] Sarnoff Corporation, ``Sarnoff JND vision model", J. Lubin (Ed.), Contribution to IEEE G-2.1.6 Compression and Processing Subcommittee, Aug., 1997.

[21] P. Longere and X. Zhang and P. B. Delahunt and D. H. Brainaro, “Perceptual Assessment of Demosaicing Algorithm Performance”, Proc. IEEE, vol.90, no.7, pp.123-132, Jan, 2002.

[22] D.M. Tan, H. R. Wu and Z. H. Yu, “Perceptual coding of digital monochrome images”, IEEE Signal Processing Letters, Vol. 11( 2), pp.239 – 242, Feb. 2004.

[23] B. Watson, “DCTune: A technique for visual optimization of DCT quantization matrices for individual images”, Society for Information Display Digest of Technical Papers XXIV, pp. 946-949, 1993.

[24] I. Hontsch, and L. J. Karam, “Adaptive image coding with perceptual distortion control”, in IEEE Trans. on Image Processing, vol. 11, No. 3, pp. 213-222, 2002.

[25] C.-H. Chou and Y.-C. Li, “A perceptually optimized 3-D subband codec for video communication over wireless channels,” in IEEE Trans. Circuits Syst. Video Technol., vol.6, no.2, pp. 143- 156, 1996.

[26] J. Malo, J. Gutierrez, I. Epifanio, F.J. Ferri and J. M. Artigas, ``Percetual feedback in multigrid motion estimation using an improved DCT quantization", IEEE Trans. Image Processing, vol. 10, No. 10, pp. 1411-1427, October, 2001.

[27] X.K. Yang, W. Lin, Z.K. Lu, E.P. Ong and S.S.Yao, ``Perceptually-adaptive Hybrid Video Encoding Based On Just-noticeable-distortion Profile", SPIE 2003 Conference on Video Communications and Image Processing (VCIP), Vol.5150, pp.1448-1459, 2003.

[28] X. Yang, W. Lin, Z. Lu, X. Lin, S. Rahardja, E. Ong and S. Yao, “Rate Control for videophone using perceptual sensitivity cues”, IEEE Trans. Circuits and Systems for Video Technology, vol 15(4), pp.496-507, April, 2005.

[29] Y. J. Chiu and T. Berger, ``A Software-only Videocodec Using Pixelwise Conditional Differential Replenishment and Perceptual Enhancement", IEEE Trans. Circuits Syst. Video Technol., vol. 9, No. 3, pp. 438-450, April, 1999.

[30] R. J. Safranek, ``A JPEG compliant encoder utilizing perceptually based quantization", Proc. SPIE Human Vision, Visual Proc., and Digital Display V, Vol. 2179, pp. 117-126, Feb. 1994.

[31] R. B. Wolfgang, C. I. Podilchuk, and E. J. Delp, ``Perceptual Watermarks for Digital Images and Video", Proc IEEE, 87( 7), pp.1108-1126, July 1999.

[32] A. E. Savakis, S. P. Etz and A. C. Loui, “Evaluation of image appeal in consumer photograph”, Proc. SPIE, Human Vision and Electronic Imaging V, vol. 3959, pp. 111-120, 2000.

[33] H.R. Wu, Z. Yu and B. Qiu, “Multiple reference impairment scale subjective assessment method for digital video”, International Conference on Digital Signal Processing (DSP2002), pp. 185-189, July 2002.

[34] M. Tapiovaara, “Objective measurement of image quality in fluoroscopic X-ray equipment: FluoroQuality”, STUK-A196, May 2003, http://www.stuk.fi/julkaisut/stuk-a/stuk-a196.pdf

[35] W. Lin, L. Dong and P.Xue, “Visual Distortion Gauge Based on Discrimination of Noticeable Contrast Changes ”, IEEE Trans. Circuits and Systems for Video Technology, vol.15(7), pp. 900- 909, July, 2005.

[36] J. Caviedes and S. Gurbuz, “No-reference sharpness metric based on local edge kurtosis”, Proc IEEE Int’l Conf. Ima. Proc .(ICIP), vol. 3, pp. 53-56, 2002.

[37] E. Ong, X. Yang, W. Lin, Z. Lu, S. Yao, X. Lin, S. Rahardja and C. Boon, “Perceptual Quality and Objective Quality Measurements of Compressed Videos”, Journal of Visual Communication and Image Representation, accepted, 2005.

[38] Z. Lu, W. Lin, Z. Li, K. P. Lim, X. Lin, S. Rahardja, E. Ong and S. Yao, “Perceptual Region-of-interest (ROI) based Scalable Video Coding”, ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6, 15th Meeting, JVT-O056, Bushan, Korea, April, 2005.[39] M. P. Eckert and A. P. Bradley, “Perceptual quality metrics applied to still image compression”, Signal Processing, Vol. 70, 1998, pp.177-200.

[40] L. M. J. Meesters, W. A. Ijsselsteijn and P. J. H. Seuntiens, “A survey of perceptual evaluations and requirements of three-Dimensional TV”, IEEE Trans. Circuits Syst. Video Technol., Vol. 14, No. 3, Mar. 2004, pp. 381-391.

[41] N. Suresh and N. Jayant, “Mean time between failures: a functional quality metric for consumer video”, First International Workshop on Video Processing and Quality Metrics for Consumer Electronics, Scottsdale, Arizona, USA, 23-25 January 2005.

[42] Z. Lu, W. Lin, X. Yang, E. Ong, S. Yao, C. S. Boon and S. Kato, “Measuring the negative impact of frame dropping on perceptual visual quality”, SPIE Human Vision and Electronic Imaging X, eds, B. E. Rogowitz,, T. N. Pappas, and S. J. Daly, Vol. 5666, pp.554-562, 2005.

[43] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error measurement to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004.[44] J. Caviedes and F. Oberti, “No-reference quality metric for degraded and enhanced video”, Proc PPIE, vol. 5150, pp. 621-632, 2003.

[45] H. R. Sheikh, A. C. Bovik and L. Cormack, “No-reference quality assessment using natural scene statistics: JPEG2000”, IEEE Trans. Image Processing, Vol.14(11), pp.1918 – 1927, Nov. 2005.

[46] M. C. Q. Farias, S. K. Mitra and J. M. Foley, “Perceptual contributions of blocky, blurry and noisy artifacts to overall annoyance”, IEEE International Conference on Multimedia and Expo (ICME), Vol. I, pp. 529-532, 2003.

[47]M. Yuen, and H.R. Wu, “A survey of MC/DPCM/DCT video coding distortions”, Signal Processing, Vol. 70, No. 3, Nov. 1998, pp. 247-278.

[48] L. Itti, “Automatic foveation for video compression using a neurobiological model of visual attention”, IEEE Trans. Image Processing, Vol.13(10), pp.1304 – 1318, Oct. 2004.

[49] X. Yang, W. Lin, Z. Lu, E. Ong and S.Yao, “Motion-compensated Residue Pre-processing in Video Coding Based on Just-noticeable-distortion Profile”, IEEE Trans. Circuits and Systems for Video Technology, vol.15(6), pp.742-750, June, 2005.