Objective Assessment of Perceived Sharpness ofProjection Displays with a Calibrated Camera

Ping Zhao, Yao Cheng, Marius PedersenGjøvik University College, Norway

Email: ping.zhao@hig.no

Abstract—Sharpness is an important image quality attributefor projection displays. However, it has not been well studiedfor projection displays in the existing literature. In this paper,we conduct an experimental study of perceived sharpness onprojection displays in a controlled environment. The basic ideais to simulate the optical blurring process with Gaussian filteringand apply them to selected natural images. We project thesedistorted images onto the screen, and we invite a group of humanobservers to give perceptual ratings. We also use a calibratedcamera to capture them in order to measure the sharpness witheight state-of-art image quality metrics. The correlations betweenthe objective and subjective results indicate that SSIM, FSIM andVIF metrics give excellent average performance.

Keywords—image quality, sharpness, camera calibration, pro-jection display, image quality metric

I. INTRODUCTION

Nowadays, modern digital imaging with advanced tech-nologies composes an essential part of our daily life. It is easyfor consumers to capture what they see and record what theyexperience with portable imaging devices without professionaltraining. Sharing stories with friends can be achieved byclicking a single button afterward. One common way to do thisis via projection systems which are typically configured withhigh definition displays to visualize image reproductions. Com-paring to other display technologies, projection systems haveunique advantages like portability, flexibility for deployment,and large screens for sharing information to a crowd. In a fewscenarios, multiple projections can be tiled up to produce largeperceptual seamless images which visualize information to thetarget audiences [1], [2]. In recent years, there is an increasinggeneral interest on embedding projectors into portable devicesto further enhance the continuity experiences between mobileimaging devices and socialization over the Internet [3], [4]. Theinvestigation [5] showed that users are willing to project con-tent with other people around in social spaces, which indicatesthe projection system have a good potential to become morepopular in the coming future. Hence, image quality assessmentof projection displays becomes an increasingly interestingtopic for both scientific research and industrial communities,because it defines a systematic approach to measure andevaluate the quality of image reproductions.

Image quality is mainly evaluated with respect to theperception. The ultimate goal of image quality assessmentis to correlate the objective results to the subjective results,and thereby eliminate the demand of human observers. In thiscontext, image quality attributes are used. These are termsof perception, such as, but not limited to, lightness, contrast,color accuracy, sharpness, artifacts (including noises), and

physical properties (screen dimension, display resolution andrefreshing rate, etc.). For the researches of imaging systemsin various domains, the selection of the most important imagequality attributes may have different priorities. For printing,Pedersen et al. [6] suggested that all these image qualityattributes mentioned above are important; Johnson [7] speciallyremarked color accuracy, sharpness, and contrast for printing.For information displays, You et al. [8] and Lehtimaki et al. [9]pointed out noise, sharpness and perceived depth are prioritiesfor stereoscopic imaging. However, for projection displays,limited works have been done so far. Thomas et al. [10] andStrand et al. [11] remarked lightness and color accuracy, whileMajumder et al. [12], [13] indicated that lightness is moreimportant than the color accuracy. With respect to the literatureabove, it is clear that despite of specific research domains andimaging technologies involved, sharpness is commonly rec-ognized as an important image quality attribute for perceptualevaluation, and it is closely associated with other image qualityattributes like lightness and contrast. Since sharpness definesthe amount of details in image reproductions, it is commonlyreferred as the counterpart of blur which is one of the mosttypical image quality distortions. The human visual system hasa remarkable capability to detect image blur without seeing theoriginal image, but unfortunately the underlying mechanismsare not well understood [14], [15].

In this paper, we conduct an experimental study of per-ceived sharpness on projection displays in a controlled envi-ronment. The goal is to evaluate the performance of state-of-art image quality metrics measuring image sharpness, anddetermine their performance with respect to the correlationsbetween perceived and measured sharpness. The results can bepotentially used by manufacturers to improve their product set-ting without doing subjective investigations or the consumersto optimize their projection displays accordingly.

The rest of this paper is organized as follows. First, inSection II, we present state-of-art image quality metrics usedto measure image sharpness. Then, in Section III, we presentthe experimental setup. In Section IV, we demonstrate thesubjective and objective results. Last, in Section V, conclusionsand future works are presented.

II. SHARPNESS METRICS

Conventionally, sharpness was largely evaluated based onthe definition of edges in images. The main idea is to locateedges in local regions, compute the quality scores in theseregions at the detected edges, and pool them to generate afinal score representing the global sharpness quality [16]. Theedge features represent the quality of optical components in

2015 Colour and Visual Computing Symposium (CVCS)978-1-4799-1765-5/15/$31.00 c©2015 IEEE

an imaging system, so the measured edge responses can beused as an estimate of the modulation transfer function [17].Many sharpness metrics are based on the clarity of detailsin the image reproductions. They are slightly more advancedthan edge base metrics, because they are suitable to measurehighly degraded images. Recently, there is also an increasinglyinterest on developing perceptual models to simulate humanvisual system on evaluating image sharpness. Nevertheless,sharpness metrics can be classified into two main categoriesdepending on the availability of the original image.

A. Full Reference Metrics

The ”Structure Similarity Index” (SSIM) [18] is commonlyused to predict the degradation of structures in images. Al-though it was not originally developed to evaluate sharpness,it estimates the visibility of detail preservation which is implic-itly associated with sharpness. Marziliano et al. [19] proposedtwo full reference based metrics to evaluate the sharpnessof JPEG2000 compressed images, not merely on ringingartifacts, but also on blurring. Zhang et al. [20] proposed a fullreference metric Feature Similarity based Index Metric (FSIM)to measure image sharpness. Firstly, they generated a localimage quality map with phase congruence and image gradientmagnitude as features, and then utilized the phase congruenceinformation again as a weighting function to derive the finalimage quality score. Another commonly referred image qualitymetric ”Visual Information Fidelity” (VIF) was proposed bySheikh et al. [21]. This metric was derived from a statisticalmodel for natural scenes, a model for image distortions, and ahuman visual system model in an information-theoretic setting.The ”Visual-Signal-to-Noise-Ratio” (VSNR) was presented byChandler et al. [22]. This metric uses low-level human visualsystem properties of contrast sensitivity and visual maskingfirst via a wavelet-based model to determine if the distortionsare below the threshold of visual detection. If the distortionsare supra-threshold, the low-level human visual system prop-erty of perceived contrast and the mid-level human visualsystem property of global precedence are taken into accountas an alternative measure of structural degradation.

B. No Reference Metrics

Caviedes et al. [23] developed a content independentno-reference sharpness metric based on the local frequencyspectrum around edges, however this method has problemsto predict image quality when artifacts become dominant.Maalouf et al. [24] defined a metric based on the eigenvaluesof the wavelet-based multi-scale structure tensor to accumulatemulti-scale gradient information of local regions. The structuretensor has the advantage to identify edges in spite of thepresence of noises, so the metric is suitable to measure thesharpness of color edges. Cao et al. [25] introduced a metricwhich takes the advantage of anisotropic diffusion to buildup a preliminary map of ringing artifacts and refined it byconsidering the property of ringing structure. The proposedmetric was reported to work well for JPEG2000 compressedimages. Samira et al. [26] proposed a method to measure colordifferences to determine the sharpness in local regions. Thismethod is good in the cases of which the color managementis critical to the applications. Vu et al. [27] presented ablock-based metric to quantify the local perceived sharpness

within and across images. Both spectral and spatial propertiesof images are utilized to build up indexes for the standarddeviation of the impulse response used in Gaussian blurring.

Hassen et al. [15] developed a metric ”Local Phase Coher-ence based Sharpness Index” (LPC-SI) to identify sharpness asstrong local phase coherence in the complex wavelet transformdomain. They incorporated this metric into a framework thatallows for computation of local phrase coherence in arbitraryfractional scales. Leclaire et al. [28] introduced a metric S-Index which can be used to measure the sharpness in aprobabilistic scene using the small variation of an imagecompared to that of certain associated random-phase fields.In addition, Narvekar et al. [29] presented a metric based on”Cumulative Probability of Blur Detection” (CPBD), whichdiscretizes visual sharpness into several regions and for eachregion a distinct quality class or qualitative score is assigned.Then a training base method was proposed to determine thecentroids of the quality classes for the assigned scores, andfinally the index of image quality class is assigned as themeasured image quality. Narvekar et al. [30] proposed a no-reference metric based on a cumulative probability of blurdetection. Comparing to the saliency-weighted foveal poolingbased metric developed by Sadaka et al. [31], their metric doesnot require additional visual attention or salience maps. In theformer case, the computational complexity is largely reduced.Besides, Ferzli et al. [32] derived from the measured just-noticeable blurs to develop a perceptual-based sharpness metricwhich is applied to 8x8 blocks instead of the entire image. Themetric took into account the response of the human visualsystem to sharpness at different contrast levels. Wang et al.[33] proposed a metric to predict wavelet coefficients of localphase coherence structures across scale and space in a coarse-to-fine manner. Another no-reference metric sharpness metric”Just Noticeable Blur Metric” (JNBM) proposed by Ferzli etal. [34] integrated the concept of just noticeable blur into aprobability summation model. The metric was reported to beable to predict the relative amount of blurriness in images withdifferent content.

III. EXPERIMENTAL SETUP AND PROCEDURE

In our experiment, we use a calibrated camera to captureall pixels on the projection screen in one shot, and theperformance of eight selected state-of-art sharpness metrics areevaluated with respect to the perceptual ratings on the capturedimages which are registered with their original ones.

The experiments take place in a controlled dark room,which simulates the condition of home theater. We use aportable three chip LCD projector SONY APL-AW15 (throwratio: 1.5) to produce projections on a planar screen whichis naturally hanging on the ceiling. The projector is put on atable placed in front of the projection screen about 3 metersaway with respect to the throw ratio of the projector. A remotecontrolling laptop is connected to the projector via a HDMIcable in order to generate full screen projections which haveresolution 1920×1080 in pixels. On the screen, the dimensionof projection area is approximately 2 × 1.2 in meters. Weuse a DLSR Nikon D610, which has an imaging resolution6048× 4016 in pixels with a Sigma VR 24-105mmf/4G (VRoff) lens to capture the projections. The camera is fixed on atripod placed in front of the projection screen about 4 meters

Fig. 1. The selected test images from the Colourlab Image Database: Image quality, and they are Gaussian blurred to generate 6 levels of sharpness distortions.

away. Pictures are taken remotely with a software withoutphysically touching the camera. The pictures are saved in rawformat and rendered with aliasing minimization and zipperelimination demosaicing algorithm [35] without automaticvignetting correction, brightness adjustment, gamma correctionand noise reduction etc. We select 7 test images (Figure 1)from the Colourlab Image Database: Image Quality [36] togenerate 6 levels of Gaussian blur with kernel size 11 andstandard deviation 0, 0.5, 1, 1.5, 2, and 3 respectively. Theselection criteria of the test images is established based on thecoverage of different image features such as hue, saturation,lightness, contrast, skin color, sky, grass, size of neutral grayareas, color transition, fine details, and text presence etc.

The cameras need to be calibrated in advance because theyare known to have optical and electronic issues. Vignetting ef-fect is an optical phenomenon which stands for the undesirablegradual intensity fall off from the image center to its externallimits. It corrupts every picture taken from the cameras. Insome cases, the camera sensors are not necessary to producecompletely linear intensity responses. In this paper, we incor-porate existing methods proposed in our previous researches[37] to eliminate the vignetting effect. The basic idea is touse a hazy sky as a closely uniform illuminant to create avignetting mask from multiple rotated shots of the same scene,and apply the mask to the every picture we take subsequently.The camera settings are also optimized, so the sensor responsesare linearized in all circumstances. Meanwhile, we incorporatethe image registration algorithm [38] to register the capturedprojection with its original digital copy in order to applyfull reference metrics. For reduced reference and no-referencemetrics, we also apply the registered images, so the test imagesare identical to all metrics under evaluation.

We calculate the objective sharpness using eight state-of-art image quality metrics, which are commonly referred thein existing literature: SSIM [18], VSNR [22], VIF [21], FSIM[20], LPC-SI [15], S-Index [28], CPBD [29], and JNBM [34].

IV. EXPERIMENTAL RESULTS

A. Subjective Results

In the subjective experiment session, we invite 15 humanobservers (recommended by CIE [39] and ITU [40]) to giveperceptual ratings to the projections of image distortions.Each observer is placed at exactly the same position as thecamera. The viewing condition is similar to a home theater likeenvironment where the room is completely dark and the visualangle from the projection boundaries to the principal axis ofobservation is approximately 15 degrees. The blurred imagesare displayed in a randomized order to observers, and eachtime only one image is displayed. The experiment is set up asa category judgment experiment. For each displayed image, the

observers are asked to indicate the overall perceptual sharpnesswith a category label which stands for the rank between noblurring at all and completely blurred corresponding to theratings numbers ranging from 1 to 9 respectively. After theexperiment, the observers are asked to share their evaluationstrategies on the image sharpness.

The perceptual ratings collected in the experiment arescaled to generate Z-scores [41] (Figure 2). In this figure, itis clear that the rank order of perceived sharpness decreasesas the blur level increases. However, the sharpness perceptionshould not be simply fitted into a linear regression model,since for the average Z-scores for test image 1, 4, 6 and 7appear to have a flat region between the first and the secondblur levels. Another observation is that the general tendency ofmean Z-scores for all test images are fairly similar and theirvalue ranges are almost identical. The variation of Z-scores oncertain blur levels (the blur level 4 for test image 1) are slightlylarger and observers have contrary arguments on the perceivedsharpness (the blur level 1 for test image 5). This observationsuggests that observers have closely the same perception onsharpness despite of image content.

B. Objective Results

The main purpose is to know which metric works thebest for the captured images of projection displays. Theperformance of the image quality metrics are evaluated withrespect to the Pearson correlation coefficients between themetric results and the mean Z-scores of perceptual ratings(Figure 3a). It is clear that the correlations coefficients ofobjective and subjective results are high. In most cases, theirvalues are above 0.9 and all metrics have a correlation above0.7. S-Index and VSNR have relative poorer performance fortest image 2; S-Index, CPBD and JNBM have relative poorerperformance for test image 4. Obviously, the performance ofSSIM, LPC-SI, FSIM, JNBM and VIF have good performanceon prediction in general. In this case, we generate a box plotof all metrics for the correlation coefficients for all test images(Figure 3b). In this figure, we can find that VIF has the bestperformance on prediction of sharpness in general. Both of itsmedian and mean values are very high, and the variation ofcorrelations is very compact. In other words, this metric hasboth good as well as stable performance despite of the contentof test images. This observation can also be applied to SSIMand FSIM, both have slightly larger variations.

Another interesting conclusion can be made based oncomparing the difference in performance between the full ref-erence metrics and no reference metrics. Among these metrics,the metric SSIM, FSIM, VSNR and VIF are full referencemetrics, while the rest are no-reference metrics. Obviously, theaverage performance of full reference metrics is higher than

Blur Levels

1 2 3 4 5 6

Z-s

co

res

-2

0

2

Test Image 1

Blur Levels

1 2 3 4 5 6Z

-sco

res

-2

0

2

Test Image 2

Blur Levels

1 2 3 4 5 6

Z-s

co

res

-2

0

2

Test Image 3

Blur Levels

1 2 3 4 5 6

Z-s

co

res

-2

0

2

Test Image 4

Blur Levels

1 2 3 4 5 6

Z-s

co

res

-2

0

2

Test Image 5

Blur Levels

1 2 3 4 5 6

Z-s

co

res

-2

0

2

Test Image 6

Blur Levels

1 2 3 4 5 6

Z-s

co

res

-2

0

2

Test Image 7

Z-scores of Perceptual Ratings for All Test Images

Fig. 2. The Z-scores of perceptual ratings collected from 15 human observers based on 6 blur levels of 7 test images. The red dots stand for mean Z-scoresfor each blur levels over all human observers, while the red bars stand for the median. The blue box stand for the 25% inner quantiles of Z-scores, and theblue bars stand for 75% outer quantiles of Z-scores. The red crosses stand for outliers with respect to the 75% outer quantiles. All plots are scaled to have theidentical Z-score value range from -2.5 to 2.5.

the no-reference metrics, and the full reference metrics tendto give more stable outcomes with respect to the performancevariance. Since the captured images in the experiments areall registered with their original ones, the full reference andno-reference metrics have the identical input captured imageswith the exactly the same dimension, content and opticaldegradation. In addition, there is no extra disturbance like theouter border of projection area or the background introducedin the captured images. Then the performance differencesmust root in the metrics themselves. One important purposeof incorporating metrics in image quality assessment is tomaximize the prediction accuracy of image quality attributes.So, for image quality assessment of projection displays, weshould always incorporate full reference metrics over no-reference metrics in the first place since the original digitalcopies are available in most cases.

C. Perceptual Evaluation Strategies

After the subjective rating session, each human observeris asked to share his/her perceptual evaluation strategies onthe image sharpness. In the experiment, almost all observersimplicitly insist to focus on only a few specific areas to locateand observe the fine details assuming that the global sharpnessis identical to the local areas. The areas without fine detailsare largely ignored by the human observers according to theexperimental records. We rank these areas by the count ofobservers who really pay attentions to and put the top twofor each test image in the Table I. From this table, we cansee that the fine details that observers actually pay attentionare located in the areas where the images are sharpest in theoriginal images. When it comes to comparing two images,

the observers are sensitive to the changes of these sharpestareas either due to the transition of blurred edges, lightness orcolors in these areas. The changes of less sharp areas appearto be less visible to the human observers. This explains whythe human observers largely ignore the unfocused backgroundbut only pay attention to the well focus foreground objects.In this case, for the design of a good image qualityF metricpredicting sharpness, we should develop algorithms to separatewell focused and non-focused areas, and take the advantage ofthe original images to discover the changes of lightness andcolors in the sharpest areas.

TABLE I. TOP TWO SALIENCY AREAS OF PERCEIVED SHARPNESS

Test Image Saliency Area Count of Observers

1 Walls in the center 7Chairs 4

2 Flower center 12Flower staments 6

3 Green grass 11River 10

4 Human face (highlight area) 7Human face (shadow area) 4

5 Peacock feathers 5Peacock eyes 5

6 Car body 8Eaves 5

7 Texts 12Bird 4

V. CONCLUSION

In this paper, we conduct an experimental study of per-ceived sharpness on projection displays in a controlled en-

(a)

(b)

Fig. 3. The Pearson correlation coefficients between the objective and subjective results over all test images (a), and performance of sharpness metrics over alltest images (b). In the experiment, we have 7 test images blurred with Gaussian filtering with kernel size 11 and standard deviation 0.5. We invite 15 humanobservers in the subjective experiments to give perceptual ratings. In (b), The red dots stand for mean Pearson correlation coefficient, while the red bars standfor the medians. The blue box and bars stand for the 25% inner and 75% outer quantiles of correlation values respectively. The red crosses stand for outliers.FR and NR indicate full and no reference metrics respectively.

vironment. The perceptual results suggest that the perceivedsharpness follows a nonlinear tendency pattern but its rankorder remain the same as the unversed ranked blur levels.The correlations between the metric results and perceptualresults indicate that the VIF metric perform well for mosttypes of distorted natural images. However, SSIM, FSIM andVIF give excellent prediction performance on average in mostcases, both terms of absolute mean values and variance ofPearson correlation coefficients. There is an indication thatfull reference metrics outperform no-reference metrics in ourexperimental environment. In the experiments, we ask humanobservers to share their perceptual evaluation strategies. Theirideas indicate that the human observers are only sensitive to

the changes of lightness and colors in the sharpest areas in theoriginal images. In the coming future, experiments should bedone to investigate the impact of different viewing conditions.

ACKNOWLEDGMENT

We would like to acknowledge the contributions of a largenumber of volunteers in the subjective experiments. This workis part of the HyPerCept research program funded by theResearch Council of Norway.

REFERENCES

[1] O. Bimber and A. Emmerling, “Multifocal Projection: A Multipro-jector Technique for Increasing Focal Depth.,” IEEE Transactions on

Visualization and Computer Graphics, vol. 12, no. 4, pp. 658–67, 2006.

[2] B. Sajadi, M. Lazarov, A. Majumder, and M. Gopi, “Color Seamlessnessin Multi-projector Displays Using Constrained Gamut Morphing,” IEEETransactions on Visualization and Computer Graphics, vol. 15, no. 6,pp. 1317–25, 2009.

[3] O. Bimber, A. Emmerling, and T. Klemmer, “Embedded Entertainmentwith Smart Projectors,” IEEE Computer, vol. 38, no. 1, pp. 48–55, Jan.2005.

[4] W. Zou and H. Xu, “Colorimetric Color Reproduction Framework forScreen Relaxation of Projection Display,” Displays, vol. 32, no. 5, pp.313–319, Dec. 2011.

[5] Yun Zhou, Tao Xu, Bertrand David, and Rene Chalon, “Where is mobileprojection interaction going? the past, present and future of the mobileprojected interface,” in Human-Computer Interaction. Applications andServices, pp. 189–198. Springer, 2014.

[6] M. Pedersen, N. Bonnier, J. Y. Hardeberg, and F. Albregtsen, “Attributesof Image Quality for Color Prints,” Journal of Electronic Imaging, vol.19, no. 1, pp. 011016–1–011016–13, Jan. 2010.

[7] G. M. Johnson, Measuring images: Differences, quality, and appear-ance, Ph.D. thesis, Rochester Institute of Technology, 2003.

[8] J. You, L. Xing, A. Perkis, and X. Wang, “Perceptual Quality Assess-ment for Stereoscopic Images based on 2D Image Quality Metrics andDisparity Analysis,” in International Workshop on Video Processingand Quality Metrics for Consumer Electronics, Scottsdale, AZ, USA,2010, pp. 1–6.

[9] T. M. Lehtimaki, K. Saaskilahti, T. Pitkaaho, and T. J. Naughton,“Evaluation of Perceived Quality Attributes of Digital HologramsViewed with A Stereoscopic Display,” in Euro-American Workshopon Information Optics, Helsinki, Finland, July 2010, pp. 1–3, IEEE.

[10] J. B. Thomas and A. M. Bakke, “A Colorimetric Study of SpatialUniformity in Projection Displays,” in Computational Color Imaging,A. Tremeau, R. Schettini, and S. Tominaga, Eds., vol. 5646, pp. 160–169. Springer Berlin Heidelberg, 2009.

[11] M. Strand, J. Y. Hardeberg, and P. Nussbaum, “Color Image Qualityin Projection Displays: A Case Study,” in Image Quality and SystemPerformance II, Proceedings of SPIE, R. Rasmussen and Y. Miyake,Eds., San Jose, CA, USA, Jan. 2005, pp. 185–195, SPIE.

[12] A. Majumder and R. Stevens, “Perceptual Photometric Seamlessnessin Projection-based Tiled Displays,” ACM Transactions on Graphics,vol. 24, no. 1, pp. 118–139, Jan. 2005.

[13] I. Juricevic, L. Land, A. Wilkins, and M. A. Webster, “VisualDiscomfort and Natural Image Statistics,” Journal of Vision, vol. 9,no. 8, pp. 1046–1046, Mar. 2010.

[14] V. L. Jaya and R. Gopikakumari, “IEM : A New Image EnhancementMetric for Contrast and Sharpness Measurements,” InternationalJournal of Computer Applications, vol. 79, no. 9, pp. 1–9, 2013.

[15] R. Hassen, Z. Wang, and M. M. A. Salama, “Image SharpnessAssessment Based on Local Phase Cohernce,” IEEE Transactions onImage Processing, vol. 22, no. 7, pp. 2798–2810, 2013.

[16] K. Juha and H. Saarelma, “Objective Quality Potential Measures ofNatural Color Images,” Journal of Imaging Science and Technology,vol. 42, no. 3, pp. 250–263, 1998.

[17] D. Williams and P. D. Burns, “Measuring and Managing Digital ImageSharpening,” in Proceedings of IS&T Archiving Conference, Bern,Swizerland, 2008, pp. 89–93, SPIE.

[18] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “ImageQuality Assessment: from Error Visibility to Structural Similarity,”IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612,Apr. 2004.

[19] P. Marziliano, F. Dufaux, S. Winkler, and T. Ebrahimi, “Perceptual Blurand Ringing Metrics: Application to JPEG2000,” Signal Processing:Image Communication, vol. 19, no. 2, pp. 163–172, Feb. 2004.

[20] L. Zhang, X. Mou, and D. Zhang, “FSIM: A Feature SimilarityIndex for Image Quality Assessment.,” IEEE Transactions on ImageProcessing, vol. 20, no. 8, pp. 2378–2386, Aug. 2011.

[21] H. R. Sheikh and A. C. Bovik, “Image Information and Visual Quality,”IEEE Transactions on Image Processing, vol. 15, no. 2, pp. 430–444,Feb. 2006.

[22] D. M. Chandler and S. S. Hemami, “VSNR: A Wavelet-Based VisualSignal-to-Noise Ratio for Natural Images,” IEEE Transactions on ImageProcessing, vol. 16, no. 9, pp. 2284–2298, Sept. 2007.

[23] J. Caviedes and F. Oberti, “A New Sharpness Metric Based on LocalKurtosis, Edge and Energy Information,” Signal Processing: ImageCommunication, vol. 19, no. 2, pp. 147–161, Feb. 2004.

[24] A. Maalouf and M.-C. Larabi, “A No Reference Objective Color ImageSharpness Metric,” in European Signal Processing Conference, Aalborg,Denmark, 2010, pp. 1019–1022.

[25] Z. Cao, Z. Wei, and G. Zhang, “A No-Reference Sharpness MetricBased on Structured Ringing for JPEG2000 Images,” Advances inOptical Technologies, vol. 2014, pp. 1–13, 2014.

[26] B. Samira and M. Lindsay, “Colour Difference Metrics and ImageSharpness,” in Color and Imaging Conference Final Program andProceedings, Scottsdale, AZ, USA, 2000, pp. 262–267.

[27] C. T. Vu and D. M. Chandler, “S3: A Spectral and Spatial SharpnessMeasure,” in International Conference on Advances in Multimedia,Colmar, France, July 2009, pp. 37–43, IEEE.

[28] A. Leclaire and L. Moisan, “No-reference Image Quality Assessmentand Blind Deblurring with Sharpness Metrics Exploiting Fourier PhaseInformation,” Journal of Mathematical Imaging and Vision, 2014.

[29] Niranjan D. Narvekar and Lina J. Karam, “A No-Reference Image BlurMetric Based on the Cumulative Probability of Blur Detection,” IEEEtransactions on image processing : a publication of the IEEE SignalProcessing Society, vol. 20, no. 1, pp. 2678–2683, 2011.

[30] N. D. Narvekar and L. J. Karam, “A No-reference Perceptual ImageSharpness Metric based on A Cumulative Probability of Blur Detec-tion,” in International Workshop on Quality of Multimedia Experience,K. O. Egiazarian, S. S. Agaian, A. P. Gotchev, J. Recker, and G. Wang,Eds., San Diego, CA, USA, July 2009, pp. 87–91, IEEE.

[31] N. G. Sadaka, L. J. Karam, R. Ferzli, and G. P. Abousleman, “ANo-reference Perceptual Image Sharpness Metric based on Saliency-weighted Foveal Pooling,” in IEEE International Conference on ImageProcessing, San Diego, CA, USA, 2008, pp. 369–372, IEEE.

[32] R. Ferzli and J. L. Karam, “Human Visual System Based No-ReferenceObjective Image Sharpness Metric,” in International Conference onImage Processing, Atlanta, GA USA, 2006, pp. 2949–2952, IEEE.

[33] Z. Wang and E. P. Simoncelli, “Local Phase Coherence and the Per-ception of Blur,” in Neural Information Processing Systems, Whistler,Canada, 2003, number December 2003, pp. 9–11, MIT Press.

[34] R. Ferzli and L. J. Karam, “A No-reference Objective Image SharpnessMetric based on The Notion of Just Noticeable Blur (JNB),” IEEETransactions on Image Processing, vol. 18, no. 4, pp. 717–28, Apr.2009.

[35] E. Martinec and P. Lee, “AMAZE Demosaicing Algorithm,” 2010.[36] X. Liu, M. Pedersen, and J. Y. Hardeberg, “CID:IQ - A New Image

Quaity Database,” in International Conference on Image and SignalProcessing, Abderrahim Elmoataz, Olivier Lezoray, Fathallah Nouboud,and Driss Mammass, Eds., Cherbourg, Normandy, France, 2014, vol.8509 of Lecture Notes in Computer Science, pp. 193–202, SpringerInternational Publishing.

[37] P. Zhao, M. Pedersen, J. Thomas, and J. Y. Hardeberg, “PerceptualSpatial Uniformity Assessment of Projection Displays with a CalibratedCamera,” in Color and Imaging Conference, Boston, MA, USA, 2014,pp. 159–164, Society for Imaging Science and Technology.

[38] P. Zhao, M. Pesersen, J. Y. Hardeberg, and J. B. Thomas, “ImageRegistration for Quality Assessment of Projection Displays,” in IEEEInternational Conference on Image Processing, Paris, France, 2014, pp.3488–3492, IEEE.

[39] CIE, “Guidelines for the evaluation of gamut mapping algorithms,”Technical Report ISBN: 3-901-906-26-6, CIE TC8-03, 156:2004.

[40] International Telecommunication Union, “Recommendation itu-rbt.500-11: Methodology for the subjective assessment of the quality oftelevision pictures,” Technical report, International TelecommunicationUnion/ITU Radiocommunication Sector, 2009.

[41] P. G. Engeldrum, Psychometric Scaling: A Toolkit for Imaging SystemsDevelopment, Imcotek Pr, 2000.