lecture 9 perceptual image quality assessment 09-perceptual iq… · lin zhang, sse, 2016 lecture 9...
TRANSCRIPT
-
Lin ZHANG, SSE, 2016
Lecture 9Perceptual Image Quality Assessment
Lin ZHANG, PhDSchool of Software Engineering
Tongji UniversityFall 2016
-
Lin ZHANG, SSE, 2016
Contents
• Problem definition• Full reference image quality assessment• No reference image quality assessment• Summary
-
Lin ZHANG, SSE, 2016
Problem DefinitionPlease rank these images according to their visual quality
(a) (b)
(c) (d)
A subjective process
Can we have some algorithms to measure the image quality? And the results is highly consistent with the human judgments
Our goal in this lecture!
-
Lin ZHANG, SSE, 2016
Problem Definition
• The goal of the IQA research is to develop objective metrics for measuring image quality and the results should be consistent with the subjective judgments
• Classification of the IQA problem• Full reference IQA (FR‐IQA)
• The distortion free image is given. Such an image is considered to have a perfect quality and is called reference image. A set of its distorted versions are also provided. Your task is to devise an algorithm to evaluate the perceptual quality of distorted images
-
Lin ZHANG, SSE, 2016
Problem Definition
• The goal of the IQA research is to develop objective metrics for measuring image quality and the results should be consistent with the subjective judgments
• Classification of the IQA problem• Reduced reference IQA (RR‐IQA)
• The distorted image is given; The reference image is not available; however, partial information of the reference image is known
-
Lin ZHANG, SSE, 2016
Problem Definition
• The goal of the IQA research is to develop objective metrics for measuring image quality and the results should be consistent with the subjective judgments
• Classification of the IQA problem• Reduced reference IQA (RR‐IQA)
-
Lin ZHANG, SSE, 2016
Problem Definition
• The goal of the IQA research is to develop objective metrics for measuring image quality and the results should be consistent with the subjective judgments
• Classification of the IQA problem• No reference IQA (NR‐IQA)
• Only the distorted image is given. Or more accurately in such a case, we cannot call it as "distorted" image since we do not know the corresponding distortion‐free reference image. You need to design an algorithm to evaluate the quality of the given image
-
Lin ZHANG, SSE, 2016
Contents
• Problem definition• Full reference image quality assessment
• Application scenarios• Problem of the classical FR‐IQA metric—MSE• Error visibility method• Structural Similarity (SSIM)• Feature Similarity (FSIM)• Performance metrics
• No reference image quality assessment• Summary
-
Lin ZHANG, SSE, 2016
Application Scenarios
• Quantify the performance of de‐noising algorithms
I
simulation
'IAI
algoA
BIalgoB
denoising results
Which algorithm is better?
has better quality than
We need to design a metric function f having the following property:
( , ) ( , )A Bif f I I f I Ihas better quality than ; AI BI
otherwise, BI AISuch an f is our desired FR‐IQA metric
-
Lin ZHANG, SSE, 2016
Application Scenarios
• Quantify the performance of compression algorithms
I
Which compression algorithm is better?
AI
algoA
BI
algoB
compression results
We also need a FR‐IQA metric
-
Lin ZHANG, SSE, 2016
Application Scenarios
• FR‐IQA metrics usually can be used in the following applications• Measure the performance of some image enhancement or restoration algorithms, such as algorithms for denoising, deblurring, dehazing, etc
• Measure the performance of image compression algorithms• Used to adjust parameters of some image processing algorithms
-
Lin ZHANG, SSE, 2016
Contents
• Problem definition• Full reference image quality assessment
• Application scenarios• Problem of the classical FR‐IQA metric—MSE• Error visibility method• Structural Similarity (SSIM)• Feature Similarity (FSIM)• Performance metrics
• No reference image quality assessment• Summary
-
Lin ZHANG, SSE, 2016
Problem of the Classical FR‐IQA Metric—MSE
• MSE (mean squared error) is a classical metric to measure the similarity between two image signals
• MSE is a point‐to‐point based measure
• Advantages• Easy to compute• Easy to optimize• Clear physical meaning: energy
• What’s the problem?
image ximage y
1/221
i iiN
x y
MSE
-
Lin ZHANG, SSE, 2016
• MSE is point‐to‐point and doesn’t care about ordering
MSE = 1600, MSSIM = 0.6373 MSE = 1600, MSSIM = 0.0420
MSE thinks that the similarity between I1and I2 and the similarity between I3and I4 are the same; this contradicts with the human intuition
1I
2I
reorder
3I
reorder
4I
Problem of the Classical FR‐IQA Metric—MSE
-
Lin ZHANG, SSE, 2016
1/ 221
i ii
x yN
+ 30
+ (rand sign)* 30
MSE = 900
SSIM = 0.9329
MSE = 900
SSIM = 0.2470
Don’t care about the sign
Problem of the Classical FR‐IQA Metric—MSE
-
Lin ZHANG, SSE, 2016
Problem of the Classical FR‐IQA Metric—MSE
• Mean Squared Error
1/ 221
i ii
E x yN
signal samplesare independent
signal sampleshighly correlated
• Natural Images
highly structured
Conflict
-
Lin ZHANG, SSE, 2016
Contents
• Problem definition• Full reference image quality assessment
• Application scenarios• Problem of the classical FR‐IQA metric—MSE• Error visibility method• Structural Similarity (SSIM)• Feature Similarity (FSIM)• Performance metrics
• No reference image quality assessment• Summary
-
Lin ZHANG, SSE, 2016
Error Visibility Method: Idea
• Representative work• Frequency weighting[Mannos & Sakrison ’74]• Sarnoff model [Lubin ’93]• Visible difference predictor [Daly ’93]• Perceptual image distortion [Teo & Heeger ’94]• DCT‐based method [Watson ’93]• Wavelet‐based method [Safranek ’89, Watson et al. ’97]
distorted signal = reference signal + error signal
Quantify error signal perceptually
-
Lin ZHANG, SSE, 2016
Error Visibility Method: Framework
• Goal: simulate relevant early HVS components• Structures motivated by physiology• Parameters determined by psychophysics
-
Lin ZHANG, SSE, 2016
• Contrast sensitivity function
CSF
10-2
10-1
100
spatial frequency (cycles/degree)
no
rma
lize
d s
en
sit
ivit
y
10-1
100
101
102
In this image, the contrast amplitude depends only on the vertical coordinate, while the spatial frequency depends on the horizontal coordinate. Observe that for medium frequency you need less contrast than for high or low frequency to detect the sinusoidal fluctuation
Error Visibility Method—HVS Properties Modeling
-
Lin ZHANG, SSE, 2016
Error Visibility Method—HVS Properties Modeling
• Masking
highly visible
weak masking
hardly visible
strong masking
-
Lin ZHANG, SSE, 2016
Error Visibility Method—Difficulties
• Natural image complexity problem• Based on simple‐pattern psychophysics
• Quality definition problem• Error visibility = quality ?
-
Lin ZHANG, SSE, 2016
Contents
• Problem definition• Full reference image quality assessment
• Application scenarios• Problem of the classical FR‐IQA metric—MSE• Error visibility method• Structural Similarity (SSIM)• Feature Similarity (FSIM)• Performance metrics
• No reference image quality assessment• Summary
-
Lin ZHANG, SSE, 2016
Structural Similarity (SSIM)
Purpose of vision: extract structural information
Quantify structural distortion
• Questions:• How to define structural/nonstructural distortions?• How to separate structural/nonstructural distortions?
-
Lin ZHANG, SSE, 2016
Structural Similarity (SSIM)
• What are structural/non‐structural distortions?
non‐structural distortions
luminance change contrast change
Gamma distortion spatial shift JPEG blocking wavelet ringing
blurring noise contamination
structural distortions
-
Lin ZHANG, SSE, 2016
Structural Similarity (SSIM)
• What are structural/non‐structural distortions?
distortedimage
originalimage
-
Lin ZHANG, SSE, 2016
Structural Similarity (SSIM)
• What are structural/non‐structural distortions?
structuraldistortion
distortedimage
originalimage
nonstructuraldistortion
-
Lin ZHANG, SSE, 2016
Structural Similarity (SSIM)
• What are structural/non‐structural distortions?
structuraldistortion
+
distortedimage
originalimage
nonstructuraldistortion
-
Lin ZHANG, SSE, 2016
Structural Similarity (SSIM)
• What are structural/non‐structural distortions?
structuraldistortion
+
distortedimage
originalimage
+
nonstructuraldistortion
-
Lin ZHANG, SSE, 2016
Structural Similarity (SSIM)—Computation
For two corresponding local patches x and y in two images
Luminance Comparison
Contrast Comparison
Structure Comparison
CombinationSimilarity Measure
( )x y
( , )l x y
( , )c x y
( , )s x y
is the mean intensity of x (y),
( )x y is the standard deviation of x (y),
xy is the covariance of x and y,
Assume that x and y are vectorized as 1 2, ,..., Nx x xx 1 2, ,..., Ny y yyand
1
1 Nx i
ix
N
1/ 2
2
1
1 Nx i x
ix
N
1
1 Nxy i x i y
ix y
N
-
Lin ZHANG, SSE, 2016
Structural Similarity (SSIM)—Computation
12 2
1
2( , ) x y
x y
Cl
C
x y 22 22
2( , ) x y
x y
Cc
C
x y 33
( , ) x yx y
Cs
C
x y, ,
1 2
2 2 2 21 2
2 2( , ) ( , ) ( , ) ( , ) x y xy
x y x y
C CSSIM l c s
C C
x y x y x y x y
Then, the structure similarity between x and y are defined as 1 2 3, ,C C C are fixed constants, and usually set 3 2 / 2C C
If the image contains M local patches (defined by a sliding window), the overall image quality is
1
1SSIM ( , )M
i ii
SSIMM
x y
-
Lin ZHANG, SSE, 2016
Structural Similarity (SSIM)—Computation
[Wang & Bovik, IEEE Signal Proc. Letters, ’02][Wang et al., IEEE Trans. Image Proc., ’04]
distortion/similaritymeasure withinsliding window
original image
distorted image
quality map
pooling
quality score
1 22 2 2 2
1 2
(2 )(2 )( , )
( )( )x y xy
x y x y
C CSSIM
C C
x y
-
Lin ZHANG, SSE, 2016
Structural Similarity (SSIM)—Computation
original image
Gaussian noise corrupted image
absolute error map
SSIM index map
-
Lin ZHANG, SSE, 2016
Structural Similarity (SSIM)—Computation
JPEG2000 compressed image
original image
SSIM index map
absolute error map
-
Lin ZHANG, SSE, 2016
Structural Similarity (SSIM)—Computation
JPEG compressed image
original image
SSIM index map
absolute error map
-
Lin ZHANG, SSE, 2016
Comparison between MSE and SSIM
MSE=0, SSIM=1 MSE=309, SSIM=0.928 MSE=309, SSIM=0.987
MSE=309, SSIM=0.580 MSE=309, SSIM=0.641 MSE=309, SSIM=0.730
original Image
-
Lin ZHANG, SSE, 2016
Comparison between MSE and SSIM
referenceimage
initialimage
converged image(best SSIM)
equal-MSEcontour
converged image(worst SSIM)
-
Lin ZHANG, SSE, 2016
Summary about SSIM
• Structural similarity (SSIM) metric measures the structure distortions of images
• In implementation, SSIM measures the similarity of two local patches from three aspects, luminance, contrast, and structure
• The quality scores predicted by SSIM is much more consistent with human judgments than MSE
• SSIM is now widely used to gauge image processing algorithms
In the next section, you will encounter an even more powerful IQA metric, FSIM
-
Lin ZHANG, SSE, 2016
Contents
• Problem definition• Full reference image quality assessment
• Application scenarios• Problem of the classical FR‐IQA metric—MSE• Error visibility method• Structural Similarity (SSIM)• Feature Similarity (FSIM)
• Phase congruency• Feature similarity index (FSIM)
• Performance metrics
• No reference image quality assessment• Summary
-
Lin ZHANG, SSE, 2016
Phase Congruency
• Why is phase important?
2 ( )( ) ( ) ( ) ( )i ux i uf x F u f x e dx A u e Fourier transform( )u is called the Fourier phase or the global phase
• Phase is defined for a specified frequency• The Fourier phase indicates the relative position of the frequency components
• Phase is a real number between
-
Lin ZHANG, SSE, 2016
Phase Congruency
• Why is phase important?
Fourier
Hilbert
Reconstruction results
From Fourier’s amplitude
From Fourier’s phase
From Fourier’s phase + Hilbert’s amplitude
-
Lin ZHANG, SSE, 2016
Phase Congruency
• Local phase analysis
Question: What are the frequency components (and the associated phases) at a certain position in a real signal f(x) ?
Fourier transforms cannot answer such questions
-
Lin ZHANG, SSE, 2016
Phase Congruency
• Local phase analysis
( ) ( ) ( )A Hf x f x if x Analytic signal needs to be constructed
where1( ) ( )* ( ), ( )Hf x h x f x h x x
is called the Hilbert transform of f(x)( )Hf x
Instantaneous phase: ( ) arctan 2 ( ), ( )Hx f x f x
Instantaneous amplitude: 2 2( ) ( ) ( )HA x f x f x
seems local, but not so since HT is a global transform( )x
-
Lin ZHANG, SSE, 2016
Phase Congruency
• Local phase analysisThus, local complex filters whose responses are analytic signals themselves are used instead
That isIf is a complex filter and ( ) ( ) ( )e og x g x ig x
( )* ( ) ( )* ( ) ( )* ( )e og x f x g x f x ig x f x is an analytic signal, then, the local phase (instead of the instantaneous phase) of f(x)is defined as
( ) arctan 2 ( ) * ( ), ( ) * ( )o ex g x f x g x f x The local amplitude is
2 2( ) ( ) * ( ) ( ) * ( )e oA x g x f x g x f x
-
Lin ZHANG, SSE, 2016
Phase Congruency
• Local phase analysisThus, local complex filters whose responses are analytic signals themselves are used instead
That isIf is a complex filter and ( ) ( ) ( )e og x g x ig x
( )* ( ) ( )* ( ) ( )* ( )e og x f x g x f x ig x f x is an analytic signal,
and are called a quadrature pairogeg
What are the commonly used quadrature pair filters?See the next sections!
-
Lin ZHANG, SSE, 2016
Phase Congruency
• Gabor filter
' 2 ' 2
'2 2
1( , ) exp exp 22 x y
x yG x y i fx
where ' 'cos sin , sin cosx x y y x y
(1)
-
Lin ZHANG, SSE, 2016
Phase Congruency
• Gabor filter
' 2 ' 2
'2 2
1( , ) exp exp 22 x y
x yG x y i fx
where ' 'cos sin , sin cosx x y y x y
(1)
John Daugman, University of Cambridge, UK
Denis Gabor, 1900~1979, Nobel Prize Winner
-
Lin ZHANG, SSE, 2016
Phase Congruency
• Gabor filter
' 2 ' 2
'2 2
1( , ) exp exp 22 x y
x yG x y i fx
where ' 'cos sin , sin cosx x y y x y
(1)
Primary Cortex
-
Lin ZHANG, SSE, 2016
Phase Congruency
• Gabor filter
' 2 ' 2
'2 2
1( , ) exp exp 22 x y
x yG x y i fx
where ' 'cos sin , sin cosx x y y x y
(1)
J. G. Daugman, Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two‐dimensional visual cortical filters, Journal of the Optical Society of America A, 2(7):1160–1169, 1985.
-
Lin ZHANG, SSE, 2016
Phase Congruency
• Log‐Gabor filter• It is also a quadrature pair filter; defined in the frequency domain
2202 2 2
log /( , ) exp exp
2 2j
jr
G
where is the orientation angle, is the center frequency, controls the filter’s radial bandwidth, and determines the angular bandwidth
/j j J 0r
radial part angular part Log‐Gabor
-
Lin ZHANG, SSE, 2016
Phase Congruency—Motivation
• Gradient‐based feature detectors• Roberts, Prewitt, Sobel, Canny et al…..• Find maximum in the gradient map • Sensitive to illumination and contrast variations• Poor localization, especially with scale analysis• Difficult to use—threshold problem. One does not know in advance what level of edge strength corresponds to a significant feature
-
Lin ZHANG, SSE, 2016
Phase Congruency—Motivation
• Gradient‐based feature detectors• Roberts, Prewitt, Sobel, Canny et al…..• Find maximum in the gradient map • Sensitive to illumination and contrast variations• Poor localization, especially with scale analysis• Difficult to use—threshold problem. One does not know in advance what level of edge strength corresponds to a significant feature
-
Lin ZHANG, SSE, 2016
Phase Congruency—Motivation
Harris corners, Harris corners, 1 7
-
Lin ZHANG, SSE, 2016
Phase Congruency—Motivation
• Phase congruency is proposed to overcome those drawbacks• Totally based on the local phase information• A more general framework for feature definition• Invariant to contrast and illumination variation• Offers the promise of allowing one to specify universal feature thresholds
-
Lin ZHANG, SSE, 2016
Phase Congruency—Definition
• First appears in [1] • It is more like the human visual system• It postulates that features are perceived at points of maximum phase congruency
[1] M.C. Morrone, J. Ross, D.C. Burr, and R. Owens, Mach bands are phase dependent, Nature, vol. 324, pp. 250‐253, 1986
[all the following discussions will be based on this observation]
-
Lin ZHANG, SSE, 2016
Phase Congruency—Definition
• Features from the PC view. Fourier components are all in phase in the two cases
-
Lin ZHANG, SSE, 2016
Phase Congruency—Computation
• Now the widely used to method to compute phase congruency is [1]
• In [1], Kovesi proposed a framework to compute PC by using quadrature pair filters
[1] P. Kovesi, Image features from phase congruency, Videre: Journal of Computer Vision Research, vol. 1, pp. 1‐26, 1999
-
Lin ZHANG, SSE, 2016
Phase Congruency—Computation
denote the even‐symmetric and odd‐symmetric wavelets at a scale
,e on nM Mn
( ), ( ) ( )* , ( )*e on n n ne x o x I x M I x M
( ) ( ), ( ) ( )n nn n
F x e x H x o x
The amplitude and phase of the transform at a given wavelet scale is given by
2 2( ) ( ) ( )n n nA x e x o x
and can be estimated as:( )F x ( )H x
( )( )( )n
n
E xPC xA x
2 2( ) ( ) ( )E x F x H x
( )( )( )
nn
n
o xx arctge x
-
Lin ZHANG, SSE, 2016
Phase Congruency—Example
-
Lin ZHANG, SSE, 2016
Phase Congruency—Example
-
Lin ZHANG, SSE, 2016
Phase Congruency—Example
-
Lin ZHANG, SSE, 2016
Phase Congruency—Example
-
Lin ZHANG, SSE, 2016
Contents
• Problem definition• Full reference image quality assessment
• Application scenarios• Problem of the classical FR‐IQA metric—MSE• Error visibility method• Structural Similarity (SSIM)• Feature Similarity (FSIM)
• Phase congruency• Feature similarity index (FSIM)
• Performance metrics
• No reference image quality assessment• Summary
-
Lin ZHANG, SSE, 2016
Feature Similarity Index (FSIM)
• A state‐of‐the‐art method proposed in [1]
[1] Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang, FSIM: A feature similarity index for image quality assessment, IEEE Trans. Image Processing, vol. 20, pp. 2378‐2386, 2011
-
Lin ZHANG, SSE, 2016
Feature Similarity Index (FSIM)
• A state‐of‐the‐art method proposed in [1]• Motivations
• Low‐level feature inspired• Visual information is often redundant• low‐level features convey most crucial information• Image degradations will lead to changes in image low‐level features
Thus, an IQA index could be devised by comparing the low‐level features between the reference image and the distorted image
What kinds of features?
-
Lin ZHANG, SSE, 2016
Feature Similarity Index (FSIM)
• Phase congruency • Physiological and psychophysical evidences• Measure the significance of a local structure
• Gradient magnitude• PC is contrast invariant. However, local contrast indeed will affect the perceptive image quality
• Thus, we have to compensate for the contrast• Gradient magnitude can be used to measure the contrast similarity
-
Lin ZHANG, SSE, 2016
Feature Similarity Index (FSIM)
• Phase congruency—An example
-
Lin ZHANG, SSE, 2016
Feature Similarity Index (FSIM)
• Gradient magnitude
1 10 * ( ), 0 0 0 * ( )16 16
3 3 10 3x yG f G f
x x
Scharr operator to extract the gradient
Gradient magnitude (GM):
2 2x yG G G
-
Lin ZHANG, SSE, 2016
Feature Similarity Index (FSIM)
• FSIM computationGiven two images, f1 and f2Their PC maps, PC1 and PC2Their GM maps, G1 and G2
PC similarity 1 2 12 21 2 1
2 ( ) ( )( )( ) ( )PC
PC PC TSPC PC T
x xx
x x
GM similarity 1 2 22 21 2 2
2 ( ) ( )( )( ) ( )G
G G TSG G T
x xx
x x
( ) ( ) ( )FSIM
( )PC G m
m
S S PCPC
x
x
x x xx
where 1 2( ) max ( ), ( )mPC PC PCx x x
T1 is a constant
T2 is a constant
-
Lin ZHANG, SSE, 2016
Feature Similarity Index (FSIM)
• Extended to a color IQASeparate the chrominance from the luminance
0.299 0.587 0.1140.596 0.274 0.3220.211 0.523 0.312
Y RI GQ B
I1(I2) and Q1(Q2) be the I and Q channels of f1 and f2
1 2 32 2
1 2 3
2 ( ) ( )( )( ) ( )I
I I TSI I T
x xx
x x1 2 4
2 21 2 4
2 ( ) ( )( )( ) ( )Q
Q Q TSQ Q T
x xx
x x
( ) ( ) ( ) ( ) ( )FSIM
( )PC G I Q m
Cm
S S S S PCPC
x
x
x x x x xx
-
Lin ZHANG, SSE, 2016
Feature Similarity Index (FSIM)—Schematic diagram
-
Lin ZHANG, SSE, 2016
Summary
• FSIM is a HVS‐driven IQA index• HVS perceives an image mainly based on its low‐level features
• PC and gradient magnitude are used• PC is also used to weight the contribution of each point to the overall similarity of two images
• FSIM is extended to FSIMC, a color IQA index
• FSIM (FSIMC) outperforms all the other state‐of‐the‐art IQA indices evaluated
-
Lin ZHANG, SSE, 2016
Contents
• Problem definition• Full reference image quality assessment
• Application scenarios• Problem of the classical FR‐IQA metric—MSE• Error visibility method• Structural Similarity (SSIM)• Feature Similarity (FSIM)• Performance metrics
• No reference image quality assessment• Summary
-
Lin ZHANG, SSE, 2016
Performance Metrics
• How to evaluate the performance of IQA indices?• Some benchmark datasets were created
• Reference images (quality distortion free) are provided• For each reference image, a set of distorted images are created; they suffer from kinds of quality distortions, such as Gaussian noise, JPEG compression, blur, etc; let’s suppose that there are altogether N distorted images
• For each distorted image, there is an associated quality score, given by subjects; thus, altogether we have N scores
• For distorted images, we can compute their objective quality scores by using an IQA index f; we can get N quality scores
• f’s performance can be reflected by the rank order correlation coefficients between and
1{ }N
i is
1{ }N
i io
1{ }N
i is 1{ }N
i io
-
Lin ZHANG, SSE, 2016
Performance Metrics
• How to evaluate the performance of IQA indices?
Spearman rank order correlation coefficient (SRCC)
2
12
61
( 1)
N
ii
dSRCC
N N
where di is the difference between the ith image's ranks in the subjective and objective evaluations.
Note: in Matlab, you can compute the SROCC by usingsrcc = corr(vect1, vect2, 'type', 'spearman')
-
Lin ZHANG, SSE, 2016
Performance Metrics
• How to evaluate the performance of IQA indices?
Kendall rank order correlation coefficient (KRCC)
0.5 ( 1)c dn nKRCC
N N
where nc is the number of concordant pairs and nd is the number of discordant pairs
Note: in Matlab, you can compute the SROCC by usingkrcc = corr(vect1, vect2, 'type', ‘kendall')
-
Lin ZHANG, SSE, 2016
Performance Metrics
• Popular used benchmark datasets for evaluating IQA indices
Database name
Reference Images
Distorted images
Observer numbers
Distortion types
TID2013 [1] 25 2000 971 24
TID2008 [2] 25 1700 838 17
CSIQ [3] 30 866 35 6LIVE [4] 29 779 161 5
[1] http://www.ponomarenko.info/tid2013.htm[2] http://www.ponomarenko.info/tid2008.htm[3] http://vision.okstate.edu/?loc=csiq[4] http://live.ece.utexas.edu/research/Quality/
-
Lin ZHANG, SSE, 2016
Performance Metrics—Comparison of IQA Indices
FSIM FSIMC MS‐SSIM VIF SSIM IFC VSNR NQM
TID2013
SRCC 0.8015 0.8510 0.7859 0.6769 0.7417 0.5389 0.6812 0.6392
KRCC 0.6289 0.6665 0.6047 0.5147 0.5588 0.3939 0.5084 0.4740
TID2008
SRCC 0.8805 0.8840 0.8528 0.7496 0.7749 0.5692 0.7046 0.6243
KRCC 0.6946 0.6991 0.6543 0.5863 0.5768 0.4261 0.5340 0.4608
CSIQSRCC 0.9242 0.9310 0.9138 0.9193 0.8756 0.7482 0.8106 0.7402
KRCC 0.7567 0.7690 0.7397 0.7534 0.6907 0.5740 0.6247 0.5638
LIVESRCC 0.9634 0.9645 0.9445 0.9631 0.9479 0.9234 0.9274 0.9086
KRCC 0.8337 0.8363 0.7922 0.8270 0.7963 0.7540 0.7616 0.7413
Note: For more details about full reference IQA, you can refer to http://sse.tongji.edu.cn/linzhang/IQA/IQA.htm
-
Lin ZHANG, SSE, 2016
Contents
• Problem definition• Full reference image quality assessment• No reference image quality assessment
• Background introduction• Our proposed method: IOUML
• Summary
-
Lin ZHANG, SSE, 2016
Background introduction—Problem definition• No reference image quality assessment (NR‐IQA)
•Devise computational models to estimate the quality of a given image as perceived by human beings
• The only information an NR‐IQA algorithm receives is the image whose quality is being assessed itself
-
Lin ZHANG, SSE, 2016
Background introduction—Problem definition• No reference image quality assessment (NR‐IQA)
How do you think the quality of these two images?
Though you are not provided the ground‐truth reference images, you may judge the quality of these two images as poor
-
Lin ZHANG, SSE, 2016
Background introduction—Problem definition• No reference image quality assessment (NR‐IQA)
How do you think about the qualities of these images?Rank them
Remember that you DONOT know the ground‐truth “high quality” reference image
-
Lin ZHANG, SSE, 2016
Background introduction—Typical methods• Opinion‐aware approaches
• These approaches require a dataset comprising distorted images and associated subjective scores
• At the training stage, feature vectors are extracted from images and then the regression model, mapping the feature vectors to the subjective scores, is learned
• At the testing stage, a feature vector is extracted from the test image, and its quality score can be predicted by inputting the feature vector to the learned regression model
-
Lin ZHANG, SSE, 2016
Background introduction—Typical methods• Opinion‐aware approaches
feature vectors
-
Lin ZHANG, SSE, 2016
Background introduction—Typical methods• Opinion‐aware approaches
• BIQI [1]• BRISQUE [2]• BLIINDS [3]• BLIINDS‐II [4]• DIIVINE [5]• CORNIA [6]• LBIQ [7]
Proposed by Bovik’s group, Univ. Texas
http://live.ece.utexas.edu/
-
Lin ZHANG, SSE, 2016
Background introduction—Typical methods• Opinion‐aware approaches
– [1] A. Moorthy and A. Bovik, A two‐step framework for constructing blind image quality indices, IEEE Sig. Process. Letters, 17: 513‐516, 2010
– [2] A. Mittal, A.K. Moorthy, and A.C. Bovik, No‐reference image quality assessment in the spatial domain, IEEE Trans. Image Process., 21: 4695‐4708, 2012
– [3] M.A. Sadd, A.C. Bovik, and C. Charrier, A DCT statistics‐based blind image quality index, IEEE Sig. Process. Letters, 17: 583‐586, 2010
– [4] M.A. Sadd, A.C. Bovik, and C. Charrier, Blind image quality assessment: A natural scene statistics approach in the DCT domain, IEEE Trans. Image Process., 21: 3339‐3352, 2012
– [5] A.K. Moorthy and A.C. Bovik, Blind image quality assessment: from natural scene statistics to perceptual quality, IEEE Trans. Image Process., 20: 3350‐3364, 2011
– [6] P. Ye, J. Kumar, L. Kang, and D. Doermann, Unsupervised feature learning framework for no‐reference image quality assessment, CVPR, 2012
– [7] H. Tang, N. Joshi, and A. Kapoor. Learning a blind measure of perceptual image quality, CVPR, 2011
-
Lin ZHANG, SSE, 2016
Background introduction—Typical methods• Opinion‐unaware approaches
• These approaches DONOT require a dataset comprising distorted images and associated subjective scores
• A typical method is NIQE [1]• Offline learning stage: constructing a collection of quality‐aware features from pristine images and fitting them to a multivariate Gaussian (MVG) model
• Testing stage: the quality of a test image is expressed as the distance between a MVG fit of its features and
[1] A. Mittal et al. Making a “completely blind” image quality analyzer. IEEE Signal Process. Letters, 20(3): 209-212, 2013.
-
Lin ZHANG, SSE, 2016
Contents
• Problem definition• Full reference image quality assessment• No reference image quality assessment
• Background introduction• Our proposed method: IL‐NIQE
• Motivations and our contributions• NIS‐induced quality‐aware features• Pristine model learning• IL‐NIQE index• Experimental results
• Summary
-
Lin ZHANG, SSE, 2016
Motivations[1]
• Opinion‐unaware approaches seems appealing, so we want to propose an opinion‐unaware approach
• Design rationale• Natural images without quality distortions possess regular statistical properties that can be measurably modified by the presence of distortions
• Deviations from the regularity of natural statistics, when quantified appropriately, can be used to assess the perceptual quality of an image
• NIS‐based features have been proved powerful. Any other NIS‐based features?
[1] Lin Zhang et al., A feature‐enriched completely blind image quality evaluator, IEEE Trans. Image Processing 24 (8) 2579‐2591, 2015
-
Lin ZHANG, SSE, 2016
Contributions• A novel “opinion‐unaware” NR‐IQA index, IL‐NIQE (Integrated Local‐NIQE)• A set of prudently designed NIS‐induced quality‐aware features
• Bhattacharyya distance based metric to measure the quality of a local image patch
• A visual saliency based quality score pooling scheme• A thorough evaluation of the performance of modern NR‐IQA indices
-
Lin ZHANG, SSE, 2016
Contents
• Problem definition• Full reference image quality assessment• No reference image quality assessment
• Background introduction• Our proposed method: IL‐NIQE
• Motivations and our contributions• NIS‐induced quality‐aware features• Pristine model learning• IL‐NIQE index• Experimental results
• Summary
-
Lin ZHANG, SSE, 2016
• Statistics of normalized luminance• The mean subtracted contrast normalized (MSCN) coefficients have been observed to follow a unit normal distribution when computed from natural images without quality distortions [1]
• This model, however, is violated when images are subjected to quality distortions; the degree of violation can be indicative of distortion severity
IL‐NIQE—NIS‐induced quality‐aware features
( , )nI x y
[1] D.L. Ruderman. The statistics of natural images. Netw. Comput. Neural Syst., 5(4):517-548, 1994.
-
Lin ZHANG, SSE, 2016
• Statistics of normalized luminance
IL‐NIQE—NIS‐induced quality‐aware features
( , ) ( , )( , )( , ) 1n
I x y x yI x yx y
,( , ) ( , )K L
k lk K l L
x y I x k y l
2,( , ) ( , ) ( , )K L
k lk K l L
x y I x k y l x y
where
Conforms to Gaussian
-
Lin ZHANG, SSE, 2016
• Statistics of normalized luminance•We use a generalized Gaussian distribution (GGD) to model the distribution of
IL‐NIQE—NIS‐induced quality‐aware features
( , )nI x y
( ; , ) exp
2 1 /x
g x
Density function of GGD,
Parameters are used as quality‐aware features which can be estimated from {In(x, y)} by MLE
,
-
Lin ZHANG, SSE, 2016
• Statistics of MSCN products• The distribution of products of pairs of adjacent MSCN coefficients, In(x, y)In(x, y+1), In(x, y)In(x+1, y), In(x, y)In(x+1, y+1), and In(x, y)In(x+1, y-1), can also capture the quality distortion
IL‐NIQE—NIS‐induced quality‐aware features
-
Lin ZHANG, SSE, 2016
• Statistics of MSCN products• They can be modeled by asymmetric generalized Gaussian distribution (AGGD),
IL‐NIQE—NIS‐induced quality‐aware features
exp / , 01 /
( ; , , )exp / , 0
1 /
ll r
l r
rl r
x xg x
x x
The mean of AGGD is 2 / / 1 /r l
, , ,r l are used as “quality‐aware” features
-
Lin ZHANG, SSE, 2016
• Statistics of partial derivatives and gradient magnitudes• We found that when introducing quality distortions to an image, the distribution of its partial derivatives, and gradient magnitudes, will be changed
IL‐NIQE—NIS‐induced quality‐aware features
-
Lin ZHANG, SSE, 2016
• Statistics of partial derivatives and gradient magnitudes
IL‐NIQE—NIS‐induced quality‐aware features
1(a)
1(b)
1(d)
1(c)
1(e)
-
Lin ZHANG, SSE, 2016
• Statistics of partial derivatives and gradient magnitudes
IL‐NIQE—NIS‐induced quality‐aware features
-0.015 -0.01 -0.005 0 0.005 0.01 0.0150
1
2
3
4
5
6
partial derivative (normalized)
Perc
enta
ge (%
)
Fig. 1(a)Fig. 1(b)Fig. 1(c)Fig. 1(d)Fig. 1(e)
0 0.005 0.01 0.0150.5
1
1.5
2
2.5
3
3.5
gradient magnitude (normalized)
Perc
enta
ge (%
)
Fig. 1(a)Fig. 1(b)Fig. 1(c)Fig. 1(d)Fig. 1(e)
-
Lin ZHANG, SSE, 2016
• Statistics of partial derivatives and gradient magnitudes
IL‐NIQE—NIS‐induced quality‐aware features
Partial derivatives* ( , ), * ( , )x x y yI I G x y I I G x y
where, 2 24 2
2 2
4 2
( , ) exp2 2
( , ) exp2 2
x
y
x x yG x y
y x yG x y
Gradient magnitudes2 2( , ) x yGM x y I I
-
Lin ZHANG, SSE, 2016
• Statistics of partial derivatives and gradient magnitudes• We use a GGD to model the distributions of Ix (or Iy) and take its parameters as features
• We use a Weibull distribution [1] to model the distribution of the gradient magnitudes and use the parameters as features,
IL‐NIQE—NIS‐induced quality‐aware features
1 exp , 0
; ,
0, 0
aa
a
a xx xh x a b b b
x
a and b are used as features
[1] J.M. Geusebroek and A.W.M. Smeulders. A six-stimulus theory for stochastic texture. Int. J. Comp. Vis., 62(1): 7-16, 2005.
-
Lin ZHANG, SSE, 2016
• Statistics of image’s responses to log‐Gabor filters• Motivation: neurons in the visual cortex respond selectively to stimulus’ orientation and frequency, statistics on the images’ multi‐scale multi‐orientation decompositions should be useful for designing a NR‐IQA model
IL‐NIQE—NIS‐induced quality‐aware features
-
Lin ZHANG, SSE, 2016
• Statistics of image’s responses to log‐Gabor filters• For multi‐scale multi‐orientation filtering, we adopt the log‐Gabor filter,
IL‐NIQE—NIS‐induced quality‐aware features
2
20
22
log
222 ,
j
rG e e
where is the orientation angle, is the center frequency, controls the filter’s radial bandwidth, and determines the angular bandwidth
/j j J 0r
radial part angular part Log‐Gabor
-
Lin ZHANG, SSE, 2016
• Statistics of image’s responses to log‐Gabor filters
IL‐NIQE—NIS‐induced quality‐aware features
With log‐Gabor filters having J orientations and N center frequencies, we could get response maps
, ,{( ( ), ( )) :| 0,..., 1, 0,..., 1}n j n je o n N j J x xwhere and represents the image’s response to the real and imaginary part of the log‐Gabor filter
, ( )n je x , ( )n jo x
We extract the quality‐aware features asa) Use a GGD model to fit the distribution of {en,j(x)} (or {on,j(x)}) and take
the model parameters α and β as features.b) use a GGD to model the distribution of partial derivatives of {en,j(x)} (or
{on,j(x)}) and also take the two model parameters as features.c) Use a Weibull model to fit the distribution of gradient magnitudes of
{en,j(x)} (or {on,j(x)}) and take the corresponding parameters a and b as features
-
Lin ZHANG, SSE, 2016
• Statistics of colors• Ruderman et al. showed that in a logrithmic‐scale opponent color space, the distributions of the image data conform well to Gaussian [1]
IL‐NIQE—NIS‐induced quality‐aware features
[1] D.L. Ruderman et al. Statistics of cone response to natural images: implications for visual coding. J. Opt. Soc. Am. A, 15(8): 2036-2045, 1998.
-
Lin ZHANG, SSE, 2016
• Statistics of colors
IL‐NIQE—NIS‐induced quality‐aware features
RGB to logarithmic signal with mean subtracted,( , ) log ( , ) log ( , )
( , ) log ( , ) log ( , )( , ) log ( , ) log ( , )
x y R x y R x yx y G x y G x yx y B x y B x y
where means the mean of logX(x,y)>
to opponent color space
1
2
3
( , ) ( ) / 3
( , ) ( 2 ) / 6
( , ) ( ) / 2
l x y
l x y
l x y
For natural images, l1, l2, and l3 conform well to Gaussian
-
Lin ZHANG, SSE, 2016
• Statistics of colors
IL‐NIQE—NIS‐induced quality‐aware features
-0.2 -0.1 0 0.1 0.20
1
2
3
4
5
6
7
8
Perc
enta
ge (%
)
Fig. 3(a)Fig. 3(b)Fig. 3(c)
l1 coefficients-0.2 -0.1 0 0.1 0.20
2
4
6
8
10
12
Perc
enta
ge (%
)
Fig. 3(a)Fig. 3(b)Fig. 3(c)
l2 coefficients
-0.2 -0.1 0 0.1 0.20
2
4
6
8
10
12
14
16
18
20
Perc
enta
ge (%
)
Fig. 3(a)Fig. 3(b)Fig. 3(c)
l3 coefficients
3(a)
3(b)
3(c)
-
Lin ZHANG, SSE, 2016
• Statistics of colors
IL‐NIQE—NIS‐induced quality‐aware features
We use Gaussian to fit the distribution of l1, l2, and l3,2
22
1 ( )( ; , ) exp22xf x
For each l1, l2, and l3 channel, we estimate the two parameters ζ and ρ2 and take them as quality‐aware features
-
Lin ZHANG, SSE, 2016
Contents
• Problem definition• Full reference image quality assessment• No reference image quality assessment
• Background introduction• Our proposed method: IL‐NIQE
• Motivations and our contributions• NIS‐induced quality‐aware features• Pristine model learning• IL‐NIQE index• Experimental results
• Summary
-
Lin ZHANG, SSE, 2016
• The pristine model acts as a “standard” for representing characteristics of high quality images
• It is learned from a pristine image set collected by us, which contains 92 high quality images
Pristine model learning
Sample high quality images
-
Lin ZHANG, SSE, 2016
• Step 1: for each pristine image, it is partitioned into patches
• Step 2: high contrast patches are selected based on local variance field
• Step 3: for each selected patch, the quality‐aware features are extracted. Thus, we can get a feature vector set,
Pristine model learningP P
1{ :| 1,..., }, di ii M x x
where M is the number of patches and d is the feature dimension
d is very large, so we need a further dimension reduction operation
-
Lin ZHANG, SSE, 2016
• Step 4: dimension reduction by PCA
Pristine model learning
Suppose is the dimension reduction matrix,d m m d
After the dimension reduction,1d
ix ' 1T mi i
x x
• Step 5: feed into a MVG model and regard it as the pristine model
'1{ }
Mi ix
1/2 1/2
1 1( ) exp22
Tmf
x x v x v
where v is the mean vector and is the covariance matrix The mean vector and the covariance matrix of the pristine model are denoted as v1 and 1
-
Lin ZHANG, SSE, 2016
Contents
• Problem definition• Full reference image quality assessment• No reference image quality assessment
• Background introduction• Our proposed method: IL‐NIQE
• Motivations and our contributions• NIS‐induced quality‐aware features• Pristine model learning• IL‐NIQE index• Experimental results
• Summary
-
Lin ZHANG, SSE, 2016
• Step 1: partition the test image into patches• Step 2: for each patch, we extract from it a feature vector; thus, we can get a feature vector set,
IL‐NIQE indexP P
1{ :| 1,..., }, di t ii M y y
where Mt denotes the number of patches extracted from test image
• Step 3: reduce the dimension of yi as' ' 1,T mi i i
y y y
• Step 4: fit a MVG from and denote its covariance matrix as
'1{ } t
Mi iy
2
-
Lin ZHANG, SSE, 2016
• Step 5: the quality qi of patch i is measured as
IL‐NIQE index
1
' '1 21 12
T
i i iq
v y v y
Such a metric is inspired from the Bhattacharyya distance
• Step 6: visual saliency guided quality pooling• High salient patches are given high weights• Patch saliency si is computed as the sum of saliency values covered by patch i
• For saliency computation, we use the Spectral Residual approach [1]
1 1
/t tM M
i i ii i
q q s s
[1] X. Hou and L. Zhang. Saliency detection: a spectral residual approach. CVPR’07, 1-8, 2007.
-
Lin ZHANG, SSE, 2016
Offline pristine model learning
…
pristine images
n high-contrast patches
…
patch extraction
n feature vectors
feature extraction
MVG parameters and
MVG fitting
Online quality evaluation of a test image
test image
k image patches
…
patch extraction
k feature vectors
feature extraction
quality score computation for each patch
1 2, ,..., kq q qfinal quality score pooling
1/k iiq q k
μ
-
Lin ZHANG, SSE, 2016
Contents
• Problem definition• Full reference image quality assessment• No reference image quality assessment
• Background introduction• Our proposed method: IL‐NIQE
• Motivations and our contributions• NIS‐induced quality‐aware features• Pristine model learning• IL‐NIQE index• Experimental results
• Summary
-
Lin ZHANG, SSE, 2016
Protocol• Protocol for experiments
• Experiments are conducted on TID2013, CSIQ, LIVE, LIVE Multiply‐Distortion
• Spearman rank order correlation coefficient (SRCC) and Pearson linear correlation coefficient (PLCC)
YES1225LIVE MD2YES1225LIVE MD1NO5799LIVENO6866CSIQYES243000TID2013
Contains multiply‐distortions?
Distortion Types No.
Distorted Images No.Dataset
Benchmark image datasets used
-
Lin ZHANG, SSE, 2016
Protocol• IL‐NIQE was compared with
• “opinion‐aware” approaches• BIQI, BRISQUE, BLIINDS2, DIIVINE, and CORNIA
• “opinion‐unaware” approaches• NIQE and QAC
-
Lin ZHANG, SSE, 2016
Cross‐datasets evaluation• Drawback of single‐database evaluation strategy
• It cannot faithfully measure the prediction performance of NR‐IQA indices since it cannot reflect the “blindness”
• At the training stage the “opinion aware” approaches had already met all the possible distortion types that would appear in the testing stage
• Consequently, we will train the “opinion aware” approaches on one dataset and test their performances on other rest datasets
-
Lin ZHANG, SSE, 2016
Cross‐datasets evaluation—Training on LIVE
TID2013 CSIQ MD1 MD2SRCC PLCC SRCC PLCC SRCC PLCC SRCC PLCC
BIQI 0.394 0.468 0.619 0.695 0.654 0.774 0.490 0.766BRISQUE 0.367 0.475 0.557 0.742 0.791 0.866 0.299 0.459BLIINDS2 0.393 0.470 0.577 0.724 0.665 0.710 0.015 0.302DIIVINE 0.355 0.545 0.596 0.697 0.708 0.767 0.602 0.702CORNIA 0.429 0.575 0.663 0.764 0.839 0.871 0.841 0.864NIQE 0.311 0.398 0.627 0.716 0.871 0.909 0.795 0.848QAC 0.372 0.437 0.490 0.708 0.396 0.538 0.471 0.672
IL‐NIQE 0.493 0.586 0.813 0.852 0.891 0.902 0.882 0.895
Evaluation results when being trained on LIVE
-
Lin ZHANG, SSE, 2016
Cross‐datasets evaluation—Training on LIVE
BIQI BRISQUE BLIINDS2 DIIVINE CORNIA NIQE QAC IL‐NIQE
SRCC 0.458 0.424 0.424 0.435 0.519 0.429 0.402 0.598PLCC 0.545 0.548 0.525 0.595 0.643 0.512 0.509 0.672
Weighted‐average performance derived from last table
-
Lin ZHANG, SSE, 2016
Cross‐datasets evaluation—Training on TID2013
Evaluation results when being trained on TID2013
LIVE CSIQ MD1 MD2SRCC PLCC SRCC PLCC SRCC PLCC SRCC PLCC
BIQI 0.047 0.311 0.010 0.181 0.156 0.175 0.332 0.380BRISQUE 0.088 0.108 0.639 0.728 0.625 0.807 0.184 0.591BLIINDS2 0.076 0.089 0.456 0.527 0.507 0.690 0.032 0.222DIIVINE 0.042 0.093 0.146 0.255 0.639 0.669 0.252 0.367CORNIA 0.097 0.132 0.656 0.750 0.772 0.847 0.655 0.719NIQE 0.906 0.904 0.627 0.716 0.871 0.909 0.795 0.848QAC 0.868 0.863 0.490 0.708 0.396 0.538 0.471 0.672
IL‐NIQE 0.898 0.903 0.813 0.852 0.891 0.902 0.882 0.895
-
Lin ZHANG, SSE, 2016
Cross‐datasets evaluation—Training on TID2013
Weighted‐average performance derived from last table
BIQI BRISQUE BLIINDS2 DIIVINE CORNIA NIQE QAC IL‐NIQE
SRCC 0.074 0.384 0.275 0.172 0.461 0.775 0.618 0.860PLCC 0.250 0.491 0.349 0.251 0.527 0.821 0.744 0.881
-
Lin ZHANG, SSE, 2016
• We have the following findings• “Opinion aware” indices depend much on the training dataset; it can be seen that these approaches perform better when being trained on LIVE than when being trained on TID2013
• The proposed method IL‐NIQE can achieve the best results nearly in all cases
• The prominent performance of IL‐NIQE indicates that if being designed properly, an “opinion unaware” approach could obtain much better prediction performance than their “opinion aware” counterparts
Cross‐datasets evaluation
-
Lin ZHANG, SSE, 2016
Contents
• Problem definition• Full reference image quality assessment• No reference image quality assessment• Summary
-
Lin ZHANG, SSE, 2016
• The research in IQA aims to propose computational models to compute the image quality in a subjective‐consistent manner
• IQA problems can be classified as FR‐IQA, RR‐IQA, and NR‐IQA problems according to the availability of the reference information
• Quality scores predicted by the modern FR‐IQA methods can be highly consistent with the subjective ratings
• There is still a large room for development of NR‐IQA methods
Summary
-
Lin ZHANG, SSE, 2016
Thanks for your attention