image quality assessment: ssim
TRANSCRIPT
![Page 1: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/1.jpg)
Image Quality Assessment: From Error
Visibility to Structural Similarity
Zhou Wang
![Page 2: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/2.jpg)
MSE=0, MSSIM=1 MSE=225, MSSIM=0.949 MSE=225, MSSIM=0.989
MSE=215, MSSIM=0.671 MSE=225, MSSIM=0.688 MSE=225, MSSIM=0.723
Motivationoriginal Image
![Page 3: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/3.jpg)
Define PerceptualIQA Measures
Optimize IP Systems &Algorithms “Perceptually”
PERCEPTUAL IMAGE PROCESSING
Application Scope: essentially all IP applications image/video compression, restoration, enhancement,
watermarking, displaying, printing …
Perceptual Image Processing
Standard measure (MSE) does not agree with human visual perception
Why?
![Page 4: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/4.jpg)
• Goal—Automatically predict perceived image quality
• Classification— Full-reference (FR); No-reference (NR); Reduced-reference (RR)
• Widely Used Methods—FR: MSE and PSNR
—NR & RR: wide open research topic
• IQA is Difficult
Image Quality Assessment
MSELPSNR
2
10log10=
![Page 5: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/5.jpg)
• VQEG (video quality experts group)1. Goal: recommend video quality assessment standards
(TV, telecommunication, multimedia industries)2. Hundreds of experts
(Intel, Philips, Sarnoff, Tektronix, AT&T, NHK, NASA, Mitsubishi, NTIA, NIST, Nortel ……)
• Testing methodology1. Provide test video sequences2. Subjective evaluation3. Objective evaluation by VQEG proponents4. Compare subjective/objective results, find winner
VQEG (1)
![Page 6: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/6.jpg)
• Current Status1. Phase I test (2000):
Diverse types of distortions10 proponents including PSNRno winner, 8~9 proponents statistically equivalent, including PSNR!
2. Phase II test (2003):Restricted types of distortions (MPEG)Result: A few models slightly better than PSNR
3. VQEG is extending their directions:FR/RR/NR, Low Bit RateMultimedia: video, audio and speech …
VQEG (2)
![Page 7: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/7.jpg)
• Representative work– Pioneering work [Mannos & Sakrison ’74]– Sarnoff model [Lubin ’93]– Visible difference predictor [Daly ’93]– Perceptual image distortion [Teo & Heeger ’94]– DCT-based method [Watson ’93]– Wavelet-based method [Safranek ’89, Watson et al. ’97]
Philosophydistorted signal = reference signal + error signal
Assume reference signal has perfect qualityQuantify perceptual error visibility
Standard IQA Model: Error Visibility (1)
![Page 8: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/8.jpg)
• MotivationSimulate relevant early HVS components
Referencesignal
Distortedsignal
Quality/DistortionMeasure
Standard IQA Model: Error Visibility (2)
ChannelDecomposition
ErrorNormalization
.
.
.
ErrorPooling
Pre-processing
.
.
.
• Key featuresChannel decomposition linear frequency/orientation transforms
Frequency weighting contrast sensitivity function
Masking intra/inter channel interaction
ββ
/1
, ⎥⎦
⎤⎢⎣
⎡= ∑∑
l kkleE
![Page 9: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/9.jpg)
Standard IQA Model: Error Visibility (3)
• Quality definition problem– Error visibility = quality ?
• The suprathreshold problem– Based on threshold psychophysics– Generalize to suprathreshold range?
• The natural image complexity problem– Based on simple-pattern psychophysics– Generalize to complex natural images?
[Wang, et al., “Why is image quality assessment so difficult?” ICASSP ’02][Wang, et al., IEEE Trans. Image Processing, ’04]
![Page 10: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/10.jpg)
New Paradigm: Structural Similarity
• How to define structural information?
• How to separate structural/nonstructural information?
PhilosophyPurpose of human vision: extract structural information
HVS is highly adapted for this purposeEstimate structural information change
Classical philosophy New philosophyBottom-up Top-down
Predict Error Visibility Predict Structural Distortion
![Page 11: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/11.jpg)
++
_
distortedimage
originalimage
Separation of Structural/nonstructural Distortion
![Page 12: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/12.jpg)
++
_
structuraldistortion
distortedimage
originalimage
nonstructuraldistortion
Separation of Structural/nonstructural Distortion
![Page 13: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/13.jpg)
Separation of Structural/nonstructural Distortion
++
_
structuraldistortion
+
distortedimage
originalimage
nonstructuraldistortion
![Page 14: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/14.jpg)
++
_
structuraldistortion
+
distortedimage
originalimage
+
nonstructuraldistortion
Separation of Structural/nonstructural Distortion
![Page 15: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/15.jpg)
++
_structuraldistortion
+
distortedimage
originalimage +
nonstructuraldistortion
Adaptive Linear System
![Page 16: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/16.jpg)
++
_
= + +...
...
structuraldistortion
+
distortedimage
originalimage
= + +
+
nonstructuraldistortion
cK +1.
c1 .
cK +2.
c2 .
cM .
cK .+
+
nonstructural distortioncomponents
structural distortioncomponents
Adaptive Linear System
![Page 17: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/17.jpg)
++
_
+ +...
...distortedimage
originalimage
+ + +cK +1.
c1 .
cK +2.
c2 .
cM .
cK .+
+
nonstructural distortioncomponents
structural distortioncomponents
Adaptive Linear System
=
overcomplete, adaptive basis in the space of all images[Wang & Simoncelli, ICIP ’05, submitted]
![Page 18: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/18.jpg)
i
k
j
x
xi + xj + xk = 0
x - x
O
luminancechange
contrastchange
structuralchange
xi = xj = xk
),(),(),(),( yxyxyxyx sclSSIM ⋅⋅=
122
12),(
CC
lyx
yx
++
+=
μμμμ
yx
222
22),(
CC
cyx
yx
++
+=
σσσσ
yx
3
3),(C
Cs
yx
xy
+
+=
σσσ
yx
Structural Similarity (SSIM) Index in Image Space
[Wang & Bovik, IEEE Signal Processing Letters, ’02][Wang et al., IEEE Trans. Image Processing, ’04]
![Page 19: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/19.jpg)
O O O
O O
Minkowski (MSE) component-weighted magnitude-weighted
Model Comparison
magnitude and component-weighted SSIM
![Page 20: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/20.jpg)
original image
JPEG2000 compressed
image
absolute error map
SSIM index map
![Page 21: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/21.jpg)
original image
Gaussian noise
corrupted image
absolute error map
SSIM index map
![Page 22: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/22.jpg)
original image
JPEG compressed
image
absolute error map
SSIM index map
![Page 23: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/23.jpg)
MSE=0, MSSIM=1 MSE=225, MSSIM=0.949 MSE=225, MSSIM=0.989
MSE=215, MSSIM=0.671 MSE=225, MSSIM=0.688 MSE=225, MSSIM=0.723
Demo Images
![Page 24: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/24.jpg)
Validation LIVE Database
PSNR MSSIM
0.4 0.5 0.6 0.7 0.8 0.9 10
10
20
30
40
50
60
70
80
90
100
MSSIM (Gaussian window, K1 = 0.01, K2 = 0.03)
MO
S
JPEG images JPEG2000 images Fitting with Logistic Function
15 20 25 30 35 40 45 500
10
20
30
40
50
60
70
80
90
100
PSNR
MO
S
JPEG images JPEG2000 images Fitting with Logistic Function
Dataset JP2(1) JP2(2) JPG(1) JPG(2) Noise Blur
82 145
PSNR 0.934 0.895 0.902 0.914 0.987 0.774 0.881
0.971
145
0.9360.967
Error
# of images 87 87 88 145
SSIM 0.968 0.965 0.986 0.944
![Page 25: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/25.jpg)
original image
initial image
initial distortion
MAD Competition: MSE vs. SSIM (1)
[Wang & Simoncelli, Human Vision and Electronic Imaging, ’04]
![Page 26: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/26.jpg)
worst SSIM forfixed MSE
best SSIM forfixed MSE
original image
MAD Competition: MSE vs. SSIM (2)
[Wang & Simoncelli, Human Vision and Electronic Imaging, ’04]
![Page 27: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/27.jpg)
worst MSE forfixed SSIM
best MSE forfixed SSIM
original image
MAD Competition: MSE vs. SSIM (3)
[Wang & Simoncelli, Human Vision and Electronic Imaging, ’04]
![Page 28: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/28.jpg)
worst SSIM forfixed MSE
best SSIM forfixed MSE
worst MSE forfixed SSIM
best MSE forfixed SSIM
original image
MAD Competition: MSE vs. SSIM (4)
[Wang & Simoncelli, Human Vision and Electronic Imaging, ’04]
![Page 29: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/29.jpg)
original image
initial distortedimage
best SSIM forfixed MSE
worst SSIM forfixed MSE
best MSE forfixed SSIM
worst MSE forfixed SSIM
![Page 30: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/30.jpg)
original image
initial distortedimage
best SSIM forfixed MSE
worst SSIM forfixed MSE
best MSE forfixed SSIM
worst MSE forfixed SSIM
![Page 31: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/31.jpg)
• Color image quality assessment
• Video quality assessment
• Multi-scale SSIM
• Complex wavelet SSIM
[Wang, et al., Signal Processing: Image Communication, ’04]
[Wang, et al., Invited Paper, IEEE Asilomar Conf. ’03]
[Wang & Simoncelli, ICASSP ’05]
[Toet & Lucassen., Displays, ’03]
Extensions of SSIM (1)
![Page 32: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/32.jpg)
Extensions of SSIM (2)
Ccc
Ccc
yx
yx
++
+⋅=
∑∑∑
22
*2),(SSIM yx
: complex wavelet coefficients in images x and y
• Complex wavelet SSIM– Motivation: robust to
translation, rotation and scaling
[Wang & Simoncelli, ICASSP ’05]
yx cc ,
complex wavelettransform
![Page 33: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/33.jpg)
Correct Recognition Rate:
MSE: 59.6%; SSIM: 46.9%; Complex wavelet SSIM: 97.7%
Database: 2430 images
Standard patterns: 10 images
Image Matching without Registration
[Wang & Simoncelli, ICASSP ’05]
![Page 34: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/34.jpg)
Using SSIMWeb site: www.cns.nyu.edu/~lcv/ssim/
SSIM Paper: 11,000+ downloads; Matlab code: 2400+ downloadsIndustrial implementation: http://perso.wanadoo.fr/reservoir/
• Image/video coding and communications– Image/video transmission, streaming & robustness [Kim & Kaveh ’02,
Halbach & Olsen ’04, Lin et al. ’04, Leontaris & Reibman ’05]– Image/video compression [Blanch et al. ’04, Dikici et al. ’04 , Ho et al. ‘03,
Militzer et al. ’03]– High dynamic range video coding [Mantiuk et al. ’04]– Motion estimation/compensation [Monmarthe ’04]
• Biomedical image processing– Microarray image processing for bioinformatics [Wang et al. ’03]– Image fusion of CT and MRI images [Piella & Heijmans ’03, Piella ‘04]– Molecular image processing [Ling et al. ’02]– Medical image quality analysis [Chen et al. ’04]
![Page 35: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/35.jpg)
Using SSIM (continued)• Watermarking/data hiding [Alattar ’03, Noore et al. ’04, Macq et al. ‘04
Zhang & Wang ’05, Kumsawat et al. ‘04]• Image denoising [Park & Lee ’04, Yang & Fox ’04 , Huang et al. ’05
Roth & Black ’05, Hirakawa & Parks ’05]• Image enhancement [Battiato et al. ’03]• Image/video hashing [Coskun & Sankur ’04, Hsu & Lu ‘04]• Image rendering [Bornik et al. ’03]• Image fusion [Zheng et al. ’04, Tsai ’04, Gonzalez-Audicana et al. ’05]• Texture reconstruction [Toth ’04] • Image halftoning [Evans & Monga ’03, Neelamani ‘03]• Radar imaging [Bentabet ’03]• Infrared imaging [Torres ’03, Pezoa et al. ‘04]• Ultrasound imaging [Loizou et al. ’04]• Vision processor design [Cembrano et al., ’04]• Wearable display design [von Waldkirch et al. ’04]• Contrast equalization for LCD [Iranli et al. ’05]• Airborne hyperspectral imaging [Christophe et al. ’05]• Superresolution for remote sensing [Rubert et al. ’05]
![Page 36: Image Quality Assessment: SSIM](https://reader031.vdocuments.mx/reader031/viewer/2022021206/620619fb8c2f7b1730049b5e/html5/thumbnails/36.jpg)
THE END
Thank you!