icip2016 image compression grand challenge
TRANSCRIPT
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
OVERVIEWANDBENCHMARKINGSUMMARYFORTHEICIP2016COMPRESSIONCHALLENGE
Evangelos Alexiou ,IreneViola,LukasKrasula,ThomasRichter,TimBruylants,AntonioPinheiro,KarelFliegel,MartinRerabek,Athanassios Skodras,PeterSchelkens andTouradjEbrahimi
AcontributionfromQualinet toJPEGcallforinformation
23rdInternationalConferenceonImageProcessingSeptember25– 28,2016,Phoenix,Arizona
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Background
• Callforinformationonstillimagecoding– IssuedbyJPEG committeeonFebruary2015– Broadscopenotonlylimitedtocompressionefficiency• Newimagingmodalities (morethan8-bit,HDR,…)• Features (scalability,randomaccess,…)• Characteristics (complexity,latency,…)
– AfirstresponseproducedduringPCS2015– Bothlossy andlossless
• ICIP2016FeatureEvent– EvaluationofcurrentandfutureImagecompressiontechnologies
• Thiscontributiononlyfocusesoncompressionefficiencyofconventionalimagesinlossy and losslesswithouttakingintoaccountothercriteria(features,complexity,delay,etc.)• Objective andsubjective evaluations inlossy casecarriedoutbyQualinet– VUB/iMinds (Belgium)– UBI(Portugal)– CTU(CzechRepublic)– UniversityStuttgart(Germany)– UniversityPatras(Greece)– EPFL(Switzerland)
• Lossless evaluations carriedoutbyUniversityofStuttgart(Germany)
2
http://www.jpeg.org
http://www.qualinet.eu
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Testmaterialinlossy evaluations• Contents:7 (1training+6test):– Resolutions - 800x1152or800x1280dependingoncontent– Subjectiveevaluationsoncroppedversionstofitdisplay– Objectivemetricsperformedonthecroppedversions
• Stimuli:– Originalimages– Compressed/decompressedimageswith10codecs• JPEG(default)• JPEG(PSNR)• JPEG (visual)• JPEG2000(PSNR)• JPEG2000(visual)• JPEGXR(444)• JPEGXR(420)• HEVC (SCCext.)• Daala• WebP
• 8bitratesforobjectivemetrics:– 0.25,0.5,0.75,1,1.25,1.5,1.75and2bpp
• 4bitratesforsubjectiveevaluations:– 0.25,0.5,0.75and1bpp or0.75,1,1.25and1.5bpp dependingoncontent
training
3
bike cafe honolulu
p08 p26 woman
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Testmaterialinlosslessevaluations 4
RGB,444,24bpp
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Croppedimagesusedinlossy evaluations
5
p08honolulu
bike cafe
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Croppedimagesusedinlossy evaluations
6
p26 woman
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Codecsusedinlosslessevaluations 7
1. JPEGXRLossless2. JPEG2000lossless3. FLIF (submittedtoICIP2016GrandChallenge)4. JPEGLSpart-1withoutcolortransformation5. JPEGLSpart-2withlosslesscolortransformation6. PNG7. WebP lossless8. JPEGXTpart8losslesswithresidualcoding9. JPEGXTpart8losslesswitharithmeticcoding10. JPEGXTpart8withlosslessDCT11. JPEG-1lossless12. JPEG-1losslesswitharithmeticcoding13. JPEGXTpart8,residualcodingwithoptimizedHuffmancoding,progressivemode14. JPEGXTpart8,losslessDCTwithoptimizedHuffmancoding,progressivemode.15. HierarchicalJPEGwiththeinitialpassaDCTcodingandthesecondlevelofthepyramidlosslesscodingand
optimizedHuffmancoding16. Sameas15,butwitharithmeticcodinginsteadofHuffmancoding17. Sameas15,exceptthatthefirstlevelofthepyramidisadownscaled¼andprocessedbytheDCT.compressed
withDCT,thenupscaled,andtheresidualiscompressedbythepredictivemodeofJPEG.18. Sameas17,butwitharithmeticcoding.
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Codecsusedinlossy evaluations
• JPEG(default):IJGimplementationofJPEGwithdefaultoptions(i.e.AnnexKquantizationsettings,420colorsubsampling,baselineconfiguration) – “cjpeg” withoutfurtheroptions.
• JPEG(PSNR):JPEGXTreferencesoftwareusingJPEGXTpart1baselinecompressorcompatiblewith10918-1.PNSRoptimizedversion:-oz-h-qt 1–v– Enabling:444subsampling,flatquantization matrix,optimized Huffman coding,progressive scan order.
• JPEG(visual): JPEGXTreferencesoftwareusingJPEGXTpart1baselinecompressorcompatiblewith10918-1.Visuallyoptimizedversion: -oz-h-qt 3-v-s1x1,2x2,2x2– Enabling: 420subsampling,ImageMagick quantization matrix,optimized Huffman coding,progressivecoding.
8
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Codecsusedinlossy evaluations
• JPEG2000(PSNR):JPEG2000compressorusingAccusoft softwarewiththefollowingcommandlineoptions: -lo -as-cn 1– Enabling: strict rateallocator,5decomposition levels,one layer,noprecincts,notiles. lossy 5/3and not9/7wavelet.
• JPEG2000(visual):JPEG2000compressorusingAccusoft software.Visuallyoptimizedversion:-lo -as-cn 1-w 1000– Enabling: features identical to JPEG2000(PSNR),but with visualweighting.
• JPEGXR(444):JPEGXRreferencesoftwarewiththefollowingoptions: -fYUV444-l1–d– Enabling: 444chromasubsampling,oneleveloverlap,derivedchromaquantization.
• JPEGXR(420):JPEGXRreferencesoftwarewiththefollowingoptions:-fYUV420-l2–d– Enabling: 420chromasubsampling,twoleveloverlap,derivedchromaquantization.
9
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Codecsusedinlossy evaluations
• HEVC(SCCext.):HEVCwithScreenContentCodingextensionconfiguredtothemainintra-profilefor420chroma input,withthefollowingcommandlineoptions: -ccfg/encoder_intra_main_scc.cfg --InputChromaFormat=420--ProgressiveSource --FrameOnly -cf 420--FrameRate=30--FramesToBeEncoded=1--QuadtreeTULog2MaxSize=5--GOPSize=1--IntraPeriod=1--ConformanceWindowMode=1--AdaptiveQP=1--RateControl=0--TransquantBypassEnable=1--CrossComponentPrediction=0--ColourTransform=0
• Daala:mozilla Daala withdefaultconfiguration.Nofurtheroptions.
•WebP:GoogleWebP (partofWebM)withthefollowingcommandlineoptions:-m6-partition_limit 50-af
10
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Subjectiveevaluationmethodology
• SubjectiveevaluationmethodologybasedonITU-TP.910• ACR-HR:AbsoluteCategoryRatingwithHiddenReference• Randomization ofpresentationorder• 5-level discretescale:bad,poor,fair,good,excellent• 10codecstestedfortheirsubjectivequality–10(codecs)x6(images)x4(bitrates)+6(originals)=246stimuli
• 21naïvesubjects participatedinVUB,UBIandEPFL labs• Eachsubject completed 3sessions of80stimuli(circa15minpersession,30minbreak)• Shorttraining forbad,fairandexcellentqualityillustrations• Display:AppleMacBookProRetina15inchorequivalent• Typicalofficeenvironment
11
Stimulus1
Vote1
Stimulus80
Vote80
time
Training
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Objectiveevaluationmetrics
• PSNR–Widelyusedqualitymetricinimageprocessingcommunity.–PerformedforbothYchannelandRGB.
• SSIM:StructuralSimilarityIndex–Meanofsimilaritybetweenanimageundertestanditsreferencebasedonstructuralinformation.
•MSSIM:MultiscaleStructuralSimilarityIndex–MultiscaleversionofSSIM.
• FSIM:FeatureSimilarityIndex–BasedonSSIM.–Addsacomparisonoflow-levelfeaturesetsbetweenthereferenceandthedistortedimages.– analyzesthehighphasecongruencyextractinghighlyinformativefeaturesandthegradientmagnitude,toencodethecontrastinformation.– ThisanalysisiscomplementaryandreflectsdifferentaspectsoftheHVSinassessingthelocalqualityofanimage.–PerformedforbothYandCchannels.
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Objectiveevaluationmetrics
• HDR-VDP2.2: HighDynamicRangeVisibleDifferencePredictor–CalibratedmetricdevelopedforHDRimages–Considersalight-adaptivecontrastsensitivityfunction,astherangesoflightadaptationcanvarysubstantially.– Includesaspecificmodelofthepointspreadfunction(PSF)oftheeyeoptics,ashumanopticallensflarecanbeverystronginhighcontrastHDRcontent.– Thefront-endamplitudenon-linearityisbasedonintegrationoftheWeber-Fechnerlaw.– Takesintoaccounttheangularresolution.–Usesamulti-scaledecomposition.–Aneuralnoiseblockisdefinedtocalculateper-pixelprobabilitiesmapsofvisibilityandthepredictedqualitymetric.
• CIEDE2000:Colordifferencemetric– Includesweightingfactorsforlightness,chroma,andhue(liketheCIE1976L*a*b*perceptualspace).–Alsoincludesfactorstohandletherelationshipbetweenchroma andhue.
• VIF:VisualInformationFidelity–Analysesthenaturalscenestatistics.–UsesanimagedegradationmodelandtheHVSmodel.–BasedonthequantificationoftheShannoninformationpresentinboththereferenceandthedistortedimages.
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
14
Thankyouforyourattention!Resultsofevaluationswillbepresentedattheendofthissession!!!
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
SUMMARYANDNEXTSTEPSFORTHEIMAGECOMPRESSIONCHALLENGE
Evangelos Alexiou,IreneViola,LukasKrasula,ThomasRichter,TimBruylants,AntonioPinheiro,Karel Fliegel,MartinRerabek,Athanassios Skodras,PeterSchelkens andTouradjEbrahimi
AcontributionfromQualinet toJPEGcallforinformation
23rdInternationalConferenceonImageProcessingSeptember25– 28,2016,Phoenix,Arizona
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Overview
• ICIP2016FeatureEvent– EvaluationofcurrentandfutureImagecompressiontechnologies
• 7Contents usedinlossy evaluations(1training+6tests)– Resolutions800x1152or800x1280
• 10codecs testedinlossy evaluations(anchorsandproponents)• 8 bitrates forobjectivemetrics– 0.25,0.5.0.75,1,1.25,1.5,1.75,2bpp
• 4bitrates forsubjectiveevaluations– 0.25,0.5,0.75,1bpp or0.75,1,1.25and1.5bpp
• SubjectiveassessmentprotocolbasedonITU-TP.910forlossy evaluations– ACR-HR:AbsoluteCategoryRatingwithHiddenReference– 21naïvesubjects inVUB,UBIandEPFL– 3sessionsof80stimuli– 5-level discretescale–Outliersdetectionbasedon boxplotalgorithm– Display: AppleMacBookProRetina15inch– Typicalofficeenvironment
• Objectiveevaluation inlossy caseusing9metrics– Cross-checked between CTU,UniversityStuttgartandUniversityPatras
16
http://www.jpeg.org
http://www.qualinet.eu
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Testimagesinlossy evaluations 17
training
bike cafe honolulu
p08 p26 woman
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Objectiveevaluation:PSNRY results18
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Objectiveevaluation:PSNRRGB results19
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Objectiveevaluation:SSIMresults 20
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Objectiveevaluation:MSSIMresults 21
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Objectiveevaluation:FSIMY results22
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Objectiveevaluation:FSIMC results23
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Objectiveevaluation:HDR-VDP-2.2results 24
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Objectiveevaluation:CIEDE2000results 25
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Objectiveevaluation:VIFresults 26
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Outlierdetection
• StatisticalanalysisusingBoxplot method.
• Foreachtestcondition,anoutlier isdefinedasadatapointthatislocatedoutsidetheinterquartilerange i.e.abovetheupperquartileorbelowthelowerquartileofthedistributionofthescoresmultipliedby1.5.
• Ifthesamesubjectisidentifiedasoutlierinmorethan20% ofthetestconditions,thecorrespondingscoresarediscarded.
• Nooutlierswerefound.
27
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Subjectiveevaluationresults 28
bike.ppm
0 0.5 1 1.5bitrate [bpp]
1
1.5
2
2.5
3
3.5
4
4.5
5
MO
S
JPEG (default)HEVC (SCC ext.)JP2K (PSNR)JP2K (visual)DaalaJPEG XR (444)WebPJPEG (PSNR)JPEG (visual)JPEG XR (420)
cafe.ppm
0.5 1 1.5 2bitrate [bpp]
1
1.5
2
2.5
3
3.5
4
4.5
5
MO
S
JPEG (default)HEVC (SCC ext.)JP2K (PSNR)JP2K (visual)DaalaJPEG XR (444)WebPJPEG (PSNR)JPEG (visual)JPEG XR (420)
bike cafe
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Subjectiveevaluationresults 29
honolulu.ppm
0 0.5 1 1.5bitrate [bpp]
1
1.5
2
2.5
3
3.5
4
4.5
5
MO
S
JPEG (default)HEVC (SCC ext.)JP2K (PSNR)JP2K (visual)DaalaJPEG XR (444)WebPJPEG (PSNR)JPEG (visual)JPEG XR (420)
p08.ppm
0 0.5 1 1.5bitrate [bpp]
1
1.5
2
2.5
3
3.5
4
4.5
5
MO
S
JPEG (default)HEVC (SCC ext.)JP2K (PSNR)JP2K (visual)DaalaJPEG XR (444)WebPJPEG (PSNR)JPEG (visual)JPEG XR (420)
honolulu p08
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Subjectiveevaluationresults 30
p26.ppm
0 0.5 1 1.5bitrate [bpp]
1
1.5
2
2.5
3
3.5
4
4.5
5
MO
S
JPEG (default)HEVC (SCC ext.)JP2K (PSNR)JP2K (visual)DaalaJPEG XR (444)WebPJPEG (PSNR)JPEG (visual)JPEG XR (420)
woman.ppm
0 0.5 1 1.5bitrate [bpp]
1
1.5
2
2.5
3
3.5
4
4.5
5
MO
S
JPEG (default)HEVC (SCC ext.)JP2K (PSNR)JP2K (visual)DaalaJPEG XR (444)WebPJPEG (PSNR)JPEG (visual)JPEG XR (420)
p26 woman
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Statisticalsignificancetest
•One-tailedWelch’st-testwithnullhypothesis at5%significancelevel.• andareMeanOpinionScoresforaspecificcontentcompressedataspecificbitratewithcodecsand•Whenthenullhypothesisisrejected,thealternativehypothesisindicatesthat,accordingtoMOS,thefirstcodecissignificantlybetteratthe5%level.• Thesetestsareperformedconsideringallcombinationsofcodecs:10x10matrix.• Foreachpairofcodecs,wesumuptheresultsforeverycontentatagivenqualitylevel:scale0-6.•Onematrixforeachqualitylevel:4figures.
31
H0 :m1 ≤m2
m1 m2c2c1
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Codecassessmentbasedonsubjectivetests 32
• Bitrate B1 correspondstothelowestquality• Bitrate B4 correspondstothehighestquality
Significant difference matrix for bitrate B1
JPEG (default)
HEVC (SCC ext.)
JP2K (PSNR)
JP2K (visual)
Daala
JPEG XR (444)
WebP
JPEG (PSNR)
JPEG (visual)
JPEG XR (420)
JPEG
(def
ault)
HEV
C (S
CC
ext
.)
JP2K
(PSN
R)
JP2K
(vis
ual)
Daa
la
JPEG
XR
(444
)
Web
P
JPEG
(PSN
R)
JPEG
(vis
ual)
JPEG
XR
(420
) 0
1
2
3
4
5
6Significant difference matrix for bitrate B2
JPEG (default)
HEVC (SCC ext.)
JP2K (PSNR)
JP2K (visual)
Daala
JPEG XR (444)
WebP
JPEG (PSNR)
JPEG (visual)
JPEG XR (420)
JPEG
(def
ault)
HEV
C (S
CC
ext
.)
JP2K
(PSN
R)
JP2K
(vis
ual)
Daa
la
JPEG
XR
(444
)
Web
P
JPEG
(PSN
R)
JPEG
(vis
ual)
JPEG
XR
(420
) 0
1
2
3
4
5
6
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Codecassessmentbasedonsubjectivetests 33
• Bitrate B1 correspondstothelowestquality• Bitrate B4 correspondstothehighestquality
Significant difference matrix for bitrate B3
JPEG (default)
HEVC (SCC ext.)
JP2K (PSNR)
JP2K (visual)
Daala
JPEG XR (444)
WebP
JPEG (PSNR)
JPEG (visual)
JPEG XR (420)
JPEG
(def
ault)
HEV
C (S
CC
ext
.)
JP2K
(PSN
R)
JP2K
(vis
ual)
Daa
la
JPEG
XR
(444
)
Web
P
JPEG
(PSN
R)
JPEG
(vis
ual)
JPEG
XR
(420
) 0
1
2
3
4
5
6Significant difference matrix for bitrate B4
JPEG (default)
HEVC (SCC ext.)
JP2K (PSNR)
JP2K (visual)
Daala
JPEG XR (444)
WebP
JPEG (PSNR)
JPEG (visual)
JPEG XR (420)
JPEG
(def
ault)
HEV
C (S
CC
ext
.)
JP2K
(PSN
R)
JP2K
(vis
ual)
Daa
la
JPEG
XR
(444
)
Web
P
JPEG
(PSN
R)
JPEG
(vis
ual)
JPEG
XR
(420
) 0
1
2
3
4
5
6
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Testmaterialinlosslessevaluations 34
RGB,444,24bpp
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Losslessevaluationresults 35
6
8
10
12
14
16
18
Lossless compression
bpp
JPEG-XR
JPEG-2000 FL
IF
JPEG-LS
JPEG-LS2PNGWebP
JPEG-LL
JPEG-LLA
JPG-SRM
JPG-PRED
JPG-ACPRD
JPG-ACP
JPG-ACPLJPG-H
JPG-ACH
JPG-H1
JPG-ACH1
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Conclusions 36
• HEVC andDaala oftenoutperformothercodecsinbothobjectivemetricsandsubjectiveevaluationsinlossy case–Daala performs bestinimagescontainingfaces
• JPEG,JPEG2000and JPEGXRperformwellin higherbitratesbasedonPSNRRGB metricinsometestedimagesinlossy case
• JPEG2000(PSNR)exhibitsgoodcolorrenditionbasedonCIEDE2000metricinseveraltestedimagesinlossy case
• JPEG(visual)performwellinhigherbitratesinlossy case
• FLIF inaverageproducesbestlosslesscompression performancewhencomparedtoallalternativestestedfortheimagestested
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
Nextsteps
• Thereisevidencethatsignificantimprovementsincompressionefficiencycanbeobtainedusinglateststateoftheartinlossy andlossless cases
• Furtherevaluationsareneededtobetterquantifythecostofsuchhigherefficiencyincompressionintermsofrequiredresourcesandotherfeatures(delay,etc.)
• Conclusionsregardingcompressionefficiencyneedtobeverifiedusingalargerdatasetandthroughmoreextensiveevaluationscampaignssuchascrowdsourcing
• Keepinmindtheseresultscompareencodersandnotcodingalgorithms(inparticulardecoders!)
•Manyofthealgorithmsundertestareindevelopmentandtheirperformancecanstillimprove
37
23rd InternationalConferenceonImageProcessing,September25–28,2016,Phoenix,Arizona
38
Thankyouforyourattention!