content based image retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · content based image...
TRANSCRIPT
![Page 1: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/1.jpg)
Content Based Image Content Based Image RetrievalRetrievalNatalia [email protected]
HP Labs Russia
© 2008 Hewlett-Packard Company
![Page 2: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/2.jpg)
Tutorial outlineTutorial outline• Lecture 1
− Introduction− Introduction
− Applications
• Lecture 2− Performance measurement− Performance measurement
− Visual perception
− Color features
• Lecture 3• Lecture 3− Texture features
− Shape features
− Fusion methods− Fusion methods
• Lecture 4− Segmentation
Local descriptors− Local descriptors
• Lecture 5− Multidimensional indexing
2/51
− Survey of existing systems
![Page 3: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/3.jpg)
Lecture 3Lecture 3
Texture featuresTexture features
Shape featuresShape features
Fusion methods
![Page 4: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/4.jpg)
Lecture 3: Outline
• Texture features
Lecture 3: Outline
• Texture features• Texture features
− Statistical
− Spectral
• Texture features
− Statistical
− Spectral− Spectral
− Comparison
− Spectral
− Comparison
• Shape features
− Boundary based
• Shape features
− Boundary based− Boundary based
− Region based
− Comparison
− Boundary based
− Region based
− Comparison− Comparison
• Fusion methods
− Comparison
• Fusion methods
4/51
![Page 5: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/5.jpg)
Texture featuresTexture features
• What is texture?• What is texture?
5/51
Smooth Rough Regular
![Page 6: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/6.jpg)
Texture featuresTexture features
Texture features
General statistics parameters
Haralick’s co-occurrence matrices
Statistical
PWT
Spectral
Model basedTamura features
Geometrical
TWT
DCT, DST, DHT
Complex wavelets
Gabor filters
ICA filters
Markov random fields
Fractals
Model-based
Voronoi tesselation features
Structural methods
GeometricalICA filters
6/51
![Page 7: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/7.jpg)
Texture featuresTexture features
• General statistics• General statistics
Based on intensity histogram of the whole image or its regions:
– histogram of intensity, L – number of intensity levels.– histogram of intensity, L – number of intensity levels.
– central moment of order n.
– average intensity.
– variance, is a measure of contrast.
, R=0 where intensity is equal., R=0 where intensity is equal.
– a measure of histogram assimetry.
7/51
– a measure of histogram assimetry.
![Page 8: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/8.jpg)
Texture featuresTexture features
• General statistics (2)• General statistics (2)
– a measure of contrast of homogeneity
(max for homogeneous areas ).(max for homogeneous areas ).
– entropy, a measure of variability
(0 for homogeneous areas ).(0 for homogeneous areas ).
Texture Average Deviation R µ3 U Entropy
SmoothSmooth
Rough
Regular
8/51
![Page 9: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/9.jpg)
Texture featuresTexture features
Grey Level Co-occurrence Matrices (GLCM):Grey Level Co-occurrence Matrices (GLCM):
GLCM - matrix of frequencies at which two pixels, separated by a certain vector, occur in the image.
∑∑ =∆+∆+=
=N M
otherwise
jyqxpIiqpIifjiC
0
1
,
),(,),(,),( ∑∑
= = =
p q otherwisejiC
1 1 0,),(
),( yx ∆∆ – separation vector;
I(p,q) – intensity of a pixel in position (p, q).
9/51
![Page 10: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/10.jpg)
GLCM – an exampleGLCM – an example
10/51
)0,1(),( =∆∆ yx
![Page 11: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/11.jpg)
GLCM – descriptorsGLCM – descriptors
Statistical parameters calculated from GLCM values:
∑∑= jiCEnergy ),(2
Statistical parameters calculated from GLCM values:
– is minimal when all elements are equal∑∑=i j
jiCEnergy ),(
∑∑−= jiCjiCEntropy ),(log),(
– is minimal when all elements are equal
– a measure of chaos, is maximal when all elements are equal
∑∑i j
∑∑ −=i j
jiCjiContrast ),()( 2
is maximal when all elements are equal
– has small values when big elements are near the main diagonal
∑∑i j
∑∑ −+=
i j ji
jiCMomentDifferenceInverse
2)(1
),(
near the main diagonal
−+i j ji )(1
– has small values when big elements are far from the main diagonal
11/51
![Page 12: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/12.jpg)
Texture features: Tamura featuresTexture features: Tamura features
Features, which are important for visual perception:Features, which are important for visual perception:
� Coarseness Tamura image:Coarseness
� Contrast
� Directionality
Coarseness-coNtrast-Directionality –
points in 3-D space CND
� Directionality
� Line-likeness
Regularity
� Euclidean distance in 3D (QBIC)
� 3D histogram (Mars)
Features:
� Regularity
� Roughness
� 3D histogram (Mars)
12/51
![Page 13: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/13.jpg)
Texture features: spectralTexture features: spectral
Texture features
General statistics parameters
Haralick’s co-occurrence matrices
Statistical
PWT
Spectral
Model basedTamura features
Geometrical
TWT
DCT, DST, DHT
Complex wavelets
Gabor filters
ICA filters
Markov random fields
Fractals
Model-based
Voronoi tesselation features
Structural methods
GeometricalICA filters
13/51
![Page 14: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/14.jpg)
Texture features: wavelet basedTexture features: wavelet basedWavelet analysis – decomposition of a signal:
)()( ,
,
xxf kj
kj
kψα∑=
Basis functions: Basis functions:
)()(,,
)2(2
2
2/
,
RLxkj
kxjj
kj
∈Ζ∈
−=
ϕ
ϕψ – scaling function
– mother wavelet
A set of basis functions – filters bank
Image
– mother wavelet
Filter 1
Image
Energy 1Filter 1
Filter 2
Filter N
Energy 1
Energy 2
Energy N
Feature vector
14/51
Filter N Energy N
![Page 15: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/15.jpg)
Texture features: Gabor filtersTexture features: Gabor filters
Mother wavelet: Gabor functionMother wavelet: Gabor function
+
+−
= jWxyx
yxg π21exp
1),(
22
+
+−
= jWx
yxyxg
yxyx
πσσσπσ
22
1exp
2
1),(
22
1,-S0,1,...,minteger,,,1),,(),( ==>′′= − nmayxgayxg m
mn
Filters bank:
),cossin(
),sincos(
1,-S0,1,...,minteger,,,1),,(),(
Θ+Θ−=′Θ+Θ=′
==>′′=
−
−
yxay
yxax
nmayxgayxg
m
m
mn
),cossin( Θ+Θ−=′ yxay
Kn /π=Θ)1/(1)/( −−= S
lh UUa
К – a number of directions, S – a number of scales,Uh, Ul – max and min of frequencies taken into consideration.
15/51
lh
![Page 16: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/16.jpg)
Texture features: ICA filtersTexture features: ICA filtersFilters are obtained using Independent Component Analysis
I1
dist(I1,I2) = KLH(H1i , H2i)Σi=1
N
H. Borgne, A. Guerin-Dugue, A. Antoniadis.
Representation of images for classification
with independent features. Pattern with independent features. Pattern
Recognition Letters, vol. 25, p. 141-154,
2004I2
…
N filters
16/51
N filters
![Page 17: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/17.jpg)
ICA FiltersICA Filters
17/51
![Page 18: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/18.jpg)
Lecture 3: Outline
• Texture features
Lecture 3: Outline
• Texture features
− Statistical
− Spectral− Spectral
− Comparison
• Shape features
− Boundary based− Boundary based
− Region based
− Comparison− Comparison
• Fusion methods
18/51
![Page 19: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/19.jpg)
Texture features: comparisonTexture features: comparison
In the context of In the context of image retrieval!
P. Howarth, S. Rüger. Robust texture features for still image retrieval.
19/51
P. Howarth, S. Rüger. Robust texture features for still image retrieval. In Proc. IEE Vis. Image Signal Processing, vol. 152, No. 6, December 2006
![Page 20: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/20.jpg)
Texture features: comparison (2)Texture features: comparison (2)
Gabor filters v. s. ICA filtersGabor filters v. s. ICA filters
Image classification task:
� Collection of angiographic images
� ICA filters performs better by 13%� ICA filters performs better by 13%
� Brodatz texture collection
� ICA filters perform better by 4%
Snitkowska, E. Kasprzak, W. Independent Component Analysis of Textures in Angiography Images. Computational Imaging and Vision, vol. 32, pages 367-372, 2006.
20/51
![Page 21: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/21.jpg)
Lecture 3: Outline
• Texture features
Lecture 3: Outline
• Texture features
− Statistical
− Spectral− Spectral
− Comparison
• Shape features
− Boundary based− Boundary based
− Region based
− Comparison− Comparison
• Fusion methods
21/51
![Page 22: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/22.jpg)
Shape featuresShape features
Shape featuresShape features
Boundary-based methodsRegion-based methods
Perimeter
EccentricityCentroid Distance
Complex Coordinates
Gometrical Signatures
Geometrical
Moment invariants
Global
Curvature
Axes directionality
Complex Coordinates
Curvature signature
Turning Angle
Area
Compactness
Euler number
Moment invariants
Zernike moments
Pseudo Zernike moments
Grid method
Others
Fourier Descriptors
UNL-Fourier
NFD
Wavelet Descriptors
Signature descriptors
Triangulation
Medial Axis Transform
(Skeleton Transform)
Decompasition
Chain codes
Others
Wavelet Descriptors
B-Splines
( )
Spectral descriptors
22/51
![Page 23: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/23.jpg)
Requirements to the shape featuresRequirements to the shape features
� Translation invariance
� Scale invariance� Scale invariance
� Rotational invariance� Rotational invariance
� Stability against small form changes
Low computation complexity� Low computation complexity
� Low comparison complexity� Low comparison complexity
23/51
![Page 24: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/24.jpg)
Boundary-based featuresBoundary-based features
Shape features
Boundary-based methodsRegion based methodsRegion-based methods
Geometrical
Signatures GeometricalGlobal
Signature desciptors
Geometrical
Decomposition Others
Fourier Descriptors
NFD
...
Chain codes
24/51
![Page 25: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/25.jpg)
Chain codesChain codesDirections for 4-connected and 8-connected chain codes:
A: 03001033332322121111 A: 03001033332322121111
B: 70016665533222
Example:Example:
0 0
0 03 1 3
31
10 0
1 6
62
2 7
Starting point invariance: minimal code
70016665533222 -> 00166655332227
22
22
3
3
31
1
1 6
5
53
3
270016665533222 -> 00166655332227
Rotation invariance: codes subtractiona) б) в)
A B
Rotation invariance: codes subtraction
00166655332227 -> 01500706070051
25/51
![Page 26: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/26.jpg)
Fourier descriptorsFourier descriptors1. Signature calculation (2D -> 1D):
� Centroid – contour distance� Centroid – contour distance
� Complex coordinates: z(t) = x(t) + iy(t)
� ...
2. Perform the discrete Fourier transform, take coefficients (s(t) – signature):
∑−
−=1
/2)(1 N
Nntjetsu π∑=
−=0
/2)(1
t
Nntj
n etsN
u π
3. Normalization (NFD – Normalized Fourier Descriptors):
0
1
0
2
0
1,...,,
u
u
u
u
u
u N−
2
12
)(∑ −=cN
n
J
n
I ffd
4. Comparison:
26/51
0
)(∑=
−=n
JI ffd
![Page 27: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/27.jpg)
Region-based featuresRegion-based features
Shape featuresShape features
Boundary-based methodsRegion-based methods
Geometrical
Signatures
Geometrical
Global
Signature representation
Moment invariants
Zernike moments
Pseudo Zernike moments
Grid method
DecompositionOthers
27/51
![Page 28: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/28.jpg)
Grid-methodGrid-methodА: 001111000 011111111 111111111 111111111 111110111 0111000011А
Б: 001100000 011100000 111100000 111101111 111111110 001111000Б
Invariance:
Normalization by major axe:
� direction;
� scale;� scale;
� position.
28/51
![Page 29: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/29.jpg)
Moment invariantsMoment invariantsThe moment of order (p+q) for a two-dimension continuous function:
∫∫= dxdyyxfyxm qp
pq ),(
Central moments for f(x,y) – discrete image:
mmqp ==−−=∑∑µ00
01
00
10 ,),,()()(m
my
m
mxyxfyyxx
x y
qp
pq ==−−=∑∑µ
Seven scale, translation and rotation invariant moments were derived based
Feature vector:
Seven scale, translation and rotation invariant moments were derived based
on central normalized moments of order p + q = 2; 3.
29/51
![Page 30: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/30.jpg)
Lecture 3: Outline
• Texture features
Lecture 3: Outline
• Texture features
− Statistical
− Spectral− Spectral
− Comparison
• Shape features
− Boundary based− Boundary based
− Region based
− Comparison− Comparison
• Fusion methods
30/51
![Page 31: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/31.jpg)
Shape features comparisonShape features comparison
Mehtre B. M., Kankanhalli M. S., Lee W. F. Shape measures for content based image retrieval: a comparison. Inf. Processing and Management, vol. 33, No. 3, pages 319-337, 1997.
31/51
comparison. Inf. Processing and Management, vol. 33, No. 3, pages 319-337, 1997.
![Page 32: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/32.jpg)
Lecture 3: Outline
• Texture features
Lecture 3: Outline
• Texture features
− Statistical
− Spectral− Spectral
− Comparison
• Shape features
− Boundary based− Boundary based
− Region based
− Comparison− Comparison
• Fusion methods
32/51
![Page 33: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/33.jpg)
Data fusion in CBIRData fusion in CBIR
color (2)annotations
texture shapecolor
� Combined search
fusion
� Combined search
(different features)
� Refine search results fusion� Refine search results
(different algorithms for the same
feature)
result� Supplement search results
(different datasets)
33/51
![Page 34: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/34.jpg)
Fusion of retrieval result sets
Fusion of weighted lists with ranked elements:
(x11, r
11), (x
12, r
12), … , (x1
n, r1n)ω1
(x21, r
21), (x
22, r
22), … , (x2
k, r2n)ω2
…?
(xm1, r
m1), (x
m2, r
m2), … , (xm
l, rm
l)ωm
Existing approaches in text retrieval:Existing approaches in text retrieval:
� CombMax, CombMin, CombSum
� CombAVG� CombAVG
� CombMNZ = CombSUM * number of nonzero similarities
� ProbFuse
34/51
� ProbFuse
� HSC3D
![Page 35: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/35.jpg)
Fusion function: propertiesFusion function: properties
1) Depend on both weight and rank1) Depend on both weight and rank
2) Symmetric
3) Monotony by weight and rank
},...,,max{},...,,min{)()()()0()()()( 2121 NN rrrrrrr
αααααα ≤≤
3) Monotony by weight and rank
4) MinMax condition /CombMin, CombMax, CombAVG/:
},...,,max{},...,,min{)()()()0()()()( 2121 NN
xxxxxxx rrrrrrrαααααα ≤≤
5) Additional property – “conic” property: non-linear dependency from 5) Additional property – “conic” property: non-linear dependency from
weight and rank; high weight, high rank – influence bigger to the
result than several inputs with low weight, low rank.
35/51
![Page 36: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/36.jpg)
Weighted Total with Gravitation Function
CombAVG as a base, but use gravitation function instead of weight:CombAVG as a base, but use gravitation function instead of weight:
where
36/51
![Page 37: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/37.jpg)
WTGF: some resultsWTGF: some results
• Experiments on search in semi annotated collections and • Experiments on search in semi annotated collections and of color and texture fusion (compare with CombMNZ)
• WTGF is good when:• WTGF is good when:
− There are a lot of viewpoints.
− Viewpoints are very different (different opinions regarding the rank − Viewpoints are very different (different opinions regarding the rank of the same element).
− Viewpoints have different reliability.
• CombMNZ is good when:
− Viewpoints have the same reliability.
Viewpoints have similar opinions.− Viewpoints have similar opinions.
Natalia Vassilieva, Alexander Dolnik, Ilya Markov. Image Retrieval. Combining multiple search
37/51
Natalia Vassilieva, Alexander Dolnik, Ilya Markov. Image Retrieval. Combining multiple search
methods’ results. In "Internet-mathematics" Collection, 46—55, 2007.
![Page 38: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/38.jpg)
Adaptive merge: color and textureAdaptive merge: color and texture
Dist(I, Q) = α*C(I, Q) + (1 - α)*Т(I, Q),Dist(I, Q) = α*C(I, Q) + (1 - α)*Т(I, Q),
C(I, Q) – color distance between I and Q;
T(I, Q) – texture distance between I and Q; T(I, Q) – texture distance between I and Q;
0 ≤ α ≤ 1
Hypothesis:
Optimal α depends on features of query Q. It is possible Optimal α depends on features of query Q. It is possible to distinguish common features for images that have the same “best” α.
Ilya Markov, Natalia Vassilieva, Alexander Yaremchuk. Image retrieval. Optimal weights for color
and texture fusion based on query object. In Proceedings of the Ninth National Russian Research
38/51
and texture fusion based on query object. In Proceedings of the Ninth National Russian Research Conference RCDL'2007
![Page 39: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/39.jpg)
Example: texture searchExample: texture search
39/51
![Page 40: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/40.jpg)
Example: color searchExample: color search
40/51
![Page 41: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/41.jpg)
Mixed metrics: semantic groupsMixed metrics: semantic groups
2a
1a
a41/51
3a
![Page 42: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/42.jpg)
Experimental results 1Experimental results 1
• It is possible to select the best value of a• It is possible to select the best value of a
0,6
0,4
0,5
Pre
cis
ion
0,2
0,3
полнота Кластер 6
Кластер 7
Кластер 8
Pre
cis
ion
Cluster 7
Cluster 6
Cluster 8
0,1
0,2
0
0 0,25 0,5 0,75 1
коэффициент смешанной метрики (а)Value of a
42/51
![Page 43: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/43.jpg)
Experimental results 2Experimental results 2
• Adaptive mixed-metrics increase precision • Adaptive mixed-metrics increase precision
43/51
![Page 44: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/44.jpg)
Adaptive merge: color and colorAdaptive merge: color and color1,2
0,8
1
0,6
0,8
Pre
cis
ion
0,2
0,4
0
0 1 2 3 4 5 6 7 8 9 10
Top N
Moments, Clouds Moments, Fields
HSL, 6*2*2, Clouds HSL, 6*2*2, Fields
44/51
![Page 45: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/45.jpg)
Adaptive merge: color and colorAdaptive merge: color and color
1,2
1
1,2
0,6
0,8
Pre
cis
ion
0,2
0,4
Pre
cis
ion
0
0 1 2 3 4 5 6 7 8 9 10
Top NTop N
Moments, People Moments, Bears
HSL, 6*2*2, People HSL, 6*2*2, Bears
45/51
![Page 46: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/46.jpg)
Color fusionColor fusion
CombMNZ (Moments + HSL histogram)CombMNZ (Moments + HSL histogram)
0,8
0,6
0,7
0,4
0,5
Moments
HSL, 6*2*2
Color mixed
0,2
0,3
Color mixed
0
0,1
46/51
0
0 1 2 3 4 5 6 7 8 9 10
![Page 47: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/47.jpg)
Ranked lists fusion: application area
� Search by textual query in semi annotated image � Search by textual query in semi annotated image collection
Textual query
…tr1by annotations
base
d
TextResult1, textrank1…tr2
conte
nt-
base
d
Result
TR2, tr2,
... …conte
nt
47/51
![Page 48: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/48.jpg)
Retrieve by text: fusion resultsRetrieve by text: fusion results
∑
⋅=Moverlap
xRMxR
)()(
)0(
α∑M
i
xR i )()(α
∑
⋅=M
i
overlap
xR
xRMxR
i )(
)()(
)(
)0(
α
⋅= xNMxN
)()(
)0(
Size of input lists ∑=
M
i
overlap
xN
xN
i )(
)()(α
48/51
![Page 49: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/49.jpg)
Lecture 3: ResumeLecture 3: Resume
• Texture features• Texture features
− Statistics (Haralik’s co-occurance matrices, Tamura features)
− Spectral features are more efficient (Gabor filters, ICA filters)− Spectral features are more efficient (Gabor filters, ICA filters)
• Shape features
− Boundary-based (Fourier descriptors)
− Region-based (Moment invariants)
• Fusion methods
Are very important− Are very important
− Need to choose based on a particular fusion task
49/51
![Page 50: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/50.jpg)
Lecture 3: BibliographyLecture 3: Bibliography
• Haralick R. M., Shanmugam K., Dienstein I. Textural features for image classification. In IEEE Transactions on Systems, Man and Cybernetics, classification. In IEEE Transactions on Systems, Man and Cybernetics, vol. 3(6), pp. 610 – 621, Nov. 1973.
• Tamura H., Mori S., Yamawaki T. Textural features corresponding to visual perception. In IEEE Transactions on Systems, Man and visual perception. In IEEE Transactions on Systems, Man and Cybernetics, vol. 8, pp. 460 – 472, 1978.
• Tuceryan M., Jain A. K. Texture analysis. The Handbook of Pattern Recognition and Computer Vision (2nd Edition), by C. H. Chen, L. F. Pau, Recognition and Computer Vision (2nd Edition), by C. H. Chen, L. F. Pau, P. S. P. Wang (eds.), pp. 207-248, World Scientific Publishing Co., 1998.
• Tuceryan M., Jain A. Texture segmentation using Voronoi polygons. In IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, No 2, pp. 211 – 216, February 1990.
• Walker R., Jackway P., Longstaff I. D. Improving co-occurrence matrix feature discrimination. In Proc. of DICTA’95, The 3rd Conference on feature discrimination. In Proc. of DICTA’95, The 3rd Conference on Digital Image Computing: Techniques and Applications, pp. 643 – 648, 6-8 December, 1995.
50/51
![Page 51: Content Based Image Retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · Content Based Image Retrieval Natalia Vassilieva ... − Survey of existing systems. Lecture 3 Texture](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f603164ba79135b5e2c0225/html5/thumbnails/51.jpg)
Lecture 3: BibliographyLecture 3: Bibliography• Li B., Ma S. D. On the relation between region and contour representation. In Proc.
of the IEEE International Conference on Pattern Recognition, vol. 1, pp. 352 – 355, of the IEEE International Conference on Pattern Recognition, vol. 1, pp. 352 – 355, 1994.
• Lin T.-W., Chou Y.-F. A Comparative Study of Zernike Moments for Image Retrieval. In Proc. of 16th IPPR Conference on Computer Vision, Graphics and Image Processing (CVGIP 2003), pp. 621 – 629, 2003.Image Processing (CVGIP 2003), pp. 621 – 629, 2003.
• Loncaric S. A survey of shape analysis techniques. In Pattern Recognition, vol. 31(8), pp. 983 – 1001, 1998.
• Luren Y., Fritz A. Fast computation of invariant geometric moments: A new method • Luren Y., Fritz A. Fast computation of invariant geometric moments: A new method giving correct results. In Proc. of IEEE International Conference on Image Processing, 1994.
• Zakaria M. F., Vroomen L. J., Zsombor-Murray P. J. A., van Kessel J. M. H. M. Fast algorithm for the computation of moment invariants. In Pattern Recognition, vol.
• Zakaria M. F., Vroomen L. J., Zsombor-Murray P. J. A., van Kessel J. M. H. M. Fast algorithm for the computation of moment invariants. In Pattern Recognition, vol. 20(6), pp. 639 – 643, 1987.
• Zernike polynomials. Wikipedia, the free encyclopedia. http://en.wikipedia.org/wiki/Zernike_polynomialshttp://en.wikipedia.org/wiki/Zernike_polynomials
• Zhang D., Lu G. Shape-based image retrieval using generic Fourier descriptor. In Signal Processing: Image Communication, vol. 17, pp. 825 – 848, 2002.
• Zhang D., Lu G. A Comparative Study on Shape Retrieval Using Fourier
51/51
• Zhang D., Lu G. A Comparative Study on Shape Retrieval Using Fourier Descriptors with Different Shape Signatures. In Proc. of the International Conference on Multimedia, 2001.