multiple reflection symmetry detection via linear-directional kernel density estimation

IntroductionRelated WorkMethodology

Results and Discussion

Multiple Reflection Symmetry Detection viaLinear-Directional Kernel Density Estimation

M. Elawady1, O. Alata1, C. Ducottet1, C. Barat1, P. Colantoni2

1Universite de Lyon, CNRS, UMR 5516, Laboratoire Hubert Curien,Universite de Saint-Etienne, Jean-Monnet, F-42000 Saint-Etienne, France

2Universite Jean Monnet, CIEREC EA n0 3068, Saint-Etienne, France

17th International Conference on Computer Analysis of Images and Patterns

UMR • CNRS • 5516 • SAINT-ETIENNE

M. Elawady, O. Alata, C. Ducottet, C. Barat, P. Colantoni Hubert Curien Laboratory, Jean Monnet University, France 1 / 33



Table of Contents

1 IntroductionBackgroundApplicationsProblem Definition

2 Related WorkIntensity-based MethodsEdge-based Methods

3 MethodologyMotivationAlgorithm Details

4 Results and Discussion




BackgroundApplicationsProblem Definition

Table of Contents









Bilateral Symmetry

1Image from book: The Photographer’s Eye by Michael Freeman





Bilateral Symmetry in Computer Vision

Medial Image Segmentation [1]

Aerial-based vehicle detection [2]





Detection of Global Symmetries

Axis Legend: Strong, Weak




Intensity-based MethodsEdge-based Methods

Table of Contents









Baseline (Loy 2006) and its Successor (Mo 2011)

The general scheme (Loy and Eklundh 2006 [3]) consists of:

Disadvantages:

Depending mainly on the properties of hand-crafted features (i.e. SIFT).

For example: (smooth objects with noisy background)little feature points =⇒ lost symmetry.

(Mo and Draper 2011 [4]) proposed refinements in the general scheme.





First Work (Cic 2014)

Instead of SIFT, the general idea (Cicconet et al. 2014 [5]) is extracting aregular set of wavelet segments with local edge amplitude and orientation.

Disadvantages:

Lacking neighborhood’s information inside the feature representation.

Depending on the scale parameter of the edge detector.

For example: (high texture objects with noisy background)inferior symmetrical info =⇒ incorrect symmetry.





State-of-Art (Ela 2016)

(Single Symmetry) Investigating Cicconet’s edge features [5] within Loy’sscheme [3] by adding neighboring-pixel information.

(1) Mul�scale Edge Segment Extrac�on

(2) Triangula�on based on Local Symmetry Weights:

• Geometry Edge Orienta�ons (Cic)• Local Texture Histogram (Loy)

(3) Vo�ng Space for Peak Detec�on with Handling Orienta�on Discon�nuity.

θ

ρ0

π

Legend: Groundtruth, Our2016, Loy2006, Mo2011, Cic2014




MotivationAlgorithm Details

Table of Contents









Proposed Idea





Symmetry Detection Algorithm: Main Steps

Input Image

d) Symmetry Selection

a) Feature ExtractionScale 1 Scale S

Mag

nit

ud

eO

rien

tati

on

His

togr

am

Point 1 Point 2 Point P

c) Kernel Density Estimator

b) Feature TriangulationEdge Magnitude

Symmetry Coefficient

Neighborhood Texture





Multiscale Edge Segment Extraction I

Input Image

Scale 1

Scale 2

Scale S

Orientation 1

Orientation O/2

Orientation O

Amplitude Map

Orientation Map





Multiscale Edge Segment Extraction II

A feature point pi and its local edge characteristics (J i , φi ) are extractedwithin each cell using a Morlet wavelet ψk,σ of constant scale σ andvarying orientation {φo , o = 1 . . .O}.

Amplitude Map Orientation Map

MaxAmplitude

CorrespondingOrientation





Multiscale Edge Segment Extraction III

Neighboring textural histogram hi of size B:

hi (b) =∑

r∈D(pi )

J rδφb−φr , φb ∈ {bπ

B, b = 0 . . .B − 1, 8 ≤ B ≤ 32} (1)

where hi is l1 normalized and circular shifted respect to the maximummagnitude J i among the neighborhood window D(pi ).

0 36 72 108 1440

0.5

1

1.5

2

2.5

3

3.5

4

4.5#106

Magnitude Histogram

108 144 0 36 720

0.1

0.2

0.3

0.4

0.5

0.6

Histogram Count (hi)

0 36 72 108 1440

500

1000

1500

2000

2500

3000

Frequency Histogram





Symmetry Triangulation I

(Textural Information) Symmetry degree of the two regions around i andj can be measured by comparing their corresponding local orientationhistograms hi and hj (reverse histogram of hj). Texture-based symmetrymeasure is given by:

d(i , j) =B∑

b=1

min(hi (b), hj(b)) (2)

108 144 0 36 720

0.1

0.2

0.3

0.4

0.5

0.6

Histogram Count (hp)

72 36 0 144 1080

0.1

0.2

0.3

0.4

0.5

0.6

Histogram Count (hq*)

1 2 3 4 50

0.1

0.2

0.3

0.4

0.5

0.6

Histogram Intersection





Symmetry Triangulation II

(Edge Information) Semi-dense edge magnitude m(i , j) and mirrorsymmetry coefficient c(i , j) {similar to cosine distance} are defined as [5]:

m(i , j) = J iJ j (3)

c(i , j) = |τ iS(T⊥ij )τ j | (4)

where τ i = [cos(φi ), sin(φi )], and S(.) is a reflection matrix.





Symmetry Triangulation III

The candidate axis is parametrized by angle θn, and displacement ρn andhas pairwise symmetry weight ωn = ωi,j (i 6= j and σi = σj) is defined as:

ωn = ω(pi , pj) = m(i , j) c(i , j) d(i , j) (5)





Kernel Density Estimation I

(Linear: Displacement) Linear kernel density estimator fl(.) is defined as

fl(x ; g) =1

Ng

N∑n=1

G(x − ρn

g) | G(u) =

1

(2π)12

e−12|u|2 (6)

where G(.) is a Gaussian kernel with bandwidth parameter g .

(Directional: Angle) Directional kernel density estimator fd(.) is defined as:

fd(y ; k) =C(k)

N

N∑n=1

L(yTµn; k) | L(x ; k) = ekx , C(k) =1

2πS(0, k)(7)

where L(.) is a von-Mises Fisher kernel with concentration parameter k, and

normalization constant C(k). S(.) is the modified Bessel function of the first

kind.





Kernel Density Estimation II

Thanks for the linear-directional density estimator fl,d(.) [6]. We define

the extended weighted version fl,d(.) as:

fl,d(x , y ; g , k) =C (k)

Ng

N∑n=1

ωnG (x − ρn

g)L(yTµn; k) (8)

y = [cos(θ), sin(θ)], µn = [cos(θn), sin(θn)]

assuming that linear and directional data are independent resulting dotproduct between accompanying kernels.

B

A2 A1 A3B

B

A2 A1 A3





Kernel Density Estimation III

B

A2 A1 A3

Input Image

A2

A1

A3

Linear KDE fl (.)

[A1,A2,A3]B B

Directional KDE fd (.)

Edge Magnitude mn(.) Symmetry Coefficient cn(.) Neighborhood Texture dn(.)




Table of Contents








Evaluation Details

Datasets:PSUm dataset: Liu’s vision group proposed a symmetry groundtruth[7, 8] for Flickr images (# images = 142, # symmetries = 479) inECCV2010, CVPR2011 and CVPR2013.

NYm dataset: Cicconet et al. [9] presented a new symmetrydatabase (# images = 63, # symmetries = 188) in 2016.

Evaluation Metrics:True Positive [8]:

ang(SC ,GT ) < 10◦ (9)

dist(CenSC ,CenGT ) < 20%×min(LenSC , LenGT ) (10)

Precision, Recall, and Maximum F1 Score




Quantitative Results I

(Precision-Recall Curves)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1Loy2006 (0.29211)Cicconet2014 (0.15883)Elawady2016 (0.27744)Our2017 (0.32828)

PSUm (2010-2013)

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1Loy2006 (0.33657)Cicconet2014 (0.2365)Elawady2016 (0.38788)Our2017 (0.43373)

NYm (2016)

X-axis: Recall, Y-axis: Precision




Quantitative Results II

(Statistical Comparison)

Max F1 Score and its equivalent Precision and Recall rates




Qualitative Results I - PSUmColumns: (1) GT, (2) Our, (3) Ela2016 [10], and (4) Loy2006 [3]

Top 5 detections: red, yellow, green, blue, and magenta.




Qualitative Results II - NYmColumns: (1) GT, (2) Our, (3) Ela2016 [10], and (4) Loy2006 [3]

Top 5 detections: red, yellow, green, blue, and magenta.




Conclusion

Summary:1 A weighted joint density estimator is proposed to handle both orientation and

displacement information.

2 A reliable detection framework is developed for global multiple symmetries.

Future work:1 The proposed detection can be improved using a continuous maximal-seeking

technique to avoid over-extended axes.

2 Entropy-based balance measure can be introduced to describe the existence anddegree of global axes inside an image.

3 Possibility of integration within retrieval systems for artistic photographs andpaintings in museums




References I

[1] F. Abdolali, R. A. Zoroofi, Y. Otake, and Y. Sato, “Automatic segmentation ofmaxillofacial cysts in cone beam ct images,” Computers in biology and medicine,vol. 72, pp. 108–119, 2016.

[2] S. Ram and J. J. Rodriguez, “Vehicle detection in aerial images using multiscalestructure enhancement and symmetry,” in Image Processing (ICIP), 2016 IEEEInternational Conference on, pp. 3817–3821, IEEE, 2016.

[3] G. Loy and J.-O. Eklundh, “Detecting symmetry and symmetric constellations offeatures,” in Computer Vision–ECCV 2006, pp. 508–521, Springer, 2006.

[4] Q. Mo and B. Draper, “Detecting bilateral symmetry with feature mirroring,” inCVPR 2011 Workshop on Symmetry Detection from Real World Images, 2011.

[5] M. Cicconet, D. Geiger, K. C. Gunsalus, and M. Werman, “Mirror symmetryhistograms for capturing geometric properties in images,” in Computer Visionand Pattern Recognition (CVPR), 2014 IEEE Conference on, pp. 2981–2986,IEEE, 2014.




References II

[6] E. Garcıa-Portugues, R. M. Crujeiras, and W. Gonzalez-Manteiga, “Kerneldensity estimation for directional–linear data,” Journal of Multivariate Analysis,vol. 121, pp. 152–175, 2013.

[7] I. Rauschert, K. Brocklehurst, S. Kashyap, J. Liu, and Y. Liu, “First symmetrydetection competition: Summary and results,” tech. rep., Technical ReportCSE11-012, Department of Computer Science and Engineering, ThePennsylvania State University, 2011.

[8] J. Liu, G. Slota, G. Zheng, Z. Wu, M. Park, S. Lee, I. Rauschert, and Y. Liu,“Symmetry detection from realworld images competition 2013: Summary andresults,” in Computer Vision and Pattern Recognition Workshops (CVPRW),2013 IEEE Conference on, pp. 200–205, IEEE, 2013.

[9] M. Cicconet, V. Birodkar, M. Lund, M. Werman, and D. Geiger, “A convolutionalapproach to reflection symmetry,” Pattern Recognition Letters, 2017.

[10] M. Elawady, C. Barat, C. Ducottet, and P. Colantoni, “Global bilateral symmetrydetection using multiscale mirror histograms,” in International Conference onAdvanced Concepts for Intelligent Vision Systems, pp. 14–24, Springer, 2016.




Questions?




Appendix - Why Feature Normalization?

The feature points are normalized with keeping aspect ratio as following:

pi =pi − cW ,H

max(W ,H)(11)

where cW ,H represents the original image center (W2

, H2

).

Without Normalization With Normalization

Voting: unified space and independent parameters
