texture analysis and segmentationanalysis and segmentation...
TRANSCRIPT
![Page 1: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/1.jpg)
Texture Analysis and SegmentationTexture Analysis and Segmentation using Modulation Models
Department of Mathematics, UCLAI P i S iImage Processing Seminar
Iasonas KokkinosDepartment of Statistics, UCLA
Joint work with Georgios Evangelopoulos and Petros Maragos,
Department of ECE, National Technical University of Athens, Greece
![Page 2: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/2.jpg)
Presentation Outline
Amplitude Modulation- Frequency Modulation (AM-FM) models
2-D AM-FM Model2-D AM-FM ModelEnergy Separation Algorithm, Regularized DemodulationDominant Component Analysis (DCA)
Filtering and modelling
Model-based interpretation of Gabor filtering Model based interpretation of Gabor filtering Alternative models for edge and smooth signalsTexture / edge / smooth classification via model comparison
Applications to Segmentation
Variational Image Segmentation using AM-FM featuresWeighted Curve Evolution for cue combination
![Page 3: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/3.jpg)
1-D AM-FM Models
AM FM AM-FM
Applications: Telecommunications, Speech Analysis ...
![Page 4: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/4.jpg)
2-D AM-FM models
Monocomponent AM-FM signal
Multicomponent AM-FM signals
= +
![Page 5: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/5.jpg)
AM-FM models for Natural Images
Man-made structures
Results of natural processes
![Page 6: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/6.jpg)
AM-FM Demodulation: Energy Separation Algorithm
Given, recover s.t.Assume bandpass modulating signalsTeager-Kaiser Energy Operator:
Energy Separation Algorithm:
Compared with Hilbert transform: locality
Refs. 1-D: Maragos, Quattieri & Kaiser, IEEE TSP ‘92, 2-D: Maragos & Bovik, JOSA ‘95
![Page 7: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/7.jpg)
Natural Image Demodulation
Problems:Natural images do not satisfy ESA assumptionsNatural images do not satisfy ESA assumptionsDecomposition into AM-FM components: ill-posed problemEffects of noise and approximations of derivatives
G b filt i l tiGabor filtering solution:Break signal into simple components by Gabor filtering
ertic
al F
requ
ency
Fourier transform
isocurves
Horizontal Frequency
Verisocurves
Demodulate individual outputsUse derivative-of-Gabor filters to avoid differentiation
![Page 8: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/8.jpg)
Channelized & Dominant Component AnalysisHavlicek & Bovik IEEE TIP ’00Havlicek & Bovik, IEEE TIP 00
DCA:
![Page 9: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/9.jpg)
DCA reconstruction of textured signals
![Page 10: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/10.jpg)
Presentation Outline
Amplitude Modulation- Frequency Modulation (AM-FM) models
2-D AM-FM Model2-D AM-FM ModelEnergy Separation Algorithm, Regularized DemodulationDominant Component Analysis (DCA)
Filtering and Modelling
Model-based interpretation of Gabor filtering Model based interpretation of Gabor filtering Alternative models for edge and smooth signalsTexture / edge / smooth classification via model comparison
Applications to Segmentation
Variational Image Segmentation using AM-FM featuresWeighted Curve Evolution for cue combination
![Page 11: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/11.jpg)
Motivation: deciding when to trust texture features
Input Image DCA Features
Model-based approachDetermine where the model fits the image wellWell = better than alternatives: Bayesian approach
`Special treatment’ for textured regions:F lk Shi & M lik N li d C t f S t tiFowlkes, Shi & Malik, Normalized Cuts for Segmentation Meyer, Vese, Osher, U+V decompositionGuo, Wu, Zhu, Texture + Sketch for reconstruction
![Page 12: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/12.jpg)
Bayesian approach
Synthesis model for each class
Adopt probabilistic error model I t t t t t b ti lik lih d i lIntegrate out parameters to express observation likelihood given class
Derive class posterior using Bayes’ rule
![Page 13: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/13.jpg)
Model 1 D profile along principal orientation:
Texture Model: sinusoidModel 1-D profile along principal orientation:
Rewrite as expansion on linear basis:
Typical Matched filtering: Project signal on sine/cosine basis (convolution with sine/cosine filters)
Gabor filtering: Filters have falloff (local analysis)
![Page 14: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/14.jpg)
Probabilistic formulation of locality
Leave distant data for a background model
b ti t i tobservation at point xmodel-based predictionprobability that observationi d t f d d lis due to foreground model
![Page 15: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/15.jpg)
Lower bound of likelihood
Likelihood for independent errors
White Gaussian noise: weighted least squares
![Page 16: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/16.jpg)
Gabor filtering as a weighted projection on a linear basis
Rewrite lower bound in matrix form
1Texture model components
Weighted least squares estimate
0.5
1DCEvenOddCertainty
Weighted least squares estimate
−30 −20 −10 0 10 20 30−0.5
0
For diagonal : parameters obtained by Gabor/Gaussian responses at
Relation between Amplitude and bound
![Page 17: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/17.jpg)
Alternative HypothesesCast edge detection in same setting:
Phase congruency model for edges & lines:
Rewrite as expansion on basis:
Edge model components
0.4
0.6
0.8
1
1.2Edge model components
DCEvenOddCertainty
−30 −20 −10 0 10 20 30−0.4
−0.2
0
0.2
Iterate previous stepsConnection with Energy-based edge detection - QFPs
Morrone & Owens ‘87, Perona & Malik ’90,
Smooth signal:
![Page 18: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/18.jpg)
Structure captured by the Edge and Texture models
Input Edge Reconstruction Texture Reconstruction
![Page 19: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/19.jpg)
Texture/Edge/Smooth discrimination in 2D imagesFor each scale/orientation combination use all three models
Use Gabor/Edge/Gaussian filters to estimate model parameters
Quantify gain of Edge/Texture hypothesis vs Smooth hypothesisQuantify gain of Edge/Texture hypothesis vs. Smooth hypothesis
Normalize for scale invariance: per-pixel gain
Compute class posteriors
![Page 20: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/20.jpg)
Text/Edge/Smooth Hypothesis ClassificationIntensity Texture Amplitude Edge Amplitude
Posterior Probabilities
Prob(Smooth) Prob(Texture) Prob(Edge)
![Page 21: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/21.jpg)
Texture vs. Edge discriminationIntensity Prob(Texture) Prob(Edge)( ) ( g )
![Page 22: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/22.jpg)
Presentation Outline
Amplitude Modulation- Frequency Modulation (AM-FM) models
2-D AM-FM Model2-D AM-FM ModelEnergy Separation Algorithm, Regularized DemodulationDominant Component Analysis (DCA)
Probabilistic Aspects
Model-based interpretation of Gabor filtering Model based interpretation of Gabor filtering Alternative models for edge and smooth signalsTexture / edge / smooth classification via model comparison
Applications to Segmentation
Variational Image Segmentation using AM-FM featuresW i ht d C E l ti f bi tiWeighted Curve Evolution for cue combination
![Page 23: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/23.jpg)
Variational Image Segmentation
Mumford & Shah ’89 Zhu & Yuille, ’96: Region Competition Functional
Level Set framework:Chan & Vese, Scale-Space ’99, Yezzi, Chai & Willsky, ICCV ’99Yezzi, Chai & Willsky, ICCV 99Paragios & Deriche, ICCV ’99, ECCV ’00
Combination with Geodesic Active Contours (Paragios & Deriche):
![Page 24: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/24.jpg)
Features for Variational Texture Segmentation
Filterbank-based methodsZhu & Yuille, PAMI ‘96: Small filterbank, few results on textureZhu & Yuille, PAMI 96: Small filterbank, few results on textureParagios & Deriche, IJCV ‘02: SupervisedSagiv, Sochen et al. , ‘02. Sandbert Chan & Vese et al, ‘02 : Feature selection
HistogramsKim, Fisher & Willsky, ICIP `01: Nonparametric estimate of intensity Tu & Zhu PAMI ’02: Histograms of intensity + model calibrationTu & Zhu, PAMI 02: Histograms of intensity + model calibration
Low dimensional descriptorsZ H li k A t & P tti hi ICIP ‘01 M d l ti f t l t iZray, Havlicek, Acton & Pattichis, ICIP ‘01: Modulation features + clusteringVese & Osher, JSC ’02, features from decompositionRousson, Brox & Deriche, CVPR ‘03: Anisotropic diffusion + structure tensor.
![Page 25: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/25.jpg)
Modulation features via Dominant Component Analysis
DCA
![Page 26: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/26.jpg)
Variational Segmentation with Modulation Features
3-dimensional feature vectorAmplitude function: ContrastMagnitude of frequency vector: ScaleMagnitude of frequency vector: ScaleAngle of frequency vector: Orientation
Smooth, low-dimensional descriptorGaussian distribution for von-Mises forGaussian distribution for , von Mises for
Initialize segmentation randomly and iterate: Estimate region parameters using current segmentationEstimate region parameters using current segmentation Modify segmentation by curve evolution
![Page 27: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/27.jpg)
Cue Combination TaskIntensity Prob(Smooth)
Texture Features P b(T t )Texture Features Prob(Texture)
Edge Strength Prob(Edge)Edge Strength Prob(Edge)
![Page 28: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/28.jpg)
Classifier Combination Approach
Treat probabilistic balloon force of RC as log-odds of two-class classifier
Decide about pixel label by comparing feature likelihoods Consider separate classifiers based on texture/intensity/edge cues
`Supra –Bayesian’ classifier combination, a.k.a. `stacking’Treat classifier outputs themselves as random variables
Ideally, Consider joint distribution of vector of classifier log-odds.For independent classifiers s.t. decision is given by
![Page 29: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/29.jpg)
Weighted Curve Evolution
Last slide summary: give higher weight to log-odds of better classifier
Adaptation to curve evolution: set weights equal to class posteriorsWeighted curve evolution:Weighted curve evolution:
Compare to Geodesic Active RegionsCompare to Geodesic Active Regions
Geodesic Active Regions Weighted Curve Evolution
![Page 30: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/30.jpg)
Segmentation Result ComparisonsIn
put
)D
CA
(pla
in)
Dox
et.
al.
Bro
WC
ED
CA
+
![Page 31: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/31.jpg)
Quantitative EvaluationBerkeley Benchmark: 100 hand-segmented images (test-set)Bidirectional Consistency Error
At each pixel: normalized set difference of machine- and user- regions
Make symmetric, take minimum over users, and averagey g
Precision-Recall
![Page 32: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/32.jpg)
Berkeley Dataset Segmentations
![Page 33: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/33.jpg)
Conclusions & Future Work
AM FM models: naturally suited for modelling oscillationsEfficient and reliable parameter estimationL di i l d i tLow-dimensional descriptors
Model-based interpretation of feature extractionGabor filteringEnergy-based feature detection
Cue Combination for Curve Evolution
Future workAM FM models: synthesis, PDE methods (G. Evangelopoulos)I t t ith th t tIntegrate with other structures
Crosses, junctions, blobs, ridgesUse segmentation to drive object detection
U t l t i t tUse segments as elementary image structures Construct segment-based object representations
![Page 34: Texture Analysis and SegmentationAnalysis and Segmentation ...vision.mas.ecp.fr/Personnel/iasonas/slides/AMFM_math_UCLA.pdf · Texture Analysis and SegmentationAnalysis and Segmentation](https://reader031.vdocuments.mx/reader031/viewer/2022020214/5a9db2ef7f8b9a28388d3470/html5/thumbnails/34.jpg)
Synthetic signal reconstruction Constant Edge Texture
nt 0.7
0.8
0.9
1Weighted MSE: 0.401
0.7
0.8
0.9
1Weighted MSE: 0.308
0.7
0.8
0.9
1Weighted MSE: 0.388
Con
sta
50 60 70 80 90 1000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
50 60 70 80 90 1000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
50 60 70 80 90 1000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Edg
e
50 60 70 80 90 100
0.4
0.5
0.6
0.7
0.8
0.9
1Weighted MSE: 1.893
0.5
0.6
0.7
0.8
0.9
1Weighted MSE: 0.138
0.4
0.5
0.6
0.7
0.8
0.9
1Weighted MSE: 1.507
Ee
50 60 70 80 90 1000
0.1
0.2
0.3
0.4
50 60 70 80 90 1000
0.1
0.2
0.3
0.4
0.5
50 60 70 80 90 1000
0.1
0.2
0.3
0.4
1Weighted MSE: 1.216
1Weighted MSE: 1.665
1Weighted MSE: 0.327
Text
ure
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
50 60 70 80 90 1000
0.1
0.2
50 60 70 80 90 1000
0.1
0.2
50 60 70 80 90 1000
0.1
0.2