Perceptual Organization of Perceptual Organization of Curvilinear StructuresCurvilinear Structures
Laurent AlquierLaurent Alquier
Research Director : Research Director : Chabane OussalahProfessor
Thesis Advisor :Thesis Advisor :Philippe MontesinosAssistant Professor
- UNIVERSITE MONTPELLIER II -- Laboratoire de Génie Informatique et d 'Ingénierie de
Production, Nîmes -- September, 30th 1998 -
1 September 1999Laurent ALQUIER -
Image AnalysisImage Analysis
Intensity Image
Contours Images
Detection of visual cues
Construction of a symbolic representation
1 September 1999Laurent ALQUIER -
Visual PerceptionVisual Perception
Theories of Visual PerceptionTheories of Visual Perception Continuous flow of visual informationContinuous flow of visual information
Necessity to guide perceptionNecessity to guide perception
1 September 1999Laurent ALQUIER -
Perceptual OrganizationPerceptual Organization PrinciplesPrinciples
Pre-attentive phenomenon of groupingPre-attentive phenomenon of grouping Global visual properties : Global visual properties : SaliencySaliency Imposed to perception Imposed to perception beforebefore interpretation interpretation Elementary groupsElementary groups
Principle of "good shape"Principle of "good shape" Simplicity, closure, familiarity : Simplicity, closure, familiarity : StabilityStability
Useful properties Useful properties Generic, robust - Qualitative organization Generic, robust - Qualitative organization
Gestalt Theory : Wertheimer, Koffka - 1923Gestalt Theory : Wertheimer, Koffka - 1923
References
1 September 1999Laurent ALQUIER -
OverviewOverview
Introduction.Introduction.
Context and Objectives.Context and Objectives.
Organization using Saliency Networks.Organization using Saliency Networks.
High Levels of Organization.High Levels of Organization.
Contributions and Perspectives.Contributions and Perspectives.
1 September 1999Laurent ALQUIER -
ObjectivesObjectives Complete system of image analysisComplete system of image analysis
Segmentation - hypotheses - interpretationSegmentation - hypotheses - interpretation Application of psycho-vision principlesApplication of psycho-vision principles
Analysis of complex scenesAnalysis of complex scenes Restriction to shapes from contoursRestriction to shapes from contours Validation with artificial scenesValidation with artificial scenes Application to images of real scenesApplication to images of real scenes
Goal :Goal : Extract a set of elements of representationExtract a set of elements of representation Remain open to possible needs from future Remain open to possible needs from future
applicationsapplications
1 September 1999Laurent ALQUIER -
Overview of the systemOverview of the system
Edge detectionEdge detection
Three levels of organizationThree levels of organization Selection of salient structuresSelection of salient structures Extraction of elementary hypothesesExtraction of elementary hypotheses Organization into complex hypothesesOrganization into complex hypotheses
Hierarchic relationship between hypothesesHierarchic relationship between hypotheses
Application Application Detection of junctionsDetection of junctions Structural matchingStructural matching
1 September 1999Laurent ALQUIER -
OverviewOverview
Introduction.Introduction.
Context and Objectives.Context and Objectives.
Organization using Saliency Networks.Organization using Saliency Networks.
High Levels of Organization.High Levels of Organization.
Contributions and Perspectives.Contributions and Perspectives.
1 September 1999Laurent ALQUIER -
Detection of Curvilinear StructuresDetection of Curvilinear Structures Goal :Goal :
Select the most regular contoursSelect the most regular contours Complete discontinuitiesComplete discontinuities Extract stable structures ( noise, scale )Extract stable structures ( noise, scale )
1 September 1999Laurent ALQUIER -
Grouping by SaliencyGrouping by Saliency
PrinciplesPrinciples Estimate the visual quality of a structureEstimate the visual quality of a structure optimise this quality functionoptimise this quality function
Direct ApproachDirect Approach Extension Fields (Guy and Medioni, 1996)Extension Fields (Guy and Medioni, 1996) Stochastic Completion Fields (Williams and Jacobs, 1994)Stochastic Completion Fields (Williams and Jacobs, 1994)
Approaches by optimisationApproaches by optimisation Neural Networks (F. Mangin, 1994)Neural Networks (F. Mangin, 1994) Mean Field Annealing (L. Hérault, 1991)Mean Field Annealing (L. Hérault, 1991) Saliency Network (Shashua et Ullman, 1989) Saliency Network (Shashua et Ullman, 1989)
1 September 1999Laurent ALQUIER -
Saliency Network (2)Saliency Network (2)
Selection of a PrimitiveSelection of a Primitive Definition of visual properties.Definition of visual properties.
Definition of Local NeighborhoodDefinition of Local Neighborhood
Network of locally connected elements.Network of locally connected elements.
Quality function for a groupQuality function for a group Estimate compatibility between elements of a group.Estimate compatibility between elements of a group. "Extensible" functions."Extensible" functions.
Definition of SaliencyDefinition of Saliency Quality of the best possible group for a given primitiveQuality of the best possible group for a given primitive
1 September 1999Laurent ALQUIER -
Local NeighborhoodLocal Neighborhood
Relationships between primitives.Relationships between primitives. Primitives linked using Primitives linked using Elements of Elements of
Connection.Connection. Properties of Properties of ProximityProximity and and CompatibilityCompatibility.. Method suited for Method suited for curved groupscurved groups..
Importance of neighborhood.Importance of neighborhood. Initialize optimisation of the network.Initialize optimisation of the network. Sets complexity of optimisation.Sets complexity of optimisation. Influence visual quality of groups.Influence visual quality of groups.
1 September 1999Laurent ALQUIER -
Quality FunctionQuality Function
Properties of ContinuityProperties of Continuity Proximity, smoothness, similarity.Proximity, smoothness, similarity.
FormalismFormalism Linear combination of opposed constraints.Linear combination of opposed constraints. "Internal" relationships."Internal" relationships.
Visual properties of groups expected.Visual properties of groups expected. "External" relationships."External" relationships.
Imposed by contours from image.Imposed by contours from image.
external internal
)F(j
jjk
kk RR
1 September 1999Laurent ALQUIER -
Structural SaliencyStructural Saliency
Recursive expressionRecursive expressionF is supposed to be an "extensible function"
Formal definition.Formal definition. Best sum of contributions around a primitive Best sum of contributions around a primitive
according to two directions.according to two directions.
)()(Max )( vv)( )v (v,
FFPSPV 2
)),,(Max,(Max),,(Max 21)(
11)(
1)( 2
111
NNee
NNe
eeeeeNN
1 September 1999Laurent ALQUIER -
Structural Saliency (2)Structural Saliency (2)
Example of recursive expressionExample of recursive expressionLocal term.
Contribution of neighbors.
Iterative optimisation.Iterative optimisation. Research of best neighbors.Research of best neighbors. Update of contributionsUpdate of contributions
)(),(
)(),(
)(
)(3
13
2
221
eRevP
eRevP
vQ
vRnkk
nkk
nk
nk
1 September 1999Laurent ALQUIER -
Structural Saliency (3)Structural Saliency (3)
PropertiesProperties Local measurements - Global optimisation.Local measurements - Global optimisation. Completion of discontinuities.Completion of discontinuities. Saliency Map.Saliency Map. Grouping possible by following connections.Grouping possible by following connections.
Only one optimal group for each primitive.Only one optimal group for each primitive.
1 September 1999Laurent ALQUIER -
Structural Saliency (4)Structural Saliency (4)
ApplicationApplication Organization of Pixels.Organization of Pixels.
Static neighborhood.Static neighborhood. Heavy computation.Heavy computation. Slow optimisation.Slow optimisation.
Organization of Chains.Organization of Chains. Dynamic Neighborhood.Dynamic Neighborhood. Reduced complexity.Reduced complexity. Fast optimisation. Fast optimisation.
1 September 1999Laurent ALQUIER -
Consequence of groupingConsequence of grouping
Normal edge linkingNormal edge linking
Organization of chainsOrganization of chains
1 September 1999Laurent ALQUIER -
Elementary GroupsElementary Groups
Classes of groupsClasses of groups Starting points along a global structure.Starting points along a global structure. "Attraction" phenomenon."Attraction" phenomenon.
Evaluation of groups Evaluation of groups Local saliency.Local saliency. Global saliency ( sum of saliency of primitives )Global saliency ( sum of saliency of primitives ) Accumulation of votesAccumulation of votes
SelectionSelection Threshold from evaluations.Threshold from evaluations.
1 September 1999Laurent ALQUIER -
OverviewOverview
Introduction.Introduction.
Context and Objectives.Context and Objectives.
Organization using Saliency Networks.Organization using Saliency Networks.
High Levels of Organization.High Levels of Organization. Primary Hypotheses.Primary Hypotheses. Complex groups and applications.Complex groups and applications.
Contributions and perspectives.Contributions and perspectives.
1 September 1999Laurent ALQUIER -
Primary Hypotheses Primary Hypotheses
Structural HypothesesStructural Hypotheses
Straight parts : Segments.Straight parts : Segments. Curved parts : Arcs.Curved parts : Arcs. Special points : Junctions, inflection points, corners.Special points : Junctions, inflection points, corners.
Principles of extractionPrinciples of extraction Detection from each elementary groupDetection from each elementary group
Scale, sensitiveness.Scale, sensitiveness. Fusion of primary hypotheses.Fusion of primary hypotheses.
Similarity - elimination of duplicates.Similarity - elimination of duplicates.
1 September 1999Laurent ALQUIER -
Primary Hypotheses (2)Primary Hypotheses (2)
Elements of symbolic representationElements of symbolic representation
Set of HypothesesSet of Hypotheses Role of ambiguities and errors.Role of ambiguities and errors. Certain amount of duplicates tolerated.Certain amount of duplicates tolerated.
Problems to solveProblems to solve DiscretizationDiscretization Structures with different scales within the same scene.Structures with different scales within the same scene. Output:Output:
Similar groups superimposedSimilar groups superimposed Exceptions to properties of continuity (occlusions and Exceptions to properties of continuity (occlusions and
junctions)junctions)
1 September 1999Laurent ALQUIER -
Hypotheses - Segments Hypotheses - Segments
Salient GroupsSalient GroupsSalient GroupsSalient Groups
Points of Points of interestinterest
Points of Points of interestinterest
Recursive division Recursive division (tolerance Es)(tolerance Es)
Recursive division Recursive division (tolerance Es)(tolerance Es)
Fusion of similar segments Fusion of similar segments (length,orientation)(length,orientation)
Fusion of similar segments Fusion of similar segments (length,orientation)(length,orientation)
Grouped segmentsGrouped segmentsGrouped segmentsGrouped segments
Elementary segments
1 September 1999Laurent ALQUIER -
Hypotheses - ArcsHypotheses - Arcs
Salient GroupsSalient GroupsSalient GroupsSalient Groups
Points of Points of InterestInterest
Points of Points of InterestInterest
Division according to Extremes of Division according to Extremes of Curvature Curvature
(Scale Ea)(Scale Ea)
Division according to Extremes of Division according to Extremes of Curvature Curvature
(Scale Ea)(Scale Ea)
Fusion of similar arcs Fusion of similar arcs (classe,superposition)(classe,superposition)
Fusion of similar arcs Fusion of similar arcs (classe,superposition)(classe,superposition)
Grouped ArcsGrouped ArcsGrouped ArcsGrouped Arcs
Classes of Elementary Arcs
1 September 1999Laurent ALQUIER -
OverviewOverview
Introduction.Introduction.
Context and Objectives. Context and Objectives.
Organization using Saliency Networks.Organization using Saliency Networks.
High Levels of Organization.High Levels of Organization. Primary Hypotheses.Primary Hypotheses. Complex groups and applications.Complex groups and applications.
Contributions and perspectives.Contributions and perspectives.
1 September 1999Laurent ALQUIER -
Complex groupsComplex groups
Example of applicationExample of application Extraction of multiple junctionsExtraction of multiple junctions Structural matching of junctionsStructural matching of junctions
MotivationsMotivations
Rich structural informationRich structural information Location of center, orientation of branchesLocation of center, orientation of branches Robust matching possibleRobust matching possible Precise location possible after matchingPrecise location possible after matching
Structures difficult to extractStructures difficult to extract Alterations from contour detectionAlterations from contour detection
Few publications in that areaFew publications in that area
1 September 1999Laurent ALQUIER -
Organization of junctionsOrganization of junctions
SegmentsSegmentsSegmentsSegments PointsPointsPointsPoints
Detection of Double JunctionsDetection of Double JunctionsDetection of Double JunctionsDetection of Double Junctions
Fusion of Similar JunctionsFusion of Similar Junctions(proximity of centers, similar branches)(proximity of centers, similar branches)Fusion of Similar JunctionsFusion of Similar Junctions(proximity of centers, similar branches)(proximity of centers, similar branches)
Multiple JunctionsMultiple JunctionsMultiple JunctionsMultiple Junctions
Intersections
Confirmation of center
1 September 1999Laurent ALQUIER -
Structural MatchingStructural Matching
PrinciplePrinciple
Consistent labeling of two sets of Consistent labeling of two sets of junctionsjunctions
RequiresRequires
Direct comparison of junctionsDirect comparison of junctions
Evaluate motion (transformation)Evaluate motion (transformation)
Between two imagesBetween two images
Within a single imageWithin a single image
Conditional comparisonConditional comparison
1 September 1999Laurent ALQUIER -
Structural Matching (2)Structural Matching (2)
Method in two stagesMethod in two stages Temporal MatchingTemporal Matching
Elimination of improbable matchesElimination of improbable matches Spatial MatchingSpatial Matching
Elimination of improbable groups Elimination of improbable groups
PropertiesProperties Matching as perceptual groupingMatching as perceptual grouping Tolerate important differences between imagesTolerate important differences between images Mutual reinforcement of two type of organizationMutual reinforcement of two type of organization
1 September 1999Laurent ALQUIER -
OverviewOverview
Introduction.Introduction.
Context and Objectives.Context and Objectives.
Organization using Saliency Networks.Organization using Saliency Networks.
High Levels of Organization.High Levels of Organization.
Contributions and perspectives.Contributions and perspectives.
1 September 1999Laurent ALQUIER -
ContributionsContributions
Saliency NetworksSaliency Networks Generic FormalismGeneric Formalism
Generalized to Organization of ChainsGeneralized to Organization of Chains Configurable Quality FunctionConfigurable Quality Function More stable optimisationMore stable optimisation
Choice of the most important connections Choice of the most important connections Extraction of final groupsExtraction of final groups
Selection of the most salient structuresSelection of the most salient structures
Shashua and Ullman, 1991 - Alter and Basri, 1997Shashua and Ullman, 1991 - Alter and Basri, 1997
References
1 September 1999Laurent ALQUIER -
Contributions (2)Contributions (2)
Organization of Geometric HypothesesOrganization of Geometric Hypotheses SegmentsSegments ArcsArcs Points of InterestPoints of Interest
Modular approachModular approach Global strategy of organizationGlobal strategy of organization Specialized modulesSpecialized modules
Hypotheses defined for only one scaleHypotheses defined for only one scale
Application to numerous types of scenesApplication to numerous types of scenes
Mohan and Nevatia, 1992 - Sarkar and Boyer, 1993 - Mohan and Nevatia, 1992 - Sarkar and Boyer, 1993 - Gao and Wong, 1993Gao and Wong, 1993
References
1 September 1999Laurent ALQUIER -
Contributions (3)Contributions (3)
Perceptual Organization of JunctionsPerceptual Organization of Junctions Detection of Elementary JunctionsDetection of Elementary Junctions Organization into Multiple JunctionsOrganization into Multiple Junctions
Structural MatchingStructural Matching Cooperation between Spatial Perceptual Cooperation between Spatial Perceptual
Organization and Temporal Matching.Organization and Temporal Matching.
Matas and Kittler, 1993 - Chang and Aggarwal, 1997Matas and Kittler, 1993 - Chang and Aggarwal, 1997
References
1 September 1999Laurent ALQUIER -
ResultsResults
Stable results Stable results Generic parameters for classes of scenesGeneric parameters for classes of scenes Robust in case of perturbationsRobust in case of perturbations
Reasonable computation time on low-end systemsReasonable computation time on low-end systems ExampleExample
PC - Pentium 100 - 65 Mo RAMPC - Pentium 100 - 65 Mo RAM Image 800x600 pixelsImage 800x600 pixels 500 chains in a scene500 chains in a scene Grouping with Saliency Network : 30sGrouping with Saliency Network : 30s Global computation time : 5 minGlobal computation time : 5 min
1 September 1999Laurent ALQUIER -
Short term extensionsShort term extensions
Organization of Generic ObjectsOrganization of Generic Objects
Quantitative evaluation of resultsQuantitative evaluation of results Qualitative results only for nowQualitative results only for now Numerous parametersNumerous parameters Empirical definition for nowEmpirical definition for now
Automatic validation of hypothesesAutomatic validation of hypotheses
Multi-scale detectionMulti-scale detection
Hierarchical Structural Matching Hierarchical Structural Matching Top-down approachTop-down approach
1 September 1999Laurent ALQUIER -
PerspectivesPerspectives
Integration with other visual cuesIntegration with other visual cues Visual attention - trackingVisual attention - tracking
Laurent Iti, 1997 Caltech - Roch and Ullman, 1985Laurent Iti, 1997 Caltech - Roch and Ullman, 1985
References