image segmentation of multiple objects and their compartments jerry l. prince image analysis and...
TRANSCRIPT
Image Segmentation of Multiple Objects and Their Compartments
Jerry L. PrinceImage Analysis and Communications
Laboratory (IACL)http://iacl.ece.jhu.edu
Johns Hopkins University© 2010
Acknowledgments• Chenyang Xu• Dzung Pham• Xiao Han• Duygu Tosun• Bai Ying• Daphne Yu• Kirsten Behnke• Xiaodong Tao• Sarah Ying• Xian Fan
• Susan Resnick• Mike Kraut• Maryam Rettmann• Christos Davatzikos• Nick Bryan• Aaron Carass• Ulisses Braga-Neto• Lotta Ellingsen• Pierre-Louis Bazin
Funding sources: NSF, NIH/NINDS, NIH/NIA
Outline• Introduction• Deformable models• TGDM: topology-preserving geometric
deformable model• MGDM: multi-object geometric deformable
model• Conclusion
Outline• Introduction• Deformable models• TGDM: topology-preserving geometric
deformable model• MGDM: multi-object geometric deformable
model• Conclusion
Conventional Structural Image
www.medical.philips.com
Segmentation of Brain Structures
Volumetric MR Data
Subcortical Structures
Cortex
TOADS CRUISE
Bazin and Pham, TMI, 2007
Bazin and Pham, MedIA, 2008
Xu et al., TMI, 1999
Han et al., NeuroImage, 2004
Tosun et al., NeuroImage, 2006
Thalamic nuclei
Cerebellar lobules
Multi-Compartment Anatomy
Cell counting
Circuit board inspection
Retinal examination Traffic camera
Satellite imageryAerial photographs
Other Multi-Object Scenarios
Outline• Introduction• Deformable models• TGDM: topology-preserving geometric
deformable model• MGDM: multi-object geometric deformable
model• Conclusion
Cortical Surface Segmentation
Partial Inflation
Ventricle Segmentation
• Parametric deformable models (PDMs)
─explicit parameterization
• Geometric deformable models (GDMs)– implicit
representation
Deformable Models
Parametric to Geometric[Osher & Sethian 1988]
Level Set PDE:
Contour Deformation:
Visual Concept of GDM
Properties of GDMs• Advantages:
– Produce closed, non-self-intersecting contours– Independent of contour parameterization– Easy to implement: numerical solution of PDEs on
regular computational grid– Stable computations– Automatically changes topology
• Potential disadvantage:– Does not maintain topology
• GDM cannot control topology• TGDM (ours) preserves topology
Topology Behavior
GDM: Standard GeometricDeformable Model
TGDM: Topology-preserving Geometric Deformable Model
Why Maintain Topology?
GDM: Standard GeometricDeformable Model
TGDM: Topology-preserving Geometric Deformable Model
Outline• Introduction• Deformable models• TGDM: topology-preserving geometric
deformable model• MGDM: multi-object geometric deformable
model• Conclusion
Marching Cubes Isosurface• Where is the boundary
defined by a level set function?
• Consider voxel values on corners of a cube
• Label as– above isovalue– below isovalue
• Determine position of triangular mesh surface passing through the cubes by linear interpolation
> 0.5
< 0.5
Voxel values
Digital Connectivity
• Consistent pairs: (foreground,background) → (6,18), (6,26), (18,6), (26,6)
6-connectivity
18-connectivity 26-connectivity
Digital Embedding of Contour Topology
White Points:
Black Points:
• Contour topology is determined by signs of the level set function at pixel locations
• Topology of the implicit contour is the same as the topology of the digital object
Connectivity Rule of Contour
• Topology of digital contour determined by connectivity rule
Same digital object, different topologies
Topology Preservation Principle
• Preserving surface topology is equivalent to maintaining the topology of the digital object
• The digital object can only change topology when the level set function changes sign at a grid point
• Which sign changes can be allowed, and which cannot?
• To prevent the digital object from changing topology, the level set function should only be allowed to change sign at simple points
[Han et al., PAMI, 2003]
Simple Point• Definition: a point is simple if adding or removing
the point from a binary object will not change the object topology
• Determination: can be characterized locally by the configuration of its neighborhood (8- in 2D, 26- in 3D) [Bertrand & Malandain 1994]
SimpleNon-
Simple
x is a Simple Point
0)( x
x
0)( x
xx
x is Not a Simple Point
0)( x 0)( xX
X
Topology Preserving Geometric Deformable Model (TGDM)
• Evolve level set function according to GDM• If level set function is going to change sign, check
whether the point is a simple point– If simple, permit the sign-change– If not simple, prohibit the sign-change (replace the grid value by epsilon with same sign)– (Roughly, this step adds 7% computation time.)
• Extract the final contour using a connectivity consistent isocontour algorithm
Ambiguous Faces
Two possible tilings:
Ambiguous Cubes
Two possible tilings:
Connectivity Consistent MC Algorithm
• (black,white)• (18,6) choose b, f• (26,6) choose b, e
(a) (b) (c)
(d) (e) (f)
AmbiguousFace
AmbiguousCube
• (6,18) choose c, f• (6,26) choose c, f
Nested Deformable Surfaces
Pial Surface
Inner Surface
Central Surface
TGDM-3
Initial WM Isosurface
TGDM-2TGDM-1
TGDM for Inner Surface
Initial WM Isosurface Evolving GM/WM Interface
[Han et al., NeuroImage, 2004]
IACL
TGDM for Central Surface
Initialize with GM/WM surface Evolving toward Central Surface
TGDM for Outer Surface
Evolving toward Outer SurfaceStart from Central Surface
Results—Visual Inspection
Sagittal
• surfaces overlaid on cross-sections of the original image
Axial
Outline• Introduction• Deformable models• TGDM: topology-preserving geometric
deformable model• MGDM: multi-object geometric deformable
model• Conclusion
Multiple Object Challenges1. Maintenance of
multiple level sets
2. Maintenance of object’s individual topologies and relationships between objects
Anatomical parcellation is not arbitrary
Prior Strategies• N level set functions for N objects
[Paragios00, Brox06, …]– Pros: Flexibility between objects– Cons: Objects might overlap or form gaps;
large memory and computational demands
• Multi-phase [Vese02] (4-color theorem)– Pros: Log(N) level set functions for N
objects; low computational complexity; no overlaps or gaps;
– Cons: Forces limited to region and length terms; little control over individual object forces; no 3D equivalent
Principle of MGDM• Simple point criterion can
be replaced by digital homeomorphism criterion
• Movement of collection of objects occurs primarily at: – edges between two objects
or – junctions between three
objects
• Higher-order relationships can be ignored
Objects are not digitally homeomorphic.
Fan et al., CVPR’08, MMBIA’08
Level Set Function Decomposition• N objects Oi, i=1,…N• Distance to objects:
• Label functions:
L0 = Object
L1 = Nearest neighbor
L2 = 2nd nearest neighbor
Distance and Level Set Functions• Distance-based functions:
• Reconstruction of level set functions:
0(x)
1(x)
2(x)
^
Approximation: valid assuming 3 objects max per junction
2D
3D
Evolution• Recall GDM:
• Required evolutions:
• Distance-based functions:
• Assume1. Compute forces2. Find “third” neighbor:
3. Compute:
If then setSet
MGDM Algorithm (2D case)4. Compute:
If then setand• Digital topology and
homeomorphism are readily added
Simulation Experiments• Compare algorithms:
– multiphase (MP)– coupled level sets (CLS)– ours (MGDM)
• Objective function (classic Mumford Shah energy; also Chan-Vese for GDM)
• Evaluate:– convergence, memory usage, computation
time, and misclassification percentage
Visual Comparison
Convergence ComparisonE
Iteration
Quantitative Results
Experiment I: Whole Brain SegmentationStructure memberships from TOADS [Bazin 07] (Topology-preserving, Anatomy-driven segmentation)
ii uf 5.0iu
Membership function:
Force:
Balloon force Smooth force
Sulcal CSF
Cerebral GM
Cerebral WM Cerebellar GM
Cerebellar WM
CaudateThalamus PutamenVentriclesBrainstem
Whole Brain Segmentation: 2D Visualization
(a) Original Image (c) No Topology or Smooth
(b) Result from Toads (d) No Topology but Smooth (f) Group Topology
(e) Single Topology
Topology is preserved with DHC
Whole Brain Segmentation: 3D Visualization
Object topology and relationships between objects can be preserved.
(a) No Topology or Smooth (c) Single Object Topology
(b) No Topology but Smooth (d) Group Objects Topology
Experiment II: Thalamic Nuclei Parcellation
MP-RAGE image
Thalamus Membership
TOADS
FA with PEV color map
co-registered
Homogeneous Orientation
Force for the thalamus boundary.
Force for the thalamus nuclei.
Apply to voxels whose label or the first neighbor is the background.
Apply to voxels whose label or the first neighbor belongs to thalamic nuclei.
Different forces applied to different parts of an object
F
Thalamic Nuclei Parcellation
iii xxxf )()(v)( V
Mean principal orientation from region i
)()(5.0)( xxuxf
The force for thalamus nuclei parcellation is designed based on the assumption that the orientations for each nuclei is homogeneous.
The force for thalamus boundary is a combination of balloon and smooth terms.
Membership function of thalamus at voxel x
Thalamic Nuclei Parcellation: Result
(b) Initialization (d) Result(c) Principal Orientation of Thalamus
(a) Membership Function for Thalamus
Front View Off Diagonal ViewOff Left ViewBack View
Outline• Introduction• Deformable models• TGDM: topology-preserving geometric
deformable model• MGDM: multi-object geometric deformable
model• Conclusion
General Principles• Object topology can be strictly preserved in
geometry deformable models: TGDM• Multiple objects can be simultaneously
segmented and– topology preserved– object relationships preserved– memory efficient– all conventional forces can be applied– guaranteed to have no overlaps or gaps
Remaining Concerns and the Future• How to establish initial object or collection?
– digital topology is not always preserved under simple transformations such as rotation
– recent work on manual skeleton is promising, but tedious
– automatic topology correction is known only for spherical topology, and it is not globally optimum
• Problem of objects getting “stuck”– not so bad in TGDM– much worse in MGDM
Questions?