Segmentation of Medical Images under Topology Constraints
Florent Ségonne
Computer Science and Artificial Intelligence Laboratory, MIT
Athinoula Martinos Center for Biomedical Imaging, MGH, Boston
Most macroscopic brain structures have the topology of a sphere.
Cortical surface C can be considered to have the topology of a sphere S
GOAL: Achieve Accurate and Topologically Correct Segmentations of Anatomical
Structures from Medical Images
Definition of Topological Defects
Background
TOPOLOGICAL DEFECT: deviation from spherical topology
• Handles or holes
• Cavities
• Disconnected components
Importance of Accurate, Topologically-Correct Segmentations
Local functional organization of the cortex is largely 2-dimensional
From (Sereno et al, 1995, Science)
Cortical parcellation
Analysis of functional activitySpherical AtlasVisualization
• Shape Analysis • Presurgical Planning• Statistical analysis of morphometric properties
Aging Neurodegenerative diseases
Longitudinal studies of structural changesHemispheric asymmetry
• Image artifacts
- Partial voluming a single voxel may contain more than one tissue type.
- Bias field effective flip angle or sensitivity of
receive coil may vary across space.
- Tissue inhomogeneities even within tissue type (e.g. cortical gray matter), intrinsic properties such as T1,
PD can vary (up to 20%).
• Topological notions are difficult to integrate into
a discrete numerical framework
Difficulties of Achieving
Accurate, Topologically Correct Segmentations
Assigning tissue classes to voxels can be difficult
Partial volume effect is often the cause for an incorrect topology
Previous Work
Essentially two types of approaches:
1) Segmentation under topological constraint
2) Retrospective topology correction of segmentations
Previous Work
Local decision to preserve topology might lead to large geometrical errors
1) Segmentation under Topological Constraints
• Active contours: Triangulations: self-intersection
Dale et al.-99Davatzikos-Bryan-96MacDonald-et-al.-00
Level sets and Topologically constrained level-sets Zeng-Staib-Schultz-Duncan-99Han-Xu-Prince-03
• Homotopic digital deformations Mangin-et-al.-95
Poupon-et-al.-98Bazin-Pham-05
• Segmentation by registration and vector fields Karacali-Davatzikos-04Christensen-et-al.-97
Final surfaceDeformationInitial condition
Correct segmentation
Incorrect segmentation
Sensitivity to the initialization
Incorrect segmentation
Strict topology preservation is
restrictive
Previous Work
1) Limitations of Segmentations under Topological Constraints
• Sensitivity to initialization
• Strict topology preservation is restrictive
cannot distinguish handles from disconnected components or cavities
Section I
Location of the topological defects is hard to controlCorrection of the defects might not be optimal
Previous Work
2) Retrospective Topology Correction of Segmentations
• Digital binary images Shattuck-Leahy-01Han-Xu-Braga-Neto-Prince-02Kriegeskorte-Goeble-01
• Triangulations Guskov-Wood-01
Fischl-Liu-Dale-01
Topological defect Inaccurate correction Accurate correction
• Necessity to integrate additional information
• Solution is not necessarily obvious
Previous Work
Difficulty of finding the correct solution!
Topological defect Sagital view Corrected defect Sagital view
1) Limitations of Retrospective Topology Correction Methods
• Do not use all the available information
corrections based on the size of the defect only
• Do not generate several potential solutions to find the best one
cutting a handle or filling the corresponding hole may not generate the valid correction
Section II
Motivation
1) Existing methods are limited
- segmentation under strict topology preservation is too restrictive.
- retrospective topology correction methods do not use of additional information and cannot guarantee optimal topological corrections.
2) Discrete topological tools are restricted
Introduction
Background
Section I - Novel Approaches in Digital Topology
Section III - Manifold Surgery
Road Map
Background
Topology in Discrete Imaging
• General Notions of Topology
• Topology in Discrete Imaging
Topology: study of shape properties preserved through deformations, twistings, and stretchings, but no tearings. [Massey 1967]
=
Background
General Notions of Topology
• Continuous theory
• Different levels of equivalence
Background
Difficult to adapt into a discrete framework
• Intrinsic Topology: - properties preserved by homeomorphisms
- ignore the embedding space
• Homotopy type: - continuous transformations in the embedding space
Euler-characteristic , genus g of a surface S with V: # vertices E: # edges F: # faces
in any polyhedral decomposition of S
Background
= V-E+F
Very useful to check the topology type of a triangulation
= 8-12+6=2 = 16-32+16=0
Euler-characteristic is a topological invariant
But no localization of the topological defectsRelated to the genus of a surface = 2-2g
Genus of a surface is equivalent to the number of handles
• Adapt concept of continuity to a discrete framework
• Surfaces and 3D images
- Tessellations
- 3D digital images
Background
Topology in Discrete Imaging
use the notion of connectivity instead
Euler-characteristic
Theory of Digital topology
Theory of Digital Topology
• Replace the notion of continuity by one of connectivity
Choice of connectivity n=8 or n=4
NEED TO WORK WITH A PAIR OF CONNECTIVITIES TO AVOID TOPOLOGICAL PARADOXES.Intersecting curves or not?
Background
One connectivity for the object X and one compatible connectivity for the inverse object X
Background
4 different pairs of compatible connectivities for the foreground F and Background B
F B
6 26
26 6
6+ 18
18 6+
n=6 n=18 n=26
( , )n n
3D digital images: 3 different types of connectivity
How to characterize the topology type of a given voxel ?
How to detect topology changes ?
• Characterize the topology type of a given voxel• Simple point: point that can be added or removed from a digital object without changing its topology. They are characterized by:
•Local computations using the 3D neighborhood only Fast
Topological numbers (see Bertrand, 1994, Pat. Rec. Let.)
Background
and n nT T
1),(),( XxTXxT nn
Homotopic deformation: addition or deletion of simple points.
Background
simple point in red – non simple point in yellow
Problem: concept of simple point cannot distinguish different topological changes
(formation of handles ≠ formation of cavities ≠ formation of disconnected components)
Homotopic deformations are often too restrictive
Section I
Novel Approaches in Digital Topology
• On the Characterization of Multisimple Points
• Genus Preserving Level Sets
Strict preservation of the topology is a strong constraint
• Notion of simple point is too restrictive
Cannot distinguish different topological changes.
• Difficult to use topology preserving methods
Sensitivity to initialization, noise, unexpected cavities, …
• Interested in handles
Handles are hard to detect and retrospectively correct.
Disconnected components, cavities are less problematic.
Requires an extension of the concept of simple point
On the Characterization of Multisimple Points
Goal: Develop digital techniques to prevent formation of handles but not the generation of cavities or of disconnected components.
Definition: a multisimple point is a point that can be added or removed from a digital object X without changing the number of holes in X or the complement .
On the Characterization of Multisimple Points
X
Need to find a characterization of multisimple points.
On the Characterization of Multisimple Points
Can we use the topological numbers to characterize different topological changes ?
Topology type0 Isolated point
0 Interior point
1 1 Border point ( = Simple Point )
2 1 Curve point
>2 1 Curves junction
1 2 Surface point
1 >2 Surfaces junction
>1 >1 Surface(s)-curve(s) junction
nTnT
Voxel topology type and topological numbers
)18,6(),( nn
1),(0),( XxTXxT nn
1),(2),( XxTXxT nn
• Topological numbers are not able to characterize multisimple points
• Topological numbers are computed locally: no information about the global connectivity of the digital object
extension of the concept of Topological Numbers
• We introduce the set of neighboring components of a point x:
• We define two Extended Topological Numbers:
),( XxCn
( , ) and ( , )n n n nT C x X T C x X
On the Characterization of Multisimple Points
How to integrate information about the global connectivity of the digital object ?
On the Characterization of Multisimple Points
A point is multisimple if and only if:
andn n n nT T T T
Necessary and Sufficient Condition for a Point to be Multisimple ?
• Simple characterization of multisimple points.
• ‘Low’ computational complexity.
• deformations that do not generate/suppress any handles in the digital objects
On the Characterization of Multisimple Points
1 2
1 1
n n
n n
T T
T T
2 2
1 1
n n
n n
T T
T T
Example 1),(2),( XxTXxT nn
• Computational complexity - Implementation using a grid of labels
assign a label to each component
- local computations using the grid of labels
merging = assigning the same label
splitting = assigning different labels (using region growing algorithms)
• New sets of transformations under topology control - Preservation of the genus
Components can split/merge/appear/disappear without generating any handles
On the Characterization of Multisimple Points
Topology constrained Topology controlled with one single initial component
Topology controlled with several initial components
simple point
non simple point
Multisimple point
1) Multisimple Points preserve the genus of a digital object
2) Level sets require an underlying digital image to be implemented
The negative/positive grid points of a level set define set of connected components of a digital object
Apply the concept of multisimple point to
level sets in order to design a
Genus Preserving Level Set Framework
Genus Preserving Level Sets
Genus Preserving Level Sets
Level set evolution equation Level set evolution equation with outward normal speed with outward normal speed ββ::
What is the level set method ?
Representing a manifold implicitly as the zero level set of a Representing a manifold implicitly as the zero level set of a higher-dimensional functionhigher-dimensional function (Osher-Sethian, 1988) (Osher-Sethian, 1988)
+ No parameterization
+ Easy computation of intrinsic geometric properties (e.g. normal, curvature)
+ Mathematical proofs and numerical stability
+ Topology changes handled automatically
+ No self-intersection in the resulting mesh
- Computationally expensive
narrow band algorithm
- Need for an isocontour extraction step
marching cubes algorithm
Level sets vs. explicit representations
Genus Preserving Level Sets
Why using the level set methods ?
Genus Preserving Level Sets
How to preserve the topology of level sets ?
Topology Preserving Level Sets (Han, Xu, Prince, 2003)Algorithm: for each grid point at each time step,- Compute the new value of the level set- Sign is about to change?
No: accept the new valueYes: check if the point is simple
SIMPLE: accept the new valueCOMPLEX: set +/-ε as the new level set value
Initial conditionInitial condition Traditional level Traditional level setssets
Topological level setsTopological level sets
+ Local computations only FAST Strict Preservation of the topology RESTRICTIVE
Algorithm: for each grid point at each time step,- Compute the new value of the level set- Sign is about to change?
No: accept the new valueYes: check if the point is simple- SIMPLE: accept the new value- COMPLEX: set +/-ε as the new level set value
Genus Preserving Level Sets
Genus Preserving Level Sets
Strict Topology Preservation
Algorithm: for each grid point at each time step,- Compute the new value of the level set- Sign is about to change?
No: accept the new valueYes: check if the point is simple- SIMPLE: accept the new value- COMPLEX: set +/-ε as the new level set value
- Compute the set of neighboring connected components from the grid of labels L- Check if the point is multisimpleMULTISIMPLE: accept the new value
update the grid of labels LCOMPLEX: set +/-ε as the new level set value
Genus Preserving Level Sets
velocity field ( , ) ( ( ) ) ( , ) ( , )
( , ) 1Euler-Lagrange equation ( ) div( )
1
thres
thres
t I I H t t
tI I
t n
v x x x N x
x
Experiments: Simple Segmentation Tasks
with I(x) : intensity in the image at location x
Ithres : global intensity threshold
H(x,t) : mean curvature of active contour at location x
N(x,t) : normal of active contour at location x
: mixing parameter
Data termSmoother surfaces
Genus Preserving Level Sets
Traditional level sets Topology preserving level sets Genus preserving level sets
Segmentation of a synthetic ‘C’ shape
Trade-off between traditional level-sets and topology preserving
level-sets
Less sensitive to initial conditions
Genus Preserving Level Sets
Segmentation of blood vessels from MRA
Segmentation of blood vessels from MRA under two different initial conditions
movie
Genus Preserving Level Sets
Topology preserving level sets Genus preserving level sets
OTHER EXAMPLES: square with cavities cortical segmentation initialized with 100 seeds
Advantages of Genus Preserving Level Sets
Genus Preserving Level Sets
• Preserve the number of handles in the volume
• Components can be created or destroyed, can merge or split
• Less sensitive to initial conditions
• Less sensitive to noise/unexpected structures in the image
• ‘Low’ computational complexity (except for splits)
Section I: Contributions
Novel Approaches in Digital Topology
• Concept of Multisimple Point
• New sets of digital transformations
• Genus Preserving Level Sets
Section II
Manifold Surgery
Topology Correction of Cortical Surfaces
Surfaces with the same Euler-characteristic are homeomorphic
Manifold Surgery : Position of the problem
Segmentation under topological constraints is a difficult problem: partial voluming effect, noise, bias field, image inhomogeneity, …
many topological defects (handles) in the resulting cortical surface
GOAL: given a cortical surface C, detect and optimally correct the topological defects D
Manifold Surgery
• Approach
• Location of topological defects
• Optimal topology correction
Shrink-Wrap Methods cannot reach deep folds
Quasi-Homeomorphic mapping to locate and
correct the defects
Manifold Surgery
Manifold Surgery : Approach
Homeomorphic mapping M:C S– Transformation that is continuous, one-to-one with continuous
inverse M-1
– Jacobian is positive definite: JM > 0
– Jacobian is related to the areal distortion
– For triangulations, the areal distortion is an approximation of the Jacobian
S M CdA J dA
SM
C
AJ
A
CA SA
Manifold Surgery
Definition: Homeomorphism
Manifold Surgery
Manifold Surgery
• Approach
• Location of topological defects
• Optimal topology correction
• Cortical surface C with correct topology is homeomorphic to
the sphere S:
- continuous, one-to-one, continuous inverse
- strictly positive Jacobian
• In the presence of topological defects (i.e. handles), no such mapping exists.
Search for a mapping that minimizes the regions with negative Jacobian
Quasi-homeomorphic mapping = maximally homeomorphic
Manifold Surgery
• Try to find a mapping that is maximally homeomorphic
The areal distortion is an approximation of the Jacobian
Quasi-Homeomorphic mapping
SM
C
AJ
A
Generate a mapping that minimizes the
regions with negative area
th
th
area of i face in at
area of i face in
ti
ti
A S t
A C
1
log(1 )with
ikR tFi
M i i oi i
AeE R R
k A
1) Initialize the mapping by inflation and projection
2) Minimize EM by gradient descent
Manifold Surgery
Location of topological defects
different steps
- inflation
- projection
- minimization of
negative regions
- defect = set of
overlapping faces
- back-projection
Manifold Surgery
Location of topological defects
Cortical surface with incorrect topology
Topological defect
Spherical projection
Manifold Surgery
Manifold Surgery
• Approach
• Location of topological defects
• Optimal topology correction
The retessellation problem
defect in original cortical surface
corresponding spherical projection of defect
quasi-homeomorphi
c mapping
1) Discard all faces and edges in each defect D
2) Find a valid retessellation D for each defect D
Manifold Surgery
Finding a valid retessellation D for each defect D
- Geometrically Accurate, i.e. smoothness, location, self-intersection
Use Spherical Representation S
Use Cortical Representation C
- Geometrically Accurate, i.e. smoothness, location, self-intersection
Manifold Surgery
The retessellation problem
Constraining the Topology on S
Discard edges and faces
Retessellate
Find a retessellation with no self-intersecting edges on the sphere
D = 1
Manifold Surgery
Many potential retessellations exist!
Sedges={ …} ={e0 ,e1 ,…,eN} set of all potential edges
where N=n(n-1)/2 and n is the number of vertices in the defect
One candidate retessellation corresponds
to an ordering OD of the set Sedges
Retessellating = iteratively adding edges
Constraining the Topology on S
Manifold Surgery
Retessellating = Knitting on the sphere
Edge ordering OD ={ …}
Different edge orderings (permutations) will generate different configurations in the original
cortical space
Constraining the Topology on S
Manifold Surgery
Measuring the accuracy of D on C
• No self-intersection
• Smoothness of D
• MRI intensity I profile inside C - and outside C +
( | , ) ( | , ) ( | )
Likelihood PriorD D D
pT C I p I C T pT C
Bayesian Framework
Manifold Surgery
Prior = smoothness
Likelihood = MRI intensity profile
#vertices
1 21
( | ) p( (v), (v)| )D
i
pT C C
#vertices
i i1
( | , ) ( ( ) | , )
( ( ) | , )
p(g(v),w (v)| , )
wD Dx C
g Dx C
Di
p I C T p I x C T
p I x C T
C T
1 2, principal curvatures
volume
surface
Measuring the accuracy of on C
Manifold Surgery
Huge space to be searched!Discrete space No gradient information
Search : Genetic Algorithm
defect with n vertices N=n(n-1)/2 potential edges
For a patch, v-e+f=1 # of used edges 3n
Size of Space of potential retessellations
2
3
2
n
n
Manifold Surgery
Use of a Genetic Algorithm to explore the space of potential retessellations OD
• introduced in the 60s by John Holland as a way to import the mechanism of natural adaptation into computer algorithm and numerical optimization
• candidate solution = chromosome
• Fitness function = measure the evolutionary viability of each chromosome
• Genetic operations = mutations, crossovers
representation = edge ordering OD
posterior probability
Search : Genetic Algorithm
Manifold Surgery
• Mutations : swap intersecting edges in OD with probability pmut
• Crossovers : randomly combine two orderings by iteratively adding edges from each of them
• Parameters :population = 20 chromosomes
elite population = 40% - mutation = 30% - crossovers = 30%
pmut= 0.1 – stop after 10 populations without any change
Genetic OperationsManifold Surgery
Results: real MRI datasets• applied to hundred of brains (part of FreeSurfer)
typical brain segmentation = 40 defects (~50 vertices)
average time = ~3 hours (~ 5 minutes / defect)
• validation on 35 brains (70 hemispheres)
comparison with expert Hausdorff distance < 0.2mm
Manifold Surgery
Topological defect
Corrected defect
Results: real MRI datasets
Manifold Surgery
Topological defect Corrected defectInitial cortical surface
Sagital view Coronal view
Results: real MRI datasets
Manifold Surgery
Topological defect
Corrected defect
Results: real MRI datasets
Manifold Surgery
Convergence• Genetic Search versus Random Search
- Boost up average fitness of population- Speed up convergence by a log factor of 2
- Converge in few iterations
Manifold Surgery
Section II: Contributions
Manifold Surgery
• Method for optimally correcting the topology of cortical surfaces
• Genetic algorithm for solving the retessellation problem
Manifold Surgery
CONCLUSION
TO BE DONE…
IT”S GREAT WORK !
Manifold Surgery
Approach: retrospective topology correction of the cortical surface C
orig MRI segmentation
tessellate
surface locate defects correct defects
skull strip
spherical mapping
Discarding vertices on S
Spherical mapping
C
S
All vertices are included in the retessellation jagged surfaces
Solution: use the iterative retessellation to eliminate late vertices
During the retessellation, vertices that are included inside temporary triangles are discarded from the
tessellation
The space of all potential retessellations depends on the spherical mapping
Limitations: the mapping problem!
Spherical mapping
C
S Discard faces and edges
impossible
Retessellate
The mapping problem!
Solution: Generate several mappings corresponding to different configuration
- identify defects with large handles (geodesic distances)
- cluster proximal vertices
- generate several mappings using spring term force
The mapping problem!
Example
Topological defect
Spherical representation
Sagital view of topological defect
The mapping problem
Example
Topological defect Corrected defect
Spherical representation
The mapping problem!
Solution: Generate several mappings corresponding to different configuration
- spherical location is not appropriate
circle in 2D
The mapping problem!
- cluster proximal vertices in original space
- generate different mappings using a spring term force
The mapping problem
Original defect Final configuration
Example
Results: synthetic data