active shape models: their training and applications cootes, taylor, et al. robert tamburo july 6,...
TRANSCRIPT
![Page 1: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/1.jpg)
Active Shape Models:Their Training and Applications
Cootes, Taylor, et al.
Robert Tamburo
July 6, 2000
Prelim Presentation
![Page 2: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/2.jpg)
Other Deformable Models
“Hand Crafted” Models Articulated Models Active Contour Models – “Snakes” Fourier Series Shape Models Statistical Models of Shape Finite Element Models
![Page 3: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/3.jpg)
Motivation – Prior Models
Lack of practicality Lack of specificity Lack of generality Nonspecific class deformation Local shape constraints
![Page 4: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/4.jpg)
Goals of Active Shape Model (ASM)
Automated Searches images for represented structures Classify shapes Specific to ranges of variation Robust (noisy, cluttered, and occluded image) Deform to characteristics of the class represented “Learn” specific patterns of variability from a training
set
![Page 5: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/5.jpg)
Goals of ASM (cont’d.)
Utilize iterative refinement algorithm
Apply global shape constraints
Uncorrelated shape parameters
Better test for dependence?
![Page 6: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/6.jpg)
Point Distribution Model (PDM)
Captures variability of training set by calculating mean shape and main modes of variation
Each mode changes the shape by moving landmarks along straight lines through mean positions
New shapes created by modifying mean shape with weighted sums of modes
![Page 7: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/7.jpg)
PDM Construction
Manual Labeling Alignment
StatisticalAnalysis
Point Distribution Model
![Page 8: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/8.jpg)
Labeling the Training Set
Represent example shapes by points Point correspondence between shapes
![Page 9: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/9.jpg)
Aligning the Training Set
xi is a vector of n points describing the the ith shape in the set:
xi=(xi0, yi0, xi1, yi1,……, xik, yik,……,xin-1, yin-1)T
Minimize:
Ej = (xi – M(sj, j)[xk] – tj)TW(xi – M(sj, j)[xk] – tj)
Weight matrix used:
11
0
−−
=
⎟⎠
⎞⎜⎝
⎛= ∑
n
lRk kl
Vw
![Page 10: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/10.jpg)
Alignment Algorithm
Align each shape to first shape by rotation, scaling, and translation
Repeat– Calculate the mean shape– Normalize the orientation, scale, and origin of the current
mean to suitable defaults– Realign every shape with the current mean
Until the process converges
![Page 11: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/11.jpg)
Mean Normalization
Ensures 4N constraints on 4N variables Equations have unique solutions Guarantees convergence Independent of initial shape aligned to Iterative method vs. direct solution
![Page 12: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/12.jpg)
Aligned Shape Statistics
PDM models “cloud” variation in 2n space
Assumptions:– Points lie within “Allowable Shape
Domain”– Cloud is hyper-ellipsoid (2n-D)
![Page 13: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/13.jpg)
Statistics (cont’d.)
Center of hyper-ellipsoid is mean shape
Axes are found using PCA– Each axis yields a mode of variation– Defined as , the eigenvectors of covariance matrix
, such that
,where is the kth eigenvalue of S
∑=
=N
iN 1
1ixx
kp
∑=
=N
i
Tidd
N 1
1xxS i
kkk pSp λ= kλ
![Page 14: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/14.jpg)
Approximation of 2n-D Ellipsoid
Most variation described by t-modes Choose t such that a small number of modes
accounts for most of the total variance
∑=
=n
kkT
2
1
λλ If total variance =
∑=
=t
iiA
1
λλ
TA λλ ≅
and the
approximated variance = , then
![Page 15: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/15.jpg)
Generating New Example Shapes
Shapes of training set approximated by: , where is the matrix of the
first t eigenvectors and is a vector of weights
Vary bk within suitable limits for similar shapes
Pbxx += )( t21 ...pppP =T
tbbb )...( 21=b
kkk b λλ 33 ≤≤−
∑=
≤=t
k k
km D
bD
1
2max
22
λ
![Page 16: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/16.jpg)
Application of PDMs
Applied to:– Resistors– “Heart”– Hand– “Worm” model
![Page 17: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/17.jpg)
Resistor Example
32 points
3 parameters capture variability
![Page 18: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/18.jpg)
Resistor Example (cont.’d)
Lacks structure
Independence of parameters b1 and b2
Will generate “legal” shapes
![Page 19: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/19.jpg)
Resistor Example (cont.’d)
![Page 20: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/20.jpg)
Resistor Example (cont.’d)
![Page 21: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/21.jpg)
Resistor Example (cont.’d)
![Page 22: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/22.jpg)
“Heart” Example
66 examples 96 points
– Left ventricle– Right ventricle– Left atrium
Traced by cardiologists
![Page 23: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/23.jpg)
“Heart” Example (cont.’d)
![Page 24: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/24.jpg)
“Heart” Example (cont.’d)
Varies Width Varies Septum
Vary LV Vary Atrium
![Page 25: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/25.jpg)
Hand Example
18 shapes
72 points
12 landmarks at fingertips and joints
![Page 26: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/26.jpg)
Hand Example (cont.’d)
96% of variability due to first 6 modes
First 3 modes
Vary finger movements
![Page 27: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/27.jpg)
“Worm” Example
84 shapes
Fixed width
Varying curvature and length
![Page 28: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/28.jpg)
“Worm” Example (cont.’d)
Represented by 12 point
Breakdown of PDM
![Page 29: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/29.jpg)
“Worm” Example (cont.’d)
Curved cloud
Mean shape:– Varying width– Improper length
![Page 30: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/30.jpg)
“Worm” Example (cont.’d)
Linearly independent
Nonlinear dependence
![Page 31: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/31.jpg)
“Worm” Example
Effects of varying first 3 parameters:
1st mode is linear approximation to curvature
2nd mode is correction to poor linear approximation
3rd approximates 2nd order bending
![Page 32: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/32.jpg)
PDM Improvements
Automated labeling 3D PDMs Nonlinear PDM
– Polynomial Regression PDMs Multi-layered PDMs Hybrid PDMs Chord Length Distribution Model Approximation problem
![Page 33: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/33.jpg)
PDMs to Search an Image - ASMs
Estimate initial position of model Displace points of model to “better fit” data Adjust model parameters Apply global constraints to keep model “legal”
![Page 34: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/34.jpg)
Adjusting Model Points
Along normal to model boundary proportional to edge strength
Vector of adjustments:T
nn dYdXdYdXd ),,...,,( 1100 −−=X
![Page 35: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/35.jpg)
Calculating Changes in Parameters
Initial position: Move X as close to new position (X + dX) Calculate dx to move X to dX
Update parameters to better fit image Not usually consistent with model constraints Residual adjustments made by deformation
csM XxX += ])[,(
)()(])[,(),1(( xXXXxx dddddssM cc +=++++
where,]))[,(,))1((( 1 xyx −−+= − ddssMd cddsM XXxy −+= })[,(
![Page 36: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/36.jpg)
Model Parameter Space
Transforms dx to parameter space giving allowable changes in parameters, db
Recall: – Find db such that – - yields
Update model parameters within limits
Pbxx +=)( bbPxxx dd ++≈+
Pbx + xbbPx dd −++ )(
xPb dd T=
![Page 37: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/37.jpg)
Applications
Medical Industrial Surveillance Biometrics
![Page 38: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/38.jpg)
ASM Application to Resistor
64 points (32 type III) Adjustments made
finding strongest edge Profile 20 pixels long 5 degrees of freedom 30, 60, 90, 120 iterations
![Page 39: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/39.jpg)
ASM Application to “Heart”
Echocardiogram 96 points 12 degrees of freedom Adjustments made
finding strongest edge Profile 40 pixels long Infers missing data (top
of ventricle)
![Page 40: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/40.jpg)
ASM Application to Hand
72 points Clutter and occlusions 8 degrees of freedom Adjustments made
finding strongest edge Profile 35 pixels long 100, 200, 350 iterations
![Page 41: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/41.jpg)
Conclusions
Sensitivity to orientation of object in image to model Sensitivity to large changes in scale? Sensitive to outliers (reject or accept) Sensitivity to occlusion Quantitative measures of fit Overtraining Occlusion, cluttering, and noise Dependent on boundary strength Real time Extension to 3rd dimension Gray level PDM
![Page 42: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/42.jpg)
MR of Brain2
Improves ASM– Tests several model hypotheses– Outlier detection adjustment/removal
114 landmark points 8 training images Model structures of brain together Model brain structures
![Page 43: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/43.jpg)
MR Brain (cont.’d)
![Page 44: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/44.jpg)
MR Brain (cont.’d)
Separate Together
![Page 45: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/45.jpg)
MR Brain (cont.’d)
![Page 46: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/46.jpg)
![Page 47: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/47.jpg)
![Page 48: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/48.jpg)
References
1 – Cootes, Taylor, et al., “Active Shape Models: Their Training and Application.” Computer Vision and Image Understanding, V16, N1, January, pp. 38-59, 1995
2 - Duta and Sonka, “Segmentation and Interpretation of MR Brain Images: An Improved Active Shape Model.” IEEE Transactions on Medical Imaging, V17, N6, December 1998
![Page 49: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/49.jpg)
“Hand Crafted” Models
Built from subcomponents (circles, lines, arcs)
Some degree of freedom May change scale, orientation, size, and
position Lacks generality Detailed knowledge of expected shapes Application specific
back
![Page 50: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/50.jpg)
Articulated Models
Built from rigid components connected by sliding or rotating joints
Uses generalized Hough transform Limited to a restricted class of variable shapes
back
![Page 51: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/51.jpg)
Active Contours – “Snakes”
Energy minimizing spline curves Attracted toward lines and edges Constraints on stiffness and elastic parameters ensure
smoothness and limit degree to which they can bend Fit using image evidence and applying force to the
model and minimize energy function Uses only local information Vulnerable to initial position and noise
back
![Page 52: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/52.jpg)
Spline Curves
Splines are piecewise polynomial functions of order d
Sum of basis functions with applied weights
Spans joined by knots∑
−
=
=1
0
)()(BN
nnn sBxsx
back
![Page 53: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/53.jpg)
Fourier Series Shape Models
Models formed from Fourier series
Fit by minimizing energy function (parameters) Contains no prior information Not suitable for describing general shapes:
– Finite number of terms approximates a square corner– Relationship between variations in shape and parameters is not
straightforward
∑ ++=n
nn naxx )sin(0 φθ
∑ ++=n
nn nbyy )sin(0 φθ
back
![Page 54: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/54.jpg)
Statistical Models of Shape
Register “landmark” points in N-space to estimate:– Mean shape– Covariance between coordinates
Depends on point sequence
back
![Page 55: Active Shape Models: Their Training and Applications Cootes, Taylor, et al. Robert Tamburo July 6, 2000 Prelim Presentation](https://reader038.vdocuments.mx/reader038/viewer/2022102808/56649e205503460f94b0b161/html5/thumbnails/55.jpg)
Finite Element Models
Model variable objects as physical entities with internal stiffness and elasticity
Build shapes from different modes of vibration Easy to construct compact parameterized
shapes
next