alignment of dynamic plantar pressure image sequencestavares/downloads/publications... · 2011. 7....

Alignment of Dynamic Plantar Pressure

Image Sequences

Francisco P. M. Oliveira, João Manuel R. S. Tavares

[email protected]

Outline

♦ Introduction

– Plantar pressure images

– Registration and applications

♦ Registration of static plantar pressure images

– Methodologies: I. Matching of the feet external contours

II. Direct maximization of the cross-correlation (using Fourier transform)

III. Direct minimization of the sum of squared differences (using Fourier transform)

IV. Phase correlation (using Fourier transform)

V. Iterative optimization (using Powell’s method)

– Results and Discussion

♦ Registration of dynamic plantar pressure image sequences

– Methodology

– Results and Discussion

♦ Conclusions

Alignment of Dynamic Plantar Pressure Image Sequences 2 F. Oliveira & J. Tavares

3

A plantar pressure image is a data set that conveys the interaction between foot sole / ground

Introduction: Plantar pressure images

F. Oliveira & J. Tavares Alignment of Dynamic Plantar Pressure Image Sequences

Static pressure images: from a plate with an array of piezoelectric sensors (left) and an light reflection device (right)

4

A dynamic plantar pressure image sequence represents the interaction foot sole / ground for a complete step



Example of footstep sequence obtained at normal walking speed:

EMED® plate and an image sequence

5



Example of footstep sequence obtained at normal walking speed:

Scheme of an light reflection device and an original and the segmented image sequences

camera

mirror

contact layer

+ glass

reflected light glass

pressure opaque layer

lamp

lamp transparent

layer

F. Oliveira & J. Tavares 6

The registration of plantar pressure images is useful in laboratories and

clinics

• It facilitates the automatic computation of several statistical

measures that can be used to study foot pressure distributions

(e.g. diabetic foot)

• It allows the building of mean plantar pressure images and image

sequences that are more accurate to represent the pressure

distribution than single images/sequences

• It simplifies usual diagnosis tasks, such as foot classification, foot

main regions identification, comparison between feet of different

subjects

Introduction: Registration and applications

Alignment of Dynamic Plantar Pressure Image Sequences

The goal of our work has been the development of fast and accurate

methodologies for the automatic registration of plantar pressure images

(of the same subject and of different subjects, statics and dynamics)


Template image I0 Source image I1

I1 registered

Extract the contours

Assemble the cost matrix

Establish the optimal matching

Compute the geometric transformation

Register I1

The cost matrix is built based on geometric features

The matching is established based on the minimization of the sum of the costs associated to the possible correspondences

To searching for the best matching is used an optimization algorithm based on dynamic programming that preserves the circular order of each contour points

Registration of static plantar pressure images: Methodology I - Matching of the feet external contours


Oliveira F, Tavares J, Pataky T (2009) Journal of Biomechanics 42(15):2620-2623


Smoothing

and

threshold

Boundary

points

detection

Contours

extraction

Contours extraction methodology:

Matching example I (images from a Footscan® device):




Matching example II (images from a light reflection device):

Template image and segmented external contours

Source image and segmented external contours

Optimal matching found




Registration example I (images from a light reflection device):

Alignment fully automated

Processing time: 0.125 s (using an AMD Turion64, 2.0 GHz, 1.0 GB of RAM)

Images dimensions: 160x288 pixels

Template image

Overlaped images before registration

Overlapped images after registration

Mean image obtained after registration

Difference image after registration

Source image




Assumption: The higher the cross-correlation between the plantar pressure images, the better the registration is

dxaxIxIaCC II 1010

Cross-correlation between I0

and I1 in function of a shift a:

That can be written as a convolution: aIIdxxaIxIaCC II 1010 *

10

From the convolution Theorem, one have: 1010 * IIII FFF

Thus, computing the product of the Fourier transform of I0 and , and then its inverse Fourier transform, the Cross-correlation can be obtained for all shifts

1I

( represents the convolution operation and F represents the Fourier transform)

*

Registration of static plantar pressure images: Methodology II - Direct maximization of the cross-correlation



The scaling and rotation are obtained from the spectrum images after their conversion to the log-polar coordinate system

The fundaments of this methodology to get the scaling and rotation are based on the shift, scaling and rotation properties of the Fourier transform



Oliveira F, Pataky T, Tavares J (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(6):731-740


Images from the same foot

Images from different feet

Processing time: 0.04 s (using an AMD Turion64, 2.0 GHz, 1.0 GB of RAM)


Rigid transfromation (shift and rotation)

Similarity transformation (shift, rotation and uniform scaling)

Template image

Source image

Overlapped images before and after registration

Registration example (images from a Footscan® device):




Assumption: The lower the sum of squared differences (SSD) between the plantar pressure images, the better registered the images are

dxaxIxIaSSD II 2

1010

Sum of squared differences between I0 and I1 in

function of a shift a:

That can be written as:

dxaxIxI

dxaxIdxxIaSSD II

10

2

1

2

0

2

10

The first two terms of this equation can be directly evaluated, and the third term can be transformed into a convolution and then efficiently evaluated using the Fourier transform

The algorithm implemented is quite similar to the cross-correlation based algorithm. The main difference is the similarity measure considered

Registration of static plantar pressure images: Methodology III - Direct minimization of the sum of squared differences



The algorithm implemented is also similar to the cross-correlation based algorithm

This technique is essentially based on the shift property of the Fourier transform:

If:

Then:

001 xxIxI

uxIeuxIuxi

0

2

10FF

To estimate the shift between the input images, the inverse of the Fourier transform of the cross-power is computed:

Cross-power:

02

10

10 uxie

II

II

*

*

FF

FF

(* represents the complex conjugate)

Registration of static plantar pressure images: Methodology IV - Phase correlation technique



The algorithm ends when a stop criterion is reached, usually related to the evolution of an image (dis)similarity measure

It is robust only against small misalignments → a pre-registration of the input images can be needed

This methodology is based on the searching for the parameters of the geometric transformation that optimize the (dis)similarity measure used

Source image Template image

Pre-alignment (optional)

Source image resampling

Images (dis)similarity measure computation

Optimization algorithm

Geometric transformation computation

Registration of static plantar pressure images: Methodology V - Iterative optimization



Because the convergence to the optimal value is highly dependent on the initial guess for the geometric transformation, we consider one of the previous methodologies to compute the initial guess

As optimization algorithm, we use the Powell’s method

Three image (dis)similarity measures have been considered:

• mean squared error (MSE), mutual information (MI), a dissimilarity measure based on the exclusive-or (XOR)

The geometric transformations allowed are:

• rigid, similarity, affine, projective and polynomials up to 4th degree

Oliveira F, Tavares J (2011) Medical & Biological Engineering & Computing 49(3):313-323




Oliveira F, Tavares J (2011) Medical & Biological Engineering & Computing 49(3):313-323

Registration example (images from a Footscan® device):

Processing time:

- rigid: 0.08 s, 0.09 s, 0.15 s

-projective: 0.13 s, 4.3 s, 0.4 s

- 2nd degree: 0.3 s, 4.4 s, 0.6 s

(using an AMD Turion64, 2.0 GHz, 1.0GB of RAM)





Residual errors obtained after registration of a data set of 30 image pairs (45x63 pixels) using a control geometric transformation (shift: 12.7 mm, -16.256 mm, rotation: 12º):

Methodology Mean residual

error

[mm]

Maximum

residual error

[mm]

Mean

processing

time [ms]

Contours based 1.52 3.01 24

Cross correlation based 0.21 0.44 39

Sum of squared differences based 0.21 0.44 48

Phase correlation based 0.31 0.41 45

Iterative optimization (minimizing the MSE) 3.96e-05 1.07e-04 65

Iterative optimization (maximizing the MI) 3.76e-02 0.17 146

Iterative optimization (minimizing the XOR) 0.12 0.38 70

(Implementation in C++ and tested on a notebook PC with an AMD Turion64 2.0 GHz microprocessor, 1.0 GB of RAM and running Microsoft Windows XP)

Registration of static plantar pressure images: Results and Discussion



Input: template and source image sequences

(III) Estimate the initial linear temporal shift and scaling T1

(I) For each image sequence, build the peak pressure image

(II) Compute the spatial transformation G1 that best aligns the peak pressure images

(IV) Compute the final transformations by optimizing an image similarity measure and using T1 and G1 as initial transformations

(V) Apply the optimal spatial and temporal transformation to the source image sequence

Output: registered source image sequence

Registration of dynamic plantar pressure image sequences: Methodology


Oliveira F, Sousa A, Santos R, Tavares J (2011) Medical & Biological Engineering & Computing 49(7):843-850


The methodology is based on the searching for the parameters of the geometric transformation that optimize an (dis)similarity measure used:

• mean squared error (MSE), mutual information (MI), a dissimilarity measure based on the exclusive-or (XOR)

The geometric transformations allowed are:

• rigid, similarity, affine and projective

For the temporal alignment, four polynomials models are available:

• 1st degree (shift and linear temporal scaling), 2nd, 3rd and 4th degrees (shift and curved temporal scaling)

As the optimization algorithm, the Powell’s method is considered

The initial geometrical alignment is obtained by one of the registration algorithms presented for the alignment of static plantar pressure images

Two optimization schemas were considered: unconstrained and constrained (the limit frames of a sequence must agree with the limit frames of the other sequence)

Registration of dynamic plantar pressure image sequences: Methodology




Before registration

After registration

Registration example I (slow motion, sequences from an Emed® system):

Template sequence

Source sequence

Overlapped sequences

Image similarity measure: MSE

Processing time: 4 s (using an AMD Turion64, 2.0 GHz, 1.0 GB of RAM)

Sequences dimension: 32x55x13, 32x55x18

Emed® system: 25 fps, resolution 2 pixels/cm2

Registration of dynamic plantar pressure image sequences: Results and Discussion



Template sequence

Source sequence



Source sequence aligned using a temporal

transformation of 1st degree

Source sequence aligned using a temporal

transformation of 4th degree

Registration example I (sequences from an Emed® system):


Image similarity measure: MSE

Processing time: 1 min (using an AMD Turion64, 2.0 GHz, 1.0 GB of RAM)

Sequences dimension: 160x288x22 160x288x25

Light reflection device: 25 fps, resolution 30 pixels/cm2

Template sequence

Source sequence

Overlapped sequences

Registration example II (slow motion ,sequences from a light reflection device):

Before registration

After registration






Applied temporal

transformation

Polynomial model

used in the

temporal

alignment

Unconstrained

optimization Constrained optimization

Maximum

spatial RE

[mm]

Maximum

temporal RE

[s]

Maximum

spatial RE

[mm]

Maximum

temporal RE

[s]

f1

1st 0.01 0.0002 0.26 0.0112

2nd 0.01 0.0002 0.08 0.0083

3rd 0.01 0.0003 0.05 0.0052

4th 0.01 0.0003 0.05 0.0049

f2

1st 0.44 0.0501 6.38 0.2211

2nd 0.02 0.0002 0.16 0.0124

3rd 0.02 0.0003 0.13 0.0104

4th 0.02 0.0020 0.10 0.0073

f3

1st 0.07 0.0127 0.82 0.0435

2nd 0.08 0.0080 0.26 0.0200

3rd 0.02 0.0002 0.02 0.0025

4th 0.02 0.0014 0.02 0.0019

f4

1st 0.16 0.0540 0.82 0.0860

2nd 0.48 0.0340 0.53 0.0485

3rd 0.04 0.0056 0.13 0.0104

4th 0.03 0.0030 0.14 0.0095

Residual errors obtained using control geometric and temporal transformations (sequences resolution 2 pixels/cm2, 40 experiments):





Mean MSE values obtained after registration considering real plantar pressure sequences (168 different sequences pairs were used):


(28 subjects - 3 pairs of the left foot and 3 pairs of the right foot per subject)


For the registration of static plantar pressure images, we can point out that:

I. The methodology based on the iterative optimization was the most

accurate. This was already expected since it started with good initial

registrations obtained using one of the remainder methodologies

II. The methodologies based on the direct optimization of the cross-

correlation and sum of the squared differences and the phase

correlation technique achieved good and identical results

III. The methodology based on the matching of the feet external

contours was the fastest; but, its accuracy was the lowest

IV. The methodologies based on the cross-correlation, sum of the

squared differences, phase correlation and matching of the

contours are robust to arbitrary shifts and rotations and have

considerable robustness to linear scaling

Conclusions



For the registration of static plantar pressure images, we can point out that:

V. The lowest residual error was obtained by minimizing the mean

squared error, indicating that this dissimilarity measure is a good option

when registering plantar pressure images

VI. The methodology based on the iterative optimization allows the

computation of geometric transformations beyond the rigid and

similarity transformations computed by the other methodologies that

can be specially useful in inter-subject (non-rigid) registration

VII. For rigid geometric transformations, the results obtained by

optimizing the MSE, XOR and MI are similar. For non-rigid

registration, care must be taken when the MSE is minimizing since

large image deformations can occur

Conclusions



For the registration of dynamic plantar pressure image sequences, we can point out that :

I. Considering control sequences, the best temporal accuracies were obtained using 3rd and 4th temporal transformation models

II. Using real misaligned sequences, the best accuracy were obtained using a 4th degree polynomial for the temporal model; however, the visual results obtained are indistinguishable from the ones obtained considering a 3rd degree polynomial as temporal model

III. The unconstrained optimization lead always for better accuracy than the constrained optimization

Conclusions


Acknowledgments

This work was partially done in the scope of the following projects financially supported by Fundação para a Ciência e a Tecnologia (FCT), in Portugal :

• “Methodologies to Analyze Organs from Complex Medical Images – Applications to Female Pelvic Cavity”, PTDC/EEA-CRO/103320/2008

• “Aberrant Crypt Foci and Human Colorectal Polyps: mathematical modelling and endoscopic image processing”, UTAustin/MAT/0009/2008

• “Cardiovascular Imaging Modeling and Simulation - SIMCARD”, UTAustin/CA/0047/2008

The first author would like to thank Fundação Gulbenkian, in Portugal, for his PhD grant

F. Oliveira & J. Tavares 30 Alignment of Dynamic Plantar Pressure Image Sequences

Alignment of Dynamic Plantar Pressure

Image Sequences

Francisco P. M. Oliveira, João Manuel R. S. Tavares

[email protected]

alignment of dynamic plantar pressure image sequencestavares/downloads/publications... · 2011. 7....

Documents