deformable model acquisition and validation · 2011-11-22 · deformable model acquisition and...
TRANSCRIPT
Deformable Model Acquisition and Validation
by
Jochen Lang
M.Sc., York University, Toronto, Canada, 1996
B.Eng.(Hons), University of Plymouth, Uk, 1992
Diplom-Ingenieur (FH), Fachhochschule Ulm, Germany, 1992
A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF
THE REQUIREMENTS FOR THE DEGREE OF
Doctor of Philosophy
in
THE FACULTY OF GRADUATE STUDIES
(Department of Computer Science)
We accept this thesis as conformingto the required standard
The University of British Columbia
October 2001
c© Jochen Lang, 2001
Abstract
Objects deform in response to contact forces exerted on them. The deformationdepends on material properties, the geometry of the object and external forces.This thesis develops a robotic system for automatically acquiring observations of adeforming object and for estimating a model of the deformation from these obser-vations.
Models of deformable objects are in wide-spread use in simulation, computergraphics and virtual reality. Deformation, impact and fitting simulation aid man-ufacturing. In computer graphics deformable objects are designed and animated.Medical simulators incorporating physical models of organs and tissue are a signifi-cant emerging virtual reality application.
The material properties of deformable models are often assigned based onmechanical (and other) testing of material samples. Material samples do not repre-sent commonly simulated objects well if there is a high variability in the materialof the object, e.g., due to mixed material, unknown material, or material with im-perfections. In contrast to material sampling, this thesis develops a method to scanthe deformation behavior of a complete object. The scanning is analogous to thescanning of the visual appearance of an object. The scan captures the individualresponse of a complete object to contact forces. The result of the scan is not only adeformable model but also data which serves to validate the model. This validationstep gives a qualitative and sometimes quantitative assessment of the suitability ofa model.
ii
Contents
Abstract ii
Contents iii
Acknowledgements vii
1 Introduction 1
1.1 Problem Statement and Overview . . . . . . . . . . . . . . . . . . . . 5
1.1.1 Hypothesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1.2 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1.3 Computational Deformation Model . . . . . . . . . . . . . . . 6
1.1.4 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2.1 Deformable Models . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2.2 Inverse Problems in Elasticity . . . . . . . . . . . . . . . . . . 12
1.2.3 Theoretical Results . . . . . . . . . . . . . . . . . . . . . . . . 12
1.2.4 Elastic Parameter Estimation . . . . . . . . . . . . . . . . . . 13
1.3 Summary of Related Work . . . . . . . . . . . . . . . . . . . . . . . . 14
2 Elastostatic Object Behavior 15
iii
2.1 Discrete Green’s Functions Matrix . . . . . . . . . . . . . . . . . . . 16
2.2 Deformation Models in Continuum Mechanics . . . . . . . . . . . . . 18
2.2.1 Strain Tensor . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.2 Stress Tensor . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.2.3 Constitutive Equation . . . . . . . . . . . . . . . . . . . . . . 21
2.2.4 Navier’s Equations . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3 Solution Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3.1 Finite Element Model . . . . . . . . . . . . . . . . . . . . . . 23
2.3.2 Boundary Element Model . . . . . . . . . . . . . . . . . . . . 25
2.4 Other Physics Based Deformable Models . . . . . . . . . . . . . . . . 28
2.4.1 Particle Systems . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4.2 Deformable Superquadrics . . . . . . . . . . . . . . . . . . . . 30
2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3 The ACME Facility 32
3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.1.1 ACME Devices . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.1.2 Control Architecture . . . . . . . . . . . . . . . . . . . . . . . 37
3.1.3 Programming ACME . . . . . . . . . . . . . . . . . . . . . . 38
3.1.4 Sensor Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.2 ACME Deformation Experiment . . . . . . . . . . . . . . . . . . . . 41
3.3 Geometric Modeling with ACME . . . . . . . . . . . . . . . . . . . . 44
3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4 Deformation Measurement 47
4.1 Robust Flow Estimation . . . . . . . . . . . . . . . . . . . . . . . . . 49
iv
4.1.1 Robust Range-Flow by Additional Imagery . . . . . . . . . . 51
4.1.2 Implementation Details . . . . . . . . . . . . . . . . . . . . . 54
4.1.3 Results for Rigid Body Motion . . . . . . . . . . . . . . . . . 56
4.1.4 Examples of Object Deformation . . . . . . . . . . . . . . . . 62
4.2 Estimation of Vertex Displacement . . . . . . . . . . . . . . . . . . . 62
4.2.1 Displacement Results . . . . . . . . . . . . . . . . . . . . . . 63
4.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
5 Estimation of Discrete Green’s Functions 68
5.1 Set-Up of Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5.2 Local Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.2.1 Least Squares . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
5.2.2 Measurement Design Matrix . . . . . . . . . . . . . . . . . . 73
5.2.3 Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.3 Global Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
5.3.1 Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . 82
5.3.2 Outliers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
5.3.3 Hole Filling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.4 Mesh Refinement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.5.1 Plush Toy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
5.5.2 Anatomic Soft-Tissue Human Wrist Model . . . . . . . . . . 94
6 Inverse Problem Solution 102
6.1 Boundary Element Method for an Inverse Elastic Problem . . . . . . 103
6.1.1 Elastic Parameter Estimation . . . . . . . . . . . . . . . . . . 103
v
6.1.2 Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
6.1.3 Results for Half Ball . . . . . . . . . . . . . . . . . . . . . . . 106
6.2 Approximate Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
6.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
7 Conclusions and Future Work 113
7.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
Appendix A Linear Triangular Elements 117
Appendix B Geometric Modeling with ACME 120
Bibliography 125
vi
Acknowledgements
I would like to thank my thesis supervisors Dr. Robert Woodham and Dr. DineshPai. I am grateful for Bob’s thoughtful consideration and questions, as well as hisencouragement. I like to thank Dinesh for his enthusiasm and support in exploringthe ideas of this thesis. Also, I would like to thank my committee member Dr. UriAscher for his valuable comments.
I am thankful for many helpful discussions with Doug James on deformablemodels and haptic rendering. Dr. John Lloyd’s diligent work on the University ofBritish Columbia (UBC) Active Measurement facility (ACME) enabled the exper-imental work of this thesis. This thesis also profits from the work of all the otherACME group (former) members: Rod Barman, Timothy Edmunds, Julian Fong,Stewart Kingdon, Paul Kry, Josh Richmond, and Som Yau. I am also thankful toDon Murray at Point Grey Research (PGR) for help with the PGR Triclops StereoVision SDK. Furthermore, I like to acknowledge Dr. Gerhard Roth at the NationalResearch Council, Canada for making his volumetric mesh creation software avail-able to me. Jesse Hoey, Paul Kry and John Lloyd provided helpful comments on adraft of (parts of) this thesis.
I am grateful for the financial support of my doctoral studies by the Uni-versity of British Columbia and by the National Sciences and Engineering ResearchCouncil of Canada. I would like to acknowledge IRIS Precarn Centres of Excellenceand their support of the reality-based modeling project (ACME) at UBC.
Jochen Lang
The University of British ColumbiaOctober 2001
vii
Chapter 1
Introduction
Interactive simulation of physical objects is desirable for numerous applications in
the areas of virtual and augmented reality. The goal in interactive simulation is to
ensure virtual objects resemble their real counterparts as closely as possible but at
the same time maintain interactivity. The consequence of this trade-off is the need
to employ an approximate model of the physical behavior of objects. In general, the
realism mathematical models of objects can achieve depends on the proper choice
of the physical parameters of the model. How should these physical parameters be
chosen? This thesis argues that realistic simulated behavior requires parameters
which are based on measurements and that these measurements can be acquired
efficiently. The physical behavior explored is the deformation of solid objects due to
contact forces. This thesis shows how to efficiently scan this behavior in a robotic
measurement facility very similar to the scanning of object geometry in a shape
scanner.
Simulations gain realism if the effect of forces on an object is represented
realistically. Forces govern many behaviors of real world objects. In particular,
forces deform an object depending on its geometric and material properties. The
1
simulation of realistic virtual deformations necessitates the inclusion of forces, ob-
ject geometry and realistic material properties. Realistic deformation of virtual
objects is required in medical training simulators [31], in haptic interfaces to virtual
worlds [113], and in design of robotic tasks involving soft materials [59, 60]. Med-
ical simulators allow students to practice [123], and help to predict and plan some
surgical procedures [67]. Medical robotics can also profit from user interfaces with
haptic feedback [112].
Force simulation plays a very important role in the creation and animation
of virtual worlds [40]. Forces are a very natural way for a designer to interact with
virtual objects. Realistic force responses are intuitive because they copy the behavior
of our surrounding physical world. A realistic force response is a powerful method
to manipulate complex surfaces while meeting other constraints. An example is the
editing of a model of a human face, the skin is modified by the user but forces due
to bone, muscle and tissue structure resist the changes and ensure the modification
is in accordance with physical limits [126]. Another example is the Virtual Clay
system of McDonnell, Qin and Wlodarczyk [79] where forces during sculpting are
simulated in real time allowing the creation of virtual sculptures.
Many computer graphics techniques have evolved over the years to model
deformations [40]. These techniques vary from full continuum models for elastic
materials, through mass-spring particle systems, to very ad-hoc methods. But even
the most elaborate deformation model will be unrealistic if the simulated material
does not match the real physical object. The measurement of these material prop-
erties has so far only been done in a “material oriented” way, e.g., in physical and
mechanical testing laboratories. As a result, many computer graphics methods do
not model material properties for deformation simulation of physical objects. Fur-
2
thermore, simulation results with a specific computational model are commonly not
validated with respect to physical object behavior [40, 31].
The key for future increase in realism is observations. These observations
of physical phenomena need to be sufficient to develop a computational model for
objects to be simulated. The combination of a measurement approach, a compu-
tational model and a robotic measurement system is required. This combination
creates a scanner for deformation behavior of objects which is the goal of this the-
sis. Scanning deformation and other physical interaction behavior has recently been
described by Pai et al. [95].
Scanning the geometry of objects is common practice today. Shape infor-
mation for a wide class of everyday objects can be acquired with scanners typically
based on optical triangulation. Another physical property beside shape which can
be scanned is the reflectance of an object represented by its bidirectional reflectance
distribution function (BRDF) [116, 20]. Given the ubiquitous graphics rendering
capabilities of today’s computers, reality-based animations of three-dimensional ob-
jects are commonplace. Up to very recently, the physical behavior of objects due to
contact forces could not be scanned. This is part of the reason why these behaviors
are rarely rendered.
Scanning physical object behavior due to contact forces may be understood
as an inverse problem. In an inverse problem a model is determined based on obser-
vations. (Given a parametric model that applies, the process of determining model
parameters forms the field of parameter estimation.) Inverse problems related to de-
formable models are being researched in the mechanics of solids and structures [66].
In mechanics, computational methods are often more elaborate than in computer
graphics since accuracy of simulation results is critical in many applications (e.g., in
3
structural analysis in civil engineering). Surprisingly, the verification and validation
of computational methods in inverse problems in mechanics are often neglected as
well. The performance of computational methods for inverse problems is commonly
evaluated by comparison with either an analytic solution or by comparison with the
same method using finer discretization of the model domain (or boundary). Direct
experimental validation provides a more meaningful characterization of a method.
The scanning method of this thesis makes experimental validation simple.
However, scanning of object deformation does not need to be approached as
an inverse problem. Deformation behavior can be directly observed and rendered.
The interior stress and strain, often the quantities of interest in mechanics, are not
directly necessary to render deformation behavior. The purpose of the deformable
model in computer graphics is to allow physically realistic rendering of the defor-
mation of a real object under force. The model has to encode the response of the
object as a whole to a force. What interior structure causes that response is irrele-
vant. However, the scanning of object deformation is also the first step in an inverse
problem solution. This thesis also contains a method to estimate physical model
parameters based on scanned behavior. The model assumption in this case is that
the object behaves as a homogeneous linear isotropic elastostatic body.
Next, in Section 1.1, a concise problem statement is given which may guide
the reader through the following chapters. Section 1.2 reviews work related to this
thesis.
4
1.1 Problem Statement and Overview
1.1.1 Hypothesis
The central hypothesis is: A framework which acquires and validates a computational
model of object deformations by displacement and force measurements enhances the
utility of the model. This utility is in the rapid availability of the model, and in
the knowledge of the applicability and limitations of the model available from the
comparison between model behavior and actual object behavior.
1.1.2 Approach
This thesis develops a measurement apparatus, a measurement method and an esti-
mation procedure to fit a computational deformable model to an object’s measured
deformation behavior. A result of the fitting process is a model and a residual which
characterizes the ability of the model to explain the measurements. The model is
able to synthesize (novel) object behavior which allows one to analyze and verify
model behavior. The approach taken in this thesis may be characterized as an
analysis by synthesis framework [130].
This thesis develops and utilizes the UBC active measurement facility (ACME)
for object deformation measurements. ACME is a robotic facility which automates
object measurement tasks. Object deformation is characterized by surface displace-
ments and contact forces, and potential topological changes in the case of inelastic
deformation. A measurement facility for object deformation has to be capable of
applying and measuring contact forces, as well as measuring the displacement at
the contact itself and over the free surface of the object.
The measurement method of this thesis is a direct method in which the
5
behavior of the object to be modeled is directly measured. The forces over the
surface of the object and the displacement of the free surface of the object are
observed. The force and displacement are measured by a robotic probe while it
actively deforms the object. The displacement of the free surface is observed visually
with a trinocular stereo vision system. The three-dimensional displacement vectors
over the surface are obtained with simultaneous (geometric) stereo and optical flow.
The measurement of the initial (undeformed) shape of an object is also based on
stereo range data.
The estimation procedures developed in this thesis are applicable with dif-
ferent prior knowledge. A procedure with very weak assumptions about the object
behavior is developed, as is a method with very restrictive assumptions about the
behavior. Robustness is emphasized during the estimation process. Outliers typical
of many visual techniques are handled by measurements which overconstrain the
solution and by regularization.
1.1.3 Computational Deformation Model
In this thesis the deformable model is a discrete linear elastostatic boundary model.
The boundary model is based on a triangular surface tessellation of the object. The
model linearly relates the traction on surface elements to the displacement of surface
elements. The model coefficients depend on the object shape and its elastic material
properties. The model is adequate to render small deformations1 which an object
undergoes if a force is exerted on it. In the case of a homogeneous, isotropic and
linearly elastic object the above linear boundary model corresponds to a discretized1Here, small means the small deformation strain tensor for linear-elastic material is
employed neglecting second order effects (see Section 2.2.1). Also, the initial tessellation ofthe objects surface is unchanged during the deformation limiting the approximation of thediscrete representation of object geometry.
6
continuum mechanics model. In this case, the coefficients of the model can be
derived with a triangular surface tessellation in a finite element formulation of the
object’s elastostatics (see Section 2.3). The efficient haptic and visual rendering of
this type of boundary model is developed in the thesis work of James [56].
1.1.4 Overview
Work related to this thesis can be found in Section 1.2. The reader unfamiliar
with elasticity may find helpful explanations regarding the cited work in Chapter 2.
Chapter 2 describes elastostatic object behavior and the deformable model of this
thesis in detail. Chapter 3 summarizes the capabilities of the ACME robotic mea-
surement system and gives details of features specific to deformation modeling. The
visual deformation measurement approach is introduced in Chapter 4. This Chapter
includes results and ground truth comparison for the method. The next two chap-
ters describe the estimation method: Chapter 5 explains the procedure for acquiring
estimates for the linear boundary model; Chapter 6 describes the inverse problem
solution to elastic parameter estimation for objects made of isotropic homogeneous
linear elastostatic material. This thesis concludes with Chapter 7. Possibilities for
continuation of this work are discussed.
1.2 Related Work
Related work to thesis has been done in a number of fields with varying goals. In
computer graphics the efficient representation of deformation behavior of solids has
been surveyed by Gibson [40], while Section 1.2.1 reviews deformation models in
interactive simulation. The focus of the review is on how parameters for the em-
ployed model are chosen. Section 1.2.2 introduces work on related inverse problems
7
in elasticity. Theoretical results have been obtained for the inverse problem of elastic
parameter estimation and are summarized in Section 1.2.3. Section 1.2.4 discusses
numerical and experimental methods in elastic parameter estimation. Finally, Sec-
tion 1.3 gives a brief summary of related work.
1.2.1 Deformable Models
This section focuses on deformation models for physically realistic interactive sim-
ulation.
At the Forschungszentrum Karlsruhe in Germany, the development of medi-
cal training simulators has been ongoing since the early 1990’s [67]. The simulation is
based on KISMET (Kinematic Simulation, Monitoring and Off-Line Programming
Environment for Telerobotics). The Karlsruhe Endoscopic Surgery Trainer employs
two types of deformable models: particle systems [125] and linear elastostatic finite
element models with and without condensation [18]. The stiffness of living tissue is
estimated based on ultrasound measurements and compared to stress-strain curves
of animal samples obtained in tension test machines and with a compression test [78].
The stress-strain relation is fit to a third order polynomial σ = Eε(1 + P1ε + P2ε2)
where σ is the stress, ε the strain, E is a scaling parameter (Young’s modulus) and
P1, P2 are non-linear shape factors. This relationship is simulated with a particle
system.
Szekely et al. report [122] a Laparoscopic Surgery Simulator (LASSO) at
the ETH Zurich, Switzerland, which includes deformable models of human organs.
The organs are modeled as hyper-elastic material. The simulation is based on finite
elements computed on a specifically designed parallel computer with 64 DEC Alpha
processors. The material parameters are estimated based on measurements with an
8
aspiration tube; a novel instrument which applies a tensile load to a small area of
the tissue by suction and observes the displacement visually [129]. The material law
employed in simulation and estimation is based on a modification of the Mooney-
Rivlin strain energy function W = µ(J1− 3) + α(J2 − 3) + 12κ(J3 − 1)2 where J1, J2
and J3 are the reduced invariants of the right Cauchy-Green deformation tensor and
µ, α and κ are the material parameters. The estimation is based on a finite element
model of the area surrounding the probe without the inclusion of bulk modulus κ.
The non-linear least squares error is minimized based on displacement and tension
force measured over time. However, due to instabilities, α is arbitrarily set to zero
and only µ is estimated.
Within the Sharp project at INRIA, Rhone-Alpes, C. Laugier and his group
modeled human tissue for medical simulators [27, 29]. The force interaction between
an echo-graphic sensorhead and the human thigh is modeled. The human thigh is
represented with a particle system including linear and non-linear springs. The
parameters of the springs are derived from measurements with a robotic probe [27].
The model is optimized to match the measured local force response within 5%. The
authors noted the stiff behavior of the human thigh and the difficulty of modeling
this behavior with a particle system. A haptic interface to the deformable model
has been presented in [26]. A human liver is also modeled however no measurements
of the behavior are reported [29].
The Epidaure project at INRIA, Sophia-Antipolis, is also modeling deformable
objects for medical simulation. Bro-Nielsen and Cotin [18] introduced finite element
modeling with condensation into interactive simulation. The model applies to lin-
ear elastostatic simulation. Their model representation is strongly related to the
presentation employed in this thesis as detailed in Section 2.3.1. More recently,
9
Cotin, Delingette and Ayache [25] utilized a tensor-mass model solved with finite
elements for soft tissue in a dynamic simulation. This model is combined with an
elastostatic model in a hybrid approach. The topology of the model can be changed
in real-time by removing complete tetrahedra. Very recently Picinbono, Delingette
and Ayache [99] have adapted a dynamic model with a non-linear large deformation
strain tensor. The model is solved with finite elements and mass-lumping with an
external incompressibility constraint. A comparison of the behavior of a physical
brain sample in a compression test to a simulation is reported by Cotin, Delingette
and Ayache [24].
Keeve et al. [64] employ deformation models of facial tissue to predict the
outcome of cranofacial surgery. Simulation results are compared with the actual
outcome of the operation. Two types of simulation are reported: one based on
particle systems with piecewise linear springs; the other based on finite element
models with a non-linear strain-stress relationship. The finite element solution is
not interactive.
Basdogan et al. [7] model the deformation behavior of soft tissue with free
form surfaces. Later, De and Srinivasan [28] substituted a physical boundary model.
They model the finger-pad as a membrane in plane strain filled with incompressible
fluid. The model is linear but varies over time. Simulation results are compared to
experimental measurements.
Deformable models in VESTA (Virtual Environments for Surgical Training
and Augmentation) at the University of California2 are computed with finite ele-
ments with mass lumping [131] as in [136]. The non-linear material law is based
on Mooney-Rivlin and a Neo-Hookean model. The computation is sped up with a2The project is a joint cooperation between the University of California, San Francisco,
Berkeley and Santa Barbara.
10
geometrically adaptive scheme. The group is also investigating soft tissue behavior
during a surgical procedure (needle driving) and under tension [19].
Koch et al. [65] report a craniofacial surgery simulation. This system models
tissue behavior with a surface finite element method attached to a volumetric particle
system. The model is based on individual bone, fat and soft tissue distribution
deduced from a CT scan of the head. The simulation result is compared qualitatively
to the result of the surgery. The interactive surgical cutting simulator of Biesler,
Maiwald and Gross [11] employs particle systems only.
Haptic manipulation of soft objects and tissue based on the finite element
method with the small deformation linear strain tensor is reported in [50] and [88].
Debunne at al. [30] describe a space and time adaptive scheme for dynamic simu-
lation of soft objects. The solution method employed is finite elements with mass
lumping as in the off-line simulation of O’Brien and Hodgins [91]. Other interactive
deformation models can be found in [104, 36, 15, 23].
In the context of haptic simulation the capabilities of the human sensory
system can provide some guidance to how realistic simulation needs to be. In [12]
the capability of the human haptic system is summarized. The stiffness necessary
for an object to be perceived rigid is at least 25N/mm. The JND (just noticable
difference) of compliance is about 5−15% if the displacement is fixed during the trial
and the object surface is rigid. The maximum force that can be exerted controllably
by a human finger pad is about 100N with 0.04N or 1% resolution, whichever is
higher.
11
1.2.2 Inverse Problems in Elasticity
Inverse problems aim to discover the cause of a physical phenomenon from its man-
ifestation. Problems in elasticity are usually labeled inverse when the unknowns of
the problem are not displacements or strains, and not traction or forces [66]. The
identification of cracks, cavities or flaws based on some deformation of the object
(e.g., [54, 9, 90, 121]) form one set of inverse problems. The estimation of elastic co-
efficients given undeformed and deformed object state as well as the corresponding
deformation forces (e.g., [117, 39, 107, 45, 97]) is a problem of parameter estima-
tion. The boundary conditions may be unknown on part of the boundary. The
estimation of these boundary conditions from, e.g., other over-specified boundaries
(e.g., [10]) or interior measurements is a problem occurring in non-destructive test-
ing. Another set of inverse problems deals with inverse geometry, the calculation
of the undeformed object shape given the forces and the deformed shape of the
object (e.g., [44, 43]). Applications of inverse elasticity problems can be found,
amongst others, in medical imaging [37], non-destructive testing (e.g., [9]), compos-
ites (e.g., [46]), civil engineering (e.g., [4]) and earth sciences (e.g., [41]).
1.2.3 Theoretical Results
Related theoretical research in elastic parameter estimation considers the question:
do boundary conditions provide enough information to deduce the elastic properties
of a material? Various properties of the map of traction and displacement boundary
conditions over an inhomogeneous medium have been proven. In an inhomogeneous
medium the elastic constants are varying over the medium, i.e., they become func-
tions over the domain. Akamatsu, Nakamuru and Steinberg [1] prove that a map of
complete boundary conditions determines the Lame functions λ, µ in the case of a
12
two-dimensional, isotropic, linear-elastic, inhomogeneous material (some conditions
on the Lame functions and the smoothness of the elastic domain apply). A similar
result has later been derived for higher dimensions [85]. Furthermore, Nakamura
and Uhlmann [86] also showed that the functions are unique. Additionally, a generic
anisotropic linear-elastic tensor in two dimensions can be determined in two dimen-
sions [87]. Unfortunately, this result can not be extended to higher dimensions [84].
These results may give some guidance in designing a measurement procedure
and apparatus. They also support the idea that modeling of anisotropic, inhomo-
geneous deformable object from boundary measurements is better approached as a
direct scanning problem rather than an inverse problem.
1.2.4 Elastic Parameter Estimation
Solution strategies for inverse problems of elasticity may be classified based on the
need for observations inside a domain of the elastic continuum. Interior observa-
tions require a more elaborate experimental setup than observations of the bound-
ary alone. Parameter estimation using domain methods is applied to beam struc-
tures [114, 4] and for two-dimensional domains [107, 46] and if special measurement
techniques (like ultrasound) allow for domain interior measurements [41, 61, 103].
(The solution procedure for all the domain methods in [117, 114, 4, 107, 46, 41, 61,
97] is based on finite elements, except for the finite difference approach of [103].) If
the inverse problem involves a three-dimensional continuum and no interior measure-
ments are available, the problem is more difficult. The boundary element method
has been applied in three-dimensions even if interior measurements are not avail-
able [9].
Elastic coefficients can be estimated with at least two different inverse ap-
13
proaches (as defined by S. Kubo [66]): a direct method, or an optimization method
based on either force error minimization or displacement error minimization [4]. One
may isolate the elastic parameters for an homogeneous isotropic material [66, 107]
resulting in a direct method. A direct method may also be obtained by re-arranging
the boundary element expression [71] (see Section 6). Displacement or strain error
minimization are applied in [39, 114, 4] and a sensitivity analysis of each respective
method is provided in [39, 115, 5]. In a case-study of a beam structure [5], Banan,
Banan and Hjelmstad find that force error minimization leads to very similar results
as displacement error minimization.
1.3 Summary of Related Work
The trend in interactive simulation of deformable models has been towards phys-
ically more accurate models. Finite element solutions to problems with physical
constitutive relations of the continuum are often employed. The computational re-
quirements limit the size and the complexity of the stress-strain relationship which
can successfully be modeled at interactive update rates. Dynamic models with large
deformation are especially challenging. The need for an appropriate choice of elas-
tic parameters for these increasingly realistic models has been recognized. Elastic
parameter estimation is researched in various fields, however no generally accepted
solution for three-dimensional continua exists. The scanning of deformation behav-
ior developed in this thesis represents an alternative approach to acquire realistic
deformation models.
14
Chapter 2
Elastostatic Object Behavior
Deformation of solids has been studied extensively in physics, civil engineering, me-
chanical engineering and biomechanics. The theory of elasticity provides analytical
tools to describe the deformation of elastic solids. Linear elasticity theory forms
the basis of empirical methods which describe the tension, shear and compression
of engineering solids.
Section 2.1 states the model of linear static elasticity which this thesis em-
ploys. The model is a discrete Green’s functions matrix which is based on continuum
mechanics. Section 2.2 contains a brief introduction into linear elastostatic models in
continuum mechanics which the reader familiar with continuum mechanics may wish
to skip. In Section 2.3, solution strategies of the boundary value problem associated
with elastic solids are discussed to the extent that they are the basis of this thesis.
Section 2.3 details how either the Finite Element Method or the Boundary Element
Method may be employed to derive a discrete Green’s functions matrix. Again,
the reader familiar with these methods may wish to skip these sections. Finally,
some common modeling approaches used in computer vision, computer graphics
and haptics are summarized in Section 2.4.
15
2.1 Discrete Green’s Functions Matrix
The elasticity of a solid determines its deformation when external forces act on it.
In static equilibrium, the net effect of the external forces is zero and the solid’s
boundary remains stationary (Section 2.2.4 gives a mathematical description). The
domain of the problem is a solid Ω with its boundary Γ. The boundary has two
regions of boundary conditions, Γ0 and Γ1, such that Γ0 ∪ Γ1 = Γ and Γ0 ∩ Γ1 = 0.
On the boundary Γ0 displacements are prescribed and on the boundary Γ1 tractions
are prescribed, (see Figure 2.1), i.e., Γ0 is fixed while Γ1 is free. A traction p is
a force f normalized over the surface area A it is affecting, i.e., p = f/A. (Note,
that instead of tractions, forces may be incorporated directly since the undeformed
object surface is a constant. Details can be found in Section 2.3.) The Green’s
functions relate a field of displacement vectors to a field of traction vectors on the
boundary of the elastic solid.
The boundary in the model of this thesis is discretized into a triangular mesh
with k = 0 . . . n vertices (see Figure 2.1). The displacement and traction vector at
each vertex are denoted by uk and pk, respectively. The basis functions of the
displacement and traction fields are hat functions φ centered at a vertex. The hat
function’s magnitude is one at the vertex, linearly decays to zero at surrounding
vertices, and is zero elsewhere on the surface (see Appendix A for a mathematical
definition).
The displacement and traction vectors are combined into block vectors u =
[uT1 , . . . ,uT
n ]T and p = [pT1 , . . . ,pT
n ]T , respectively. For a given boundary configu-
ration, the block vectors u and p can be rearranged such that all prescribed values
are collected in a block vector v. Entries vk = uk if the vertex k is on the sur-
face Γ0, while vk = pk if the vertex k is on the surface Γ1. The complementary
16
k
l
k
u
kp
1
pl
ΩΓ
Γ0u = 0l
Figure 2.1: Discrete Boundary Value Problem associated with an Elastic Solid
displacement and traction vectors are entered into a block vector v. The matrix
relating the prescribed values v and v is the discrete Green’s functions matrix is Ξ
(see Equation 2.1). Equation 2.1 is central to this thesis.
v = Ξv (2.1)
In simulation, a discrete Green’s functions matrix Ξ and the input vector of
prescribed tractions and displacements v allows one to calculate the resulting de-
formed shape of the solid as well as the reaction forces. In estimation, the Green’s
functions matrix Ξ is to be determined from over-specified displacement and trac-
tion boundary conditions. The block vectors v and v are measured or known in the
estimation of the Green’s functions matrix Ξ. This measurement and estimation
problem is the topic of the remainder of this thesis. The next sections explore in
detail the assumption in the derivation of the Green’s functions matrix. In Sec-
tion 2.2, the basic definitions of elasticity are reviewed concluding with the Navier’s
equations. Section 2.3 shows how the Navier’s equations can be solved resulting in
the discrete Green’s functions matrix.
17
2.2 Deformation Models in Continuum Mechanics
Mathematical descriptions in continuum mechanics are based on some definition of
the material stress state and the corresponding strain of the material. In general, a
stress is a force density acting at an internal section of material area while strain is
the relative change in dimension of a material. Various mathematical definitions of
stresses and strains are possible, see Sections 2.2.1 and 2.2.2. The strain of a material
is caused by its stress state (and potentially its stress history). The constitutive
equations of the material relate stress and strain. Constitutive equations together
with an equilibrium condition lead to Navier’s equations for an elastic solid. The
following brief Sections 2.2.1, 2.2.2 and 2.2.3 give more concise definitions.
2.2.1 Strain Tensor
The deformation of a solid is the change in the location of an element relative to other
elements of the solid. For the definition of a strain tensor, define a displacement
vector u from the position x of a point on a body in undeformed state to the same
point y on the body in the deformed state. The displacement is u = y − x. Then,
we can define a Jacobian from undeformed to deformed state as
J = I + J =
1 + ∂u1∂x1
∂u1∂x2
∂u1∂x3
∂u2∂x1
1 + ∂u2∂x2
∂u2∂x3
∂u3∂x1
∂u3∂x2
1 + ∂u3∂x3
.
The above Jacobian may be employed to define a strain tensor. Two points
on the body P and Q are a squared distance (δl0)2 = (dx)T dx apart if the body
is undeformed. In the deformed state, they are a distance (δl)2 = (dy)T dy apart.
Since y = x + u it follows that dy = dx + ∂u/∂x dx = (I + ∂u/∂x)dx. However
18
the Jacobian is defined as J = I + J = I + ∂u/∂x and it follows that the change in
squared distance between point P and Q is
(δl)2 − (δl0)2 = (dy)T dy − (dx)T dx = (dx)T JTJdx− (dx)T dx
= (dx)T [(JT + J) + JT J]dx
= (dx)T 2εdx =3∑
i=1
3∑j=1
dxi2εi,jdxj (2.2)
Equation 2.2 defines the symmetric strain tensor εi,j and because of its sym-
metry may be written as a strain vector ε = [ε1,1 ε2,2 ε3,3 ε1,2 ε1,3 ε2,3]T . Equation
2.2 can be expressed as ε = Du where D is a differential operator. In engineering
applications, the strain is usually assumed to be small, i.e., the Jacobian product
JT J ' 0 and the strain tensor simplifies εm ' 12 [JT + J]. The differential operator
D reduces then to three tensile and three shear strains.
D =12
2 ∂∂x1
0 0
0 2 ∂∂x2
0
0 0 2 ∂∂x3
∂∂x2
∂∂x1
0
∂∂x3
0 ∂∂x1
0 ∂∂x3
∂∂x2
(2.3)
The Green-Lagrange strain tensor of Equation (2.2) is independent of the
path of deformation, which is adequate for linear elasticity. Note however, that if
rotational components are significant, JT J 0, the approximation no longer holds.
The approximation is also not suitable for plastic deformation which depend on the
path of deformation. Rate-of-deformation and spin tensor are employed in these
cases [135].
19
2.2.2 Stress Tensor
f
nσ
12
σ11
σ13 δ S
σσσ
3233
31
σ21σ22
σ23
Figure 2.2: Stress on Infinitesimal Volume Element
The stress tensor relates forces aligned along the axes of the coordinate sys-
tem to forces on a arbitrary oriented stress plane inside the material. In order to
derive the stress tensor, consider the diagram in Figure 2.2. Define a normal sur-
face force as f = pδS with the surface traction p acting on the area δS (shaded
triangle in Figure 2.2). The unit normal of the stress plane is n and the surface
traction is therefore p = σn. The remaining sides of the elementary tetrahedron
are aligned with the coordinate axis. These remaining sides have reaction forces
acting on them to counterbalance the acting force f . These reaction forces fi are in
general not aligned with the normal of their respective sides. The stress on these
sides is σi = fi/δSi. However, simple geometry leads to δSi = niδS because of the
alignment with the respective coordinate axis. Newton’s second law for the infinites-
imal tetrahedron (i.e., its volume and therefore forces due to its mass tend to zero)
becomes
f1 + f2 + f3 = f .
This yields Cauchy’s stress formula (Eqn. 2.4) and the stress tensor σij .
pj = σnj = σijnj (2.4)
20
The stress tensor is symmetric which allows for a vector notation σ =
[σ1,1 σ2,2 σ3,3 σ1,2 σ1,3 σ2,3]T . The Cauchy stress tensor does not incorporate a stress
history nor does it allow for large1 deformations. Other definitions of stress tensors
in use are the second Piola-Kirchhoff pseudo stress tensor, the Jaumann stress rate
and the Green-Naghdi rate [135] which are less restrictive than the Cauchy stress
tensor. The stress and strain tensor are related by the constitutive equations for a
material which are discussed next.
2.2.3 Constitutive Equation
Constitutive equations relate stresses to strains and are material dependent. The
constitutive equation of linear-elastic material is given by the generalized Hooke’s
law:
σij =3∑
k=1
3∑l=1
(Cijklεkl) . (2.5)
Equation 2.5 contains a maximum of 36 independent material parameters due
to the symmetry of stress and strain tensor. However, for homogeneous isotropic
materials the matrix of elastic coefficients has only two independent variables; e.g.,
the Lame constants λ and µ, or alternatively the Young’s modulus E = µ(2µ +
3λ)/(µ + λ) and the Poisson’s ratio ν = λ/(2(µ + λ)), or the Shear modulus γ = µ
and the Poisson’s ratio ν; the latter is used in this thesis. Equation 2.6 is the
tensor notation of the spring constant of the generalized Hooke’s law with δij being
the Kronecker delta (δij = 1 if i = j; 0 otherwise). For comparison, Equation 2.7
expresses the spring constant in matrix notation (σ = C ε).
Cijkl =2γν
1− 2νδijδkl + γ(δikδjl + δilδjk) (2.6)
1Here, large implies geometric non-linearities can not be ignored.
21
C =
2γ(1−ν)1−2ν
2γν1−2ν
2γν1−2ν 0 0 0
2γν1−2ν
2γ(1−ν)1−2ν
2γν1−2ν 0 0 0
2γν1−2ν
2γν1−2ν
2γ(1−ν)1−2ν 0 0 0
0 0 0 2γ 0 0
0 0 0 0 2γ 0
0 0 0 0 0 2γ
(2.7)
2.2.4 Navier’s Equations
In general, the forces acting on an infinitesimal material cubic element under de-
formation are stresses and volume-dependent body forces bj, as well as acceleration
forces ρuj. The dynamic equilibrium is given by
ρuj =∂σij
∂xi+ bj
and the static equilibrium then is given by
∂σij
∂xi+ bj = 0 (2.8)
Equations 2.8, 2.2 and 2.5 can be manipulated to derive the Navier’s equations. For
a homogeneous isotropic material, Navier’s equations are given by Equation 2.9.
γ∂uj
∂xk∂xk+
γ
1− 2ν∂uk
∂xk∂xj+ bj = 0 (2.9)
2.3 Solution Methods
Navier’s equations (Equation 2.9) can be numerically solved in at least two ways: a
volumetric approach based on energy conservation and a boundary approach based
on the application of Green’s theorem. The potential energy of the deformation over
22
the discretized domain of the solid can be numerically evaluated with the Finite
Element Method (FEM). The result of the FEM is a block matrix for the solid’s
stiffness which relates the applied forces to the solid’s deformation. The method of
condensation yields a discrete Green’s functions matrix from this stiffness matrix.
This is briefly outlined in Section 2.3.1. Application of Green’s theorem to the elastic
equilibrium equation leads to a boundary integral equation, which can be evaluated
by a weighted residual statement. The discretization of the residual statement is the
starting point of the direct Boundary Element Method. The BEM solution directly
leads to the matrix of Green’s functions. This approach is described in some detail
in Section 2.3.2.
2.3.1 Finite Element Model
The virtual work of elastic forces can be derived from Newton’s second law. In the
following body forces are ignored for clarity but they could be included trivially.
Equation 2.8 relates the forces on an element of an elastic solid. Integration of the
equation of equilibrium over the domain Ω of the deformable body yields∫Ω
σij ∂ui dΩ = 0
Employing the Cauchy stress formula and divergence theorem one can arrive (see
[118, pp. 247-249]) at∫Γ
pi∂ui dΓ−∫Ω
σij∂εij dΩ = 0.
The virtual work of the elastic forces is the volume integral of the product of the
stress tensor and the strain tensor.
δW =∫Ω
σij∂εij dΩ =∫Ω
σT ∂ε dΩ
23
=∫Ω
εTCT ∂ε dΩ
=∫Ω(Du)TCD∂u dΩ (2.10)
Equation 2.10 can be made discrete by employing finite volume elements
throughout the domain Ω of the solid. The displacements and forces are approx-
imated with polynomial trial functions over the elements. One choice of a finite
element is a tetrahedral element with four hat-functions centered at the vertices of
the tetrahedron (see Figure 2.3). Note that the boundary of the solid is represented
as a triangular mesh and the displacement field and force field over the boundary
is represented with hat-shaped basis functions. This is identical to the Boundary
Element Model described in Section 2.3.2 (with the exception that it is customary
to employ tractions instead of forces in the BEM).
Interior Vertices
Exterior Vertices Exterior Vertices
Figure 2.3: Discrete Domain and Boundary (2D)(a) FEM (b) BEM
The Finite Element Model for an elastostatic solid has the form of Equa-
tion 2.11. The displacement block vector u is related to the force block vector f by
the stiffness block matrix K.
Ku = f (2.11)
The displacement and force block vector can be reordered into block vectors
associated with the interior of the solid uΩ and fΩ and into vectors associated with
24
the boundary uΓ and fΓ and it follows KΓ,Γ KΓ,Ω
KΩ,Γ KΩ,Ω
uΓ
uΩ
=
fΓ
fΩ
. (2.12)
The displacement of the boundary uΓ may be calculated from by condensation of
Equation 2.12 as
[KΓ,Γ −KΓ,ΩK−1
Ω,ΩKΩ,Γ
]uΓ = fΓ −KΓ,ΩK−1
Ω,ΩfΩ . (2.13)
In the scenario of this thesis where internal forces do not occur and internal displace-
ments are irrelevant the condensed Equation 2.13 becomes KuΓ = fΓ. Here, K can
be reordered into a discrete Green’s functions matrix Ξ by inverting the elements
of block columns of K associated with unknown displacements.
The Finite Element Method with condensation has been applied in real-time
simulation by Bro-Nielsen and Cotin [18].
2.3.2 Boundary Element Model
The boundary element method (BEM) is a discretization of the boundary integral
equation for an elastic solid. The boundary integral equation can be derived with
the method of virtual work. This derivation will be quickly outlined in this section.
Next, the discretization with the linear triangular boundary elements is employed
to derive a discrete Green’s functions matrix.
Equation 2.8 for the static equilibrium holds over the complete body. As
mentioned before, in the course of this thesis, we will ignore body forces b inside
the solid Ω. Multiplying Equation 2.8 with a virtual displacement field u? yields
Equation 2.14. The right-hand-side boundary terms of Equation 2.14 are the work
of the virtual displacement on the boundary. Note, that the virtual displacement
plays a role as a weighting term.
25
∫Ω
∂σjk
∂xju∗kdΩ =
∫Γ1
(pk − pk)u∗kdΓ +∫Γ0
(uk − uk)p∗kdΓ (2.14)
Integrating Equation 2.14 by parts twice, as well as employing the Cauchy
stress formula 2.4 yields
∫Ω
∂σ∗jk∂xj
udΩ = −∫Γ1
pku∗kdΓ−
∫Γ0
pku∗kdΓ +
∫Γ1
ukp∗kdΓ +
∫Γ0
ukp∗kdΓ.
Given the fundamental solution for ∂σ∗jk/∂xj +4 = 0 where 4 is an unit impulse,
we arrive at the boundary integral equation 2.15.
∫Γ
u∗ij(ξ, χ)pj(χ)dΓ(χ) = cij(ξ)uj(ξ) +∫Γ
p∗ij(ξ, χ)uj(χ)dΓ(χ) (2.15)
Equation 2.15 is the boundary integral equation written as a weighted resid-
ual statement over the complete surface Γ of the elastic body. The location of the
load application is ξ, while the response location is χ and the distance between
them is r. Coordinate i is associated with the point load, while j is the coordinate
of the response. The weights of the equations, the terms p∗ij(ξ, χ) and u∗ij(ξ, χ), are
Kelvin’s fundamental solution for the 3-D Navier’s equations for an infinite body
with the same material properties subjected to an impulse point load. The fun-
damental solution is Equation 2.16 and Equation 2.17 where the surface normal is
denoted by n and δij is the Kronecker delta.
u∗ij(ξ, χ) =1
16π(1 − ν)γr
((3− 4ν)δij +
∂r
∂xi
∂r
∂xj
)(2.16)
p∗ij(ξ, χ) =−1
8π(1 − ν)r2
[((1− 2ν)δij + 3
∂r
∂xi
∂r
∂xj
)∂r
∂n
−(1− 2ν)(∂r
∂xinj − ∂r
∂xjni)
](2.17)
26
More details on the derivation of the boundary integral equation can be
found in many textbooks including [16, 33, 17]. The integrals in Equation 2.15
can be numerically evaluated over linear triangular boundary elements with Gaus-
sian quadrature. In the following the traction vector field and the displacement
fundamental solution tensor are written as p∗(ξ, χ) and u∗(ξ, χ) respectively. The
left-hand side of Equation 2.15 at a mesh node k is
(Gp)k =∫Γu∗(ξk, χ)p(χ)dΓ(χ).
The right-hand side of Equation 2.15 at a mesh node k is
(Hu)k =∫Γp∗(ξk, χ)u(χ)dΓ(χ).
The traction vector field p(ξ) over the boundary is approximated with shape func-
tions p(χ) =∑N
i=1 φi(χ)pi where pi are the traction values at a node. Individual
3× 3 block elements Gkm of the fundamental solution matrix are
Gkm =∫Γ
u∗(ξk, χ)φm(χ)dΓ(χ). (2.18)
Individual 3× 3 block elements Hkm of the fundamental solution matrix are
Hkm =∫Γ
p∗(ξk, χ)φm(χ)dΓ(χ). (2.19)
The integrals are numerically evaluated by Gaussian quadrature. The com-
putation is performed by a change to local barycentric coordinates for each element.
Appendix A contains details of these steps. The discretized boundary integral equa-
tion is the matrix equation Hu = Gp. The equation has to be rearranged for the
specific boundary condition at hand. The prescribed traction and displacements are
collected in the block vector v while the right-hand side is the complementary set
of tractions and displacements v. Exchanging elements in vectors p and u to yield
27
v and v requires exchanging block columns in matrices G and H to yield G and H.
The modified equation Hv = Gv is solved with the Green’s functions matrix Ξ as
follows:
G−1Hv = Ξv = v.
This completes the derivation of the Green’s functions matrix. Next ap-
proximate physics-based models are compared to the formulation of this thesis of
Equation 2.1.
2.4 Other Physics Based Deformable Models
Physics based deformable models have been employed extensively in the computer
vision, computer graphics and medical imaging communities. However, the term
physics based modeling is not employed consistently. In this Section, we briefly
describe the central equations in two of the most common approximate models. In
Section 2.4.1 the set-up of particle systems is described, while in Section 2.4.2 the
model of deformation in deformable superquadrics is summarized.
2.4.1 Particle Systems
The modeling of elastic solids by particle systems has been applied in computer
graphics since 1981 [100]. The particle system approach divides the solid into in-
dividual mass points inter-connected by springs (see Figure 2.4). Usually the mass
points are arranged in a regular lattice borrowing from the atomic structure of crys-
tals. The springs are typically linear and usually a viscous damping term is added.
A linear elastostatic solid modeled as a particle system gives rise to an equa-
tion Ku = f [40]. The stiffness matrix K is composed of the individual spring
28
spring
mass
Figure 2.4: Mass-Spring or Particle Model
stiffnesses and their rest lengths, u is the displacements of the mass points from
their rest state and f is the external force vector. The stiffness matrix is banded
determined by the connectivity of the lattice.
Various researchers attempt to set the spring constants to physical properties.
Lee, Terzopoulos and Waters set the spring constants of their facial models according
to typical human anatomy [73]. The actual constants are arbitrary while the material
distribution (e.g., muscle, facial tissue, epidermis etc.) is determined by the typical
human anatomy. Koch et al. set the spring constants according to the material
distribution of a individual human face based on a CT scan [65]. Again the actual
constants are arbitrary. Reznik and Laugier tune the spring in their model of a
rubber tip of a robotic finger according to the Young’s modulus of rubber [106].
Identification from observation for dynamic systems is described in [60]. In [77] a
mass-spring cloth model is fit to observation of hanging cloth behavior. D’Aulignac,
Cavusoglu and Laugier [27] model the elastic behavior of the human thigh with
linear and non-linear springs from observations. The observations consist of force
and displacement measurements in response to pressure probing with a robotic arm.
A general difficulty with mass-spring models is the lack of a suitable general-
purpose tuning algorithm for the springs [40]. The springs have no physical connec-
tion to the behavior of an elastic solid. Mass-spring models do not model various
29
physical behavior of solids well including incompressibility and large stiffness. How-
ever, mass-spring models can be rendered fast if the springs are reasonably soft and
have been utilized in computer graphics with success.
2.4.2 Deformable Superquadrics
Terzopoulos formulated regularization terms based on physical analogy of a mem-
brane under tension and a thin plate under tension [124]. Metaxas and Terzopoulos
extended this work into deformable superquadrics and introduced it to the com-
puter vision and computer graphics fields and later to the medical area [82, 81].
The deformable superquadrics of Metaxas and Terzopoulos are based on Lagrangian
dynamics.
Their stiffness matrix K combines global and local stiffness coefficients. The
position of a point on the surface of the model relative to the center depends on
the global deformation g and the local deformation l, i.e., u = g + l. This needs
to be expressed in generalized coordinates in order to be utilized in the Lagrangian
dynamics formulation. The generalized coordinates chosen by Metaxas and Ter-
zopoulos for the global deformation are the parameters of a superquadric. Also,
the center of mass of the superquadric and the rotation angles serve directly as
generalized coordinates. The generalized coordinates of the local deformation d are
not independent of other generalized coordinates which introduces a time-varying
Jacobian.
Metaxas and Terzopoulos represent the local deformation of a body with
finite elements over the surface of the solid. They use bilinear quadrilaterals with
four nodes and right-angled triangles with three nodes. All of these nodes are
on the surface of the solid. In particular, this local finite element model is not
30
a discretization of the physics of a deformable solid. Therefore, the deformable
superquadrics approach is not compatible with this thesis.
2.5 Summary
This chapter details the deformation model employed by this thesis. The model
is a discrete continuum mechanics model of linear elastostatic object behavior, ex-
pressed in terms of a Green’s functions matrix which can be derived by the Bound-
ary Element Method for homogeneous isotropic linear-elastic material. However, the
Green’s functions matrix representation does not depend on the BEM; in particular,
FEM with condensation leads to similar formulation. The definitions and assump-
tions in continuum mechanics for a linear elastostatic solid have been summarized
in this chapter. Limitations of the model are due to geometric approximation, i.e.,
first-order deformation and material approximation, i.e., linear elasticity.
The remainder of this thesis utilizes Equation 2.1 in the estimation task. The
estimation task is finding the discrete Green’s functions matrix based on observation
of vertex displacements and tractions at a vertex.
31
Chapter 3
The ACME Facility
The Active Measurement Facility (ACME) is an integrated robotic facility designed
to acquire measurements of interactions with objects. At the core of ACME is a
contact manipulation system (CMS) which executes contact interactions with an
object to be measured. These interactions can be recorded with various sensors
including a force-torque sensor, a trinocular stereo system, a microphone and a high
quality 3-CCD color video camera. Sensors which are employed to measure the
response of an object from a distance are combined in a field measurement system
(FMS).
ACME is implemented as a robotic facility because of the ability of robots
to perform tasks with high repeatability and consistent accuracy. This is extremely
important if a large number of registered measurements of an object are to be
collected. The test station is a positioning robot to move the object to be measured
with precision. The CMS is built around a robot arm positioned by a motion stage.
A gantry robot places the FMS relative to an object. All these robots are commercial
devices but their assembly, control and programming are unique to ACME. As a
result ACME is a tele-robotic system with a total of fifteen degrees of freedom.
32
Force Sensor
TestStation
CMS
FMS
TriclopsCamera3−CCD
Laser
TOF
Figure 3.1: ACME Facility Overview
This thesis employs ACME for the acquisition of measurements during active
object deformation. While the design and implementation of ACME is a group ef-
fort, some features are especially designed to enable the acquisition and verification
of deformable models. These feature are discussed in some detail in this chapter,
while only an overview is provided about ACME in general. The remainder of this
chapter is organized as follows. In Section 3.1 a system overview is given including
the actuators and sensors of ACME (Section 3.1.1), the control architecture (Sec-
tion 3.1.2), the programming of ACME (Section 3.1.3) and the sensor programming
model (Section 3.1.4). The structure of an ACME experiment directing ACME to
actively deform an object and record the deformation is discussed in Section 3.2.
The geometric modeling capability of ACME is summarized with examples utilized
in this thesis in Section 3.3.
33
3.1 Overview
The design goal of ACME is to make it efficient to build reality-based models [92], i.e.,
interactive computational models of real, physical objects based on actual measure-
ments. The physical layout of the facility with its variety of sensors and actuators
is shown in Figure 3.1. The grouping of these sensors and actuators (ACME de-
vices) is outlined in Section 3.1.1. The control of these devices is distributed over
a number of computers and layers. This allows one to employ commercial con-
trollers when possible but adding advanced features as required. In particular, the
control architecture provides the foundation for the high-level control of the sys-
tem which makes programming it simple. The architecture is based on tele-robotic
control and is described further in Section 3.1.2. The simplicity in programming
arises from an environment which integrates a kinematic simulator with the tele-
robotic paradigm in a number of ways. On the one hand the simulator serves as
a debugging tool for ACME experiments. An ACME experiment is the top-level
user program written in Java1. On the other hand the simulator is available during
execution to verify motion requests. This allows for a simple generate-and-test ap-
proach to motion planning. Furthermore, the user programming of ACME is aided
by a client program which can execute independently from the facility or act as a
terminal for ACME. The client in stand alone mode helps considerably in sharing
ACME and eases program development. Section 3.1.3 contains more information on
the programming environment, while Section 3.2 discusses the ACME deformation
experiment. The interested reader is also directed to [94, 93] for a more complete
description of ACME.1Java is a trademark of Sun Microsystems Inc., MountainView, Ca., USA
34
Figure 3.2: Measurement of contact force and position with CMS
3.1.1 ACME Devices
The actuators and sensors within ACME are collectively referred to as devices.
They are conceptually divided into the following subsystems (see Figure 3.1 for an
overview):
• The Contact Manipulation System (CMS) is based on a Puma 260 robot arm
with a wrist mounted 6-axis force/torque sensor (ATI Mini 40). The Puma
260 has been used to apply sustained contact forces of a least 10N to 15N .
The ATI Mini has a maximum force limit of 40N to 120N (depending on the
axis) and a maximum torque of 2Nm with a resolution of 0.04N to 0.12N
and 1.0Nmm, respectively. A 1.267m(4ft) long linear motion stage moves
the robot arm to and from its application area and increases the robot arm’s
workspace. The control of the robot arm has been modified to enable the
application of forces to a test object. The robot arm can work in impedance
control mode or execute guarded moves monitoring impact forces at its tip.
The CMS is capable of measuring object properties such as friction and local
stiffness and of applying a force impulse to generate contact sounds. Figure 3.2
shows the CMS acquiring force data while in contact with a test object.
35
Figure 3.3: ACME FMS Sub-System
• The Field Measurement System (FMS) holds sensors for measuring the light
and sound fields around the test object (see Figure 3.3). The unit is positioned
with a gantry robot (aluminum structural elements by “Item Industrietechnik
und Maschinenbau mbH” powered by Parker Hannifin Corp. brushless DC
motors) and oriented by a two degree-of-freedom pan-and-tilt unit (Sagebrush
Model-20). Two vision sensors are part of the sub-system: a high quality
3-CCD color video camera with computer controlled zoom and focus (Sony
DXC 950) and a trinocular color stereo vision system (Point Grey Research
Color Triclops). A microphone is also attached to the unit.
• The 3-DOF Test Station presents the test object to the sensing equipment.
The test-object can be mounted rigidly to the test station which positions the
object in the plane. The test-station consists of x- and y-translational motion
stages, as well as a rotation stage. The stages (by Daedal) possess a linear
and rotary resolution of ±0.00635mm and ±10 arc-min, respectively.
• Miscellaneous devices include a time-of-flight laser range finder (Acuity Accu-
Range 3000 LIR) and various lights which can be switched under computer
control.
36
Java VM
Any OS
Java VM
Any OS
ACME Client
Java VM
RCCL
Solaris
MC 8 RS232
Robot Arm GantryMotion Stages
Controller
Force Sensor
Motion Stage
CMS FMSTS
Control
Stereo−HeadPan−Tilt TOFLaser
Camera
Control
ACME Server
Java RMI
Internet
RS
232
Java VM
Dig
itiz
er
RS
232
Dig
itiz
er
SCSI
Java VM
LinuxWin NT
Figure 3.4: ACME Server: Overview of Control
3.1.2 Control Architecture
The software and hardware control architecture of ACME is briefly outlined in this
Section. An overview of the architecture is shown in Figure 3.4. The control is
client-server based, i.e., an ACME client with which a user interacts and an ACME
server which controls the robotic subsystems introduced above. The ACME server
system is distributed between several computers; separating low-level closed-loop
control, trajectory generation, data acquisition, and networking.
The four layers of the ACME server software from top to bottom are the
37
user’s Experiment, the Java ACME device classes, the native bindings to the devices
and finally the low-level device controller. For an example of the top-most layer, the
user’s Experiment, see Section 3.2. The remaining layers are described below. The
layer of high level Java objects called Devices grouped into Actuators and Sensors
is next. These Actuators are commanded to move by Motion objects. This is
important, since it decouples the functionality of a specific actuator from a motion
specification. Motions are grouped into a MotionPlan which is executed as a whole.
This design feature allows a MotionPlan to be verified, actuators to be synchronized
and run in a common but separate thread. The Actuator objects match the physical
actuators of ACME described in Section 3.1.1. The programming of Sensors is
separately described in Section 3.1.4.
The next lower layer of software is the native bindings to devices. The
native bindings make code exclusive to the ACME server and is neither available
nor necessary to the ACME client. These layers can be replaced by a simulator. The
actuator native bindings are through the Robot Control C Library (RCCL) [75].
RCCL generates real-time trajectories for each actuator device from a MotionPlan.
The set-points of these trajectories are sent to the actual controllers of the various
actuators. The Puma is controlled by Unimation’s control unit while the rest of the
devices are controlled by an 8 axis motion control card (Precision Micro Dynamics
MC-8) except for the RS-232 interface to the stepper motor of the FMS’s pan-tilt
head.
3.1.3 Programming ACME
The ACME client is the stand alone user interface to ACME. The client is the pro-
gram development environment for ACME and is composed of a three-dimensional
38
Figure 3.5: ACME 3-D Display
A collision event produced by the simulation is rendered with the colliding boxeshighlighted.
display of the facility, a simple editor and access to the Java compiler. The client
has access to the complete ACME Java code which includes a kinematic simulator
of the facility. This allows the user to develop, debug and verify all Experiments
without access to the ACME server. Once a compiled Experiment is ready to be
executed on the ACME server, the ACME client connects to the server over the In-
ternet, uploads the compiled Experiment and receives any data requested. Because
an Experiment is any Java object which adheres to the Experiment interface, the
user has complete freedom to implement any desired routines. The great advan-
tage of this model-based tele-programming approach [76] is that all control-loops
are closed solely on the server side ruling out communication delays between server
and client. Data processing routines can also be run on the server which allows data
to be reduced to its essential size before being transmitted to the client.
The simulator models the entire ACME facility with an enhanced Java3D
39
scene graph. The scene graph contains the geometric, positional, and appearance
information used in simulation on the client, run-time checking on the server and
graphic display on either server or client. An important feature of this scene graph
is a simple on-line collision detection using oriented bounding boxes [42]. Bound-
ing boxes are placed around principal system components and can be tailored for
different experiments. Figure 3.5 shows an example of a collision event rendered on
the graphic display of the client.
3.1.4 Sensor Model
ACME is a robotic system designed to collect data from sensors. The programming
abstraction of these sensors is an important design feature of ACME. The high-
level description of sensors is related to the description of actuators discussed in
Section 3.1.1. Sensors in ACME are widely distributed on the computers that are
the ACME server. All actuators provide a position sensor functionality which al-
lows the recording of an actuator’s position at 100Hz. This functionality is tightly
coupled with the RCCL layer of the server. This is also true of the force sensor
of the robotic arm. The video digitizer of the color 3-CCD camera and its serial
command interface on the other hand are quite separate from ACME server. The
same is true for trinocular stereo-head, the laser range-finder and the microphone.
Nevertheless, all these devices are ACME Sensor objects. This is achieved by adding
another client-server layer. An individual SensorServer executes on the computer
hosting the sensor hardware, while the ACME server acts as a client to these in-
dividual SensorServers. This design not only unifies the access to sensors from an
ACME Experiment but it also enables asynchronous real-time data acquisition. An
Experiment specifies SensorTrigger events which are uploaded and checked on the
40
server of the sensor. After being started, the server of the sensor continuously ac-
quires data checks the trigger condition and buffers data to be sent to the ACME
server and eventually to the ACME client. In this fashion, communication delays
are eliminated from the data acquisition loop.
For example, the Triclops server incorporates the control of the video dig-
itizer, the Point Grey Research Color Triclops control software and access to the
Point Grey Research Triclops stereo vision SDK (software development kit). This
allows the user to specify the stereo processing to be executed on the Triclops server.
Trigger events can take full advantage of the stereo library, e.g., by monitoring depth
in the stereo imagery, or implement whatever image processing is required. As the
trigger activates, the SensorServer may proceed to send raw images or images which
are the result of processing in various formats.
3.2 ACME Deformation Experiment
An ACME deformation experiment as employed in this thesis applies a contact force
to an object with the CMS while monitoring the global deformation with the FMS.
This section details the methods utilized in planning the contact with the object,
executing the probing and acquiring measurement data. The structure of the ACME
DeformExperiment is described here in detail since it is the Experiment used in this
thesis to acquire measurements and it also serves as an example of an ACME Ex-
periment. The Experiment does assume that a geometric model of the object under
test is available. The next Section 3.3 describes how the models for this thesis are
acquired. Also not part of the DeformExperiment are the measurement processing
routines. This separation into data acquisition and analysis was convenient for the
research methods explored in this thesis. The processing of measurements is de-
41
DeformExperiment
PokeIt
Experiment
PokeGoalPlanner GuardedMoveinZ SinglePoke
Interface Abstract Classcontains Classextends
CmsPositioning
FmsPositioning
TsPositioningScanPoistion
TestObject
Figure 3.6: Structure of an ACME Deformation Experiment
scribed in the following chapters, Chapter 4 to Chapter 6. The overall structure of
a deformation experiment is depicted in Figure 3.6.
The DeformExperiment takes the specification probing, which is object-
specific, from a data-base object which implements TestObject. Examples of speci-
fication include the maximum force exerted on the object while probing, the max-
imum probing depth, the number of steps for the deformation measurement (see
Chapter 4) and various files and directories to store data in. The other data-base
object implements the ScanPosition interface which contains pre-planned viewpoints
and calibration information for the trinocular stereo-head within the FMS. Based
on these specifications the DeformExperiment positions the FMS, initializes the
PokeGoalPlanner and starts the probing process. A simplified execution flow of the
DeformExperiment is shown in Figure 3.6.
The probing process executes as follows. The PokeGoalPlanner generates
42
1. Position TestStation, FMS and CMS
2. Initialize PokeGoalPlanner
3. Loop over all positions specified in ScanPosition
• Loop over all specified vertices to be probed
– Request probe specification from planner– While probing not successful∗ Simulate probing∗ Execute GuardedMoveInZ∗ Execute SinglePoke∗ If not successful change configuration
– Process and save all data
4. Move FMS and CMS to park position
Figure 3.7: Simplified Execution Flow of ACME Deformation Experiment
the next position and orientation for the probe based on the mesh of the object
under test and the specification in TestObject. It checks limits in order to rule out
goals which would bring the robot-arm too close to the TestStation. It also does ray-
intersection with the object in order to verify that the probe is contacting the object
at the expected location. Then the DeformExperiment calls the simulation with the
necessary motion to reach the goal in a default configuration. On success it executes
this motion. On failure the configuration is changed with a heuristic, either changing
the orientation of the object by rotating the TestStation or modifying the distance
of the robot from the object by moving it with the long linear stage. A successfully
simulated motion is executed in a three-step process. First an approximate position
away from the object is reached. Second, a GuardedMoveInZ is executed to find the
actual position of the surface of the object. All measurements are made relative to
this position reducing the influence of positioning errors of the robot arm. Third a
SinglePoke is executed which contacts the surface, applies force, records force and
43
displacements, and triggers the image capture by the stereo-head.
This experiment is run in simulation when it is applied to an object for the
first time. The simulation performs collision checking and the rendering of the mo-
tion provides a visual confirmation of the experiment. If this performs satisfactory,
ACME acquires the deformation data autonomously.
3.3 Geometric Modeling with ACME
Geometric modeling of objects in ACME is necessary for various tasks including
contact planning, object modeling and registration. This thesis relies on ACME
being able to acquire geometric models of the object under test. First, a surface
mesh of the object under test is required as discretization for the discrete Green’s
functions matrix described in Section 2.1. A surface mesh is also required for the
deformation data acquisition described in Section 3.2 above. Furthermore, the sur-
face mesh is also utilized in the deformation measurement described in Section 4.
Finally, the estimation of the discrete Green’s functions matrix in Chapter 5 and
Chapter 6 relies on the discrete surface representation as well. It is evident that
geometry is essential to this thesis. Despite this, no novel method for initial model
acquisition is proposed. Unfortunately, geometry acquisition techniques are still far
from being fully automated and general. During the research for this thesis distance
measurement with a time-of-flight laser range-finder has been investigated [70], as
well as a novel stereo-vision matching technique [69]. In the end, a collection of
techniques based on data obtained from the trinocular stereo-head have been found
to be effective. The techniques succeed in obtaining a closed (or water-tight) mesh
with subdivision hierarchy (see Figure 3.8) for the objects in this thesis. The process
is only semi-automated and some mesh editing is necessary to arrive at a water-tight
44
multi-resolution surface mesh. A detailed description of the techniques can be found
in Appendix B.
3.4 Summary
The ACME facility is a robotic measurement facility which enables the execution of
experiments in which a probe contacts an object. The probe is mounted on a robotic
arm equipped with a force sensor. This contact manipulation system (CMS) is able
to exert a force onto the surface of an object at a specified location. The trinocular
stereo-head within the field measurement system (FMS) can be placed to observe
the object during deformation from various viewpoints. A robot such as ACME can
execute repetitive tasks with high repeatability which is important during exhaustive
contact sampling of an object’s surface. The user interface of ACME combined
with its tele-robotic design make programming contact interactions convenient. A
further feature of ACME necessary for this thesis is the ability to produce three-
dimensional point-clouds registered in an object reference frame. These point-clouds
are combined in a mesh generation step which produces a hierarchy of meshes with
subdivision connectivity which can also be texture-mapped for better viewing.
45
(a) Raw Scan Mesh (b) Level 0 (149v, 294f)
(c) Level 1 (590v, 1176 f) (d) Level 2 (2354 v, 4704 f)
(e) Level 3 (9410v,18816f) (f) Level 2 Mesh Texture-Mapped
Figure 3.8: Mesh Reconstruction
The raw scan mesh 3.8(a) with a total of 15076 vertices(v) and 29095 faces(f)including various external stray polygons. The subdivision hierarchy (3.8(b),
3.8(c), 3.8(d) and 3.8(e)) is generated using displaced subdivision surface stylepiercing of the raw scan mesh.
46
Chapter 4
Deformation Measurement
The deformation measurement task is to estimate how the vertices of the unde-
formed triangular mesh are displaced during application of a load on the object.
Displacement measurement has been approached in various ways. Shape estimation
and tracking of deformable objects has been intensively studied in computer vision
(e.g., [127, 53, 34, 49, 62, 74]). In mechanical engineering, optical measurement
of strain has been developed (e.g., [83, 46, 80]), as well as commercialized (e.g.,
GOM mbH: Gesellschaft fur Optische Messtechnik, Braunschweig, Germany1). In
mechanical engineering, the strains considered are usually small and often only the
two-dimensional result is of interest. However, while strain can be measured di-
rectly visually, it may also be approached in other ways. For example, in structural
mechanics, strain gauges give local information at critical points of the structure.
This chapter describes the range-flow approach to deformation measurement
of this thesis. Range-flow is the recovery of scene changes from three-dimensional
images. It clearly solves the measurement task at hand since three-dimensional
information is recovered.1See http://www.gom.com
47
The task in range-flow is to match depth information over time. The range-
flow constraint equation [51, 132] for the depth of a surface Z(x,y,t) is
∂Z
∂xdx +
∂Z
∂ydy − dZ +
∂Z
∂tdt = 0.
The range-flow constraint equation is the three-dimensional equivalent of the optical
flow equation [52]. Similar to optical flow [6] this constraint may be utilized by
area-based matching or directly with a differential technique. This thesis employs
area-based matching. Estimating range-flow is a harder problem than optical flow
because of the increased dimensionality and the necessity for quality range data.
Nevertheless, there has been progress in range-flow estimation [132, 120], sometimes
also referred to as scene flow [128].
The approach of this thesis is to calculate range-flow based on a combination
of simultaneous 2D-optical flow with stereo-depth information. Simultaneous stereo
depth and optical flow goes back to at least Ballard and Kimball [3]. Recently this
approach has been followed successfully in [128, 133, 119]. However, in this the-
sis, the additional imagery is utilized to reduce the error in the very noise-sensitive
optical flow. This results in a purely local method and no global regularization
is employed. The high quality calibration of the commercial trinocular stereo sys-
tem [105] in ACME lets this approach succeed.
The choice of stereo vision and optical flow requires objects with sufficient
visual texture. Sufficient texture is texture which leads to an unique peak in the
area-based correlation at the correct disparity in the stereo matching and at the
correct location in the search area for optical flow. This condition can be met since
ACME is a (robotics) laboratory environment and most objects either have sufficient
visual texture or can have visual texture applied to them trivially.
In Section 4.1 methods to robustly estimate flow are briefly reviewed and in
48
Section 4.1.1 the approach of this thesis to robust range-flow estimation is outlined.
The computations involved in this method are detailed in Section 4.1.2. Results for
rigid body motion compared to ground truth data are given in Section 4.1.3, while
examples for deformable objects are shown in Section 4.1.4. Range-flow results are
not direct measurements of the vertex displacement uk needed for the estimation
task defined in Section 5.1. These displacements need to be estimated based on
range-flow results. This estimation is described in Section 4.2 and results are given
in Section 4.2.1. A summary in Section 4.3 concludes this chapter.
4.1 Robust Flow Estimation
This section gives a selective review of range-flow and optical flow estimation in light
of the deformation measurement task at hand. Most optical flow techniques can be
characterized either as a differential technique or as a region-based matching tech-
nique [6]. Differential techniques use the optical flow constraint equation which is
underdetermined (one equation in two unknowns). Local averaging or a regulariza-
tion term need to be introduced in order to find a solution. For example, the original
technique of Horn and Schunck [52] uses a global smoothness constraint. Area-based
matching techniques, on the other hand, implicitly average the flow within the area
being matched and no further constraints are strictly necessary. Nevertheless, the
results typically are noisy and need further processing [6]. This processing may in-
clude introducing a priori constraints on the solution via regularization (e.g., [68]),
fitting results to a parametric model of motion (e.g., [55]), or filtering flow results
based on a reliability measure (e.g., [2]). Multiple approaches are often used in
combination.
Since the displacement to be measured is between the undeformed and de-
49
formed object states, only two images are necessary. However, accurate derivatives
can only be calculated from a sequence of images (unless the signal is highly over-
sampled or the intensity structure is nearly linear [6]). Lack of accurate derivatives
is identified a major source of error in differential approaches [6]. Area-based tech-
niques do not require estimates of the time derivative of intensity and depth in
optical flow and range-flow respectively. Therefore, an area-based optical flow ap-
proach is implemented to avoid having to deal with a sequence of images.
Stereo-range data necessitates at least two cameras but most systems today
use either a set of stereo camera pairs or at least three cameras [63, 89, 105]. Con-
straints from additional imagery are also exploited in the computation of optical
flow in this thesis. The integration of results from all stereo cameras allows the
rejection of outliers, especially if they are generated by viewpoint dependent effects
or if they are the result of ambiguous matching results. Specular highlights (see
e.g., [13]) are a viewpoint dependent effect that produces noise in the flow estimate.
Another source of noise is the lack of texture in the matched area. This can be
identified using multiple cameras since the peak in the correlation surface will vary
between cameras. The peak will also be inconsistent at motion boundaries asso-
ciated with depth discontinuities. The peak in the correlation surface will be in
different locations since the percentage of area in the matching window supporting
the different motions will vary between cameras. Multiple cameras also help to as-
sign a confidence measure for optical flow results which has been identified as a hard
problem [6].
The additional data from multiple views does not prevent the utilization of
other approaches to compute a robust optical flow result. In particular, the ro-
bust optical flow techniques of [13] (multiple motions within an aperture) or the
50
regularization method based on the sum of squared differerences (SSD) of [68] (min-
imizing the effect of discretization errors in the flow) would be compatible with the
approach here. However, for the purposes of this thesis, the result obtained from
multiple views is sufficiently robust to serve as input to the vertex displacement
estimation stage of Section 4.2. The surface deformation measurement technique
utilizing a trinocular stereo vision system of this thesis is described in Section 4.1.1.
4.1.1 Robust Range-Flow by Additional Imagery
Robust range-flow by additional imagery tracks visual surface texture in three spatial
dimensions, i.e., it calculates the range-flow of the deformable object’s surface. The
stereo-disparity between views serves as range information as well as to register the
flow calculation. Stereo and optical flow results are both calculated with an area-
based matching technique using all three cameras of the stereo-system. An overview
of the range-flow calculation is shown in Figure 4.1. In the following, the approach is
described and design choices in its implementation clarified. Section 4.1.2 explains
implementation details.
In the algorithm, a geometric model of the undeformed surface of the object
under test is either assumed known or acquired beforehand. This geometric model
and the tracking capabilities of the system enable the segmentation of the three-
dimensional range-flow into flow at the surface of the object and other flow. With
the object undeformed, a stereo-triplet is captured. The disparity of the images
and a range image is calculated. The range-image is taken from a known position
and orientation in the ACME system. Based on this spatial information the closest
triangular surface patch for each pixel of the range-image is found. Pixels which are
within a threshold of the object’s surface are accepted as surface pixels. Optionally,
51
Reference View
Stereo Matching
Optical Flow Optical Flow Optical Flow
Combined Optical Flow
Voting
Registration andSegmentation
T = 0
T = 1 ...n
Triangular Mesh
Stereo Matching
Range−Flow
Figure 4.1: Range-flow processing: overview at T=0 (initialization) and T=1Range-flow shown is at T=4; at T=1 there is no deformation since the probe is
only just touching the object’s surface.
52
the initial registration may be improved with an iterative closest point (ICP) [8]
type algorithm. However, in our set-up this is not necessary since the initial guess is
sufficient for segmenting the range image. All image row, column, disparity triplets
associated with surface pixels are input into the tracker. Next, optical flow from
the initialization to the first time step in the reference camera is calculated for all
pixels which are tracked. In the approach of this thesis, area-based matching uses
the normalized cross-correlation. The normalized cross-correlation has been selected
because of reports of its good performance in area-based matching [35] and because
it can be calculated from correlation scores which are linear with respect to image
intensity (see Equation 4.1). The optical flow in the non-reference camera images
is calculated for surface pixels in the coordinate frame of the reference camera.
This is achieved by using the sub-pixel disparity information as a look-up table (see
Section 4.1.2 for more details). Each surface pixel then has an optical flow result
for each camera. If all results are in accordance, the flow result is accepted and the
third dimension is added based on the disparity and stereo triangulation. All points
successfully tracked from initialization to the first time step are considered surface
points for the following time step.
Before discussing the implementation details, some tracking issues need to
be discussed. It is convenient to record the initialization followed by two time steps,
even though only one is strictly necessary. The initialization is done without the
probe touching the object, while in the image triplet taken after the first time step
the probe is just touching the surface and in the final image triplet the object is
completely deformed. The separate initialization allows the range-based segmenta-
tion to succeed in image areas where the force probe otherwise would be very close
to the surface. The first time-step on the other hand allows for the correction of
53
0
row + disparity
column + disparity
Left Camera
rowcolumn
columns
Reference Camera
Top Camera
0
rows
correlation window
Figure 4.2: Triclops Stereo Image Triplet
any initial offset introduced by the force probe making contact with the object. The
short sequence of three image triplets means the tracker does not need to have new
points to be tracked added in each step. It just tracks all the initial points through
the complete sequence. However, note that after the displacement of vertices has
been estimated the deformed mesh could be utilized to add additional points into
the tracker.
4.1.2 Implementation Details
In this section, the implementation details of the range-flow calculation are described
starting with the input images. The images from the stereo cameras are rectified
such that the epipolar lines are aligned with the image axes (see Figure 4.2). The
result of the stereo processing with the Point Grey Research Triclops stereo vision
SDK is a triplet of row, column and sub-pixel disparity at each valid pixel in the
reference image. A pixel is valid if the stereo processing produces a result.
Valid pixels which are deemed to be on the object’s surface are tracked in
the reference image. The area-based matching correlates a square window centered
54
at a given pixel with a square search area in the next image of the sequence from
the same camera. The mask size of the correlation window and the search area both
influence the result of correlation. The influence of these parameters and of the
image size on the results and computation time are demonstrated in Section 4.1.3.
The optical flow correlation scores of the non-reference cameras are combined with
the ones of the reference camera with the aid of stereo disparities.
The pixels of the start and end of the flow vector in the reference image are
mapped with sub-pixel disparities into the stereo images. The correlation between
these locations is calculated with Equation 4.1. Equation 4.1 is the normalized cross
correlation NCC of two image windows at column sub-pixel location using bilinear
image intensity interpolation. The NCC is specified with pixel a at location row
ra and column ca plus 0 ≤ p < 1 and pixel b at location rb and cb plus 0 ≤ q < 1.
The NCC is calculated from the non-normalized cross-correlation CC and the self
correlation SC. Correlation of image intensities Icorr is computed over a window
with size 2 ∗ n + 1 where n a positive integer.
NCC(ra, ca, rb, cb, p, q) =
√CC(ra, ca, rb, cb, p, q)2
SC(ra, ca, p)SC(rb, cb, q)(4.1)
CC(ra, ca, rb, cb, p, q) = (1− p)(1− q)Icorr(ra, ca, rb, cb)
+(1− p)qIcorr(ra, ca, rb, cb+1)
+p(1− q)Icorr(ra, ca+1, rb, cb)
+pqIcorr(ra, ca+1, rb, cb+1)
SC(ra, ca, p) = ((1− p)(1− p)Icorr(ra, ca, ra, ca)
+2.0(1 − p)pIcorr(ra, ca, ra, ca+1)
+ppIcorr(ra, ca+1, ra, ca+1)
55
Icorr(ra, ca, rb, cb) =n∑
l=−n
n∑k=−n
[(Ira+l,ca+k − Ira,ca)(Irb+l,cb+k − Irb,cb
)]
The result of the optical flow calculation is three correlation scores for each
goal location within the search window provided a disparity value also exists. In
other words, three correlation surfaces registered with goal locations in the refer-
ence camera are found. These surfaces have missing values for locations where no
disparity value was found in the stereo processing. The maximum peak of each
surface is the flow estimate for a given camera. A voting scheme accepts the result
only if all cameras are in good accordance, i.e., the peak of the correlation surface
of all stereo images is in the 8-neighborhood of the maximum correlation score of
the reference image. This voting scheme simply rejects outliers rather than try to
find the correct flow. This strategy is in contrast to the stereo approach of sum of
the sum of squared differences (SSSD) developed by Kanade et al. [63]. Summing
the correlation scores helps to better define a peak but it is not as robust as voting.
It would be worth trying to find the combined peak to sub-pixel accuracy after the
voting step utilizing the better definition of the peak with the robustness of the
voting. However, pixel accuracy is sufficient for this thesis.
4.1.3 Results for Rigid Body Motion
In the following, the range-flow algorithm is evaluated by tracking a moving rigid
test object. This choice was made because of the difficulty of obtaining ground-truth
for surface deformation. The evaluation is done with both an object translating as
well as rotating. In Figure 4.3 the estimated optical and range-flow for translation
and rotation of the test object are shown. For simplicity, this section provides
comparisons of results for different choices of parameters for the translation only.
Note also, these comparisons are intended for relative comparison and to provide
56
(a) Optical Flow Density (b) Range-Flow Distribution
(c) Optical Flow Density (d) Range-Flow Distribution
Figure 4.3: Rigid Translation and Rotation of a Test ObjectTranslation by 5mm is shown in 4.3(a) and 4.3(b), rotation by 2.0 deg. around
the object’s z-axis (up in the image) is shown in 4.3(c) and 4.3(d). Brighterrange-flow corresponds to larger motion.
an indication of the expected error. One can expect results of slightly lesser quality
during deformation or rotation. This degradation is expected due to a significant
change in surface normal of the observed surface relative to the cameras and light
sources. Motions which will cause optical flow to vary over a correlation window will
make the matching task more difficult and introduce error in locating the match.
Note, however, that the observed translation is not fronto-parallel to the image
plane, i.e., the correct flow is changing over a correlation window.
The following tables compare the range-flow result to the correct translational
57
motion. The correct motion is obtained from reading encoder values of the high-
precision linear positioning devices in ACME. The table shows the result for tracking
the object over eight time steps plus one time step for initialization. Surface points
in the initialization time step are tracked, no points are added to be tracked at later
time steps. Therefore, the number of points tracked decreases with each time step.
The motion between steps is 2.5mm. The tables report the true and estimated values
for the orientation and length of the 3D range-flow with respect to the initialization
step. The reference camera of the stereo-head is 81.0cm away from the center of the
object coordinate frame (which is a typical distance in the scenario of this thesis
to observe deformation from). Furthermore, all results are for the test-object in
Figure 4.3.
The first two tables, Table 4.1 and Table 4.2 show results for two different
translations. The correlation window size is 7×7 pixels with a search range of 13×13
pixels and the image size is 320 × 240. The reported 3D-orientation is the angle
between the z-axis (optical axis) and the flow vectors. The variation in the angle
decreases with greater translation while the variation in vector length increases with
greater translation. The decrease in orientational variation indicates the influence
of the discretization error in the optical flow since larger flow vectors can be more
precisely oriented in the discrete image plane. The increase in variation in the length
of the flow vector is due to a reduction in the density of the flow-field combined with
larger absolute flow values. The method produces results for translation to about
12 millimeter accuracy which is sufficient for object deformation measurement. The
motion to the left-rear in the reference image (Table 4.1) produces similar results
to the motion to the rear-right (Table 4.2). In the following, the translation to the
left-rear of Table 4.1 is evaluated varying the mask size, image size and search area
58
Step Angle [deg] Length [cm]# Mean σ Mean σ Encoder Command1 67.0081 10.2836 0.2678 0.0340 0.2355 0.25002 64.7748 5.7253 0.5396 0.0402 0.4864 0.50003 64.8529 4.1890 0.8113 0.0464 0.7374 0.75004 64.8575 3.5898 1.0849 0.0571 0.9911 1.00005 64.9783 2.8942 1.3567 0.0626 1.2378 1.25006 65.0915 2.6112 1.6290 0.0697 1.4867 1.50007 65.1626 2.3755 1.9019 0.0770 1.7366 1.75008 65.2725 1.9633 2.1747 0.0827 1.9929 2.0000
Table 4.1: Range-Flow Results for TranslationMotion to the left-rear
Step Angle [deg] Length [cm]# Mean σ Mean σ Encoder Command1 48.8813 7.4717 0.2367 0.0334 0.2230 0.25002 46.2486 4.5264 0.4926 0.0461 0.4750 0.50003 46.0249 3.3112 0.7404 0.0518 0.7146 0.75004 45.4467 2.6273 0.9997 0.0507 0.9729 1.00005 45.6313 2.6246 1.2471 0.0606 1.2161 1.25006 45.4051 2.5060 1.5086 0.0780 1.4705 1.50007 45.4808 2.4960 1.7614 0.0889 1.7192 1.75008 45.2843 2.3798 2.0266 0.0937 1.9736 2.0000
Table 4.2: Range-Flow Results for TranslationMotion to the rear-right
size.
The first comparison is intended to quantify the influence of mask size and
image size. The search area is kept constant relative to the image size, i.e., the
search area is 7× 7 for the 120× 160, 13× 13 for the 320× 240 and 25× 25 for the
640× 480 image. The mask size varies for all images over 3× 3, 5× 5 and 7× 7. For
easier comparison all results are combined into a table per time step and time step
1, 4 and 8 are selected and shown in Table 4.3, Table 4.4 and Table 4.5, respectively.
The selection of a mask size, image size and search area makes various trade-offs.
The larger the mask size the more distinct is the peak in the correlation window,
59
Image Size Mask Size Angle [deg] Length [cm]Mean σ Mean σ
160, 120 3, 3 65.3530 20.5544 0.2816 0.13785, 5 64.2933 20.3311 0.2615 0.06117, 7 64.1025 19.6148 0.2630 0.0863
320, 240 3, 3 69.3456 21.9100 0.3711 0.22865, 5 64.9249 14.4413 0.2741 0.09597, 7 64.8249 13.1080 0.2761 0.0789
640, 480 3, 3 65.7337 15.5447 0.3072 0.17165, 5 63.1974 9.7911 0.2469 0.04607, 7 63.4371 9.8475 0.2446 0.0327
Table 4.3: Influence of Mask and Image Size on Flow ResultThe search area is kept constant relative to the image size. The commanded
motion is 0.25cm while the encoder value reads 0.2355cm (Step # 1 of Table 4.1).
but averaging within a larger mask may introduce a larger error since the apparent
motion in the image over the mask may change. Additionally, the larger the search
area the more likely is the inclusion of the correct flow location but the greater the
chance of a mismatch. Furthermore, a larger image improves the localization of the
peak but it also requires larger masks to make the correlation result distinct.
It is also of interest to look at the stability of the correlation peak with respect
to the size of the search area. The more distinct the intensity in the correlation
window is over the search area, the more unique the peak in the correlation surface
will be.
The time complexity of the flow computation is O(sizemasksizesearchsizeimage)
assuming the number of tracked pixels is proportional to the size of the image. For
example, the difference between the largest image with the largest mask size and
the smallest image with the smallest mask size is
7× 7× 25× 25× 640 × 4803× 3× 7× 7× 120× 160
≈ 300.
Therefore, it is crucial to be conservative in chosen mask size, search area size and
60
Image Size Mask Size Angle [deg] Length [cm]Mean σ Mean σ
160, 120 3, 3 62.8964 11.9992 0.9983 0.13875, 5 62.6035 10.9469 0.9989 0.11157, 7 62.5492 10.8381 1.0017 0.1182
320, 240 3, 3 63.4279 10.9815 1.0120 0.13505, 5 63.3405 5.2712 1.0713 0.08747, 7 63.0772 4.7647 1.0966 0.0946
640, 480 3, 3 62.1532 5.3999 0.9891 0.07145, 5 62.2516 3.1899 1.0076 0.05147, 7 62.1506 2.8523 1.0060 0.0488
Table 4.4: Influence of Mask and Image Size on Flow ResultThe search area is kept constant relative to the image size. The commanded
motion is 1.0000cm while the encoder value reads 0.9911cm (Step # 4 ofTable 4.1).
Image Size Mask Size Angle [deg] Length [cm]Mean σ Mean σ
160, 120 3, 3 63.9707 7.5256 1.9500 0.13915, 5 62.7593 7.8290 1.9893 0.14797, 7 62.7484 7.8810 1.9908 0.1429
320, 240 3, 3 61.3715 3.7981 1.9008 0.15225, 5 63.3863 3.1567 2.1400 0.12867, 7 63.1105 3.2557 2.1945 0.1330
640, 480 3, 3 65.4546 2.2675 1.9501 0.12055, 5 62.4504 1.9104 2.0245 0.06737, 7 62.1912 1.9717 2.0256 0.0793
Table 4.5: Influence of Mask and Image Size on Flow ResultThe search area is kept constant relative to the image size. The commanded
motion is 2.0000cm while the encoder value reads 1.9929cm (Step # 8 ofTable 4.1).
61
image size. Also, note, that standard methods of speeding up the calculation by
running sums will be less effective here since the evaluated pixels do not necessarily
form a closed region due to invalid stereo pixels.
4.1.4 Examples of Object Deformation
Figure 4.4 shows the range-flow and vertex displacement estimated for a deforming
object, a plush toy tiger. The intensity coding of the flow result shows the result as
quantitatively correct.
4.2 Estimation of Vertex Displacement
The final step in the deformation measurement is the estimation of vertex displace-
ments given the range-flow of surface points. This is related to the mask-size issue
in the range-flow calculation itself. The best resolution would be achieved by select-
ing the nearest flow vector to a vertex but it would be extremely sensitive to noise.
Selecting a larger aperture allows one to filter noisy individual flow estimates but
reduces the resolution. One may fit a local model of the displacement, in particular,
the underlying linear shape functions of the neighboring triangular patches. This
would couple the estimation since the one-ring neighborhood2 of a vertex has to be
taken into account. Another choice is to fit a locally rigid model to the motion.
This would require fitting a rotation matrix which is computationally expensive.
The rotation model is also in contrast to the linear modeling approach of this thesis,
i.e., the linear shape functions for displacements. Another choice is an affine motion
model. The approach followed in this thesis is one of outlier rejection followed by2The one-ring neighborhood of a vertex are all the vertices connected to this vertex by
an edge.
62
weighted averaging.
In the outlier rejection stage, flow vectors associated with a vertex are filtered.
All flow vectors which have the start point in the undeformed object state associated
with triangles at the vertex are fed into the filter. The filtering is accomplished by
finding the median of the flow values in Cartesian space. The residual of individual
flows is calculated and sorted. The median of the residuals is taken as an outlier
rejection threshold. As a result half minus one flow vectors are discarded. The
remaining flow vectors are weighed by 1/(1 + d) based on the distance d from the
vertex and averaged.
In the following, the nearest neighbor approach is compared with the robust
weighted averaging approach of this thesis.
4.2.1 Displacement Results
Table 4.6 summarizes the results for vertex displacement estimation, again for the
same translation test as in Section 4.1.3 and for a rotation test conducted in the same
way. The rotation in each step is 2.0 to a total of 16.0. The vertex displacements
shown are estimated by the filtered weighted averaging method. The error reflects
both the error in the vertex displacement as well as an error in the initial registration
of the surface. The error in the orientation is also due to the missing sub-pixel optical
flow estimation. This can be seen from the improvement in orientation as the motion
vectors become larger (especially in the orientation test).
The following comparison is between four different approaches; estimating
vertex displacement by the nearest flow vector to the vertex (method NF), the
median of flow vectors within the triangles joining at a vertex (method MF), the
weighed 1/(1 + d) average of flow vectors after median filtering (method MFWA)
63
(a) 2D Optical Flow (b) 3D Range-Flow (c) Displacement
Figure 4.4: Deformation MeasurementThe 3D range-flow shows the distribution of the estimated flow vectors withbrighter range-flow corresponding to larger motions. The vertex displacementin 4.4(c) also shows larger movement being brighter. Dark surface is either
stationary or no estimate is available.
Translation in X Translation in Y Rotation around ZAngle [deg] Angle [deg] Angle [deg] Length [cm]
Error σ Error σ Error σ Mean σ Command10.9142 23.2285 7.4402 15.0430 23.5574 47.8135 0.2998 0.0518 0.30538.6968 20.3784 5.8115 11.7741 24.8044 58.0988 0.5928 0.0884 0.60897.9037 18.9947 5.2960 10.6937 19.1940 42.1753 0.8739 0.1468 0.90528.0567 21.4754 5.0309 10.2249 17.5430 41.4193 1.1370 0.2130 1.20708.3435 23.5471 4.8527 9.9227 13.7172 37.8691 1.4744 0.2350 1.53368.0368 24.2161 4.8800 9.9940 12.2357 36.0586 1.7621 0.3434 1.86438.1894 26.5517 4.9821 10.1555 10.0026 32.9437 1.9929 0.4729 2.12977.9706 28.0945 4.9916 10.1864 8.2385 31.2005 2.1469 0.6727 2.3233
Table 4.6: Vertex Displacement Results for a Translation and RotationThe results are for the test-object in Figure 4.3. Vertex displacements are the
median-filtered weighted average 3-D flow within the triangles joining at a vertex(see text for details). The displacements are transformed into object coordinates.
The errors are the combined error over all vertices and are a result of bothregistration error and flow estimation error.
64
Step Method Orientation [deg] Magnitude [cm]Error σ Mean σ
1 NF 14.6050 18.8691 0.2388 0.0384MF 3.5504 4.4913 0.2373 0.0236
MFWA 3.0627 3.9388 0.2372 0.0191MFWAS 3.1004 3.9460 0.2373 0.0191
4 NF 4.6907 6.6571 1.0030 0.0572MF 1.8624 2.3026 1.0141 0.0245
MFWA 1.7360 2.1340 1.0107 0.0192MFWAS 1.7168 2.1106 1.0107 0.0192
8 NF 2.6801 3.0260 2.0122 0.0737MF 1.9392 2.6460 2.0325 0.0528
MFWA 1.8062 2.4697 2.0299 0.0484MFWAS 1.8042 2.4669 2.0301 0.0488
Table 4.7: Comparison of Estimates of Vertex Displacements for a TranslationThe results are for the test-object in Figure 4.3. Methods are nearest flow vector
to the vertex (method NF), the median of flow vectors within the triangles joiningat a vertex (method MF), the weighed 1/(1 + d) average of flow vectors after
median filtering (method MFWA) and median filtering followed by the weighedsquared 1/(1 + d)2 average of flow vectors (method MFWAD). See the text for an
explanation of the methods.
and median filtering followed by the weighed squared 1/(1 + d)2 average of flow
vectors (method MFWAD). Table 4.7 compares these methods for the translation
in the y-direction (to the rear-right in the image) extensively used before. The
vertex displacements are estimated from range-flow calculated in a 640× 480 image
with a mask of 5× 5 and a search area of 25× 25.
Table 4.8 compares the methods NF, MF, MFWA and MFWAD for a rotation
around the z-axis described above. The vertex displacements are again estimated
from range-flow calculated in a 640 × 480 image with a mask of 5× 5 and a search
area of 25 × 25. The rotation penalizes averaging more since the variation in flow
vectors is much higher over the set of triangles joining at a vertex.
65
Step Method Orientation [deg] Magnitude [cm]Error σ Mean σ Command
1 NF 24.2158 28.0852 0.2923 0.0695 0.2826MF 20.8808 22.5736 0.2939 0.0299 0.2732
MFWA 20.7130 22.1168 0.2939 0.0281 0.2736MFWAS 20.5988 22.0136 0.2939 0.0282 0.2738
4 NF 15.4796 16.1011 1.1682 0.1535 1.0821MF 13.3114 18.3754 1.1531 0.0911 1.0928
MFWA 13.8109 18.7523 1.1531 0.0861 1.0932MFWAS 13.7860 18.6873 1.1531 0.0839 1.0953
8 NF 7.4156 9.8216 2.3229 0.2536 2.2000MF 5.5896 9.6690 2.2861 0.1968 2.1748
MFWA 5.4954 9.5279 2.2861 0.1771 2.1867MFWAS 5.4283 9.3702 2.2861 0.1710 2.1893
Table 4.8: Comparison of Estimates of Vertex Displacements for a RotationThe results are for the test-object in Figure 4.3. Methods are nearest flow vector
to the vertex (method NF), the average of flow vectors within the triangles joiningat a vertex (method AVG), the filtered average of flow vectors (method FAVG)
and the filtered weighted average of flow vectors (method FWAVG). See the textfor an explanation of the methods.
4.3 Summary
This chapter details the vertex displacement measurement approach of this thesis.
The choices in implementing the method are discussed. The method implemented
is a robust range-flow measurement technique based on simultaneous optical flow
and stereo-range data from trinocular imagery. Robustness is achieved by addi-
tional imagery. The approach can be characterized as a purely local bottom-up
approach. The measurement of the vertex displacements is independent of the de-
formation model. This independence is convenient because it allows a comparison
of deformation models independent of displacement measurement. Even more im-
portantly, since the displacement of each vertex is estimated independently, partial
measurements of deformation is trivial. Partial measurement is paramount for all
but the simplest deformable objects. Furthermore, covering the surface of a deform-
ing object nearly always requires multiple views, each view covering different sets of
66
vertices.
Different approaches to deformation measurement are possible. In fact, many
researchers have suggested the use of deformation models to regularize various com-
puter vision tasks (e.g., [127, 98, 21, 22]). An approach employing the deformation
model as a constraint in the vertex estimation is possible (e.g., [96]).
67
Chapter 5
Estimation of Discrete Green’s
Functions
Estimation of discrete Green’s functions based on Equation 2.1 can be viewed as
a linear estimation problem. This view disregards the underlying structure of the
Green’s functions matrix and is explored in this chapter. This is in contrast to
Chapter 6 where the boundary element method derivation of the Green’s functions
matrix of Section 2.3.2 is exploited to solve an inverse problem. This chapter can
be understood as either the final step in the modeling process or as precursor to an
inverse problem solution (see Chapter 6).
First, the local behavior of an elastostatic object around the contact point is
discussed. The local behavior is dominated by the compliant motion of the object
because of the probe. Typically, the local deformation is significant but the deforma-
tion decays quickly with the distance from the contact point. (Global deformation
does not decay quickly with large scale motion of the object, e.g., due to articula-
tion of the object as with the example in Section 5.5.2). The overall response of the
68
object to the contact force is more varied than the local response across different
objects. The topology of the object as well as (in)homogeneity of the material of
the object have great influence.
The measurement set-up briefly summarized in Section 5.1 is the starting
point for the estimation procedure. The local behavior estimation is described in
detail in Section 5.2, while the global behavior estimation is discussed in Section 5.3.
This chapter concludes with two examples: estimation of the discrete Green’s func-
tions matrix of a plush toy in Section 5.5.1 and an anatomic soft-tissue human wrist
model1 in Section 5.5.2.
5.1 Set-Up of Estimation
Equation 2.1 combines the prescribed displacement and traction vectors into block
vector v and the complementary unknown displacement and traction vectors into
block vector v. Vector v consists of tractions for the free surface and displacements
for the fixed surface (see Section 2.1 for details). The displacements for the fixed
surface are zero and all tractions for the free surface are zero except for the contact
area. This structure of the problem is important and is exploited later on. The set-
up of the measurement and estimation tasks must match the above mathematical
description.
The model is fixed to a support mounted on the Test Station. The support
surface of the object corresponds to the fixed surface Γ0. The rest of the surface
is free and corresponds to the free surface Γ1. The Contact Measurement System1The anatomic soft-tissue human wrist model has a bone structure made of hard plas-
tic with a cancellous inner core. The bone structure is surrounded by soft foam with aplastic skin layer. The object is made by Sawbones, Pacific Research Laboratories, Inc.(http://www.sawbones.com).
69
(CMS) probes surface locations on the free surface corresponding to vertices of
the object’s mesh at the desired reconstruction resolution (see Section 3.3). If the
contact area of the probe of the CMS is smaller than the size of the triangles in the
mesh, it is reasonable to assume that a single vertex is affected by the probe. This
is true for all the models in this thesis. As a result the structure of the block vector
v is all zero, except for a single traction vector at the contact point at vertex k.
This leads to the structure illustrated in Equation 5.1.
Ξ1k
... never
. . . Ξkk. . . excited
...
Ξmk
unobservable
0...
pk
...
0
0
=
u1
...
uk
...
um
not observed
(5.1)
The key behavior of the model is characterized by the displacement response
of the m vertices of the free surface Γ1 due to forces applied to it. This displacement
corresponds to the measurable portion of the v block vector. The rows of v which
correspond to vertices of the fixed surface Γ0 are the tractions of the support surface.
These tractions are not measured in ACME. Therefore, the corresponding rows in
the Green’s functions matrix Ξ cannot be recovered. These same vertices correspond
to columns in Ξ which can not be actively excited since their displacement is fixed to
zero. The position of the contact probe measures the displacement of the contacted
vertex uk, while the force-torque sensor measures the contact force. The force at
a probed vertex has to be converted into surface traction for the estimation. The
observed force has to be the surface integral of tractions over the affected area.
70
The surface is represented as a triangular mesh and the shape functions for the
tractions are linear. The components of the force vector fk are the sum over all
integrated traction shape-functions on elements connected to the probed vertex, i.e.,
fk =∑
j16(p1j
k + p2jk + p3j
k ) |G|. The traction over triangle j is a linear combination
of tractions p1j . . . p1j at its vertices 1 . . . 3; while |G| is the Jacobian from local to
global coordinates (see Appendix A for more details).
The estimation problem is: given a set of block vectors v and a set of corre-
sponding block vectors v find the m×m observable submatrix of Ξ. This problem
can be broken up into m separate estimation problems for each of the m columns
due the specific structure of Equation 5.1. Each of these m estimation problems
per column consists of finding a 3× 3 block element Ξik of matrix Ξ given a set of
observed displacement vectors ui and the corresponding traction vectors pk. The
problem when i = k is the estimation of the compliance of the object at location k.
This is discussed next in Section 5.2.
5.2 Local Behavior
The local behavior or the self-compliance of a vertex is of special importance. The
local stiffness is Ξ−1kk , the inverse of the self-compliance matrix. This local stiffness
is employed in haptic simulation to map displacements of the tool tip into a force
response [58, 57]. The correct force response can be easily calculated at the high
haptic rendering rates required (typically ≈ 1000Hz). All that is required is to
multiply the local stiffness at the contact point(s) with the displacement vector(s)
due to the tool. For a single contact vertex, the equation is
Ξkkpk = uk ≡ Ξ−1kk uk = pk.
71
For this approach, it is assumed that the self-compliance elements of the Green’s
functions matrix are invertible. The compliance matrix will be invertible as long
as the response is not rigid or infinitely stiff in any direction. This local model
is exact with respect to the global model and is automatically synchronized with
the overall deformation response. The global deformation response can be updated
independently at any time (for more details and a comparison to other approaches
see [57]).
The self-compliance estimation in the formulation of Section 5.2.1 is a linear
least-squares problem given a set of displacements at the contact point and the
corresponding applied traction.
5.2.1 Least Squares
A vertex location is probed l times. These l probes are executed such that the l
directions of the vertex displacement form a spanning set in three dimensions. The
force response of the object is recorded giving l traction vectors. The standard least
squares problem to be solved for ΞTkk is then
[p1
kp2k · · ·pl
k
]TΞT
kk =[u1
ku2k · · ·ul
k
]T
In the standard notation for least squares, this becomes Ax = b. The normal
equations of the problem are then AT Ax = ATb. The solution of this system
can be found by singular value decomposition of the measurement design matrix
A = UΣVT . Here, the 3 × l matrix U and the 3 × 3 matrix V have orthonormal
column vectors [u1, · · · , ul] and [v1, · · · , v3], respectively. The 3× 3 diagonal matrix
Σ contains the ordered singular values σ1 ≥ σ2 ≥ σ3. The normal equation may
then be expressed as
VΣ2VT x = VΣUT b.
72
Reordering this equation leads to
x = VΣ−1UT b
where the diagonal matrix Σ−1 contains the ordered singular values 1/σ1 ≤ 1/σ2 ≤1/σ3. Least squares problems with a measurement design matrix A with a poor
condition number cannot be solved with the standard procedure above. Condition
numbers of the design matrix A which occur based on measured traction vectors are
considered next.
5.2.2 Measurement Design Matrix
In theory, in a numerical solution of a discrete elastostatic boundary value prob-
lem, rank deficient local stiffness matrices (Ξ−1kk ) are rarely encountered if at all.
For isotropic materials and most geometries local stiffness matrices are diagonally
dominant, i.e., if a force is exerted on an object, the objects deforms locally in the
direction of the applied force. However, measuring the response of an object is dif-
ferent since the design matrix A =[p1
kp2k · · ·pl
k
]Thas to be well conditioned. The
traction measurements plk are polluted with measurement noise and modeling error.
As mentioned previously, it is ensured that the probe directions span 3-space
in setting up the least squares problem. This guarantees that the local compliance
model is excited in all directions. Nevertheless, the measurement of the traction
response of the object may not span 3-space. One probe direction is normal to the
mesh surface of the object, while additional directions are offset from the normal.
This offset is limited by the fact that the contact between the probe and the model is
not allowed to slip for a measurement of the static force response at a given contact
point. In ACME a probe tip with high friction (grinding stone tool tip) is used in
order to use an offset as large as possible. But this setup necessarily leads to greater
73
(a) Vertex 38 (b) Vertex 55
Figure 5.1: Probe Tip at Vertex Location
Vertex σ1 σ2 σ3 σ1/σ2 σ1/σ3
38 1.6426 0.4317 0.3940 3.8047 4.169655 2.5786 0.2638 0.1762 9.7753 14.6327Vertex
√V ar1
√V ar2
√V ar3 ||P T ΞT
kk − UT ||2 ||ΞTkk||2
38 2.3513 2.0164 1.6076 0.4094 3.999355 4.1377 3.8120 3.8823 0.6973 6.8947
Table 5.1: Local Force ResponseLocation of vertices is shown in Figure 5.1. Response at vertex 38 is well
conditioned while the response at 55 is poorly conditioned.
certainty in measurement of the force response in the direction of the surface normal
than in any off-normal direction.
For the remainder of this section measurements of the traction response of
a plush toy tiger (see Figure 3.8) at two locations are considered to illustrate the
discussion. One location is vertex 38 of the base mesh on the back of the tiger as
shown in Figure 5.1(a), while the other location is vertex 55 of the base mesh on the
head of the tiger as shown in Figure 5.1(b). The traction response on the location
on the back spans 3-space well. This is apparent from the ratio of singular values
listed in Table 5.1. The traction response at the measurement location on the head
74
of the tiger is very different. Table 5.1 shows that the ratio of singular values for
vertex 55 is larger than for vertex 38. The condition number at vertex 55 is 14.6327
compared to 4.1696 at vertex 38. A condition number of 14.6327 does not cause
any numerical difficulties but here it signals a measurement problem. The larger
condition number is caused by a traction response with a dominant direction. The
dominant direction is due to the head of the tiger apparently hinging around its neck.
The dominant motion is around the axes of this apparent hinge. The directional
change of the probing does not change the direction of the force response significantly
enough. Here, not significantly enough means that the measured force response of
the normal direction is small compared to the resolution of the force sensor (see
Section 3.1.1) and the probe localization uncertainty. The probe localization error
contributes to the force error since the normal direction of the probe is changing
over the surface.
The singular values σi of the measurement design matrix determine how
sensitive the least squares solution is to errors in the associated right singular vectors
vi. This can be seen by rewriting Equation 5.2.1 as
x =n∑
i=1
uTbσi
vi.
Assuming the error is equally distributed in the components of the right-hand-side
measurement vector b, the factor 1/σi acts as an amplification of the error. A design
matrix with singular values differing by more than one order of magnitude, as in the
case of vertex 55 on the head of the plush toy tiger, causes the signal to noise ratio
to decrease by more than one order of magnitude. In Section 5.2.3 regularization is
used to deal with poorly conditioned traction responses.
75
5.2.3 Regularization
Regularization transforms a parameter estimation problem into a constrained opti-
mization problem. The goal of regularization is to find an estimate which minimizes
the residual of a parameter fit (just as in least squares) but constrained by a dis-
tance measure from an a priori known solution. This section discusses the trade-off
between choosing a solution with low residual and choosing a solution which fits
the a priori constraint in the context of estimating the local compliance matrix
ΞTkk. The direct regularization methods of truncated singular value decomposition
(TSVD) and damped singular value decomposition (DSVD) are compared with the
non-regularized solution. The TSVD (truncated at t) solves the problem
min||x||2 subject to min||Atx− b||2.
This solution is calculated based on the SVD (Equation 5.2.2) as
x =n∑
i=1
fiuTbσi
vi .
where fi is a filter factor and fi = 1 ∀ i ≤ t and fi = 0 ∀ i > t. The DSVD
employs filter factors based on a regularization threshold λ as
fi =σi
σi + λ.
Beside comparing the residual of fit and constraint error, another method for
choosing the regularization threshold λ is the generalized cross validation (GCV).
Below, the example of vertex location 55 shown in Figure 5.1(a) is evaluated. Some
of the results in this section are obtained with the Matlab package Regularization
Tools published by P.C. Hansen [48].
Consider the variance in the elements of the estimated compliance matrix.
The variance V arj in the rows j of the transposed compliance matrix ΞTkk are [102,
76
pp. 676-681]
V arj =3∑
i=1
(vji
σi
)2
.
The variance within a row of the matrix ΞTkk is identical because of the fact that
estimating ΞTkk is three similar least squares problems with three parameters each.
The reciprocal of a singular value 1/σi scales the contribution of the correspond-
ing basis vector to the variance of a parameter as well as to the estimate of the
parameter. These variances describe the uncertainty in the compliance matrix in
centimetres[cm] per Newton[N ]/centimeter2[cm2]. The variances of the three rows
are listed in Table 5.2 for the illustrative example of vertex 55 (Figure 5.1(a)).
The TSVD will weigh the different right-hand-side singular vectors v either
by the inverse of the singular value or zero in Equation 5.2.3. If the inverse of a
singular values is large, the uncertainty in the associated column of right-hand-side
singular vector is large. This leads to large variances V arj . The determination
of large corresponds to the choice of the regularization parameter. In the case of
estimating the 3 × 3 local compliance matrix, the choice can have three possible
outcomes, t = 1, t = 2 or t = 3 (no regularization). The three possible outcomes
are listed in Table 5.2. As expected, the norm of the solution decreases with the
increase in regularization.
The regularization parameter can be chosen from Table 5.2 given an expected
error (residual of fit). In the estimation problem at hand, one may be able to deduce
an expected error by comparing all the individual estimation problems. However,
the error is dependent on location and on the particulars of a measurement and set-
ting a general threshold may be difficult. One approach to deal with this difficulty
is to estimate the threshold from the data itself. The generalized cross validation
does exactly that. The idea is to find a regularized solution which is capable of
77
Vertex 55 t = 3 t = 2 t = 1√V ar1 4.1377 4.1377 4.1377√V ar2 3.8120 3.8120 0.6516√V ar3 3.8823 0.5722 0.2809||PT ΞT
kk −UT ||2 0.6973 1.3721 1.6926||ΞT
kk||2 6.8947 3.8992 2.5143σ1 = 2.5786 σ2 = 0.2638 σ3 = 0.1762
Table 5.2: Regularization with the TSVDThe table shows a comparison of the variances V ari of the rows of the localcompliance matrix ΞT
kk, residual of fit ||PT ΞTkk −UT ||2 and the norm of the
solution ||ΞTkk||2. The number of non-zero singular values is t.
1 2 0
0.005
0.01
0.015
t
G
Figure 5.2: GCV for TSVDMinimum of the generalized cross validation (GCV) for regularization with thetruncated singular value decomposition (TSVD). The GCV for each row of ΞT
kk
with k = 55 is shown separately, as well as their sum (solid here). The GCV isdiscrete at t = 1 and t = 2 (t is the number of non-zero singular values); however
function values are connected for clarity.
78
predicting an arbitrary measurement if this measurement is dropped from the esti-
mation. Additionally, the choice of regularization parameter should be independent
of a orthogonal transform of the measurement data. This leads to a GCV [48] to be
minimized. The GCV is calculated by
G =||PT ΞT
kk −UT ||22(trace(Il −PT (
∑ni=1 fi
uT bσi
vi))2
The GCV for the regularization with the TSVD is evaluated at two locations and
the smaller value in the example is at t = 2 basis vectors (see Figure 5.2).
At first, the TSVD may seem like a good fit to the estimation problem at
hand. However, the problem with the solution obtained is that the estimated com-
pliance matrix itself is rank deficient. However, rank deficiency will cause difficulty
in the haptic simulation because of the use of the inverse of the local compliance
matrix (i.e., the local stiffness matrix). The local compliance matrix will be com-
pletely rigid, i.e., non-compliant in the direction of the zeroed basis vector. The
TSVD regularization has a discontinuous filter function, i.e., it is either 1 or 0. The
damped singular value decomposition (DSVD) replaces these filter factors by the
smoother function of Equation 5.2.3. The DSVD solution for the example is listed
in Table 5.3 for regularization parameters λ resulting in the same residual as for the
TSVD of Table 5.2.
The GCV curve in case of the DSVD is increasing monotonically (see Fig-
ure 5.3). In this thesis the regularization parameter is chosen heuristically. The
heuristic is based on the empirical observation that the transition between poorly
and well-conditioned traction responses occurs at about σ1/σ3 ≈ 10. A choice of
regularization parameter of λ = σ1/10 − σ3 and λ = 0 if σ1/10 − σ3 ≤ 0 suggests
itself. This leads to a regularization parameter of λ = 0.08164 for the poorly condi-
tioned force response at vertex 55 and to λ = 0 for the well-conditioned response at
79
Vertex 55 λ = 0 λ = 0.32165 λ = 0.51677 λ = 0.08164√V ar1 4.1377 1.4809 1.0735 2.8340√V ar2 3.8120 1.7081 1.2802 2.9026√V ar3 3.8823 1.4000 1.0181 2.6656||PT ΞT
kk −UT ||2 0.6973 1.3721 1.6926 0.8489||ΞT
kk||2 6.8948 2.9310 2.4457 4.8591σ1 = 2.5786 σ2 = 0.2638 σ3 = 0.1762
Table 5.3: Regularization with the DSVDThe table shows a comparison of the variances V ari of the rows of the local stiffnessmatrix ΞT
kk, residual of fit ||PT ΞTkk −UT ||2 and the norm of the solution ||ΞT
kk||2.
0 0.2 0.4 0.60
0.005
0.01
0.015
λ
G
λ = 0.08164
Figure 5.3: GCV for DSVDMinimum (on the left) of the generalized cross validation (GCV) for regularizationwith the damped singular value decomposition (DSVD). The GCV for each row of
ΞTkk with k = 55 is shown separately, as well as their sum (solid line).
vertex 38. The heuristic achieves the goal of regularizing poorly conditioned trac-
tion responses but fitting the data optimally for well conditioned traction matrices.
All the results for the local object behavior in Section 5.5 are calculated with this
approach.
In general, the DSVD solution has the advantage of having full rank as long
as the design matrix A has full rank. This avoids difficulties in haptic simulation of
the stiffness matrices. Furthermore, all the left hand singular vectors ui are damped,
especially all off-normal vectors with their corresponding small singular values. This
80
stabilizes the solution in the face of noise in the off-normal directions.
5.3 Global Behavior
The measurement of the global behavior of the model is quite different from the mea-
surement of local behavior of Section 5.2 above. This is not obvious at first since the
structure of the problem is identical. The design matrix A of the individual least
squares problems for estimating ΞTjk when j 6= k is identical to the design matrix of
the self compliance matrix ΞTkk. However, the right hand side b of the problem, the
measurement of the vertex displacement is very different. The measured surface dis-
placement in case of the local behavior is the result of the kinematics of the contact
probe. For the global behavior the displacement is measured by the deformation
measurement technique of Chapter 4. This results in the following main differences
which need to be addressed: increased noise level, additional noise in the form of
outliers, and incomplete measurements.
Outliers can, on occasion, still be observed in the displacement measurement
despite the filtering process described in Section 4.2 (see Section 4.2 for a description
of shortcomings of the filtering process). This should not be a surprise since most
visual measurement techniques need to deal with outlier noise. Methods to deal
with these outliers are discussed in Section 5.3.2.
Incomplete measurements arise from lack of observed object surfaces due
to occlusion by the probe, self-occlusion of the object and incomplete coverage of
the object surface by the visual sensor. The problem of incomplete measurements
manifests itself both in a reduction of the number of observations per estimate at a
vertex and in a complete lack of observations for some vertices. Fewer measurements
at a single vertex can be dealt with by increasing regularization with the same
81
methods as in Section 5.2.3. In Section 5.3.1 the application of the techniques of
Section 5.2.3 to global behavior is discussed. At vertices where there is a complete
lack of observation, there is no choice but to somehow infer the behavior from
neighboring vertices or to infer the behavior from the behavior of the vertex itself
due to probing at a different but close-by location. Considering neighboring vertices
corresponds to interpolation within a column of the block matrix Ξ, while similar
probes correspond to interpolation within a row of the block matrix Ξ. Also note
that obtaining more measurements can reduce the number of vertices with complete
lack of knowledge but is not able to deal with systematic failures. Systematic failures
are the consistent failure of the deformation measurement technique for a given
vertex due to lack of visual texture or occlusion, either by the probe or by the
object itself for all observations.
Global regularization schemes are discussed in Chapter 6 but see also Sec-
tion 5.4. This section only considers interpolation schemes within a block column of
the Green’s functions matrix Ξ. Here, scattered data reconstruction is used to fill in
elements for each column individually. The interpolation of missing displacements
is achieved by solving Laplace’s equation over the set of unestimated vertices. This
hole filling approach is described in Section 5.3.3.
5.3.1 Regularization
The off-diagonal matrix elements ΞTjk with j 6= k are estimated utilizing the damped
singular value decomposition (DSVD). The method is identical to the estimation of
the diagonal elements ΞTkk described in Section 5.2.3, however the regularization
parameter λ is chosen differently. Additionally, the estimation is not attempted if
the set of displacement measurements is not the result of probe directions which
82
span 3-space. This occurs as a result of a reduction in the number of observations
for a given vertex. In this case all deformation measurements at that vertex are
disregarded and the hole filling process is left to fill them in. The estimation process
is also integrated with the outlier rejection process of Section 5.3.2. The choice
of regularization parameter in the DSVD for vertices with sufficient observation is
discussed below.
The noise in the deformation measurement effects the direction of the dis-
placement much more than the magnitude of the displacement (as discussed in
Section 4.2). Regularization with the DSVD is again well suited for this problem.
Small variations in direction should not be magnified by small singular values of
the design matrix A. The DSVD dampens these small singular values. The only
difference then, compared to the local behavior, is in the choice of regularization
parameter. A large regularization parameter filters the singular values more. Since
the noise in the visual displacement measurement is larger than the noise in the
measurement of the displacement with the CMS, a larger regularization parameter
employed for the estimation of the global behavior than for the estimation of the
local behavior. The heuristic is for the global behavior λ = σ1/5 − σ3 or λ = 0 if
σ1/5− σ3 < 0 compared to λ = σ1/10− σ3. However, the amount of regularization
may still not be sufficient at some locations with large errors. These locations need
to be identified and the regularization increased correspondingly. Another heuristic
is employed in these cases.
A way of looking for the right amount of regularization at these problem
location is to make use of the fact that the magnitudes of traction and displacement
are well defined. The local behavior defines a ratio between observed maximum
compliance max(||ulk||2/||pl
k||2) over all measurements l at vertex k and the norm
83
of the solution ||ΞTkk||2 (the largest singular value of the solution matrix). In the
case of the illustrative example of vertex 55 on the head of the tiger, the maximum
observed compliance is max(||ulk||2/||pl
k||2) = 3.1172. This gives a ratio of
ratiok =||ΞT
kk||2max(||ul
k||2/||max plk||2)
= 1.5588.
This ratio is also calculated for all other vertices j 6= k in the column, i.e., ratioj =
||ΞTjk||2/(max(||ul
j ||2/||plk||2)). The solution ΞT
jk is found with the DSVD with λ =
σ1/5 − σ3. The estimated solution at a vertex j is suspicious if the ratio is higher
than for the local compliance. In such a case, a larger regularization parameter
may produce a more desirable solution. Therefore, the DSVD is repeated with the
regularization parameter λ = σ1/5 ∗ ratiok/ratioj − σ3. This heuristic achieves the
goal of rejecting solutions with norms larger than expected but allowing arbitrary
large norms if very compliant behavior is observed.
5.3.2 Outliers
The goal of the outlier rejection process is to reject erroneous observations of the
deformation. The problem is to select a subset of measurements which does not
contain any spoiled measurements. A number of selection methods (or positive-
breakdown methods) exist within robust estimation techniques, e.g., random sample
consensus (RANSAC) [14], least median of squares (LMS) [110] and least trimmed
squares (LTS) [110].
RANSAC selects P number of samples out of M measurements where P is
the number of parameters of the model to estimate. Given a priori knowledge of
the maximum measurement error, inlier points (i.e., points which are not outliers)
are selected for a given estimate. If at least a minimum number of points are
within the expected measurement error, the estimate is accepted. Finally, a least
84
squares fit to the inlier points only is computed and this is the estimate produced
by RANSAC. RANSAC is guaranteed to succeed if the minimum number of points
H for acceptance is greater than (M + P + 1)/2. LMS proceeds identically except
it does not use an expected error to classify inlier points. Instead all possible P sets
are evaluated and the estimate with least median of squared residual is selected.
All points with a squared residual smaller than or equal to the least median of
squared residual are considered inlier points. The final least squares step and the
number of inlier points necessary for LMS to succeed are identical to RANSAC.
Finally, LTS is an improvement over LMS. In LMS the evaluation of a trial set P
depends on the errors in the data points P , i.e., the median of squared residuals
of a data set may have been considerably lower if the initial estimate would have
been improved by a least squares step. Therefore, LTS improves each subset of
points H by improving the fit with repeated least squares [111]. The least squares
iteration has converged when the H set has become stable. LTS finds the H set
with the minimum least squares residual. The LTS process again is robust even if
(M − P − 1)/2 measurements are spoiled.
This thesis employs LTS implemented very similarly to the description in [111].
One modification of the algorithm is in the selection of the initial P -set. P = 3 for
the estimation of ΞTjk since each column is represented by the same parameter (as
discussed in Section 5.2.3). There is no point to selecting an initial subset P which
is rank-deficient. Therefore, bins are employed to group traction measurements to-
gether which are obtained with the same orientation of the displacement vector.
Binning has been suggested, e.g., in [134]. The other modification is the use of
regularization by DSVD inside the least squares iteration steps of LTS.
85
5.3.3 Hole Filling
The hole filling process interpolates data within the block column k of the Green’s
functions matrix Ξjk. This is possible since the observable part of the Green’s
functions matrix contains compliance matrices only (see Equation 5.1). A column
of the compliance matrix can be viewed as the displacement due to a unit traction
applied along the axis of the object coordinate system. As a consequence, the three
columns of the observable part of Ξjk for j = 1, . . . , n all represent compatible values.
If the displacement field is smooth, the columns are smooth. This smoothness
assumption is typically violated at the contact point. Here the displacement field has
a cusp. Note that Kelvin’s fundamental solution for infinite linear-elastic isotropic
homogeneous materials decays smoothly with 1/r where r is the distance from the
contact point. The hole filling process is never necessary at the contact point since
ACME’s contact measurement system guarantees that observations are made.
Smooth hole filling can be achieved by solving Laplace’s equation over ver-
tices j with unestimated Ξjk subject to estimated values Ξlk at vertices l and
Ξwk = 0 for vertices w which are fixed. This may be computed efficiently in several
ways, however time-stepping the diffusion equation sufficiently long is effective here.
The discrete Laplacian L at time t and vertex j is calculated using data from the
vertex’s one-ring neighborhood2, Nj
L(Ξjk(t)) =2Ej
∑i∈Nj
Ξik(t)−Ξjk(t)|eji| , Ej =
∑i∈Nj
|eji|
where eji is the edge joining vertex j and i. Explicit forward Euler discretization is
computed as
Ξjk(t + ∆t) = Ξjk(t) + ∆t L(Ξjk(t))2The one-ring neighborhood of a vertex is all the vertices joined to that vertex by an
edge.
86
subject to a time-step restriction on ∆t [32]. The process is started with the un-
estimated Ξjk(t = 0) = 0. Constraints can be introduced with a weighting factor
0 ≤ β ≤ 1 and the Euler time step becomes
Ξjk(t + ∆t) = β Ξjk(0) + (1− β) (Ξjk(t) + ∆t L(Ξjk(t))) .
This simple process is quite effective for filling a limited number of holes, even
though explicit methods may require a large number of small time steps. Solving
Laplace’s equation has also been applied successfully to smooth displacement fields
and in geometric mesh fairing [32]. The result of this interpolation process is shown
in Figure 5.7.
5.4 Mesh Refinement
The subdivision mesh hierarchy can be exploited to improve rendering quality and
reduce measurement and estimation time. The object can be contacted at vertices
of a coarse resolution mesh but the global behavior may be estimated at any mesh
of level l given sufficient spatial resolution of the visual measurement of the defor-
mation. As a result the estimated Green’s functions matrix Ξ at the resolution l
of the deformation measurements lacks columns which correspond to odd vertices
inserted at that subdivision level. In order to fill in these columns, interpolation
within a row is necessary.
In this thesis, the missing columns are interpolated for rendering purposes
only. The interpolation is linear except for the one-ring neighborhood of the local
response. Here a special shifting is done to preserve the diagonally dominant struc-
ture of the overall Green’s functions matrix and prevent linear dependence columns
of Ξ (see Equation 5.2). It is stressed that global inverse problem solutions are by
87
far preferable to such a crude interpolation. However, for rendering the estimated
Green’s function for the examples presented in Section 5.5, the described approach
produced good results. Note as well that this scheme may be utilized for adaptive
sampling of the surface. Assume the mesh is probed at mesh resolution l and the
discrete Green’s functions matrix is estimated. Taking the columns of the even ver-
tices, i.e., corresponding to vertices of mesh level l− 1, the response at odd vertices
may be found by interpolation. If the measured response at odd vertices is close to
the response at these vertices found by interpolation, sufficient spatial resolution is
achieved.
Ξ =
......
...
Ξik → Ξij ← Ξim
......
...
Ξkk
. . . . . .
Ξjj
Ξmm
......
...
(5.2)
The interpolation is executed on subdivision level l. Level l has been sub-
divided once from the level l−1 and all observable block columns of Ξ are available
at level l− 1. Then, the jth odd vertex on level l has a response Ξij inferred if both
even vertices k and m of its parent edge have responses Ξik and Ξim, respectively
(see Equation 5.2). If so, Ξij is linearly interpolated from these two responses. The
local behavior, Ξjj and Ξnj when vertex n is a one-ring neighbor of j, is handled
88
differently. This local behavior is dominated by a peak at vertex location j and
linear interpolation would introduce a large error. Instead, these local responses
are computed as the weighted average of parent responses which are diagonally
shifted (see Equation 5.2, Ξnj is handled exactly as Ξjj). For example, Ξjj is
linearly interpolated from Ξkk and Ξmm and not from Ξkj nor Ξmj as for far field
responses.
5.5 Results
In this section, estimation results for two objects are presented, one a plush toy tiger
shown in Figure 3.8, the other a anatomic soft tissue human wrist model. Presen-
tation of the results on paper is somewhat limiting. Visual and haptic interaction
with the models provides a far clearer picture of the results3. Quantities which can
be presented here include the residual of fit for the local behavior estimation, i.e.,
for the diagonal of the Green’s functions matrix. The distribution of the maximum
eigenvalue of the compliances (i.e., the diagonal of Green’s functions matrix) are
shown to give a relative sense of compliance of the model. For the global behavior,
examples of the columns of Green’s functions matrices are shown graphically. This
is meaningful since these columns present the response of the model to an applied
unit traction in a given direction at a vertex location. Furthermore, the effect of the
regularization process, the outlier rejection by LTS and the hole filling by Lapla-
cian smoothing are presented for the plush toy. Finally, the effect of different mesh
resolutions is presented.3Haptic interaction with the plush toy tiger has been successfully demonstrated to au-
diences at several scientific conferences (e.g., 11th IRIS Precarn Annual Conference, June2001, Ottawa, Canada; 8th Int. Conf. of Computer Vision, July 2001, Vancouver, Canada)
89
0 200 4000
1
2
3
4
5
6
Time [10ms]
For
ce [N
] (so
lid)
Dis
tanc
e [c
m] (
dash
ed)
(a) Probe at Vertex 38 of Figure 5.1(a)
0 200 4000
1
2
3
4
Time [10ms]
For
ce [N
] (so
lid)
Dis
tanc
e [c
m] d
ashe
d)
(b) Probe at Vertex 55 of Figure 5.1(b)
Figure 5.4: Measured Magnitude of Force and Displacement
5.5.1 Plush Toy
First, results for the local behavior of the model are presented. Figure 5.4 shows
the magnitude of force and displacement at the probe tip during deformation. The
tip executes a spatial linear motion while in contact with the object. An ideal linear
elastostatic object sensed with a perfect sensor would cause the force profile to be
a multiple of the displacement. The plots in Figure 5.4 show approximately such
a behavior, however there are also differences. The force does not remain constant
after the probe stops its motion but decays exponentially to a lower value. This
decay corresponds to viscoelastic object behavior. The plush toy’s filling material
resettles to accommodate the deformation to some degree. The overall magnitude
of the effect is dominated by the magnitude due to linear elastic behavior. The
force profile is not zero at the start since before the recording of the profile, the
probe approaches the surface until the force reaches a threshold. The displacement
90
is offset by a corresponding amount not shown in the graphs. During measurement
the force profile is linearly interpolated to zero, as well as the displacement. The
force sensor causes some high frequency noise in the force profile.
The result of the estimation of the discrete Green’s functions is shown in
Figure 5.5. Figure 5.5(a) shows the maximum singular value of the diagonal of the
Green’s functions matrix. The diagonal describes the compliance of the solid. The
figure shows that the tiger is maximally compliant at the rear of the head. In general
the head is more compliant than the back. This corresponds well to the inspection
of the plush toy by hand. Figure 5.5(b) shows the residual of fit ||PT ΞTkk−UT ||2 for
the local compliance matrices. The residual is uniformly low on the back of the tiger.
The estimates for the head of tiger have increasingly larger residual further away
from the neck. Furthermore, some problem locations are on the ears of the tiger and
on the tail. The distribution of the residual is clearly different from the distribution
of the largest singular value. This indicates that the residual depends on the location
over the surface and is not just a percentage of the maximum compliance.
The results for the global behavior are illustrated with the two Green’s func-
tions block columns corresponding to vertex 38 and 55. The location of vertex 38
and 55 are shown in Figure 5.1(a) and (b), respectively. Figure 5.6 shows the ef-
fect of regularization on the y-column of the Green’s functions matrix. The x- and
z-column show similar behavior and are not shown. The behavior without regu-
larization shows large Green’s functions magnitudes with an uneven distribution.
Regularization ensures that noise in the observations do not cause solutions with
excessive norms. This can be observed in the reduction of the Green’s function
y-vectors. The estimates for vertex 38 are satisfactory without regularization, but
improve with regularization. Notice that on the head of the tiger the movement
91
(a) Solution Norm ||ΞTkk||2
(b) Residual of Fit ||PT ΞTkk −UT ||2
Figure 5.5: Local ComplianceThe figure is vertex colored, red indicates a large value while blue indicates thesmallest value. Vertices not probed due to reachability constraints result in ΞT
kk
not being estimated and are shown in black. The result shown is obtained with amaximum of eight probes per vertex location and observations from two different
viewpoints.
92
(a) Probe at Vertex 38 of Fig-ure 5.1(a)
(b) Probe at Vertex 55 ofFigure 5.1(b)
(c) Probe at Vertex 38 of Fig-ure 5.1(a)
(d) Probe at Vertex 55 ofFigure 5.1(b)
Figure 5.6: Regularization of a Green’s Functions EstimationEstimated y-columns of the discrete Green’s function without regularization areshown in 5.6(a) and 5.6(b). The result with regularization (as in Section 5.3.1) is
shown in 5.6(c) and 5.6(d) The y-column of the observable portion of a blockcolumn of the discrete Green’s function corresponds to the displacements of thefree vertices due to a unit traction in the y-direction (see Equation 5.1). Thedisplacements are color-coded where red indicates a large value while blue
indicates the smallest value.
93
gets damped while it persists around the probe location. The estimates for vertex
55 are in clear need of regularization as discussed in Section 5.2. Observe how there
are large displacements in stray directions on the head of the tiger before regular-
ization. Again, regularization dampens these directions but preserves the observed
displacements near the probed vertex. The behavior for vertex 38 is more local than
for vertex 55.
Figure 5.7 shows the effect of Laplacian smoothing described in Section 5.3.3.
The black areas corresponding to unestimated displacements due to lack of infor-
mation are smoothly filled in. Figure 5.8 shows some subtle improvements over
Figure 5.7 due to additional smoothing of existing visual observations using the
Laplacian and by utilizing LTS during the estimation. Unfortunately, this is hard
to show on paper. The major improvement is in the smoothness of the direction
of the Green’s functions over the surface. Nevertheless, the perimeter of the area
of large displacement (shown in red) is better defined after smoothing. The LTS
step brings some visible improvement for vertex 55, especially on the far side of the
tiger’s head. The difference for vertex 38 is hardly visible. Figure 5.9 shows the
same measurements with estimation done on a finer mesh. The finer mesh level is
better able to resolve the changes in the Green’s functions. As a result the magni-
tude of the y-column of the Green’s function changes more smoothly which is visible
as smoother color transitions in Figure 5.9.
5.5.2 Anatomic Soft-Tissue Human Wrist Model
As before with the plush-toy, the results for the local behavior of the soft-tissue wrist
are discussed. Two magnitude plots of forces and displacement during a probing are
shown in Figure 5.11 for the locations indicated in Figure 5.10. The force profiles
94
(a) Probe at Vertex 38 of Figure 5.1(a) (b) Probe at Vertex 55 of Figure 5.1(b)
(c) Probe at Vertex 38 of Figure 5.1(a) (d) Probe at Vertex 55 of Figure 5.1(b)
Figure 5.7: Hole fillingDiscrete Green’s functions without hole filling are shown in 5.7(a) and 5.7(b),while the result of hole filling by Laplacian smoothing ( Section 5.3.3) is shownin 5.7(c) and 5.7(d). The magnitude of displacements are shown with vertex
coloring, where red indicates a large value while blue indicates the smallest value.Black vertices are not estimated.
95
(a) Probe at Vertex 38 of Figure 5.1(a) (b) Probe at Vertex 55 of Figure 5.1(b)
(c) Probe at Vertex 38 of Figure 5.1(a) (d) Probe at Vertex 55 of Figure 5.1(b)
Figure 5.8: Hole fillingResults with Laplacian smoothing (Section 5.3.3) utilized to smooth estimatedvalues is shown in 5.8(a) and 5.8(b). The results are obtained with 25 iterationswith ∆t = 0.2, β = 1 for the probed vertex and all fixed vertices, while β = 0.75
for vertices estimated based on visual information). The results for Least TrimmedSquares (LTS) as described in Section 5.3.2 are shown in 5.8(c) and 5.8(d). The
magnitude of displacements are shown with vertex coloring, where red indicates alarge value while blue indicates the smallest value. Black vertices are not
estimated.
96
(a) Probe at Vertex 38 of Figure 5.1(a) (b) Probe at Vertex 55 of Figure 5.1(b)
(c) Probe at Vertex 38 of Figure 5.1(a) (d) Probe at Vertex 55 (Figure 5.1(b))
Figure 5.9: Discrete Green’s Functions for Different Mesh ResolutionThe result for the base mesh of the subdivision hierarchy are shown in 5.9(a)and 5.9(b). The results for the next finer mesh resolution (see Section 5.4) are
shown in 5.9(c) and 5.9(d). The estimation for both meshes is based on the sameset of probes and images. Again, the magnitude of displacements are shown withvertex coloring, where red indicates a large value while blue indicates the smallest
value. Black vertices are not estimated.
97
(a) Vertex 4 (b) Vertex 98
Figure 5.10: Probe Tip at Vertex Location
have similar structure to those of the plush toy. During the motion of the probe, the
force is linear with displacement. However, as the motion stops the displacement
is maintained with an exponentially decreasing force. This is a small viscoelastic
effect. The stiffness at the two locations shown in 5.10 is quite different. Vertex 98,
close to the mount of the object, is quite stiff while vertex 4, on the object’s hand,
is quite soft (notice the different displacement levels in Figure 5.11). The softness
on the hand results from the hinging of the hand combined with the longer lever
relative to the mounting surface.
The varying stiffness of the soft-tissue wrist is also clearly shown in in Fig-
ure 5.12(a) which is the plot of the distribution of the maximum singular values of the
compliance matrices (the diagonal of the Green’s function matrix). Figure 5.12(b)
shows the residual of fit ||PT ΞTkk − UT ||2 for the local compliance matrices. The
residual of fit is quite low and uniform over the mesh with the exception of the
thumb. The object’s internal bone structure is simplified in the fingers. The struc-
ture in the fingers contains no joints, does not extent the full length of the finger,
98
0 200 400 6000
1
2
3
4
5
6
Time [10ms]
For
ce [N
] (so
lid)
Dis
tanc
e [c
m] (
dash
ed)
(a) Probe at Vertex 4 of Figure 5.10(a)
0 100 200 3000
1
2
3
4
5
6
Time [10ms]
For
ce [N
] (so
lid)
Dis
tanc
e [c
m] (
dash
ed)
(b) Probe at Vertex 98 of Figure 5.10(b)
Figure 5.11: Magnitude of Force and Displacement
and is fractured towards the finger’s tip.
The global behavior of the stiff vertex 98 and the compliant vertex 4 is also
very different. In Figure 5.13 the corresponding y-column of the discrete Green’s
functions matrix is illustrated. The Green’s function for vertex 98 near the mounting
surface of the arm is localized in the vicinity of the vertex. However, the Green’s
function for vertex 4 on the hand of the human wrist model has no noticeable local
effect. Tractions at this vertex 4 deform the complete human wrist model. The
deformation is larger on the hand than on the lower arm. This is a result of the
lever effect due to the mounting of the object and the object’s capability to hinge
around its wrist. The physical wrist acts as a complex rotational joint causing points
on the surface of the hand to describe an approximately circular trajectory. This is
approximated as a straight-line motion by the model.
99
(a) Solution Norm ||ΞTkk||2 (b) Residual of Fit ||PT ΞT
kk −UT ||2
Figure 5.12: Local ComplianceThe figure is vertex colored, red indicates a large value while blue indicates thesmallest value. Vertices not probed due to reachability constraints result in ΞT
kk
not being estimated and are shown in black. The result shown is obtained with amaximum of 24 probes per vertex location and observations from six different
viewpoints.
100
(a) Probe at Vertex 4 ofFigure 5.10(a)
(b) Probe atVertex 98 ofFigure 5.10(b)
(c) Probe at Vertex 4 ofFigure 5.10(a)
(d) Probe atVertex 98 ofFigure 5.10(b)
Figure 5.13: Discrete Green’s Functions for Different Mesh ResolutionBase mesh level 0 of the subdivision hierarchy (5.13(a) and 5.13(b)) and level 1(5.13(c) and 5.13(d)) are shown. The estimation for both meshes is based on thesame observations (see Section 5.4). Again, the magnitude of displacements are
shown with vertex coloring, where red indicates a large value while blue indicatesthe smallest value. Black vertices are not estimated.
101
Chapter 6
Inverse Problem Solution
The discrete Green’s functions matrix estimation has been approached in Chapter 5
as a linear estimation problem. This ignored the underlying structure of the Green’s
functions matrix. The derivation given in Section 2.3.2 is based on a discretization
of the boundary integral equation. Therefore, the solution by weighted residuals
employed Kelvin’s fundamental solution for homogeneous, isotropic, linear-elastic
materials. This leads to a specific structure of the Green’s functions matrix which
is exploited below in Section 6.1 for an inverse problem solution to the estimation
problem.
The approach simplifies the estimation task significantly when applicable,
however, it is also very restrictive. The assumption of an object made entirely out
of a single homogeneous material is limiting and would be a poor approximation for
the examples of the plush toy tiger and the anatomic wrist model in Section 5.5. The
plush toy is made out of cloth with a filling and most likely the head is separate from
the body inside the cloth. The anatomic soft-tissue wrist model has an artificial bone
structure which is in itself inhomogeneous. In addition, the soft-tissue is simulated
with foam with a separate skin layer. The assumption of isotropy is also violated
102
since the toy-tiger seems to have a preferred hinging direction at the intersection
between head and body, while the human wrist model behaves similar to the human
wrist joint. (A linear elastic solid model also does in general not fit cloth.) The
question then is: can an approximate homogeneous linear elastic model help with
the estimation task? This is addressed in Section 6.2.
6.1 Boundary Element Method for an Inverse Elastic
Problem
The boundary element method is a convenient way to derive the discrete Green’s
functions matrix of the elastic behavior of a solid. The derivation is dependent on
a fundamental solution of the material behavior in case of an infinite medium with
the same material properties. Kelvin’s fundamental solution for the 3-D Navier’s
equations describes the behavior of an infinite isotropic, homogeneous, linear-elastic
material subjected to a concentrated unit point load. Kelvin’s fundamental solution
is the basis for the parameter estimation technique below, which can be considered
a type 4b) of Kubo’s [66] inverse problem classification.
6.1.1 Elastic Parameter Estimation
The weights of Equation 2.15 are the terms p∗ij(ξ, χ) and u∗ij(ξ, χ). These weights
are calculated by Kelvin’s fundamental solution of Equation 2.17 and Equation 2.16.
The key observation is the fundamental solution can be separated into terms only de-
pendent on geometry and terms only dependent on material constants. Equation 6.1
and Equation 6.2 group the factors depending on geometry and on material.
p∗ij(ξ, χ) =1− 2ν1− ν
∂r∂i nj − ∂r
∂j ni − δij∂r∂n
8πr2− 1
1− ν
3∂r∂i
∂r∂j
∂r∂n
8πr2(6.1)
103
u∗ij(ξ, χ) =3− 4ν
γ(1 − ν)δij
16πr+
1γ(1− ν)
∂r∂i
∂r∂j
16πr. (6.2)
The overall matrix equation of the boundary element formulation Hu =
Gp is Equation 6.3. Separating the geometric and material properties terms as in
Equation 6.1 and Equation 6.2 leads to Equation 6.3 (Section 2.3.2 and Appendix A
contain more detail).
[1− 2ν1− ν
HA +1
1− νHB + C
]u =
[3− 4ν
γ(1− ν)GA +
1γ(1− ν)
GB
]p (6.3)
with the diagonal matrix Cii =n∑
j=1
(1− 2δij)[1− 2ν1− ν
HAij +1
1− νHBij
]
Equation 6.3 and Equation 2.1 form the basis for the inverse problem solu-
tion. As described in Section 2.3.2, a given boundary configuration enables one to
reorder Hu = Gp into the Green’s functions matrix Equation 2.1. Equation 6.3
can be reordered in the same way. However, the separation between material and
geometric properties is lost, since the matrices H and G depend nonlinearly on the
Poisson’s ratio ν.
The boundary configuration employed in the following is the same as de-
scribed in Section 5.1. The supporting surface is fixed, i.e., a displacement of zero
is described and the free surface has its traction prescribed. As before, these values
are combined in the block vector v. The boundary configuration remains fixed as
different vertices on the free surface are probed. This enables the combination of
several probings in a common estimation step since the Green’s functions matrix
does not change. For each probing, some part of the right-hand side vector v is
observed. Again, the observation at the probing location is the local measurement
made with the robotic arm (see Section 3) while the global deformation is observed
104
by range-flow (as described in Section 4). An optimization procedure to find shear
modulus γ and Poisson’s ratio ν from Ξ(γ, ν)v = v is described in the following
Section 6.1.2.
6.1.2 Optimization
The shear modulus γ and Poisson’s ratio ν are found by an iterative optimization.
First, the surface integrals in Equation 6.3 need to be calculated. This is a very
expensive computation (O(TNM) with T the number of triangles, N the number
of vertices and M the number of integration points within an element), however,
it needs to be done only once for a given triangular mesh. Note, that this is the
same as for rendering the elastic response of an analytically derived model of an
object [58].
The objective function in the minimization is the error function
F (γ, ν) =n∑
i=0
αi (Ξ(γ, ν)v − v)2i . (6.4)
An element of the selection vector α in Equation 6.4 is one if observations of vertex
displacement are made and zero otherwise. The objective function of Equation 6.4 is
evaluated starting from an initial guess for the unknowns γ and ν. In each iteration
Equation 6.3 needs to be evaluated with the new values of γ and ν. The matrices
HA, HB, C, GA and GB remain unchanged and the fundamental solution matrices
G and H are found with Equation 6.3. In the next step the Green’s function matrix
Ξ(γ, ν) has to be calculated from the fundamental solution matrices G and H.
This calculation involves a matrix inverse in the size of Ξ (see Section 2.3.2). This
matrix inverse enables one to arrive at rows for which all measurements are available
because the complete described values v are known. The formulation is also trivially
105
expanded to multiple probings in the same boundary configuration. Equation 6.5 is
the objective function for m measurements in the same boundary configuration.
F (γ, ν) =n∑
i=0
m∑j=0
[α0 · · ·αm]ij (Ξ(γ, ν) [v1 · · · vm]− [v1 · · ·vm])2ij (6.5)
The Matlab function fmins, an implementation of the Nelder-Mead simplex
method, has little difficulty solving the minimization task at hand. However, note
that simpler methods should be sufficient because of the low dimensionality of the
problem, the limited range of possible values for Poisson’s ratio ν = 0.0 · · · 0.5 (in
practice) and the shear modulus γ acting like a scale factor.
6.1.3 Results for Half Ball
This section presents the results of the above described inverse problem solution
for a test object: half of a soft (NerfTM ) ball (see Figure 4.3). The estimation
results for the half ball are summarized in Table 6.1. The linear elastic model is a
good approximation for the ball and the model fitting is successful. The estimated
parameters ν and γ, when used in simulation, produce results which closely resemble
the recorded range-flow (see Figure 6.1).
The results in Table 6.1 are from measured displacements by range-flow and
displacement and force at a probed vertex. Table 6.1 also contains a comparison
with a destructive test. The test is a simple compression test of a cylindrical mate-
rial sample. The material sample is compressed measuring force and the reduction
in length of the cylinder. In practice, this works well for metal samples in common
testing machines, however for soft elastic material it is far less precise. For the ball
test object, the object behavior is greatly influenced by a membrane-like layer over
106
Node No. ν γ E
1 0.1183 0.7809 1.7462 0.1878 0.6308 1.4993 0.2227 0.6039 1.4774 0.2516 0.6555 1.6415 0.1857 0.6977 1.6556 0.2009 0.6696 1.6088 0.2089 0.6550 1.5849 0.2007 0.6646 1.59611 0.1868 0.7175 1.70312 0.2082 0.7585 1.83314 0.1997 0.7993 1.91815 0.2084 0.7451 1.80117 0.1192 0.7666 1.71618 0.1366 0.7562 1.71920 0.1400 0.7480 1.70522 0.1264 0.7121 1.60426 0.1442 0.9383 2.147
ALL the above 0.1730 0.7283 1.709Test: - - ≈ 1.75
Table 6.1: Material Properties Estimation ResultsThe table lists the estimation results of the material properties (shear modulus γ,
Poisson’s ratio ν and the corresponding Young’s modulus E) for the half-ball.Each row in the table contains the result for a separate estimate based on the force
probe location at the specified node(s). Using data from all probes in a singleestimation results in values for the material properties very close to an
independent compression test for the material.
107
its surface, i.e., the cylinder cut does need to have that membrane intact. The cylin-
drical sample of the ball is compressed in a milling machine. The milling machine
ensures that the cylinder is compressed in the vertical direction only. The force is
measured with an electronic balance (Ohaus CT 6000-S) under the sample. For a
compressed cylindrical material piece the Young’s modulus is E = F lAδl
, where F is
the compression force, l and δl is the height and the change in height of the cylinder,
respectively, and A the area of its circular end (see e.g., [101]). For homogeneous,
isotropic, linear-elastic material, the relation between shear modulus γ and Young’s
modulus E is γ = E2(1+ν) . This allows the comparison in Table 6.1, which shows
an excellent correspondence between the material-based compression test and the
object-based inverse problem method. The variation of the estimates between sin-
gle probes is mostly likely due to the misalignment of the probe with respect to the
surface. Combining the measurement in one estimation step averages out the noise
nicely.
A visual comparison between the measured displacement and the simulated
displacement for a probing is shown in Figure 6.1. The magnitude of surface displace-
ment shows very good correspondence between the simulation and the measurement
on the surface area not occluded by the probe.
6.2 Approximate Model
The above parameter estimation approach seems to work well if the object fits
the assumptions of the model. As previously stated, since these assumption are
fairly strong, relaxing the assumptions is desirable. One approach to overcome
the limitation of homogeneity would be to try to estimate the location of interior
boundaries between materials at the same time as the material properties. This
108
Figure 6.1: Comparison of Measured and Simulated DisplacementFrom left to right: 3D range-flow, estimated displacement, simulated Displacement.
The 3D range-flow shows the distribution of the estimated flow vectors withbrighter range-flow corresponding to larger motions. The node displacement figurealso shows larger movement being brighter. Dark surface is either stationary or noestimate is available. The estimated node displacement forms a partial ring around
the location of the probe; the location itself and the other part of the ring areoccluded by the probe.
obviously would be a formidable task but still quite restrictive. Additionally, this
interior interface between materials is not required for the simulation, only the
Green’s functions matrix is necessary.
Instead of estimating the location of an interface, the response of the object to
probing at vertex k can be modeled approximately with an apparent shear modulus
γk and an apparent Poisson’s ration νk. This allows for varying stiffness over the
object surface and variation in how global the object responds at a location on the
surface. It is limited to object responses which remain continuous over the object’s
surface and are isotropic.
The limitations of the homogeneous isotropic linear elastostatic boundary
element model are illustrated next with the example of the anatomic soft-tissue
human wrist model. An apparent shear modulus γk and an apparent Poisson ratio
νk are estimated for vertices k = 4 and k = 98 which are shown in Figure 5.10. The
estimates are based on 8 probes per vertex and only the local response is utilized.
The estimates are given in the following table.
109
Vertex No. νk γk
4 0.04658 19.28
98 0.0 1.157
This approximate model is still too restrictive for the wrist. No minimum
can be found for the response at vertex 98 and instead the limit of ν98 = 0 is selected
(This is found with constraint optimization with the Matlab function fmincon). The
apparent shear moduli γk are drastically different at the two locations.
The lower arm deforms smoothly when probed at Vertex 98 and the object’s
response is quite local with a cusp at the contact location (see Figure 6.2). The
response has also no preferred direction. However, homogeneous isotropic material
cannot produce this response given the boundary configuration. This response is
caused by the soft tissue of the object being compressed between the probe and the
object’s bone structure. Figures 6.2(a) to 6.2(b) compare the result of the param-
eter estimation with the response estimated by the Green’s function estimation of
Chapter 5. Figure 6.2(c) shows the magnitude of the difference between the two
methods for the y-column of the Green’s functions block column Ξj,98.
However, some of the response of the human wrist model above the wrist
joint can be approximated. The bone structure is not rigid enough to restrict the
deformation and the complete object bends. Figures 6.2(d) and 6.2(e) show how
both estimation method lead to a global bending of the object. The magnitude of
the difference between the estimated displacements, shown in 6.2(f), is quite even.
However, for deformation perpendicular to the axis of the joint greater differences
can be expected.
110
(a) Approx.InverseProblem
(b) Green’sFunctionsEstimation
(c) Dif. be-tween 6.2(a)and 6.2(b)
(d) Approx.InverseProblem
(e) Green’sFunctionsEstimation
(f) Dif. be-tween 6.2(d)and 6.2(e)
Figure 6.2: Inverse Problem Solution and Discrete Green’s Functions EstimationThe Green’s functions estimation method of Chapter 5 can represent deformationswhich are outside the domain of homogeneous isotropic linear elastostatic models.Figures 6.2(a), 6.2(b) and 6.2(c) show the behavior at vertex 98 of Figure 5.10(a),while 6.2(d), 6.2(e) and 6.2(f) show the comparison for vertex 4 of Figure 5.10(b).
The magnitude of displacements are shown with vertex coloring, where redindicates a large value while blue indicates the smallest value. Black vertices are
not estimated.
111
6.3 Summary
This section shows how the elasticity of an object can be estimated by solving an
inverse problem for the elastic constants of the object. The method is restricted to
objects made of isotropic, homogeneous, linear-elastic material. For objects which
fit this assumption, the model works well as shown for the test object. The material
restriction may be relaxed if the model is understood to be local and approximate
only. A local approximate model based on linear elasticity may be employed by as-
signing different elastic constants at each location on the surface. In fact, the elastic
constants become elastic functions over the surface. However, internal structure
and articulation cannot be modeled in this way. But the discrete Green’s function
estimation of Chapter 5 still provides a reasonable approximation in the case of
the anatomic soft tissue human wrist model. An explicit model of articulated soft
bodies is perhaps more promising and is the subject of future work.
112
Chapter 7
Conclusions and Future Work
7.1 Conclusions
This thesis presents a method to scan physical deformation behavior of objects. A
model of the deformation behavior is estimated based on measurements of an intact
existing object. A method to acquire these measurements including a measurement
of the global deformation response is described. The global deformation response
is observed with a computer vision technique. The robotic measurement facility
ACME is extended to enable the data acquisition. The same techniques also allow
one to validate an estimated deformable model.
ACME serves to acquire a geometric model of an object, to apply contact
forces to the model and to record the deformation of the object. ACME allows
the acquisition of a large amount of deformation data in an automatic fashion, i.e.,
ACME can scan physical deformation behavior of objects. The data acquisition
during the scan is based on two devices: a force sensor equipped robotic arm and a
trinocular stereo-vision system. The robotic arm senses the position and forces at
the contact point with the object. The deformation of the object surface is sensed
113
visually.
The technique developed for the visual sensing of the surface displacement is
based on a robust combination of trinocular optical flow and stereo. The additional
images in the calculation of the optical flow enable a consistency check of the result.
The combination of optical flow and stereo yields three-dimensional range flow. The
registration with a separately acquired surface mesh of the object leads to estimates
of the three dimensional vertex displacements during deformation. The acquired
data can be utilized to estimate and validate deformable models.
This thesis develops and implements two estimation approaches. One ap-
proach may be characterized as a data driven approach with the assumption of
linear elastostatic behavior of object. This allows one to fit a deformable model to
a wide range of objects but requires a large amount of data. This requirement of
extensive data acquisition is mitigated by the use of the measurement robot ACME.
The other estimation approach makes the stronger a priori assumptions of homo-
geneous linear elastostatic object behavior. This estimation method requires little
data because of this strong assumption. Nevertheless, it describes certain types of
objects well; which are made of a single suitable material for instance an object
made of closed-cell foam. Modeling assumptions in between these two extremes
are not investigated but are suggested. The thesis contains an example of model
fitting for a plush toy which has been demonstrated successfully to audiences at
several scientific conferences (11th IRIS Precarn Annual Conference, June 2001, Ot-
tawa, Canada; 8th Int. Conf. of Computer Vision, July 2001, Vancouver, Canada).
Another example discussed is an anatomic soft tissue human wrist model. A de-
formation model has also been estimated for a simple homogeneous foam object.
To the best of the author’s knowledge this thesis presents the first attempt to scan
114
deformation behavior of general objects.
7.2 Future Work
One goal for future work could be to embed the presented techniques into a proba-
bilistic framework based on a measurement theory. This thesis characterizes mea-
surements by a comparison with ground-truth in the case of the displacement mea-
surement in Chapter 4 and with residual of fit in the case of the estimation methods
in Chapters 5 and 6. This allows one to assess how well a model explains the
measurements. In order to be able acquire a measurement and make a quantita-
tive statement if this measurement is explained by a given model, a probabilistic
framework within a measurement theory is necessary. A measurement theory estab-
lishes the relations between observations and a physical model. The theory has to
model the measurement process, reasoning about the influence of various physical
phenomena – some in accordance with the model and some not captured by the
model. The latter are generally referred to as noise. Bayesian estimation theory is
one framework for quantitative statements about model fit and model selection.
The other suggestions for future work concern estimating dynamic models
of deformation. The trend in deformable models for medical simulation is towards
dynamic non-linear finite element models, despite the fact that current computers
can render only very small and very soft models in real-time. Dynamic model fitting
with a large deformation strain tensor would require the tracking of objects during
the deformation process at least at twice the maximum occurring frequency. The
techniques of this thesis still apply but the image acquisition with the Triclops
stereo-vision system would most likely be too slow for the deformation transient.
Higher speed camera systems and strobe lights could solve this problem.
115
Another option would be fitting articulated deformable models. This problem
is usually addressed with a priori known kinematic model. The model of articulation
could be fit while minimizing the deformation of individual links. In humans the
area around the joints exhibit large deformation (e.g., finger joints) which would be
one area to address.
116
Appendix A
Linear Triangular Elements
This Appendix gives the necessary notation to be able to evaluate integrals based
on the fundamental solutions in Equations 2.16, 2.17, 6.1 and 6.2. The integrals are
evaluated with linear triangular elements, i.e., the shape functions over a triangu-
lar boundary element are linear in the values at the vertices of the triangle. The
fundamental solutions contain the distance r between load location and response
location. A component of the distance vector ri given the vertices x1 . . . x3 of the
triangular element in anti-clockwise order can be calculated as
ri = xξi − η1x
1i − η2x
2i − (1− η1 − η2)x3
i
The distance is r =√∑3
i=1 ri, while the partial derivative of the distance is ∂r∂xi
= rir
and ∂r∂n =
∑3i=1
rinir . A component of the unit normal is ni = gi/|G| (see Equa-
tion A.7).
Next the mappings from local coordinates of an element to global coordinates
and vice versa are given. Equation A.1 maps local homogeneous surface coordinates
ηi (i.e., η3 = 1−η1−η2) of a triangular shape element to global Cartesian coordinates
xi. The vertices of the triangular element are labeled x1i . . . x3
i in anti-clockwise order.
117
xi = x1i η1 + x2
i η2 + x3i η3 (A.1)
Equations A.2, A.3 and A.4 map global to local coordinates.
η1 =−x2
1x32 + x3
1x22 − x1x2
2 + x1x32 − x2x3
1 + x2x21
−x11x
22 + x3
1x22 + x1
1x32 − x2
1x32 − x3
1x12 + x2
1x12
(A.2)
η2 =−x3
1x12 + x1
1x32 − x1x3
2 + x1x12 − x2x1
1 + x2x31
−x11x
22 + x3
1x22 + x1
1x32 − x2
1x32 − x3
1x12 + x2
1x12
(A.3)
η3 =−x1
1x22 + x2
1x12 − x1x1
2 + x1x22 − x2x2
1 + x2x11
−x11x
22 + x3
1x22 + x1
1x32 − x2
1x32 − x3
1x12 + x2
1x12
(A.4)
The displacement and traction vector over an element are expressed in Equa-
tions A.5 and A.6, respectively, using identical interpolation functions as the above
shape functions, i.e., ϕi = ηi.
ui = u1i ϕ1 + u2
i ϕ2 + u3i ϕ3 (A.5)
pi = p1i ϕ1 + p2
i ϕ2 + p3i ϕ3 (A.6)
The change to local coordinates requires the Jacobian in Equation A.7.
dΓ = |G| dη1dη2 (A.7)
with |G| =
√√√√ 3∑i=1
(gi)2
g1 =∂x2
∂η1
∂x3
∂η2− ∂x2
∂η2
∂x3
∂η1
g2 =∂x3
∂η1
∂x1
∂η2− ∂x3
∂η2
∂x1
∂η1
g1 =∂x1
∂η1
∂x2
∂η2− ∂x1
∂η2
∂x2
∂η1
118
With the local coordinates of Equation A.1, the partials in Equation A.7
become:
∂x1
∂η1= x1
1 − x31
∂x1
∂η2= x2
1 − x31
∂x2
∂η1= x1
2 − x32
∂x2
∂η2= x2
2 − x32
∂x3
∂η1= x1
3 − x33
∂x3
∂η2= x2
3 − x33
After the mappings between local and global coordinates have been estab-
lished, it remains to evaluate the integrals.
The fundamental solution integrals in Equations 2.18, 2.19 and Equation 6.3
are evaluated numerically by Gaussian quadrature. Employing L integration steps
per element weighted by a factor wl leads from Equation 2.19 to A.8. (The other
integrals are converted identically.)
Hkm =∫Γ
p∗(ξk, χ)ϕ m(χ)dΓ(χ) ≈L∑
l=1
|G|l wl(p∗km ϕm(η1, η2)l) (A.8)
The integration points and the corresponding weights for the Gaussian quadra-
ture can be found in many references, e.g., [33]. Care has to be taken with the
singular integrals for the self-effects (r −→ 0 for k = m).
119
Appendix B
Geometric Modeling with
ACME
This appendix details the initial geometric model acquisition briefly introduced in
Section 3.3.
The main shape sensor in ACME is the trinocular stereo-head within the
FMS. The sensor produces large amounts of range data in close to real time em-
ploying its commercial stereo vision SDK. However, range data from stereo is noto-
riously noisy. Another feature of stereo ranging is its dependence on image features
for matching between the stereo cameras. Additional features can be projected onto
a surface with a light or laser system. Color Triclops produces data of adequate but
modest resolution (for a typical viewpoint in ACME a depth resolution of 2-3 mm
is achieved).
Geometric object modeling requires that the entire surface of the object is
covered, which requires varying viewpoints. Registration of viewpoints in ACME is
achieved by visual calibration of the FMS with respect to an optical target, as well
120
as by exploiting the accuracy of motions of the TestStation and the repeatability
of motion commands for the FMS. The registration of the viewpoint relative to
the target is accomplished with a variant of an Iterative Closest Point (ICP) [8]
algorithm. The starting point for the minimization is derived from the kinematics
of ACME. Range images from different viewpoints are combined in a voxel-based
volumetric approach with reconstruction software provided by the National Research
Council of Canada [109]. In order for this approach to succeed the data not only has
to be in a common coordinate frame but also needs to be noise-filtered. Furthermore,
normals for the point-cloud have to be estimated.
The range data is filtered by calculating range with variable mask sizes and
combining them in a voting scheme. An example of a point-cloud before and after
filtering is shown in Figure B.1. The threshold for the filtering is set based on
experience and is far from universal. However, since the stereo-head is quite common
throughout the machine vision research community, the processing strategy and
actual setting of thresholds may be of interest to some and are reported here.
The filtering is achieved by processing a stereo image triplet twice. One stereo
range image is calculated to maximize the accuracy of the result by using a small
mask, while the other image is calculated to minimize noise with a large mask and
tight validation settings. One stereo range image is calculated with a small stereo
correlation mask of five in a 640 × 480 image. Additionally, the Triclops stereo
vision SDK’s uniqueness threshold is kept nearly ineffective with a setting of 2.0
(The uniqueness threshold is intended to filter out matches which are not consistent
between the right-left and the right-up camera pair). The second stereo range
image is calculated with a much larger correlation mask size of 13 and the tight
validation setting of an uniqueness threshold of 1.0 combined with the connected
121
(a) Before Filtering. (b) After Filtering.
(c) Before Filtering. (d) After Filtering.
Figure B.1: Point-Cloud Filtering
Two point-clouds from Triclops filtered by variable mask size processing and planefitting. Figures B.1(a) and B.1(b) shows the calibration target; Figures B.1(c) and
B.1(d) the anatonomic soft-tissue human wrist model. (See Text for details).
122
region threshold of 300 pixels with less than 0.75 disparity difference.
The images are combined in a plane fitting step simultaneously estimating
normals and validating a point. All points are rejected which are not within 1cm
between the two range images. The plane fit for each point in the noisy range image
is accomplished by singular value decomposition (SVD). A neighborhood of 5× 5 is
used when at least half (12) pixels within the neighborhood have a valid depth. The
plane fit is characterized by the ratio between smallest singular value over the sum
of the two larger singular values. Fitting is done in metric x− y − z space and a fit
with a singular value ratio of not more than 0.005 is accepted. The angles between
the viewing axes to the point and the estimated normal is checked and the point
are kept if the angle is less than 80 degrees.
The filtered range images are input in the mesh generation software and a
high-resolution mesh is generated. The resolution of the triangular mesh depends on
the sampling density of the object’s surface. The produced mesh contains typically
some triangles due to support fixtures, some due to noise, and is not watertight. A
multi-resolution subdivision mesh is built from this high-resolution raw mesh. In
the process, holes are filled, triangles edited and the method for mesh simplification
and subsequent subdivision refinement eliminates irregularities of the mesh. The
simplification is done with QSlim [38]. The remaining process is essentially a dis-
placement subdivision surface [72] or normal mesh [47] construction process. The
implementation of this subdivision construction process is due to D. James and is
described with more detail in [95]. The result for a plush toy tiger is shown in
Figure 3.8.
The stereo system also provides calibrated color imagery. These images are
used for very simple texture mapping of the object. The vertices of the subdivision
123
model are projected into the color images with the pinhole camera model. A triangle
of the mesh is assumed to be visible from a camera if all its vertices and its centroid
are visible. Visibility is calculated by ray intersection with the triangular mesh. The
intersection test is accelerated by an octree space division. The triangle is typically
visible from a number of views but with different foreshortening and resolution.
Out of the images where the triangle is visible, the one with the highest viewing
quality measure is selected for texture mapping. The viewing quality measure used
is the product of image area covered by the triangle times the cosine between view
direction and triangle normal. The relative global brightness of all these color images
is also adjusted. The produced texture map (see Figure 3.8(f)) contains many
discontinuities due to inaccuracies of the mesh, misregistration and varying local
brightness. The texture maps could be readily improved by blending of the local
textures over the entire map eliminating abrupt brightness changes at edges and
minimizing the effect of geometric errors, as in [108].
124
Bibliography
[1] M. Akamatsu, G. Nakamura, and S. Steinberg. Identification of Lame coeffi-
cients from boundary observations. Inverse Problems, 7(3):335–354, 1991.
[2] P. Anandan. A computational framework and an algorithm for the measure-
ment of visual motion. International Journal of Computer Vision, 2:283–310,
1989.
[3] D.H. Ballard and O.A. Kimball. Rigid body motion from depth and optic
flow. Computer Vision, Graphics and Image Processing, 22(1):95–115, 1983.
[4] M.R. Banan, M.R. Banan, and K.D. Hjelmstad. Parameter estimation of
structures form static response. I. Computational aspects. Journal of Struc-
tural Engineering, 120(11):3243–3258, 1994.
[5] M.R. Banan, M.R. Banan, and K.D. Hjelmstad. Parameter estimation of
structures form static response. II. Numerical simulation studies. Journal of
Structural Engineering, 120(11):3259–3283, 1994.
[6] J.L. Barron, D.J. Fleet, and S.S. Beauchemin. Performance of optical flow
techniques. IJCV, 12(1):43–77, 1994.
125
[7] C. Basdogan, C.-H. Ho, M.A. Srinivasan, S.D. Small, and S.L. Dawson. Force
interactions in laparascopic simulations: haptic rendering of soft tissues. In
Medicine Meets Virtual Reality, pages 385–391, 1998.
[8] P.J. Besl and N.D. McKay. A method for registration of 3-d shapes. IEEE
Transactions on Pattern Recognition and Machine Intelligence, 14(2):239–256,
Feb 1992.
[9] L.M. Bezerra and S. Saigal. A boundary-element formulation for the inverse
elastostatics problem (IESP) of flaw detection. International Journal for Nu-
merical Methods in Engineering, 36(13):2189–2202, 1993.
[10] L.M. Bezerra and S. Saigal. Inverse boundary traction reconstruction with
the BEM. International Journal of Solids and Structures, 32(10):1417–1431,
1995.
[11] D. Bielser, V. A. Maiwald, and M. H. Gross. Interactive cuts through 3-
dimensional soft tissue. In P. Brunet and R. Scopigno, editors, Eurographics,
Computer Graphics Forum, 18(3), pages C31–C38, Milano, Italy, 1999.
[12] J. Biggs and M.A. Srinivasan. Handbook of Virtual Environment Technology,
chapter Haptic interfaces, K.M. Stanney (editor). Lawrence Erlbaum Asso-
ciates, in press.
[13] M.J. Black and P. Anandan. The robust estimation of multiple motions:
parametric and piecewise-smooth flow fields. Computer Vision and Image
Understanding, 63(1):75–104, 1996.
126
[14] R.C. Bolles and M.A. Fischler. A RANSAC-based approach to model fitting
and its application to finding cylinders in range data. In Int. Joint Conf. on
Art. Intelligence, pages 637–643, Vancouver, Canada, 1981.
[15] S. Di Bona and O. Salvetti. A deformation model for biotissues behavior
simulation. In International Conference on Image Processing, volume 2, pages
443–446, Vancouver, Canada, Sept 2000.
[16] C.A. Brebbia. Boundary element techniques in engineering. Pentech Press,
London, Uk, 1978.
[17] C.A. Brebbia, J.C.F. Telles, and L.C. Wrobel. Boundary element techniques:
theory and applications in engineering. Springer-Verlag, Berlin, 1984.
[18] M. Bro-Nielsen and S. Cotin. Real-time volumetric deformable models for
surgery simulation using finite elements and condensation. In J. Rossignac
and F.X. Sillion, editors, Eurographics, Computer Graphics Forum, 15(3),
pages 57–66, Poitiers, France, 1996.
[19] I. Brouwer, J. Ustin, L. Bentley, A. Sherman, N. Dhruv, and F. Tendick.
Measuring in vivo animal soft tissue properties for haptic modeling in surgical
simulation. In Medicine Meets Virtual Reality, pages 69–74, Jan 2001.
[20] R. Carceroni and K. Kutulakos. Multi-view scene capture by surfel sampling:
from video streams to non-rigid 3d motion, shape and reflectance. In Inter-
national Conference on Computer Vision, volume 2, pages 60–67, 2001.
[21] S. Chaudhuri and S. Chatterjee. Estimation of motion parameters for a de-
formable object from range data. In Computer Vision and Pattern Recogni-
tion, pages 291–295. IEEE, 1989.
127
[22] S.S. Chen and M.A. Penna. Shape and motion of nonrigid bodies. Computer
Vision, Graphics and Image Processing, 36(2/3):175–207, 1986.
[23] I.F. Costa and R. Balaniuk. Lem - an approach for real-time physically based
soft tissue simulation. In International Conference on Robotics and Automa-
tion, pages 2337–2343, Seoul, South Korea, May 2001.
[24] S. Cotin, H. Delingette, and N. Ayache. Real-time elastic deformations of soft
tissues for surgery simualtion. IEEE Trans. on Visulization and Computer
Graphics, 5(1):62–73, 1999.
[25] S. Cotin, H. Delingette, and N. Ayache. A hybrid elastic model allowing
real-time cutting, deformations and force-feedback for surgery training and
simulation. The Visual Computer, 16(8):437–452, 2000.
[26] D. d’Aulignac, R. Balaniuk, and C. Laugier. A haptic interface for a vir-
tual exam of the human thigh. In International Conference on Robotics and
Automation, pages 2452–2457, San Francisco, USA, April 2000.
[27] D. d’Aulignac, M.C. Cavusoglu, and C. Laugier. Modeling the dynamics of the
human thigh for a realistic echographic simulator with force feedback. In Int.
Conf. on Medical Image Computer-Assisted Intervention, pages 1191–1198,
Cambridge, Uk, Sept 1999.
[28] S. De and M.A. Srinivasan. Thin walled models for haptic and graphical
rendering of soft tissues in surgical simulations. In Medicine Meets Virtual
Reality, pages 94–99, 1999.
128
[29] F. Boux de Casson, D. d’Aulignac, and C. Laugier. An interactive model of
the human liver. In 7th International Symposium on Experimental Robotics,
Honolulu, USA, December 2000.
[30] G. Debunne, M. Desbrun, M.-P. Cani, and A.H. Barr. Dynamic real-time
deformations using space & time adaptive sampling. In Computer Graphics,
Annual Conference Series, pages 31–36, Los Angles, Aug 2001. ACM SIG-
GRAPH.
[31] H. Delingette. Toward realistic soft-tissue modeling in medical simulation.
Proceedings of the IEEE, 86(3):512–523, 1998.
[32] M. Desbrun, M. Meyer, P. Schroder, and A.H. Barr. Implicit fairing of ir-
regular meshes using diffusion and curvature flow. In Computer Graphics,
Annual Conference Series, pages 317–324, Los Angles, USA, Aug 1999. ACM
SIGGRAPH.
[33] J. Dominguez. Boundary Elements in Dynamics. Computational Mechanics
Publications, Southhamptom, Uk, 1993.
[34] J.S. Duncan, R.L. Owen, L.H. Staib, and P. Anandan. Measurement of non-
rigid motion using contour shape descriptors. In Computer Vision and Pattern
Recognition, pages 318–324. IEEE, 1991.
[35] P. Fua. A parallel stereo algorithm that produces dense depth maps and
preserves image features. Machine Vision and Applications, 6:35–49, 1993.
[36] F. Ganovelli, P. Cignoni, C. Montani, and R Scopigno. A multiresolution
model for soft objects supporting interactive cuts and lacerations. In M. Gross
129
and F.R.A. Hopgood, editors, Eurographics, Computer Graphics Forum, 19(3),
pages C271–C282, Interlaken, Switzerland, 2000.
[37] L. Gao, K.J. Parker, R.M. Lerner, and S.F. Levinson. Imaging of the elastic
properties of tissue - a review. Ultrasound in Medicine and Biology, 22(8):959–
977, 1996.
[38] M. Garland and P.S. Heckbert. Surface simplification using quadric error
metrics. In Computer Graphics, Annual Conference Series, pages 209–216,
Los Angles, USA, Aug 1997. ACM.
[39] J.C. Gelin and O. Ghouati. Inverse method for material parameters estimation
in the inelastic range. Computational-Mechanics, 16(3):143–150, 1995.
[40] S.F.F. Gibson and B. Mirtich. A survey of deformable modeling in computer
graphics. Technical Report TR-97-19, Mitsubishi Electric Research Labora-
tory, Cambridge, MA, USA, Nov. 1997.
[41] C. Gioda. Indirect identification of the average elastic characteristics of rock
masses. In Structural foundations on rock, volume 1, pages 65–75, 1980.
[42] S. Gottschalk, M. Lin, and D. Manocha. OBB-tree: A hierarchical structure
for rapid interference detection. In Computer Graphics, Annual Conference
Series, pages 171–180, New Orleans, USA, Aug 1996. ACM SIGGRAPH.
[43] S. Govindjee and P.A. Mihalic. Computational methods for inverse finite elas-
tostatics. Computer Methods in Applied Mechanics and Engineering, 136(1-
2):47–57, 1996.
130
[44] S. Govindjee and P.A. Mihalic. Computational methods for inverse deforma-
tions in quasi-incompressible finite elasticity. Int. Journal for Numer. Meth.
in Engineering, 43:821–838, 1998.
[45] M. Grediac, C. Toukourou, and A. Vautrin. Inverse problem in mechanics
of structures: a new approach based on displacement field processing. In
M. Tanaka and H.D. Bui, editors, IUATM Symposium, Inverse Problems in
Engineering Mechanics, Berlin, Germany, 1993. Springer-Verlag.
[46] M. Grediac and A. Vautrin. A new method for determination of bending
rigidities of thin anisotropic plates. Transactions of the ASME, pages 964–
968, 1990.
[47] I. Guskov, K. Vidimce, W. Sweldens, and P. Schroder. Normal meshes. In
Computer Graphics, Annual Conference Series, pages 95–102, New Orleans,
USA, Jul 2000. ACM SIGGRAPH.
[48] P.C. Hansen. Regularization tools: A Matlab package for analysis and solution
of discrete ill-posed problems. Numerical Algorithms, 6(1/2):1–35, 1994.
[49] J.D. Helm, M.A. Sutton, and S.R. Neil. Three-dimensional image correlation
for surface displacement measurements. In S. F. El-Hakim, editor, Videomet-
rics III, volume 2350, pages 32–45. SPIE, 1994.
[50] K. Hirota and T. Kaneko. Representation of soft objects in virtual environ-
ment. In 8th Int. Conf. on Artificial Reality and Tele-Existence (ICAT), pages
59–62, Tokyo, Jp, Dec 1998.
[51] B.K.P. Horn and J.G. Harris. Rigid body motion from range image sequences.
CVGIP: Image Understanding, 53(1):1–13, 1991.
131
[52] B.K.P. Horn and B.G. Schunk. Determining optical flow. Artificial Intelli-
gence, 17(1-3):185–203, 1981.
[53] B. Horowitz and A. Pentland. Recovery of non-rigid motion and structure. In
Computer Vision and Pattern Recognition, pages 325–330. IEEE, 1991.
[54] S.-C. Hsieh and T. Mura. Nondestructive cavity identification in structures.
International Journal of Solids and Structures, 30(12):1579–1587, 1993.
[55] T.S. Huang and S.D. Blostein. Robust algorithms for motion estimation based
on two sequential stereo image pairs. In Computer Vision and Pattern Recog-
nition, pages 518–523. IEEE, 1985.
[56] D. James. Multiresolution Green’s Function Methods for Interactive Simula-
tion of Large-scale Elastostatic Objects and Other Physical Systems in Equilib-
rium. PhD thesis, University of British Columbia, expected November 2001.
[57] D. James and D.K. Pai. A unified treatment of elastostatic contact simulation
for real time haptics. Haptics-e, 2(1), 2001.
[58] D.L. James and D.K. Pai. ArtDefo accurate real time deformable objects.
In Computer Graphics, Annual Conference Series, pages 65–72, Los Angles,
USA, Aug 1999. ACM SIGGRAPH.
[59] A. Joukhadar, C. Bard, and C. Laugier. Planning dextrous operations using
physical models. In International Conference on Robotics and Automation,
pages 748–753. IEEE, 1994.
[60] A. Joukhadar, F. Garat, and C. Laugier. Parameter identification for dynamic
simulation. In International Conference on Robotics and Automation, pages
1928–1933, Albuquerque, NM, USA, 1997. IEEE.
132
[61] F. Kallel and M. Bertrand. Tissue elasticity reconstruction using linear per-
turbation method. IEEE Transactions on Medical Imaging, 15(3):299–313,
1996.
[62] C. Kambhamettu, D.B. Goldgof, and M. He. Determination of motion param-
eters and estimation of point correspondences in small nonrigid deformations.
In Computer Vision and Pattern Recognition, pages 943–946. IEEE, 1994.
[63] T. Kanade, A. Yoshida, K. Oda, H. Kano, and M. Tanaka. A stereo machine
for video-rate dense depth mapping and its new application. In Computer
Vision and Pattern Recognition, pages 196–202. IEEE, 1996.
[64] E. Keeve, S. Girod, R. Kikins, and B. Girod. Deformable modeling of facial
tissue for craniofacial surgery simulation. Computer Aided Surgery, 3(5):228–
238, 1998.
[65] R.M. Koch, M.H. Gross, F.R. Carls, D.F. von Buren, G. Fankhauser, and
Y.I.H. Parish. Simulating facial surgery using finite element models. In Com-
puter Graphics, Annual Conference Series, pages 421–428, New Orleans, USA,
Aug 1996. ACM SIGGRAPH.
[66] S. Kubo. Inverse problems related to the mechanics and fracture of solids and
structures. JSME International Journal, Series 1, 31(2):157–166, 1988.
[67] U. Kuhnapfel, H.K. Cakmak, and H. Maaß. Endoscopic surgery training using
virtual reality and deformable tissue simaulation. Computers & Graphics,
24:671–682, 2000.
[68] S.-H. Lai and B.C. Vemuri. Reliable and efficient computation of optical flow.
International Journal of Computer Vision, 29(2):87–105, 1998.
133
[69] J. Lang. Contour stereo matching by correlation in a constraint satisfaction
approach. In Vision Interface, pages 19–26, 1998.
[70] J. Lang and D.K. Pai. Bayesian estimation of distance and surface normal
with a time-of-flight laser rangefinder. In International Conference on 3-D
Digital Imaging and Modeling. IEEE, October 1999.
[71] J. Lang and D.K. Pai. Estimation of elastic constants from 3D range-flow.
In 3rd International Conference on 3-D Digital Imaging and Modeling, pages
331–338, Quebec City, Canada, 2001. IEEE.
[72] A.W.F. Lee, H. Moreton, and H. Hoppe. Displaced subdivision surfaces. In
Computer Graphics, Annual Conference Series, pages 85–94, New Orleans,
USA, Jul 2000. ACM SIGGRAPH.
[73] Y. Lee, D. Terzopoulos, and K. Waters. Realistic modeling of facial animation.
In Computer Graphics, Annual Conference Series, pages 55–62, Los Angles,
USA, Aug 1995. ACM SIGGRAPH.
[74] W.-H. Liao, S.J. Aggarwal, and J.K. Aggarwal. The reconstruction of dynamic
3D structure of biological objects using stereo microscope images. Machine
Vision and Applications, 9(4):166–178, 1997.
[75] J. Lloyd and V. Hayward. Trajectory generation in multi-RCCL. Journal of
Robotic Systems, 10(3):369–390, 1993.
[76] J.E. Lloyd, J.S. Beis, D.K. Pai, and D.G. Lowe. Model-based telerobotics
with vision. In International Conference on Robotics and Automation, pages
1297–1304. IEEE, 1997.
134
[77] J. Louchet, X. Provot, and D Crochemore. Evolutionary identification of
cloth animation models. In Computer Animation and Simulation Workshop,
Eurographics, pages 245–252, Maastricht, Netherlands, Sept 1995.
[78] H. Maaß and U. Kuhnapfel. Noninvasive measurement of elastic properties
of living tissue. In 13th Int. Congress on Comp. Assisted Radiology (CARS),
pages 865–870, 1999.
[79] K.T. McDonnell, H. Qin, and R.A. Wlodarczyk. Virtual clay: A real-time
sculpting system with haptic toolkits. In Proc. Symposium on Interactive 3D
Graphics, pages 179–190. ACM, Mar 2001.
[80] T. Merz, D. Paulus, and H. Niemann. Line segmentation for interferograms of
continously deforming objects. In F.S. Chau; C.T. Li, editor, Int. Conference
on Experimental Mechanics: Advances and Applications, volume 2921, pages
325–330. SPIE, 1997.
[81] D. Metaxas. Physics-Based Deformable Models - Applications to Computer
Vision, Graphics and Medical Imaging. Kluwer Academic Publisher, Boston,
1996.
[82] D. Metaxas and D. Terzopoulos. Shape and nonrigid motion estimation
through physics-based synthesis. IEEE Transactions on Pattern Recognition
and Machine Intelligence, 15(6):580–591, 1993.
[83] S.M. Metwalli, A.R. Ragab, A.H. Kamel, and A.A. Saheb. Determination of
plastic stress-strain behavior by digital-image-processing technique. Proc. of
the Society for Experimental Mechanics, Inc., 44:414–422, 1987.
135
[84] G. Nakamura and K. Tanuma. A nonuniqueness theorem for an inverse
boundary value problem in elasticity. SIAM Journal on Applied Mathematics,
56(2):602–610, 1996.
[85] G. Nakamura and G. Uhlmann. Identification of Lame parameters by bound-
ary measurements. American Journal of Mathematics, 115(5):1161–1187,
1993.
[86] G. Nakamura and G. Uhlmann. Global uniqueness for an inverse boundary-
problem arising in elasticity. Inventiones Mathematicae, 118(3):457–474, 1994.
[87] G. Nakamura and G. Uhlmann. Inverse problems at the boundary for an elastic
medium. SIAM Journal on Mathematical Analysis, 26(2):263–279, 1995.
[88] J.-C. Nebel. Soft tissue modelling from 3D scanned data. In Deform, Geneva,
Switzerland, Nov. 2000.
[89] J.-C. Nebel, F. J. Rodriguez-Miguel, and W. P. Cockshott. Stroboscobic stereo
rangefinder. In 3rd International Conference on 3-D Digital Imaging and
Modeling, pages 59–64, Quebec City, Canada, 2001. IEEE.
[90] B. Novotny. Some aspects of inverse problem analysis for a layered half-space.
Computational Mechanics, 14(3):266–276, 1994.
[91] J.F. O’Brien and J.K. Hodgins. Graphical modeling and animation of brittle
fracture. In Computer Graphics, Annual Conference Series, pages 137–146,
Los Angles, USA, Aug 1999. ACM SIGGRAPH.
[92] D.K. Pai, editor. Reality-based modeling and applications in reverse engineer-
ing, computer graphics and VR, Workshop. IEEE, 2000.
136
[93] D.K. Pai, J. Lang, J.E. Lloyd, and J.L. Richmond. Reality-based modeling
with ACME: A progress report. In D. Rus and S. Singh, editors, 7th Inter-
national Symposium on Experimental Robotics, LNCIS 271, pages 121–130,
Honolulu, USA, December 2000. Springer-Verlag.
[94] D.K. Pai, J. Lang, J.E. Lloyd, and R.J. Woodham. ACME, a telerobotic active
measurement facility. In P. Corke and J. Trevelyan, editors, 6th International
Symposium on Experimental Robotics, LNCIS 250, pages 391–400, Sydney,
Australia, March 1999. Springer-Verlag.
[95] D.K. Pai, K. van den Doel, D.L. James, J. Lang, J.E. Lloyd, J.L. Richmond,
and S.H. Yau. Scanning physical interaction behavior of 3D objects. In Com-
puter Graphics, Annual Conference Series, pages 87–96, Los Angles, Aug 2001.
ACM SIGGRAPH.
[96] X. Papademetris. Estimation of 3D left ventricular deformation from medical
images using biomechanical models. PhD thesis, Yale University, May 2000.
[97] H.W. Park, S. Shin, and H.S. Lee. Determination of an optimal regularization
factor in system identification with Tikhonov regularization for linear elas-
tic continua. International Journal for Numerical Methods in Engineering,
51:1211–1230, 2001.
[98] A. Pentland. Automatic extraction of deformable part models. International
Journal of Computer Vision, 4(2):107–126, 1990.
[99] G. Picinbono, H. Delingette, and N. Ayache. Non-linear and anisotropic elastic
soft tissue models for medical simulation. In International Conference on
Robotics and Automation, pages 1371–1376, Seoul, South Korea, May 2001.
137
[100] S.M. Platt and N.I. Badler. Animating facial expression. In Computer Graph-
ics, Annual Conference Series, pages 245–252, Dallas, USA, Aug 1981. ACM
SIGGRAPH.
[101] E.P. Popov. Engineering mechanics of solids. Prentice Hall, Englewood Cliffs,
NJ, USA, 1st edition, 1990.
[102] W.H. Press, S.A. Teukolsky, W.T. Vetterling, and B.P. Flannery. Numerical
recipes in C: the art of scientific computing. Cambridge University Press, 2
edition, 1992.
[103] K.R. Raghavan and A.E. Yagle. Inverse and optimal drive problems in elas-
ticity imaging of soft tissue. In Medical Imaging Conference, pages 1886–1890.
IEEE, 1993.
[104] R. Ramanathan and D. Metaxas. Dynamic deformable models for enhanced
haptic rendering in virtual environments. In Virtual Reality, pages 31–35.
IEEE, 2000.
[105] Point Grey Research. Triclops on-line manual. http://www.ptgrey.com/.
[106] D. Reznik and C. Laugier. Dynamic simulation and virtual control of a de-
formable fingertip. In International Conference on Robotics and Automation,
pages 1669–1674. IEEE, 1996.
[107] Zantout R.N. and Zheng Y.F. Geodesics: A tool for solving material properties
inverse problems. In Int. Conference on Industrial Technology, pages 391–394.
IEEE, 1994.
138
[108] C. Rocchini, P. Cignoni, C. Montani, and P. Scopigno. Multiple texture stitch-
ing and blending on 3d objects. In 10th Workshop on Rendering, Eurographics,
pages 173–180, Granada, Spain, 1999.
[109] G. Roth and E. Wibowoo. An efficient volumetric method for building closed
triangular meshes from 3-d image and point data. In Proc. Graphics Interface,
pages 173–180, 1997.
[110] P.J. Rousseeuw. Least median of squares regression. Journal of the American
Statistical Association, 79(388):871–880, 1984.
[111] P.J. Rousseeuw and K. Van Driessen. Computing LTS regression for large
data sets. Technical report, University of Antwerp, 1999.
[112] M.A. Sagar, D. Bullivant, G.D. Mallinson, P.J. Hunter, and I.W. Hunter. A
virtual environment and model of the eye for surgical simulation. In Computer
Graphics, Annual Conference Series, pages 205–212, Orlando, USA, Jul 1994.
ACM SIGGRAPH.
[113] J.K. Salisbury and M.A. Srinivasan. Phantom-based haptic interaction with
virtual objects. IEEE Computer Graphics & Applications, 17(5):6–10, 1997.
[114] M. Sanayei and M.J. Saletnik. Parameter estimation of structures form
static strain measurements. I. Formulation. Journal of Structural Engineering,
122(5):555–562, 1996.
[115] M. Sanayei and M.J. Saletnik. Parameter estimation of structures form static
strain measurements. II. Error sensitivity analysis. Journal of Structural En-
gineering, 122(5):555–562, 1996.
139
[116] Y. Sato, M. Wheeler, and K. Ikeuchi. Object shape and reflectance modeling
from observation. In Computer Graphics, Annual Conference Series, pages
379–387, Los Angles, USA, Aug 1997. ACM.
[117] D.S. Schnur and N. Zabaras. An inverse method for determining elastic-
material properties and a material interface. International Journal for Nu-
merical Methods in Engineering, 33(10):2039–2057, 1992.
[118] A.A. Shabana. Dynamics of Multibody Systems. John Wiley & Sons, New
York, USA, 1989.
[119] H. Spies, B. Jahne, and J.L. Barron. Dense range flow from depth and intensity
data. In International Conference on Pattern Recognition, volume 1, pages
131–134, Barcelona, Spain, 2000.
[120] H. Spies, B. Jahne, and J.L. Barron. Regularised range flow. In European Con-
ference on Computer Vision, LCNS 1843/44, pages 785–799, Dublin, Ireland,
2000. IEEE, Springer-Verlag, Berlin.
[121] G.E. Stavroulakis and H. Antes. Nondestructive elastostatic identification
of unilateral cracks through bem and neural networks. Computational-
Mechanics, 20(5):439–451, 1997.
[122] G. Szekely, C. Brechbuhler, J. Dual, R. Enzler, J. Hug, R. Hutter, N. Ironmon-
ger, M. Kauer, V. Meier, P. Niederer, A. Rhomberg, P. Schmid, G. Schweitzer,
M. Thaler, V. Vuskovic, G. Troster, U. Haller, and M. Bajka. Virtual reality-
based simulation of endoscopic surgery. Presence, 9(3):310–333, 2000.
140
[123] F. Tendick, M. Downes, T. Goktekin, M.C. Cavusoglu, D. Feygin, X. Wu,
R. Eyal, M. Hegarty, and L.W. Way. A virtual environment testbed for train-
ing laparoscopic surgical skills. Presence, 9(3):236–255, 2000.
[124] D. Terzopoulos. Regularization of inverse visual problems involving disconti-
nuities. IEEE Transactions on Pattern Recognition and Machine Intelligence,
8(4):413–424, 1986.
[125] D. Terzopoulos and K. Fleischer. Modeling inelastic deformation: viscoelas-
ticity, plasticity, fracture. In Computer Graphics, Annual Conference Series,
pages 269–278, Atlanta, USA, Aug 1988. ACM SIGGRAPH.
[126] D. Terzopoulos and K. Waters. Physically-based facial modeling, analysis and
animation. Journal of Visualization and Computer Animation, 1:73–80, 1990.
[127] D. Terzopoulos, A.P. Witkin, and M. Kaas. Constraints on deformable models:
Recovering 3D shape and nonrigid motion. Artificial Intelligence, 36(1):91–
123, 1988.
[128] S. Vedula, S. Baker, P. Rander, R. Collins, and T. Kanade. Three-dimensional
scene flow. In International Conference on Computer Vision, pages 722–729,
1999.
[129] V. Vuskovic, M. Kauer, G. Szekely, and M. Reidy. Realistic force feedback
for virtual reality based diagnostic surgery simulators. In International Con-
ference on Robotics and Automation, pages 1592–1598, San Francisco, USA,
April 2000.
141
[130] R.J. Woodham, E. Catanzariti, and A.K. Mackworth. Analysis by syntheis
in computational vision with application to remote sensing. Computational
Intelligence, 1(2):71–79, 1985.
[131] X. Wu, M.S. Downes, T. Gotekin, and F. Tendick. Adaptive nonlinear finite
elements for deformable body simulation using dynamic progressive meshes.
In A. Chalmers and T.-M. Rhyne, editors, Eurographics, Computer Graphics
Forum, 20(3), pages C349–C358, Manchester, Uk, 2001.
[132] M. Yamamoto, P. Boulanger, J.-A. Beraldin, and M. Rioux. Direct estimation
of range flow on deformable shape from a video rate range camera. IEEE
Transactions on Pattern Recognition and Machine Intelligence, 15(1):82–89,
1993.
[133] Y. Zhang and C. Kambhamettu. Integrated 3d scene flow and structure re-
covery from multiview image sequences. In Computer Vision and Pattern
Recognition, volume 2, pages 674–681, 2000.
[134] Z. Zhang. Parameter estimation techniques: A tutorial with application to
conic fitting. Image and Vision Computing, 15(1):59–76, 1997.
[135] Z.-H. Zhong. Finite Element Procedures for Contact-Impact Problems. Oxford
University Press, Oxford, Uk, 1993.
[136] Y. Zhuang and J. Canny. Haptic interactions with global deformations. In
International Conference on Robotics and Automation, pages 2428–2433, San
Francisco, USA, April 2000.
142