interactive animation and modeling by drawing - pedagogical applications in medicine

iMAGIS-GRAVIR / IMAG

Animation interactive et modélisation par le dessin

Applications pédagogiques en médecine

David Bourguignon

Doctorat de l’INPGSpécialité : modèles et instruments en médecine et biologie

Préparé au sein du laboratoire GRAVIRSous la direction de Marie-Paule Cani

M.C. Escher


Information Technology in Teaching

•Burgeoning field



•Burgeoning field

•Biology and medicine are visual disciplines– Three-dimensional shapes

Frog tomography data



•Burgeoning field

•Biology and medicine are visual disciplines– Three-dimensional shapes– Dynamic phenomena

Dynamic imagingleft ventricle human heart



•Burgeoning field


•Problems for teaching using real organisms– Practical availability



•Burgeoning field


•Problems for teaching using real organisms– Practical availability– Practical feasibility



•Current solutions– Anatomical databases (images, 3D models)

http://www.netanatomy.com



•Current solutions– Anatomical databases (images, 3D models)– Multimedia documents

http://www.froguts.com




•Drawbacks– No editing tools available




•Drawbacks– No editing tools available– Teacher and students in observer role



•Need for truly interactive teaching tools– User-centered



•Need for truly interactive teaching tools– User-centered– Understand shape and function of organs



•Need for truly interactive teaching tools– User-centered– Understand shape and function of organs– Create, edit, animate models



•Need for truly interactive teaching tools

•Two interdisciplinary collaborations– MENRT research action “Beating heart”




•Two interdisciplinary collaborations– MENRT research action “Beating heart”– Anatomy laboratory CHU Grenoble




•Two interdisciplinary collaborations

•Two pedagogical scenarios– Physiological anatomy course

• Build interactive samples

• Experiment




•Two interdisciplinary collaborations

•Two pedagogical scenarios– Physiological anatomy course– Structural anatomy course

• Draw or annotate

• Create or edit


Our Contributions

•Part 1: Interactive physically based animation– Animating anisotropic elastic materials

[Bourguignon and Cani, EGCAS 2000]

•Part 2: Interaction in 3D using 2D input– Drawing in 3D [Bourguignon et al., EG 2001]– Modeling by drawing


Motivation

•Manipulate interactive samples

Part 1


Motivation


•Biological materials– Dynamics– Nonlinear elasticity– Anisotropy– Incompressibility

Part 1

Computer model ofcardiac geometry andmuscle fiber (McCulloch, UCSD)


Motivation


•Biological materials– Dynamics– Nonlinear elasticity– Anisotropy– Incompressibility

Human liver (Epidaure, INRIA)

Part 1


Motivation


•Biological materials

• Intuitively and efficiently

Part 1


Previous Work

•Continuous Models– Large deformations [O’Brien and Hodgins, 1999]

Part 1


Previous Work

•Continuous Models– Large deformations [O’Brien and Hodgins, 1999]– Multiresolution [Debunne et al., 2001]

Part 1


Previous Work

•Continuous Models– Large deformations [O’Brien and Hodgins, 1999]– Multiresolution [Debunne et al., 2001]– Physical nonlinearities and transversal isotropy

[Picinbono et al., 2001]

Part 1


Previous Work



•Problems– Incompressibility

Part 1


Previous Work



•Problems– Incompressibility– Parameters setting

Part 1


Previous Work

•Discrete Models– Large deformations

Part 1


Previous Work

•Discrete Models– Large deformations– Physical nonlinearities [Lee et al., 1995]

Part 1


Previous Work


•Problems– No multiresolution [Debunne et al., 2001]

Part 1


Previous Work


•Problems– No multiresolution [Debunne et al., 2001]– Anisotropy [Ng and Fiume, 1997]

Part 1


Previous Work


•Problems– No multiresolution [Debunne et al., 2001]– Anisotropy [Ng and Fiume, 1997]– Incompressibility

Part 1


Mass-Spring Systems

• Mesh geometry influences material behavior– Undesired anisotropy

– Incorrect behavior in bending

Tetrahedral mass-spring system

Part 1


Mass-Spring Systems

• Mesh geometry influences material behavior– Undesired anisotropy

– Incorrect behavior in bending

Part 1

Tetrahedral mass-spring system


Our Approach

•Goal– As simple and efficient as mass-spring system– Speed vs precision tradeoff– Anisotropy– Incompressibility

Part 1


Our Approach

•Goal– As simple and efficient as mass-spring system– Speed vs precision tradeoff– Anisotropy– Incompressibility

•Choice– Discrete model– Uncouple forces directions and mesh geometry

[Barzel, 1992]

Part 1


Our Approach

•Data: Geometry

Surface mesh

Part 1


Our Approach

•Data: Geometry

Surface mesh

Part 1

Volume mesh


Our Approach

•Data: Vector field

Surface mesh

Part 1

Volume mesh

Vector field


Our Approach

•Elements

Surface mesh

Part 1

Volume mesh

Vector field

Barycenter

Axes of interest(mechanical characteristics)


Our Approach

•Elements

Surface mesh

Part 1

Volume mesh

Vector field

Barycenter

Axes of interest(mechanical characteristics)

For each element:1. Element deformation2. Local frame deformation3. Forces applied to local frame4. Forces applied to nodes


Forces Calculations

Stretch:Axial damped spring forces (each axis)

Shear:Angular spring forces(each pair of axes)

f1

I1’

I1

e1

f1’

f3

I1’

I1 e1

e3

I3

I3’f1

f1’

f3’

Part 1


Volume Conservation• Soft constraint [Lee et al., 1995]

• Conserve sum of barycenter-vertices distances

fC

fB

fD

fA

Part 1


Volume Conservation

•Comparison with mass-spring systems

With volume conservationforces

Mass-spring system

Without volume conservationforces

Part 1


Results• Comparison with mass-spring systems

– No more undesired anisotropy

– Correct behavior in bending

Orthotropic material (as muscle fiber)Same parameters in the 3 directions

Part 1


Results• Comparison with mass-spring systems

– No more undesired anisotropy

– Correct behavior in bending

Part 1

Orthotropic material (as muscle fiber)Same parameters in the 3 directions


Results• Different anisotropic behaviors with same tetrahedral mesh

Horizontal

Part 1



Diagonal

Part 1



Part 1

Hemicircular



Part 1

Concentric Helicoidal



Part 1

Concentric Helicoidal (top view)



Part 1

Random


Validations

• Emerging behavior [Boux de Casson, 2000]– Define a behavior at the element level

– Measure the emerging behavior at the object level

Part 1


Validations

• Emerging behaviorObject level behavior

Part 1

f1

I1’

I1

e1

f1’

Element level behavior (data points fit) +


Validations

• Multiresolution behavior [Debunne, 2000]

Part 1


Validations

• Multiresolution behavior [Debunne, 2000]

Mass-spring system Our model

Part 1


Conclusion and Future Work

•Conclusion: Pedagogical application– Build interactive samples of biological materials

• Nonlinear, anisotropic behaviors

• Soft constraint for volume conservation

• Efficient

Part 1



•Conclusion: Pedagogical application– Build interactive samples of biological materials

• Nonlinear, anisotropic behaviors

• Soft constraint for volume conservation

• Efficient

– Experiment by varying model parameters• Intuitive system image [Norman, 1988]

Part 1



•Conclusion: Pedagogical application

•Future work: “Animated sketches”– Draw sample

Part 1




•Future work: “Animated sketches”– Draw sample– Specify parameters by drawing

Part 1




•Future work: “Animated sketches”– Draw sample– Specify parameters by drawing

Animate!

Part 1


Our Contributions




Part 2


Motivation

•Most people draw– Writing alternative

• Faster

• More convenient

Part 2


Motivation

•Most people draw– Writing alternative– Minimal tool set

Part 2


Motivation

•Most people draw– Writing alternative– Minimal tool set– Since kindergarten

Part 2


Motivation

•Most people draw

•Few people sculpt– Materials difficult to handle

Part 2


Motivation

•Most people draw

•Few people sculpt– Materials difficult to handle– Simpler with computer ?

• Scanning

• Modeling

Part 2


Motivation

•Most people draw

•Few people sculpt

•Drawing application: Teaching– Example: Pr. Jean-Paul Chirossel, anatomy

laboratory CHU Grenoble

Part 2


Motivation

•Drawing characteristics– Visual abstraction

Human heart

Part 2


Motivation

•Drawing characteristics– Visual abstraction– Indication of uncertainty

Leonardo da Vinci

Part 2


Motivation

•Drawing characteristics– Visual abstraction– Indication of uncertainty– Limitation to single viewpoint

Part 2


Motivation


•Problems– Drawing with multiple viewpoints

Part 2


Motivation


•Problems– Drawing with multiple viewpoints– Modeling by drawing

Part 2


Our Contributions




Part 2.1


Previous Work

•2D-to-3D drawing: 3D Strokes– Input stroke and its shadow [Cohen et al., 1999]

• 3D curves design, no drawing

Part 2.1


Previous Work

•2D-to-3D drawing: 3D Strokes– Input stroke and its shadow [Cohen et al., 1999]– Deep canvas [Disney, 1999]

• Need a 3D model

Part 2.1


Previous Work

•2D-to-3D drawing: 3D Strokes– Input stroke and its shadow [Cohen et al., 1999]– Deep canvas [Disney, 1999]– Billboard, terrain, etc., stroke [Cohen et al., 2000]

• Drawing modes adapted to landscaping only

Part 2.1


Previous Work

•2D-to-3D drawing: 3D Strokes

•2D-to-3D drawing: 3D Objects– Reconstruction [Lipson and Shpitalni, 1996]

• No free-form drawing

Part 2.1


Previous Work

•2D-to-3D drawing: 3D Strokes

•2D-to-3D drawing: 3D Objects– Reconstruction [Lipson and Shpitalni, 1996]– Sketching interface [Igarashi et al., 1999]

• Closed strokes only

Part 2.1


Our Approach

•Drawing in 3D– Augment strokes to true 3D entities

• Line stroke: Space curve (view-independent)

Eye

Edgar Degas

Part 2.1


Our Approach

•Drawing in 3D– Augment strokes to true 3D entities

• Line stroke: Space curve (view-independent)

• Silhouette stroke: Surface contour (view-dependent)

Back

Edgar Degas

Part 2.1


Our Approach

•Drawing in 3D– Augment strokes to true 3D entities– Annotation of existing 3D models

Part 2.1


Our Approach

•Drawing in 3D– Augment strokes to true 3D entities– Annotation of existing 3D models– Illustration in 3D

Part 2.1


Our Approach

•Drawing in 3D

•Choices– Represent line stroke as space curve

Part 2.1


Our Approach

•Drawing in 3D

•Choices– Represent line stroke as space curve– Represent silhouette stroke using local surface

• Infer local surface from user input

• New silhouette from new viewpoint

Part 2.1


Silhouette Stroke

• Infer local surface from user input– Simplest: same local curvature in 3D as in 2D

Part 2.1


Silhouette Stroke

• Infer local surface from user input– Simplest: same local curvature in 3D as in 2D– Modulate width as if fitting circles along curve

Part 2.1


Silhouette Stroke

• Infer local surface from user input– Simplest: same local curvature in 3D as in 2D– Modulate width as if fitting circles along curve– Resulting surface

Part 2.1


Silhouette Stroke

•New silhouette from new viewpoint– Approximate silhouette

Part 2.1


Silhouette Stroke

•New silhouette from new viewpoint– Approximate silhouette– Represent uncertainty away from viewpoint

Part 2.1


Silhouette Stroke

•New silhouette from new viewpoint– Approximate silhouette– Represent uncertainty away from viewpoint– Manage occlusion with background color

Part 2.1


Interface for Drawing

•Two drawing modes– In empty space

Part 2.1



•Two drawing modes– In empty space– Relatively to other objects

Part 2.1



•Video

Part 2.1


Applications

•Annotation

Part 2.1


Applications

• Illustration

Part 2.1


Conclusion

•System for drawing in 3D– View-dependent strokes

Part 2.1


Conclusion

•System for drawing in 3D– View-dependent strokes– Useful for drawing simple scenes in 3D

Part 2.1


Conclusion

•System for drawing in 3D– View-dependent strokes– Useful for drawing simple scenes in 3D– Useful for annotations

Part 2.1


Conclusion

•System for drawing in 3D

•Limitations– Switching between stroke types

Part 2.1


Conclusion

•System for drawing in 3D

•Limitations– Switching between stroke types– Plane positioning can be tedious

Part 2.1


Our Contributions




Part 2.2


Motivation

• Input: Just plain strokes…– Silhouette, sharp features ?– Texture, shading ?– Open, closed, self-intersecting ?

Part 2.2


Motivation

• Input: Just plain strokes…

•Output: Manifold polyhedral surface

Part 2.2


Motivation

• Input: Just plain strokes…

•Output: Manifold polyhedral surface

•Pen-and-paper for sculptors– Painter and sculptor shading

Michelangelo BuonarrotiRembrandt van Rijn

Part 2.2


Previous Work•Painting depth as luminance [Williams, 1990]

Part 2.2

+


Previous Work•Silhouette inflation [Williams, 1991]

Part 2.2


Previous Work•Editing gradient by shading [van Overveld, 1996]

Part 2.2


Previous Work•Bump map inference [Johnston, 2002]

Part 2.2


Previous Work•Direct manipulation interface

Artisan [Alias|wavefront, 2002]

ZBrush [Pixologic, 2002]

Part 2.2


From 2D to 2.5D

•Overview

Strokes2Ddiscontinuous

Part 2.2


From 2D to 2.5D

•Overview

Strokes

Geometry

2Ddiscontinuous

Part 2.2


From 2D to 2.5D

•Overview

Strokes

Geometry Constrained triangulation

2Ddiscontinuous

Part 2.2


From 2D to 2.5D

•Overview

Strokes

Geometry Constrained triangulation

Non-convex hull

2Ddiscontinuous

Part 2.2


From 2D to 2.5D

•Overview

Strokes

Geometry

Image

Constrained triangulation

Non-convex hull

2Ddiscontinuous

Part 2.2


From 2D to 2.5D

•Overview

Strokes

Geometry

Image

Constrained triangulation

Non-convex hull

Height field

2Ddiscontinuous

2.5Dcontinuous

Part 2.2


From 2D to 2.5D•Find a non-convex hull

– Original drawing (polylines)

Part 2.2



– Original drawing (polylines)– Constrained Delaunay triangulation

Part 2.2



– Original drawing (polylines)– Constrained Delaunay triangulation– Non-convex hull [Watson, 1997]

Part 2.2


From 2D to 2.5D•Hole marks

– In comics books production

Stone #3, Avalon Studios

Part 2.2


From 2D to 2.5D•Hole marks

– In comics books production– In our system

Part 2.2


From 2D to 2.5D• Infer a height field

– Large features have large inflations

Part 2.2



– Large features have large inflations– Use geometry information to build it

Part 2.2



– Large features have large inflations– Use geometry information to build it– Use texture information to modulate it

Part 2.2


From 2D to 2.5D• Infer a height field: First step

– Euclidean distance transform

Part 2.2



– Mapping to unit sphere [Oh et al., 2001]

Part 2.2



– Adaptive low pass filter [Williams, 1991]

Part 2.2


From 2D to 2.5D• Infer a height field: Second step

– Use same filter for image

Part 2.2


From 2D to 2.5D• Infer a height field: Third step

– Use previous height field as matte for image

Part 2.2


From 2D to 2.5D• Infer a height field: Result

Part 2.2


From 2D to 2.5D

•Fast polygonal height field approximation [Garland and Heckbert, 1995]

Part 2.2


From 2D to 2.5D

•Result: Manifold polyhedral surface

Part 2.2


Results

•A simple sketch of a human heart


Conclusion

•System for modeling by drawing– Plain strokes as input– Manifold polyhedral surface as output– Using sculptor shading convention

Part 2.2


Conclusion

•System for modeling by drawing– Plain strokes as input– Manifold polyhedral surface as output– Using sculptor shading convention

•Limited to a single viewpoint

Part 2.2


Future Work

•From 2.5D to 3D: Iterative modeling process

Modeling by drawing

Changing viewpoint

Part 2.2


Future Work

•Relief metaphor– From low to high relief– From painting to sculpture

Fourth century B.C.

First century B.C.

Fifteenth century

Part 2.2


General Conclusion

•Animation of anisotropic material– Intuitive– Efficient


General Conclusion

•Animation of anisotropic material

•Three-dimensional drawing system– Use drawing characteristics– Good geometric detail vs modeling speed tradeoff


General Conclusion

•Animation of anisotropic material

•Three-dimensional drawing system

•Modeling by drawing from a single viewpoint


General Future Work

•Evaluation according to ergonomics methods

•“Drawing as a front-end to everything” [Gross and Do, 1996]


Merci de votre attention