realistic modeling of animatable faces in mpeg-4 marco fratarcangeli and marco schaerf university of...
TRANSCRIPT
Realistic Modeling of Animatable Faces in MPEG-4
Marco Fratarcangeli and Marco SchaerfUniversity of Rome “La Sapienza”
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
2
Outline
Definition of the Problem Introducing MPEG-4 FBA Facial Structure Overview Numerical Simulation Results and Conclusions
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
3
Definition of the Problem
MPEG-4 FBA standard mainly focuses on networking capabilities for sending/ receiving animations.
It allows employing model-based coding of human-like faces in a systematic manner.
It does not provide any solution on how to achieve the movements of the face/body.
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
4
Definition of the Problem
We use some already developed techniques to build a face model conforming to the anatomical structure of the human head.
Our contribution is in devising a fast and automatic building process using MPEG-4.
The resulting virtual talking head is used to perform general facial animation coming from an encoded MPEG-4 FAB stream.
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
5
Definition of the Problem
MPEG-4 FBA Encoded Stream
MPEG-4 Player
Animatable Face Model
Modeling Process
3D Mesh + Info MPEG-4 FBA
Output
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
6
Outline
Definition of the Problem Introducing MPEG-4 FBA Facial Structure Overview Numerical Simulation Results and Conclusions
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
7
Introducing MPEG-4 FBA
MPEG-4Standard dealing with efficient
transmission of multimedia on the net
FBAEfficient coding of shape and
animation of human-like faces and bodies
FDPControl points used to define
the shape of a proprietary face model
FAPSet of animation parameters
suitable to define facial movements
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
8
MPEG-4 Facial Definition Pts
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
9
MPEG-4 Facial Animation Pts
68 low-level FAPs used to express basic actions that the face can perform
20 high-level FAPs used to express more complex movements like expressions and visemes
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
10
Outline
Definition of the Problem Introducing MPEG-4 FBA Facial Structure Overview Numerical Simulation Results and Conclusions
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
11
Facial Structure OverviewFrom a VRML model
to a mass-spring structure
Skin structure{Masses [or nodes] from VRML vertices
Springs from VRML topologyActing on the skin model through muscle actions, movement is achieved
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
12
Skin is composed by 3 layers
Skin – Prism Element
subdivided in little triangular prisms
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
13
Skin – Prism Element
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
14
From FDPs
Muscles – Mapping
muscles are mapped automatically
Green points are where the muscle is attached to the skull
Red points are where the muscle is inserted in the skin
Two kinds of muscle: linear and sphincter
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
15
Muscle Mapping - Examples
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
16
Outline
Definition of the Problem Introducing MPEG-4 FBA Facial Structure Overview Numerical Simulation Results and Conclusions
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
17
Semi-Implicit Integrations
A ordinary differential equation system governs the movement of the spring-mass network.
Explicit methods are simple but unstable if applied to stiff systems [e.g. Euler steps].
Implicit methods are precise and stable but computationally expensive [e.g. Backward Euler].
Semi-Implicit Integration = implicit integration approximated with some explicit steps.
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
18
Skin nodes can be displaced by muscular force and by the elastic wave coming from the muscle movement.
Adaptive Displacement
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
19
Outline
Definition of the Problem Introducing MPEG-4 FBA Facial Structure Overview Numerical Simulation Results and Conclusions
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
20
Achieving Facial Animation
MPEG-4 FBAdata stream
MPEG-4 player(PC, PDA, mobile phones)
Animatable Face Model
anAtomic
Audio
FDPs
Static Model
Output
One action set for each MPEG-4
FAP
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
21
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
22
Conclusions - Pros
This method generates realistic facial animation encoded in a MPEG-4 FAP stream of general face meshes using only a standard PC.
It is automatic and fast. It can be used to animate talking
heads in MPEG-4 compliant applications.
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
23
Conclusions - Cons
Animation becomes inaccurate if the polygonal resolution of the face mesh is not enough fine (too few springs!) and/or the mesh topology is not regular (e.g. not simmetric).
Solution: Motion Cloning
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
24
Main References Y. Lee, D. Terzopoulos and K. Waters. ”Constructing
Physics-Based Facial Models of Individuals”
K. Waters. ”A Muscle Model for Animating Three-
Dimensional Facial Expressions”.
D. Baraff and A. Witkin. ”Large Steps in Cloth
Simulation”.
I.S. Pandzic and R. Forchheimer, editors. ”MPEG-4 Facial
Animation – The Standard, Implementation and
Applications”.
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
25
Acknowledgements
Image Coding Group, Department of Electrical Engineering, Linköping University, Sweden
Visage Technologies AB [www.visagetechnologies.com]
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
26
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
27
Skull
iiinet nnF )(
skullF 0
0 iinet nFwhe
notherwise
inetF is the net force on node i
inis the nodal normal of node i
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
28
Skull - Example
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
29
Skull – Lower Jaw
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
30
Muscle Mapping - Examples
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
31
Linear Muscles
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
32
Linear Muscles
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
33
Sphincter Muscles
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
34
Semi-Implicit Integration
txFtxF,KtxFdt
tdxD
dt
txdM musskullela
2
2
Starting from the differential system governing the net:
Equations for one mass mi:
tvdt
tdxi
i
tDvKtxFtxFdt
tdvm iielaimus
ii ,
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
35
Semi-Implicit IntegrationImplicit method (Lagrangian), it is obtained:
i
ni
niela
nimusn
ini m
DvKxFxFtvv
1111 ,
11 ni
ni
ni tvxx Non linear functions
very expensive extract xin+1 for each time step
xin+1 is approximated using an Euler step (explicit method):
ni
ni
ni tvxx 1
Trade-off between stability of the implicit and speed of the explicit
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
36
Adaptive Displacement
09 july 2004 CASA2004 - Realistic Modeling of Animatable Faces in MPEG-4
37