chap 6 motion capture (mocap)

Chap 6Motion Capture (Mocap)

Animation (U) Chap 6 Animation (U) Chap 6 Motion CaptureMotion Capture 1

CS, NCTU, J. H. Chuang2

What is motion capture? Key-frames, forward kinematics, inverse

kinematics A daunting task for creating physically realistic

motion Requires a large amount of animation talent

Record the motion and map it into a synthetic object Much easier Realistic motion

Animation (U) Chap 6 Motion CaptureAnimation (U) Chap 6 Motion Capture


What is motion capture? track motion of reference points convert to joint angles to drive

an articulated 3D model a deformable surface



What is motion capture? Objective

Reconstruct the 3D motion of a physical object and apply it to a synthetic object

Recording 3D live action, including whole body face hands animals



Applications

Animation Special effects Robot control Interactive characters Games



Computer Puppetry

Shin et al., “Computer puppetry”



VideoOptical Motion Capture in Games



Video: Lord of the rings: Gollum



Video

Facial Mocap in King Kong



Video: Irobot



Pros and Cons of Motion Capture

Pros All fine details of human motion will be recorded

-- if they can be captured Cons

Not so easy to Edit Generalize Control

Not cheap



What is captured?

What do we need to know? X,Y,Z roll, pitch, yaw

Errors cause Joints to come apart Links to grow/shrink Bad contact points



What is captured?

Large and small scale



How to use the data?

Off-line Processed by filtering, inverse kinemtics Produce libraries of motion trajectories

Choose among them Blend between them Modify on the fly

On-line (performance animation) Driving character directly based on what actor

does in real time



History of the technology Recording motion for biomechanics

High accuracy Fewer recorded pints Hand digitizing film Supplement with force plate, muscle activity

Computer animation Rotoscoping

VR tracking technology Less accuracy required Fewer sensors



Optical Motion Capture (Passive) Passive reflection

Camera Infrared, visible, or near infrared strobes High resolution (1 to 4 million pixels) 120-240 frames/sec (max 2000 frames/sec)

Not outdoors No glossy or reflective materials Tight clothing Occlusion of markers by limbs or props



Optical Motion Capture (Active)

Active output of the LED No marker confusion problem Outdoor capture 120 frames/sec (128 markers or four persons) 480 frames/sec (32 markers or single person) 1/3 the cost of passive systems



Magnetic Motion Capture

Electromechanical transducer Heavier sensors Wires on body (wireless back to base station) Limited accuracy (~10x less accuracy than

optical)



Magnetic Motion Capture (cont.)

Smaller workspace Sensors are the cost Sensitive to EMI/metal Relatively cheaper than

optical device

Ascension MotionStar Wireless



Mechanical Motion Capture Subject wears an exoskeleton No interference from light or magnetic field No marker confusions No range limit Some restriction of movement Absolute position is unknown



Mechanical Motion Capture

Data glove Bend sensor + optical tracking 6 DOF

video

http://www.vrealities.com/glove.html



Technology Issues

Resolution/range of motion Calibration Accuracy

Marker movement Sensor noise Restrictions on the environment Capture rate

Occlusion/correspondence



Markers - Examples



Skin Motion Capture Park & Hodgins, SIGGRAPH’06

Uses a conventional optical motion capture system

40-60 markers



Skin Motion Capture Uses a conventional optical motion capture system

A dense set of markers





Detailedsurface model






Data collection and cleaning






Data collection and cleaning Skin Animation



What is motion capture?Steps after motion is captured

Process 2D images to locate, identify, and correlated the markers in multiple video streams

Requires image processing techniques Reconstruct the 3D locations of the markers

Requires camera calibration Overcome numerical inaccuracies

Constrain the 3D marker locations to a model of physical system whose motion is being captured

Require satisfying constraints between relative marker positions



Camera calibration Find the locations and orientations of cameras

in world space and the intrinsic properties of the camera such as focal length, image center, and aspect ratio Use a simple pinhole camera model

An idealized model Usually sufficient for graphics and animation



Camera calibration


Pinhole camera model

3D position reconstruction

Locate a marker’s position in at least two views relative to known camera positions

Animation (U) Chap 6 Motion CaptureAnimation (U) Chap 6 Motion Capture CS, NCTU, J. H. Chuang33

3D position reconstruction Locate a marker’s position in at least two

views relative to known camera positions


. and ofmidpoint thebecan position 3D The

. and for solve and

0)()(

0)()(

:lineeach on pointsclosest theFind

intersect.exactly not willequations 2 thesepractice,In

. and for solve and

)(

)(

Form

21

21

2212

1112

21

2222

1111

PP

PP

CIPP

CIPP

kk

PCIkC

PCIkC

Fitting to the skeleton After the motion of the individual markers looks

smooth and reasonable, Attach markers to the underlying skeletal structure that

is to be controlled by the digitized motion The position of each marker is used to absolutely

position a specific joint Is not straight due to noise, smoothing, and inaccuracy Change in bone length can be significant Markers are located outside the joints at the surface


Fitting to the skeleton Given marker location and the relative distance

from marker to the joint is not sufficient since the direction of displacement is not known Put markers on both side of the joint

Work fine for simple joints, but not the complex joints (such as shoulder and spine)

Use normal of a plane formed by 3 markers Wrist-elbow-shoulder markers plane normal

Offset the elbow marker in the direction of plane normal by the amount measured from the performer


Manipulating motion capture data Processing the signal

Motion signal processing and Motion warping Considers how frequency components capture various qualities

of the motion. And warps the signal in order to satisfy user-supplied key-frame-like constraints

Retargeting the motion Map the motion onto the mismatched synthetic character

and modify it to satisfy important constraints Combining the motion

Assemble motion segments into longer sequence


Processing the signal

Motion parameter values as a time-varying signal Consider values of an individual parameter of the

captured motion as a time-varying signal Signal can be decomposed into frequencies, time-warped,

interpolated with other signals Multidimensional signal (vector-valued signal) A function of time m(t): R -> Rn

Not really in Rn

3 DOF in translation, 3 DOF in absolute orientation, many DOF in relative orientations


Processing the signal

Motion signal processing Considers how frequency components capture

various qualities of the motion. Lower frequencies – represent base activities (walking) Higher frequencies – idiosyncratic movements

(walking with limp)

Motion manipulation/editing/warping Warps the signal in order to satisfy user-supplied

key-frame-like constraints.


Processing the signalMotion signal processing

Treat motion as a multi-dimensional signal Low pass filtering

noise removal High pass filtering

style change High frequency component can be motion

details, not just noise Modify a motion through filtering is not easy

Physical constraints (joint limit, ground contact)? Naturalness?


Processing the signalMotion signal processing

Frequency bands can be extracted, manipulated, and reassembled to allow the user to modify certain qualities of the signal while leaving others undisturbed. Signal is successively convolved with expanded

versions of a filter kernel. Gains of each band are adjusted by the user

and can be summed to reconstruct a motion signal.


Motion signal processing

PROBLEM: High frequencies can be important! Getting rid of them makes motion look soggy

ANSWER: Do not over-apply LPF Small amounts of Low-Pass Filtering Noise modeling Non-linear filters


?

Motion signal processing

High frequencies are important! Don’t occur often Always significant

Impact Rapid, sudden movement Emphasis

Sensitivity of perception Eye is sensitive to high frequencies


Motion manipulation/editing/warpingWhy?

What you get is not what you want You get observations of the performance

Specific performer (a real human) Specific motion With the noise and “realism” of real sensors

You want animation A character Doing something Or something similar but not the same


Motion manipulation/editing/warping

Manipulate time Time warp / Speed control m(t) = m0( f(t) ) f : R R

Manipulate value m(t) = f(m0(t) ) f : Rn Rn


Motion manipulation/editing/warpingManipulating time

m(t) = m0( f (t) ) Time scaling

f(t) = k t Time shifting

f(t) = t + k Time warping

Interpolate a table Align events

Speed control Ease in/Ease out


Motion manipulation/editing/warpingManipulating values

Scale Shift Blending Filtering Transition between motions Cyclification Change style Constraints on the motion Concatenation


Motion manipulation/editing/warpingMotion warping


Witkin and Popovic, “Motion Warping,” SIGGRAPH’95

Keyframes as constraints in smooth deformation

Keyframe placing the ball on the racket at impact

)()()()(' tbttat )()()()(' iiiii tbttat

),( ii t

Motion manipulation/editing/warpingMotion warping


ed.interpolat becan )( and )( which from ),( ofset a

obtain will We).,( sconstraint frame-keysatisfy and

)()()()(

let i.e., ,sconstraint frame-key the

satisfy tocurve original thenslatingand/or tra scalingby

),()( warpingcurvemotion a creates

s.constraint warp time

theinginterpolatby )( mapping warp timea creates

:procedure pingMotion war

sconstraint warp timeofset a :),( frames.-key ofset a:),(

signal. original :)(

'

'

'

tbtaba

t

tbttat

tft

tgt

ttt

t

ii

ii

iiii

Manipulating motion capture dataRetargeting the motion

Map the captured motion onto the mismatched synthetic character and then modify it to satisfy important constraints Constraints:

avoid foot penetration of the floor, avoid self-penetration, no feet sliding when walking

A new motion is constructed, as close to the original motion as possible, while enforcing the constraints.

New motion is formulated as a space-time, non-linear constrained optimization problem.


Retargeting the motion



Define constraints

Apply to a new character



Add Translational Offset (approximated answer)

Solve for constraints(Nonlinear constrained optimization)


Manipulating motion capture dataCombing motions

Motions are captured in segments whose duration last a few minutes each.

Assemble motion segments into longer sections. Need to blend the end of one segment into the

beginning of the next segment Portions to be blended need to be similar Both motion signal processing and motion warping can

be used to isolate, overlap, and then blend two motion segments together.


Motion blending

“Add” two (or more) motions together Really interpolate

m(t) = a m0(t) + (1-a) m1(t)

This is a per-frame operation We’re really interpolating between a series of poses!

m(t) = a(t) m0(t) + (1-a(t)) m1(t)


Motion blending approaches

Radial Basis Function (RBF) Interpolation Rose et al., “Verbs and Adverbs,” IEEE CG&A,

1998 Rose et al., “Artist-directed IK using radial basis

function interpolation,” Eurographics’01 (shown in kinematics lecture)

align example motions use B-spline to represent a motion apply RBF to interpolate between B-spline

coefficients IK to maintain constraints


Motion blending approachesTutorial on RBF Interpolation

A radial basis function (RBF) is a real-valued function whose value depends only on the distance form the center

RBF types Gaussian Thin plate spline Multiquadric


2

)( ded

)log()( 2 ddd

22)( dd

cxd

dN

iii RRrAs

xxxxx i ,)()()(1

Motion blending approachesRBF Interpolation

Approximates a real valued function f(x) by s(x) given the set of values f = (f1, …, fN) at distinct points X = {x1, x2, …, xN}

s(x) is a weighted sum of translations of a radially symmetric basic function augmented by a polynomial term


polynomial of degree at most k

real-valued weight

radial basis function

Motion blending approachesRBF Interpolation

The coefficients in s(x) are obtained by solving a system of linear equations

RBFs are popular for interpolating scattered data as the associated system of linear equations is invertible under very mild conditions on the locations of the data points.


,...,Nifs ii 1 )( x

Motion blending approachesExample: 1D RDF Interpolation


Fit data (x, y) = {(0.15, 1), (0.3, 0), (0.75, 0)}

3

1

)()()(i

iii xxRrxAxs

Motion blendingMotion Transition


m1m0

m0 m1

m0

m1

Motion blendingMotion Transition


Often get small pieces of motion Need to connect them Very useful

motion graph (next lecture) games: concatenate short motions

Easy if motions are similar

Hard if motions are not similar

chap 6 motion capture (mocap)

Documents

motion capturecs

d motion

technologyrecording

videooptical motion

propsanimation u chap

gollumanimation u chap

irobotanimation u chap

real timeanimation u