human emotion synthesis
DESCRIPTION
Human Emotion Synthesis. David Oziem, Lisa Gralewski , Neill Campbell, Colin Dalton, David Gibson, Barry Thomas University of Bristol, Motion Ripper, 3CR Research. Project Group. Motion Ripper Project Methods of motion capture. Re-using captured motion signatures. - PowerPoint PPT PresentationTRANSCRIPT
Human Emotion Synthesis
David Oziem, Lisa Gralewski, Neill Campbell, Colin Dalton, David Gibson, Barry Thomas
University of Bristol, Motion Ripper, 3CR Research
Synthesising Facial Emotions – University of Bristol – 3CR Research
Project Group
• Motion Ripper Project
– Methods of motion capture.– Re-using captured motion signatures.– Synthesising new or extend motion sequences.– Tools to aid animation.
• Collaboration between University of Bristol CS, Matrix Media & Granada.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Introduction
• What is an emotion?
• Ekman outlined 6 different basic emotions.– joy, disgust, surprise, fear, anger and sadness.
• Emotional states relate to ones expression and movement.
• Synthesising video footage of an actress expressing different emotions.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Synthesising Facial Emotions – University of Bristol – 3CR Research
Video Textures
• Video textures or temporal textures are textures with motion. (Szummer’96)
• Schodl’00, reordered frames from the original to produce loops or continuous sequences.
– Doesn’t produce new footage.
• Campbell’01, Fitzgibbon’01, Reissell’01, used Autoregressive process (ARP) to synthesis frames.
Examples of Video Textures
Synthesising Facial Emotions – University of Bristol – 3CR Research
Autoregressive Process
• Statistical model
• Calculating the model involves working out the parameter vector (a1…an) and w.
• n is known as the order of the sequence.
y(t) = – a1y(t – 1) – a2y(t – 2) – … – any(t – n) + w.ε
Parameter vector (a1,…,an) Noise
Current value at time t
Synthesising Facial Emotions – University of Bristol – 3CR Research
Autoregressive Process
• Statistical model
• Increasing dimensionality of y drastically increases the complexity in calculating (a1…an).
y(t) = – a1y(t – 1) – a2y(t – 2) – … – any(t – n) + w.ε
Synthesising Facial Emotions – University of Bristol – 3CR Research
Autoregressive Process
PCA analysis of Sad footage in 2D
Secondary mode
Primary mode
• Principal Components Analysis is used to reduce number of dimensions in the original sequence.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Autoregressive Process
PCA analysis of Sad footage in 2D Generated sequence using an ARP
Secondary mode Secondary mode
Primary mode Primary mode
• Non-Gaussian Distribution is incorrectly modelled by an ARP.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Face Modelling
• Campbell’01, synthesised a talking head.
• Cootes and Talyor’00, combined appearance model.– Isolates shape and texture.
• Requires labelled frames.– Must label important features
on the face.
Labelled points
Synthesising Facial Emotions – University of Bristol – 3CR Research
Combined Appearance
Shape space
Hand Labelled video footage provides a point set which represents the shape space of the clip.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Combined Appearance
Shape space Texture space
Warping each frame into a standard pose, creates the texture space.
The standard pose is the mean position of the points.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Combined Appearance
Shape space Texture space
Combined spaceCombined space
Joining the shape and texture space and then re-analysing using PCA produces the combined space.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Combined Appearance
Shape space Texture space
Combined space
Reconstruction of the original sequence from the combined space.
Combined spaceCombined space
Synthesising Facial Emotions – University of Bristol – 3CR Research
Secondary mode
Primary mode
Combined Appearance
Combined Appearance sequence
Original sequence in 2D
Secondary mode
Primary mode
Change in distribution after applyingThe combined appearance technique
Synthesising Facial Emotions – University of Bristol – 3CR Research
Secondary mode
Primary mode
Combined Appearance
Generated SequenceOriginal sequence
Secondary mode
Primary mode
ARPmodelARP
model
• Visually the generated plot appears to have been generated using the same stochastic process as the original.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Copying and ARP
• Combine the benefits of copying with ARP– New motion signatures.– Handles non-Gaussian distributions.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Copying and ARP
Original inputOriginal input
Reduced inputReduced input
PCAPCA
• Important to reduce the complexity of the search process.• Need around 30 to 40 dimensions in this example.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Copying and ARP
Original inputOriginal input
Reduced inputReduced input
Segmented inputSegmented inputPCAPCA Reduced segmentsReduced segmentsPCAPCA
• Temporal segments of between 15 to 30 frames.• Need to reduce each segment to be able to train ARP’s.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Copying and ARP
Original inputOriginal input
Reduced inputReduced input
Segmented inputSegmented input Reduced segmentsReduced segmentsPCAPCA PCAPCA
ARPARP
Synthesised segmentsSynthesised segments
• Many of the learned models are unstable.• 10-20% are usable.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Copying and ARP
Original inputOriginal input
Reduced inputReduced input
Segmented inputSegmented input Reduced segmentsReduced segmentsPCAPCA PCAPCA
ARPARP
Synthesised segmentsSynthesised segmentsSegment selectionSegment selection
Outputted SequenceOutputted Sequence
Synthesising Facial Emotions – University of Bristol – 3CR Research
Example
First mode
Time t
End of generated sequence.
Possible segments.
Compared section
Synthesising Facial Emotions – University of Bristol – 3CR Research
First mode
Time t
Example
Closest 3 segmentsare chosen.
Synthesising Facial Emotions – University of Bristol – 3CR Research
First mode
Time t
Example
The segment to be copied is randomly selected from the closest 3.
Synthesising Facial Emotions – University of Bristol – 3CR Research
First mode
Time t
Example
Segments are blended together using a small overlap and averaging the overlapping pixels.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Secondary mode
Primary mode
Secondary mode
Primary mode
Copying& ARPmodel
Copying& ARPmodel
PCA analysis of Sad footage in 2D
Generated sequence
Copying and ARP
• Potentially infinitely long.• Includes new novel motions.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Results (Angry)
Source Footage Copying with ARPCombined Appearance ARP
• Combined appearance produces higher resolution frames.
• Better motion from the copying and ARP approach
Synthesising Facial Emotions – University of Bristol – 3CR Research
Results (Sad)
Source Footage Copying with ARPCombined Appearance ARP
• Similar results as with the angry footage– Copied approach is less blurred due to the reduced variance.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Comparison Results
- Combined appearance - Segment copying
• Simple objective comparison.– Randomly selected temporal segments.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Comparison
• Perceptually is it better to have good motion or higher resolution.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Combined appearance Segment Copying with ARP
Synthesising Facial Emotions – University of Bristol – 3CR Research
Other potential uses
• Self Organising Map
• Uses combined appearance– as each ARP model provides a
minimal representation of the given emotion.
• Can navigate between emotions to create new interstates.
Angry Sad Happy
Synthesising Facial Emotions – University of Bristol – 3CR Research
Conclusions
• Both methods can produce synthesised clips of a given emotion.
• Combined appearance produces higher definition frames.
• Copying and ARPs generates more natural movements.
Synthesising Facial Emotions – University of Bristol – 3CR Research
Questions