1 1 bayesian cognition winter school at chamonix, france 9.1.2008 bayesian models for computational...
TRANSCRIPT
11
Bayesian Cognition Winter School at Chamonix, France
9.1.2008
Bayesian Models for Computational Laban Movement Analysis
Jörg Rett and Jorge DiasJörg Rett and Jorge Dias
22Intro: Bayesian Models for Computational Laban Movement Analysis
Laban Movement Analysis:
Model for human behaviour
Bayesian Model:
Probabilistic model to analyse
human interaction
33
ApplicationsApplications
Intro: Human Movement Analysis
Mataric et al., Socially assistive robotics for post-stroke rehabilitationJournal of NeuroEngineering and Rehabilitation, 2007
• Rehabilitation
• Socially assistive robotics
• Social robots
Analysis• Studies on
patients
44
ApplicationsApplications
Intro: Applications
Analysis• Studies on
patients
Surveillance• Public spaces
Datasets and videos of the european project caviarhttp://homepages.inf.ed.ac.uk/rbf/CAVIAR/, 2003
55
ApplicationsApplications
Intro: Applications
Analysis• Studies on
patients
Surveillance• Public spaces
Virtual Reality• Interactive virtual
worlds
Enguerran Boissier Character animation using Maya softwareLAAS/ISR Report 05, 2005
66
ApplicationsApplications
Intro: Applications
Analysis• Studies on
patients
Surveillance• Public spaces
Virtual Reality• Interactive virtual
worlds
Control Interfaces• Gesture driven
control
C. Eberst et al., Towards Programming Robots by Gestures, Test-case: Programming Bore Inspection for Small Lotsizes, ICRA, 2006
77
ApplicationsApplications
Intro: Skills
SkillsSkills
Surveillance• Public spaces
Virtual Reality• Interactive virtual
worlds
Control Interfaces• Gesture driven
control
Human Motion Capture
R. Urtasun and P. Fua, 3D Tracking for Gait Characterization and Recognition, FGR, 2004
• Tracking
• Model based
• 3-D vs. 2-D
Analysis• Studies on
patients
88
ApplicationsApplications
Intro: Skills
SkillsSkillsHuman Motion Capture
P. Viola and M.J. Jones, Rapid Object Detection using a Boosted Cascade of Simple Features, CVPR, 2001
Face Recognition
Surveillance• Public spaces
Virtual Reality• Interactive virtual
worlds
Control Interfaces• Gesture driven
control
Analysis• Studies on
patients
99
ApplicationsApplications
Intro: Skills
SkillsSkillsHuman Motion Capture
Face Recognition
Hand Gesture Recognition
Surveillance• Public spaces
Virtual Reality• Interactive virtual
worlds
Control Interfaces• Gesture driven
control
Analysis• Studies on
patients
• Online Behaviour
• Anticipatory Behavior
J. Rett and J. Dias: Gesture Recognition Using a Marionette Model and Dynamic Bayesian Networks (DBNs), ICIAR, 2006
1010
ApplicationsApplications
Intro: Skills
SkillsSkillsHuman Motion Capture
Face Recognition
Hand Gesture Recognition
Laban Movement Analysis
Surveillance• Public spaces
Virtual Reality• Interactive virtual
worlds
Control Interfaces• Gesture driven
control
Analysis• Studies on
patients
• Expressiveness
• Semantic descriptor
J. Rett and J. Dias, Human-robot interface with anticipatory characteristics based on Laban Movement Analysis and Bayesian models, ICORR, 2007
1111
ApplicationsApplications
Intro: Methods
SkillsSkillsHuman Motion Capture
Face Recognition
Hand Gesture Recognition
Laban Movement Analysis
Surveillance• Public spaces
Virtual Reality• Interactive virtual
worlds
Control Interfaces• Gesture driven
control
Analysis• Studies on
patients
MethodsMethods
Bayesian
SVD
Neural Networks
1212Laban: Major components of LMA
Space Shape
Effort
Relation-ship
Body
Laban Movement Analysis (LMA)
• Five major components• Set of semantic descriptors (labels) for movements
Method to human movements.
observedescribenotate
interprete
1313Laban: Body
Space Shape
Effort
Relation-ship
Body component
• Which body parts are moving• How is their movement is related
to the body centre (~navel).• Locomotion• Kinematics
Body
1414Laban: Space
Shape
Effort
Relation-ship
Body
Space component
• Spatial pathways of human movements inside a frame of reference• Three Axes,• Three Planes • Vector Symbols
Space
1515Laban: Effort
Space Shape
Relation-ship
Body
Effort component
• Dynamic qualities of the movement• Inner attitude towards using energy• Four bipolar Effort qualities
Space {Direct, Neutral, Indirect}
Weight {Strong, Neutral, Light}
Time {Sudden, Neutral, Sustained}
Flow {Free, Neutral, Bound}
Neutral qualities
• Single Effort: Rare, difficult to perform• Four Effort: Rare; extreme movements• Three Effort: Most natural• Two Effort: Transitions, failure
Effort
1616Laban: Effort
Space Shape
Relation-ship
Body
Action Drive (Flow = Neutral)Action Space Weight Time
Punch Direct Strong Sudden
Slash Indirect Strong Sudden
Drives
• One Effort quality is neutralEffort
1717Laban: Shape
Space
Effort
Relation-ship
Body
Shape component
• Emerging from the Body and Space components.• Focused on the body or towards a goal in space
Shape
1818Laban: Relationship
Space Shape
Effort
• Modes of interaction with oneself• … with others• … with the external environment.
Body
Relationship
1919
-80%
-70%
-60%
0%
Space
Weight
Time
Flow
Laban: Summary
Relation-ship
Assigning semantic descriptors to the movement ‘Punch’
Hands/Head
1520
2530
-50
510
15
-20
-15
-10
-5
0
5
10
ForwardHigh
Indirect
Light
Sustained
Free
Direct
Strong
Sudden
Bound
10%
50%
80%
50%
Horizontal
Vertical
Saggital
Reach Space
Spreading
Rising
Advancing
Growing
Enclosing
Sinking
Retreating
Shrinking
Shape
Effort
Space
Body
2020Design: Process of designing a Bayesian Model
?
Affirmative, now I am going to perform
this action.
Example:Human–Robot Interaction based on gestures
2121Design: Phenomenon-description
Expressive Movements
Threading a needle
Waving away bugs
Punching
Dabbing paint on a canvas
a) Describing the Phenomenon
• What is the phenomenon?
Movements can be distinguished through their expressiveness
2222Design: Phenomenon-description
a) Describing the Phenomenon
• What is the phenomenon?
• Which features can be observed?
Interesting objects are hands and head
2323Design: Phenomenon-description
a) Describing the Phenomenon
• What is the phenomenon?
• Which features can be observed?
• How can the features be extracted?
Using:• Commercial 3-D motion capture device• Camera based colour tracker
2424Design: Phenomenon-description
1015
2025-10
-50
5
-25
-20
-15
-10
-5
0
5
-10-505
-25
-20
-15
-10
-5
0
5
3-D trajectories are represented through three principal planes. a) Describing the
Phenomenon• What is the
phenomenon?
• Which features can be observed?
• How can the features be extracted?
• How can the features be represented?
3-D Vertical plane
2525Design: Phenomenon-description
Low-level variables and their sample space.
Vector symbolsA {O, U, UR, R, DR, D, DL, L, UL}
CurvatureK {180, 135, 90, 45, 0, -45, -90, -135}
SpeedVel {Zero, Low, Medium, High}
Speed GainAcc {Zero, Low, Medium, High}
a) Describing the Phenomenon
• What is the phenomenon?
• Which features can be observed?
• How can the features be extracted?
• How can the features be represented?
2626Design: Phenomenon-description
LMA variables and their relation to the movements a) Describing the
Phenomenon• What is the
phenomenon?
• Which features can be observed?
• How can the features be extracted?
• How can the features be represented?
• How do the features relate to the phenomenon
Movement Punching
Effort.Space DirectEffort.Weight StrongEffort.Time SuddenEffort.FlowNeutral
Movement Threading a needle
Effort.Space DirectEffort.Weight LightEffort.Time SustainedEffort.FlowBound
2727Design: Phenomenon-description
a) Describing the Phenomenon
• What is the phenomenon?
• Which features can be observed?
• How can the features be extracted?
• How can the features be represented?
• How do the features relate to the phenomenon
Relation of low-level variables to LMA variables
2828Design: Bayesian model
a) Describing the Phenomenon
b) Building the probabilistic models
movement
vector symbols (atoms)
MI
frame
A B C
Bayes-net
• Bayes model for Space
2929Design: Bayesian model
a) Describing the Phenomenon
b) Building the probabilistic models
Joint distribution
P(M I A B C )= P(M) P(I) P(A | M I)
P(B | M I) P(C | M I)
• Bayes model for Space
3030Design: Bayesian model
a) Describing the Phenomenon
b) Building the probabilistic models
Random variables and their sample space
• Bayes model for Space
M {punching, ..., pointing}<n>
I {1, ..., Imax}<Imax>
A {O, F, FR, R, BR, B, BL, L, LF}<9>
B {O, U, UR, R, DR, D, DL, L, UL}<9>
C {O, U, UF, F, DF, D, DB, B , UB}<9>
3131Design: Bayesian model
a) Describing the Phenomenon
b) Building the probabilistic models
• Bayes model for Space
• Bayes model for Effort
Space Time Weight Flow
MovementM PhPhase
E.Sp E.Ti E.We E.Fl
Bayes-net (upper part)
3232Design: Bayesian model
a) Describing the Phenomenon
b) Building the probabilistic models
• Bayes model for Space
• Bayes model for Effort
Bayes-net (lower part)
Space Time Weight Flow
acce-leration
velocity
curva-ture K Vel Acc
E.Sp E.Ti E.We E.Fl
3333Design: Bayesian model
a) Describing the Phenomenon
b) Building the probabilistic models
• Bayes model for Space
• Bayes model for Effort
• Bayes model for Phase
• Bayes model for Geometry
• Bayes model for Shape
• Connecting the sub-models
• Uncertain (Soft) evidence
Full model using Space, Effort and Phase
3434Design: Learning
a) Describing the Phenomenon.
b) Building the probabilistic model.
c) Learning of the probabilities
• What needs to be learned?
movement
atoms
MI
frame
A
Question for learning
P(A | M I)
What is the probability of a vector symbol for a movement M=m at frame I=i?
3535Design: Learning
a) Describing the Phenomenon.
b) Building the probabilistic model.
c) Learning of the probabilities
• What needs to be learned?
movement
atoms
MI
frame
A
Conditional Probability Table
Asking the question for all movements M and all frames I
LearningP(A | M=m1 ... mn I=1 ... imax)
3636Design: Learning
a) Describing the Phenomenon.
b) Building the probabilistic model.
c) Learning of the probabilities
• What needs to be learned?
• How can we learn?
Histogram learning
M = pointing I = 1
Example: Davim, trial 3
O F FR R BR B BL L FL
3737Design: Learning
a) Describing the Phenomenon.
b) Building the probabilistic model.
c) Learning of the probabilities
• What needs to be learned?
• How can we learn?
Zero probability problem
Some events (Atoms) have not been observed.=>Zero probability is assigned=>Problem for later classification
O F FR R BR B BL L FL
3838Design: Learning
a) Describing the Phenomenon.
b) Building the probabilistic model.
c) Learning of the probabilities
• What needs to be learned?
• How can we learn?
Solution:Learning based on the ‚Laplace Sucession Law‘.
An
naA a
1
P*
na number of occurences of event A=a
n number of sets
|_A_| possible values of A
3939Design: Classification
a) Describing the Phenomenon
b) Building the probabilistic model
c) Learning of the probabilities
d) Defining the question for classification
Question in 2-D:
What is the probability distribution of movements m given the frame i and direction symbols of the vertical plane A?
4040Design: Classification
a) Describing the Phenomenon
b) Building the probabilistic model
c) Learning of the probabilities
d) Defining the question for classification
Question in 3-D:
What is the probability distribution of movements m given the frame i and direction symbols of all planes A, B, C?
4141Design: Continuous Update
a) Describing the Phenomenon
b) Building the probabilistic model
c) Learning of the probabilities
d) Defining the question for classification
e) Continuous update of the results
Likelihood computation
For a sequence of n observations of a.
4242Design: Continuous Update
a) Describing the Phenomenon
b) Building the probabilistic model
c) Learning of the probabilities
d) Defining the question for classification
e) Continuous update of the results
Update in 2-D:
4343Results: Confusion tables
Performance evaluation using confusion tables
6 byebye
5 pointing
4 ok
3 stretch
2 maestro
1 lunging
7 shake
8 nthrow
6 by
ebye
5 po
intin
g
4 ok
3 st
retc
h
2 m
aest
ro
1 lu
ngin
g
7 sh
ake
8 nt
hrow
Experiment 1
Using one 2-D projection (fronto-parallel view), (B atoms)
13
movem
ents
Horizontal wavingBye-bye sign
Movement 6
Testing in 13 trials of movement byebye
4444
Experiment 1
Using one 2-D projection (fronto-parallel view), (B atoms)
Results: Confusion tables
Performance evaluation using confusion tables
6 byebye
5 pointing
4 ok
3 stretch
2 maestro
1 lunging
7 shake
8 nthrow
6 by
ebye
5 po
intin
g
4 ok
3 st
retc
h
2 m
aest
ro
1 lu
ngin
g
7 sh
ake
8 nt
hrow
4 1
movem
ents
Sagittal wavingApproach sign
Movement 8
Testing in 5 trials of movement nthrow
4545
Results
Trajectories of nthrow and ok in the vertical plane
Results: Confusion tables
Performance evaluation using confusion tables
6 byebye
5 pointing
4 ok
3 stretch
2 maestro
1 lunging
7 shake
8 nthrow
6 by
ebye
5 po
intin
g
4 ok
3 st
retc
h
2 m
aest
ro
1 lu
ngin
g
7 sh
ake
8 nt
hrow
4 1
movem
ents
1015
2025-10
-50
5
-25
-20
-15
-10
-5
0
5
-10-505
-25
-20
-15
-10
-5
0
5
2025-5
05
10
-25
-20
-15
-10
-5
0
5
10
-50510
-25
-20
-15
-10
-5
0
5
10
nthrow ok
Experiment 1
Using one 2-D projection (fronto-parallel view), (B atoms)
4646
Final Results 2-D
95 trials
31 Wrong classifications
=> Reconition rate 67%
Results: Confusion tables
Performance evaluation using confusion tables
6 byebye
5 pointing
4 ok
3 stretch
2 maestro
1 lunging
7 shake
8 nthrow
6 by
ebye
5 po
intin
g
4 ok
3 st
retc
h
2 m
aest
ro
1 lu
ngin
g
7 sh
ake
8 nt
hrow
movem
ents
Experiment 1
Using one 2-D projection (fronto-parallel view), (B atoms)
4747
Experiment 2
Using the three principal planes (horizontal, vertical, sagittal), (A, B, C atoms)
Results: Confusion tables
Performance evaluation using confusion tables
6 byebye
5 pointing
4 ok
3 stretch
2 maestro
1 lunging
7 shake
8 nthrow
6 by
ebye
5 po
intin
g
4 ok
3 st
retc
h
2 m
aest
ro
1 lu
ngin
g
7 sh
ake
8 nt
hrow
5
movem
ents
Sagittal wavingApproach sign
Movement 8
Testing in 5 trials of movement nthrow
4848
Experiment 2
Using the three principal planes (horizontal, vertical, sagittal), (A, B, C atoms)
Results: Confusion tables
Performance evaluation using confusion tables
6 byebye
5 pointing
4 ok
3 stretch
2 maestro
1 lunging
7 shake
8 nthrow
6 by
ebye
5 po
intin
g
4 ok
3 st
retc
h
2 m
aest
ro
1 lu
ngin
g
7 sh
ake
8 nt
hrow
movem
entsFinal Results 3-D
95 trials
21 Wrong classifications
=> Reconition rate 78%
Final Results 2-D
=> Reconition rate 67%
4949Results: Confusion tables
Performance evaluation using confusion tables
6 byebye
5 pointing
4 ok
3 stretch
2 maestro
1 lunging
7 shake
8 nthrow
6 by
ebye
5 po
intin
g
4 ok
3 st
retc
h
2 m
aest
ro
1 lu
ngin
g
7 sh
ake
8 nt
hrow
movem
entsFinal Results 2-D perspective
95 trials
48 Wrong classifications
=> Reconition rate 48%
Final Results 2-D
=> Reconition rate 67%
Experiment 3
Using one 2-D projection but from a 22° rotated perspective, (B atoms)
Final Results 3-D
=> Reconition rate 78%
5050Results: Continuous classification
01
23
45
67
89
1011
12
34
56
0
0.2
0.4
0.6
0.8
i
0.91certain
quite certain
uncertain0.63
G
P(G)
i = 4i = 0 i = 5 i = 6 i = 8 i = 11
Anticipation and Certainty
5151Results: Human-Robot Interaction
Mov. 1. Demo of Nicole in Coimbra July 2006
Plant
Nicole
Godfather 2
Godfather 1
Coimbra Demo
5252Results: Human-Robot Interaction
Mov. 1. Demo of Nicole in Coimbra July 2006
5353Conclusions and Future Works
Conclusions
• We saw a process of designing computational LMA.
• The Bayesian approach was used for designing, learning and classification.
• Having online-classification opens the possibility for anticipatory behaviour.
• The Space model allows movement classification using a 2-D low-level feature.
• Better classification results are obtained by using features in 3-D.
• Under perspective variation the 2-D approach becomes worse.
Future Work
• Publication on the full Laban model including Effort and Shape.
• Publication on estimating the 3-D position from 2-D data using a geometric model.
• Test the usefulness of the social robot Nicole in a rehabilitation task.
• Extending the application of computational LMA to … … manipulatory movements (placing, grasping, etc.)… observation of human-human interaction (surveillance, etc.)
5454References
Inproceedings (Chi00Emote)Chi, D.; Costa, M.; Zhao, L. & Badler, N., The EMOTE model for Effort and ShapeSIGGRAPH 00, Computer Graphics Proceedings, Annual Conference Series, ACM Press, 2000, 173-182
Phdthesis (Zhao02Synthesis)Zhao, L., Synthesis and Acquisition of Laban Movement Analysis Qualitative Parameters for Communicative Gestures, University of Pennsylvania, 2002
Article (Zhao05Acquiring)Zhao, L. & Badler, N.I., Acquiring and validating motion qualities from live limb gesturesGraphical Models, 2005, 67, 1-16
Computational models for Laban parameters.
5555References
Laban Movement Analysis (LMA) for describing the effect of brain injuries on human movements.
Article (Foround06Changes)Foroud, A. & Whishaw, I.Q., Changes in the kinematic structure and non-kinematic features of movements during skilled reaching after stroke: A Laban Movement Analysis in two case studies, Journal of Neuroscience Methods, 2006, 158, 137-149
Theory of Labananalysis and its application to physical therapy, dance and dance therapy.
Book (Bartenieff80Body)Bartenieff, I. & Lewis, D., Body Movement: Coping with the Environment, Gordon and Breach Science, 1980
Inproceedings (Longstaff01Translating)Longstaff, J.S., Translating "vector symbols" from Laban’s (1926) Choreographie26. Biennial Conference of the International Council of Kinetography Laban, ICKL, Ohio, USA, 2001, 70-86