Learning and Recognizing Human Dynamics in Video Sequences

Christoph BreglerAlvina Goh

Reading group: 07/06/06

Motivation• Seeing lights attached to the joints

of an actor, humans were able to distinguish human gaits, dance styles, stair climbing, or even the gender and identity.

• This paper attempts to find the right balance of supplied structure and learned parameters.

• Guiding principles:– no early commitment to specific

hypotheses– higher level hypothesis should be

able to disambiguate lower level estimates

– low computation and representation costs

– mid and higher level models should be learnable

Motivation• Human motion is represented at many levels of abstraction.

• This paper describes a way of combining cues from the lowest level to the highest level in order to do activity recognition.

• By suggesting the idea of representing motion data by movemes (like phonemes in speech recognition), it is possible to compose a complex activity (word) out of simple movemes.

Probabilistic Compositional Framework• Low-level primitives: areas of coherent motion

– Image region belonging to a rigid body segment is one coherent motion

• Mid-level categories: simple movements

These are represented by linear dynamical systems

• High-level complex gestures: a sequence of simple movements

These are represented by Hidden Markov Models as successive phases of simple movements

Probabilistic Compositional Framework

At each pixel, represent spatio-temporal image

gradients, and the color value as a random variable

Each blob is presented with a probability

distribution over coherent motion (rigid/affine), color (HSV values), and spatial

support regions.

Temporal sequences of blob tracks are grouped to linear stochastic dynamical

models.

Each dynamical model corresponds to the

emission probability of the state of a hidden Markov

model

Probabilistic Compositional Framework• Example of one leg during a walk cycle

– One coherent blob for upper leg, another for the lower leg– One dynamical system when the leg has ground support, another

when swinging above groundState space: translation and angular velocities

– One cyclic HMM with 2 states

• Sequence of images,– need to find corresponding blob estimates, linear dynamical

systems, and HMMs for a set of different gaits, – classify using the posterior probability

ie, HMM with the highest score is the most likely complex gesture performed in the image sequence

I 1; I 2; :::I t

P (HM M i jI 1; I 2; :::; I t)

1st and 2nd Levels

At each pixel, represent spatio-temporal image

gradients, and the color value as a random variable

Each blob is presented with a probability

distribution over coherent motion (rigid/affine), color (HSV values), and spatial

support regions.

Classification of Pixels into Blobs• For each pixel location (x,y), we need to estimate the label

S(x,y), which indicates which blob the pixel belongs to. (assuming there are K blobs)

• For each one of the K blobs, we need to estimate the motion, color and spatial distribution.

• In order to estimate the labels S(x,y){1,2,...,K} and the model parameters for motion, color and spatiality simultaneously, EM is used.

Representation of Mixture of Blobs• Set of blob hypotheses for a given image frame I(t) are

represented as a mixture of multivariate Gaussians (t)

• Each k(t) contains the parameters for coherent motion and color and the center of mass and second moments in each blob. A background class with uniform distribution is also defined.

• Likelihood of an image frame I(t) conditional on a mixture of blobs hypothesis is:

We want to maximize this cost function

Spatial proximity prior for blob k

Before we can maximize the cost• We need to model

This term is defined using the spatial-temporal image gradient (motion) and color values.

• Optical flow

• How do we model the pdf for optical flow?This is done with a zero-mean Gaussian distribution as described in the paperE. Simoncelli, E. Adelson, and D. Heeger, "Probability distributions of optical flow," in Proceedings of the IEEE Computer Vision and Pattern Recognition Conference, pp. 310--315, 1991.

• This defineswhich we use for

P (I (t;x;y;)jx;y;µk(t))

r I (t;x;y) ¢v(x;y) + I t(t;x;y) = 0

P (r I (t;x;y)jmotion parameters)P (I (t;x;y;)jx;y;µk(t))

Expectation step• Estimation of the support layer for each blob, which is the

posterior probability

Note that we are calculating the expected membership.

Sk(t;x;y) = P (S(t;x;y) = kjI (t;x;y);x;y;µ(t))/ wkP (x;y; jµk(t))P (I (t;x;y)jx;y;µk(t))

Maximization step• Seek to maximize the expected log-likelihood. This is

equivalent to minimizing the following

• Minimizing (8) wrt the constraint k wk=1 is equivalent to assigning

• Minimizing (9) is equivalent to computing the weighted means and covariances for the support layer

• Minimizing (10) is done by extending the Lucas-Kanade motion estimation in the paper “Good Features to track” by Shi and Tomasi, CVPR 1994

wk =P

x;y Sk (t;x;y)Pk

Px;y Sk (t;x;y)

A side note

• Black ink = high probability

• Support map has high probability for the motion model at regions with high gradients as they can be uniquely matched to specific motion models. At non-textured regions, equal probability is assigned to several motion models.

• This approach can be viewed as an edge based tracker at regions with high edge gradients, and a region based tracker at regions with high texture.

Considering Past Estimates• Since EM converges to a local maxima only, it is important

to initialize the starting point intelligently.

• Now given past estimates of the blob parameters (t-1), Kalman filters is used to predict the mean and covariance of (t). (t) = state space of the filter

• The EM starting point is the predicted Kalman state.

3rd and 4th Levels

Temporal sequences of blob tracks are grouped to linear stochastic dynamical

models.

Each dynamical model corresponds to the

emission probability of the state of a hidden Markov

model

Classification of Blobs into Dynamical Systems

• Similar to what was done in the lower levels where we introduced the hidden variables Sk(t,x,y), indicating the probability of a blob at a pixel.

• We now introduce the variable Dm(t,k), which groups a sequence of blobs k(t), k(t-1),.. k(t-d) to a dynamical system m.

• In order to do so, we assume the following discrete 2nd order stochastic dynamical system (moveme)

The state variable Q(t) is the motion estimate of the specific blob k(t),

w is system noise, and

Cm=Bm BmT is the system covariance

Classification of Complex Gestures• Hidden Markov Models are used to represent complex geatures

composed of simple dynamical systems. The state of the HMM corresponds to the validity of the dynamic system. The emission probabilities are represented by the dynamic stochastic system.

• We want to compute the global best segmentation across time. This is done using dynamic programming.

• Estimate that a HMMi fits a track

P(Dm(t,k) is the probability that dynamical system m fits blob k at time t. trn,m is the HMM transition probability between state n and m.

• Compare across all the complex category HMMi and an outlier model HMM0, classify.

Hybrid Dynamical Models• We need to estimate the system parameters of each

dynamical model

and the entries trm,n of the HMM transition probability matrix.

• However, if we are only given the motion trajectories Q(1), Q(2),.. Q(T), we do not know the partition into subsequences. If we know the partition, calculating the system parameters is easy.

• Proceed in a EM manner by maximizing the log likelihood of a set of M dynamical systems and the corresponding HMM

Ám = [A0m;A1m;Bm]

Expectation step• Estimation of the partition of the training set.

Find the probability Dm(t) that training example Q(t) was generated by dynamical system m.

•

computed with dynamic programming with linear complexity

P (Dm(t)jQ(1);::;Q(T);Á1; :::;ÁM ;TRH M M )

Maximization step• Seek to maximize the following expected log-likelihood for

each model m

This is done by solving a linear equation.

• New estimate of the HMM transition probability is also computed with EM.

Experiments: Training and Validation of Gait Models

• 33 sequences of 5 subjects• Running, Walking, Skipping• Sequences start at different

phases• 4 dynamical models per gait• Uniform partition where each

model is assigned 1/4

Experiments: Recognizing of Gaits

• Apply the learned dynamical models and HMM on unseen data

• Outlier model with 1 state and constant velocity dynamical model

• Highest likelihood is the final gait classification

Experiments: Recognizing of Gaits

Conclusion• Decomposes the domain and incorporates different levels

of abstraction using mixture models, EM, recursive Kalman and Markov estimation.

• How much data is need to build the recognizer?

• How much computational time?