priyanshu report

7/31/2019 Priyanshu Report

1/38

An Optimization Based Framework for Pose Estimation

of Human Lower Limbs from a Single Image

Priyanshu Agarwal

December 12, 2010

Abstract

The problem of human pose estimation from a single image is a challenging problem incomputer vision. Most work dealing with the problem of pose estimation from a single image

requires user input in the form of a few clicked points on the image. In this work, we propose a

method to estimate the initial pose of human lower limbs from a single image without any user

assistance. Our method is based on the fact that if an image from a 3D model of a human can

be generated, a metric can be established by comparing this model generated image with the

original image. We use image subtraction as the means to evaluate this metric for measuring

the accuracy of the estimated pose and minimize it using Method of Multipliers in conjugation

with Powells conjugate direction method.

Keywords: Human pose estimation , constrained optimization, single image

1


2/38

Contents

1 Introduction 3

2 System Overview 3

2.1 Human Lower Limb Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2 Model Based Silhouette Generation . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.3 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.4 Optimization Based Body Position Estimation . . . . . . . . . . . . . . . . . . . . 4

2.4.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.4.2 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.4.3 Z Coordinate Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.4.4 X,Y Coordinate Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.5 Optimization Based Pose Estimation . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.5.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.5.2 Objective Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.5.3 Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.5.4 Optimization Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.5.5 Optimization Problem in Standard Form . . . . . . . . . . . . . . . . . . . 8

3 Optimization Approach 9

4 Results and Discussion 10

4.1 Initial Design Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4.2 Optimization Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4.3 Convergence Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

4.3.1 Augmented Lagrangian Method . . . . . . . . . . . . . . . . . . . . . . . 13

4.3.2 Powells Conjugate Direction Method . . . . . . . . . . . . . . . . . . . . 134.3.3 Golden Section . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

5 Conclusion 15

6 Future Work 15

Appendices 16

A MATLAB Code for overall system. 16

B MATLAB Code for Augmented Lagrangian Method implemented for Objective 1. 20

C MATLAB Code for Augmented Lagrangian Method implemented for Objective 2. 22

D MATLAB Code for Augmented Lagrangian Method implemented for Objective 3. 23

E MATLAB Code for evaluation of Objective 1. 25

1


3/38

F MATLAB Code for evaluation of Objective 2. 26

G MATLAB Code for evaluation of Objective 3. 27

H MATLAB Code for Powells Conjugate Direction Method. 28

I MATLAB Code for Golden Section Method. 29

J MATLAB Code for Swanns Bounding Method. 31

K Objective Function Output for different inputs 32

L The inputs used and output at each stage of the image processing 32

M The detailed output obtained for images in KTH dataset 34

2


4/38

1 Introduction

Human pose estimation from images is an active area of research in computer vision with various

applications such as computer interfaces to motion capture for character animation, biometrics or

intelligent surveillance [1]. Figure 1 shows the overview of a system for estimating human shape

and pose from a single image. The initialization step in the system requires some initial user inputfor determining the pose. In order to fully automate the system an automatic initialization of the

human pose is must. In this work we have addressed this problem for those images in which the

human pose is nearly parallel to the camera plane.

Figure 1: Overview of a system for estimating human shape and pose from a single image [2].

2 System Overview

2.1 Human Lower Limb DetectionWe extract silhouette from the original image by converting the original image into a binary image.

The binary image is then searched for a connected region. The centroid of the region is evaluated.

The centroid roughly lies on the waist of the human. The portion of the region lying above this

centroid is then removed to retain the lower limbs. We use standard MATLAB functions for image

processing.

3


5/38

2.2 Model Based Silhouette Generation

We have a VRML lower limbs model which can be positioned and articulated to realize any given

pose. In order to generate a similar silhouette from a VRML model, we first initialize the VRML

model with the required input and then capture a synthetic image from the model. This image is

then converted into a binary image, which provides us the model based silhouette.

2.3 Variables

z

Camera

ane

(i)

x y

13

2

4

(ii)

Figure 2: Variables used in the model. (a) Side view with respect to camera plane, (b) front view

with respect to camera plane.

Our system consists of 3 position variables and 4 angular variables. The x,y position variables

are the location of the human waist in the image coordinate frame. The z position variable is

the depth of the human waist with respect to the camera as shown in Figure 2. The four angular

variables are the right hip flexion, right knee flexion, left hip flexion, left knee flexion.

2.4 Optimization Based Body Position Estimation

We formulate the problem of determining the position of the body as two optimization subprob-

lems.

2.4.1 Problem Statement

Given a single real image with unknown camera parameters determine the pose of the human lower

limbs in sagittal plane.

4


6/38

2.4.2 Assumptions

We assume the following conditions for the developed framework:

1. Human limbs have roughly same aspect ratio for all humans.

2. Illumination conditions in the actual image are such that the human silhouette can be easilyextracted.

3. Model geometry closely represents the human body geometry. This ensures that the silhou-

ette area is roughly same as that of the actual human silhouette. This also ensures that the

centroid of the two in an image lies close to each other. However, even with slightly different

geometry the model will work successfully as the frame works on minimizing the difference.

4. The human limbs are oriented in the sagittal plane.

5. The direction of movement of the human is known and so the model is oriented appropriately.

This can also be determined using optimization. However, in this work we are interested in

determining the actual orientation of the limbs.

6. The human silhouette centroid is such that the area below is represents the human lower

limbs silhouette along with the waist.

2.4.3 Z Coordinate Estimation

The z coordinate estimation is based on the fact that an actual body will roughly occupy a similar

area in an actual image as that of the model in the synthetic image. We set up an optimization

problem based on the difference in the area of the original image and the model generated image

and try to minimize the square of this difference. We also setup an upper and a lower bound on the

z coordinate to bound its value.

Problem Statement

Given a single real image with unknown camera parameters determine the z coordinate of the

model i.e. human waist.

Objective Function

We choose the objective function to be the square of the difference in the silhouette area in the

original image and the model generated silhouette in the synthetic image as follows:

A(bz) = (AoA

m(bz))2 (1)

Constraints

We specify the limit on z coordinate after certain experimentation. The model seems to generate a

reasonable area in the synthetic image if the z coordinate is bounded in the following limts:

bzlower bz bzupper (2)

5


7/38

where bzlower=-30 meters, and bzupper=30 meters.

Optimization Problem

The optimization problem is then setup as follows:

Minimize: A(bz) = (Ao Am(bz))2 (3)

Subject to: bzlower bz bzupper (4)

2.4.4 X,Y Coordinate Estimation

The estimation of the x, y coordinate is based on the fact that once the camera depth is almost the

same the centroid of the silhouette in the original image and the model generated image should

roughly be the same.In order to estimate the x, y coordinate of the body we setup another opti-

mization problem. In this problem we try to minimize the distance between the centroid of the

original silhouette and the model generate silhouette.

Problem StatementGiven a single real image with unknown camera parameters determine the x,y coordinate of the

model i.e. human waist.

Objective Function

The objective function for this problem is the distance between the the square of the distance

between the centroid of the original silhouette and the model generate silhouette given as follows:

C(bx, by) = (xco xcm(bx, by))2 + (yco ycm(bx, by))

2 (5)

Constraints

The x,y coordinates can now be constrained based on the size of the original image. Let the size

of the original image by (m n). So, x,y must lie within following limits:

bxlower bx bxupper (6)

bylower by byupper (7)

where bxlower= -45 meters, bxupper= 45 meters, bylower= -45 meters, and byupper= 45 meters.

Optimization ProblemWe now setup an optimization problem based on above mentioned objective function and con-

straints.

Minimize: C(bx, by) = (xco xcm(bx, by))2 + (yco ycm(bx, by))

2 (8)

Subject to: bxlower bx bxupper (9)

bylower by byupper (10)

6


8/38

2.5 Optimization Based Pose Estimation

This is the primary problem which is addressed in this work. The basic idea behind this is that if

the limbs are oriented in exactly the same way as that in the original image the difference in the

original image and the model generated image should be the least. Here we try to estimate the joint

angles of the two legs by setting up an optimization problem.

2.5.1 Problem Statement

Given a single real image with unknown camera parameters determine the pose of the human lower

limbs in the sagittal plane.

2.5.2 Objective Function

The objective function here need to be evaluated by actually articulating the model with the current

variable values and then capturing an image. This image is then used to obtain the binary image

of lower limbs as explained in Section 2.1. The VRML model is also used to generate a model

based silhouette as explained in Section 2.2. These two silhouettes are then subtracted to obtain

a subtracted image. The subtracted image is not converted into an image with absolute difference

between the two. The region in the absolute subtracted image is counted, which gives a measure

of the extent of mismatch in the two images. We use this absolute area as the objective function

for estimating the pose.

f(1, 2, 3, 4) = Absolute Area of subtracted Image (11)

2.5.3 Constraints

We limit the human joint angles based on the constraints set by the human body. We refer an

example [3] in an open source software OpenSim [4] for setting the following limits:

1lower 1 1upper (12)




where 1lower= -0.6981 radians (40o), 1upper= 1.6581 radians (95

o), 2lower= -2.0943951radians (120o), 2upper= 0 radians (0

o), 3lower= -0.6981 radians (40o), 3upper= 1.6581 radians

(95o), 4lower= -2.0943951 radians (120o), and 4upper= 0 radians (0o)

2.5.4 Optimization Problem

The optimization problem is then setup using the above mentioned objective function and con-

straints.

7


9/38

Minimize: f(1, 2, 3, 4) = Absolute Area of subtracted Image (16)

Subject to: 1lower 1 1upper (17)




2.5.5 Optimization Problem in Standard Form

The optimization problem in standard form can be expressed as the following three optimization

sub-problems:

Minimize: A(x) = (Ao Am(x))2 (21)

Subject to: g11(x) = x3 bzlower 0 (22)g12(x) = bzupper x3 0 (23)

Minimize: C(x) = (xco xcm(x))2 + (yco ycm(x))

2 (24)

Subject to: g21(x) = x1 bxlower 0 (25)

g22(x) = bxupper x1 0 (26)

g23(x) = x2 bylower 0 (27)

g24(x) = byupper x2 0 (28)

Minimize: f(1, 2, 3, 4) = Absolute Area of subtracted Image (29)

Subject to: g31(x) = x4 1lower 0 (30)

g32(x) = 1upper x4 0 (31)

g33(x) = x5 2lower 0 (32)

g34(x) = 2upper x5 0 (33)

g35(x) = x6 3lower 0 (34)

g36(x) = 3upper x6 0 (35)

g37(x) = x7 4lower 0 (36)g38(x) = 4upper x7 0 (37)

where x = [bx,by,bz,1, 2, 3, 4]T, bxlower= -45 meters, bxupper= 45 meters, bylower= -

45 meters, byupper= 45 meters, bzlower=-30 meters, bzupper=30 meters, 1lower= -0.6981 radians

(40o), 1upper= 1.6581 radians (95o), 2lower= -2.0943951 radians (120

o), 2upper= 0 radians(0o), 3lower= -0.6981 radians (40

o), 3upper= 1.6581 radians (95o), 4lower= -2.0943951 radians

(120o), and 4upper= 0 radians (0o)

8


10/38

3 Optimization Approach

The initial approach to solve the optimization problem is presented in Figure 3. However, experi-

mentation shows that the nature of the underlying objective function itself changes with the results

of the first two optimizations subproblems. This brings the challenge as to how to deal with the

three the three subproblems simultaneously. Experiments showed that the optimization problem isunstable when all the parameters are optimized at once using a single objective function. To avoid

this unstable nature of the combined objective function, we choose another optimization approach

in which optimization is carried out individually at each stage.

The overall all system consists of three optimization subroutines as shown in Figure 4. We

run the optimization routines iteratively and minimize the three objective functions sequentially.

We choose to optimize the functions sequentially because the nature of the problem is such that

the optimization of individual functions does not pose contradictory requirements on the design

variables. This makes it possible to solve the problems sequentially and reach a minima.

Initialize all

Optimize for x, y ,z coordinate of human waist i.e. (bx,by,bz) and pose i.e. (1-4)

var a es

Minimize: F = AreaSub Image+1(A0-Am(bz))2+2 {(xc0-xcm (bx,by))

2 + (yc0-ycm (bx,by))2}

Subject to: bxlower bx bxupper, bylower by byupper, bzlower bz bzupper,

11ower1 1upper, 21ower2 2upper, 31ower3 3upper, 41ower4 4upper

No Optimization

has converged?

Stop

Yes

Figure 3: Flowchart of the initial optimization approach.

We use Augmented Lagrangian method i.e. Method of Multipliers for solving the optimization

problem considering its advantage over penalty methods which are less robust due to sensitivity to

penalty parameter chosen. The formulation for the three optimization sub-problems are given as

follows:

9


11/38

P1(x) = A(x) + R1[g11(x) + 112 2

11+ g12(x) + 12

2 212

]

P2(x) =C(x) + R2[g21(x) + 212 2

21+ g22(x) + 22

2 222

+ g23(x) + 232 2

23+

g24(x) + 242 2

24]

P3(x) =f(x) + R3[g31(x) + 312 2

31+ g32(x) + 32

2 232

+ g33(x) + 332 2

33+

g34(x) + 342 2

34+ g35(x) + 35

2 235

+ g36(x) + 362 2

36+

g37(x) + 372 2

37+ g38(x) + 38

2 238

]

In order to solve the ND unconstrained optimization subproblem, we use Powells conjugate

direction method as it does not require any derivative information of the objective function. For

1D optimization subproblem we employ Golden section with Swanns bounding.

However, the nature of the problem is such that there are two possible minimum. This is be-

cause of the fact that the two human limbs can be interchanged to achieve a very similar silhouette.To avoid missing the actual pose we set up the problem such that we end with two probable poses.

Figure 5 shows the flowchart of the overall system. The system takes an image from the data set

and extract the silhouette corresponding to the lower limbs of the human body as described in

Section 2.1.

The area of the silhouette is then evaluated and the model z coordinate is changed such that

the difference in the are of the model generated silhouette and the actual human silhouette is mini-

mized. The centroid of both the silhouettes is then located and optimized to lie as close as possible.

Finally, the optimization of the limb pose is carried out such that the absolute ares of the two sub-

tracted silhouette images is minimized. Once a solution is obtained the pose of the lower limbs is

swapped and the problem is solved once again to obtain a solution with the two legs now inter-

changed in pose. Both the poses are then saved as the probable poses. Some other technique thattakes into account the edges in the image need to be used for identifying the true pose. In case, the

defined optimization technique is being used for a sequence of images the pose before and after

the current image can help in establishing the true current pose. In this work, we do not consider

this problem.

We have coded all optimization routines in MATLAB and have used functions from the im-

age processing toolbox for silhouette extraction and centroid determination. We have also used

functions available in the Virtual Reality Toolbox for VRML model based image generation. Ap-

pendix A-J contain the MATLAB code implemented to solve the optimization problem.

4 Results and Discussion

We test the developed optimization approach for images from the KTH [5] data set with person

walking parallel to the camera plane.

10


12/38

waist

x,by))2

bz

pper,pper

of human

(yc0

-ycm

(

upper

upper

waist i.e.

bz))2

upper

ed Image

r

2

2u

r4 4

i.e. (bx,by

(bx,by))2

r bx b

r by by

e of huma

= (A0-Am(

er bz b

for Pose

AreaSubtrac

upper,

21owe

upper, 41ow

p

Yes

ze all

bles

timization

s has

ged?

oordinate

= (xc0

-xcm

t to: bxlow

bylowe

coordinat

ize: A(bz)

ct to: bzlow

Optimize

e: f(1-4) =

er

1

1

er3

St

Initial

vari

ll three o

proble

conve

e for x, y

: C(bx,by)

Subje

imize for

Mini

Subje

Minimi

ct to:

11o31o

Optimi

Minimiz

Op

Subj

No

Figure 4: A flowchart of the adopted optimization approach.

11


13/38

Obtain an image

rom t e ata set

Obtain the human lower limb silhouette and its

-

centroid

discussed approach

1, 3 2, 4

Again solve all three optimization sub-problems with

Save the solution from both the approaches

Figure 5: Flowchart of the overall system.

4.1 Initial Design Points

We choose an arbitrary feasible pose vector and test it on different images which acts as different

input for the system. The chosen pose vector is x = [3, 4, 0, 0.5359, -0.9626, -0.5277, -0.1271]T.

4.2 Optimization Parameters

We chose the following values for the optimization parameters:

1. 011

= 0, 012

= 0, 021

= 0, 022

= 0, 023

= 0, 031

= 0, 032

= 0, 033

= 0, 034

= 0, 035

=0, 0

36= 0, 0

37= 0, 0

38= 0.

2. = 0.1 for Swanns bounding.

3. R1 = 2, R2 = 2, R3 = 100. We choose a high value for R3 to avoid any violation of theconstrains on the pose. However, the final solution is found to be independent of the value

of R.

12


14/38

4.3 Convergence Criteria

We choose the following convergence criterion at different levels of the solution:

4.3.1 Augmented Lagrangian Method

The convergence criteria consist of both the change in the value of the function and the pose

variables which is described as follows:

Pk Pk1 fXk Xk1

x

where k is the iteration number, f1 = 10, f2 = 5, f3 = 100, x1 = 0.1, x2 = 0.1, and x3 = 0.2.The subscripts represent the values for the three optimization subproblems.

4.3.2 Powells Conjugate Direction Method

For Powells conjugate direction method the convergence criteria consists of the change in the

value of the function and a limit on the number of iterations:

Pk Pk1 f

k kmax

where k is the iteration number, f1 = 10, f2 = 5, f3 = 100 and kmax = 100.

4.3.3 Golden Section

The convergence criteria consist of both the change in the value of the function and the pose

variables, and a limit on the number of iterations.

Pk Pk1 fXk Xk1

x

k kmax

where k is the iteration number, f1 = 10, f2 = 5, f3 = 100, kmax = 100.

Such large value of the change in the function value is chosen as the convergence criteria asthe function evaluation is in terms of pixels with makes it difficult for the optimization to converge

with lower values.

Figure 6 shows the input images used and the corresponding solutions obtained. For every pose

two possible solutions are obtained. The results show that the optimization predicts the pose the

human lower limbs quite well. While the performance is very good for images where no occlusion

of a limb is present, the output is satisfactory for images having occlusion. Appendix M contains

detailed output for images in the KTH dataset.13


15/38

(i) (ii) (iii)

(iv) (v) (vi)

(vii) (viii) (ix)

(x) (xi) (xii)

(xiii) (xiv) (xv)

Figure 6: The two outputs obtained for different input images. The first column shows the input

images, the second and the third column show the output model images for the estimated pose.

14


16/38

5 Conclusion

The presented optimization based framework is a good way for estimating human pose from a

single image. This technique provides most probable poses i.e. capitalizes on finding the local

minimums of a multi-modal function and then choosing the global minimum using some other

methodology not discussed. It also performs fairly well in case of occlusion of limb. Most tech-niques present today require some user input, but the approach presented in this work alleviates

all such requirements. However, computation time to arrive at a solution was observed to be large

for certain images ( few hours on a standard PC with a 3Gb RAM) in which it took some time to

converge. We believe more efficient implementation of the subroutines in C/C++ can reduce the

solution time.

6 Future Work

The present work does not address the issue of providing the most probable pose instead it provides

two probable poses. In future using some technique that incorporates the edges in the originalimage, as a metric, can be used to finally choose one pose out of the two probable ones. This

can also be achieved by using inference or move limits from previous pose in case a sequence of

images are being processed. Also, the technique can be extended for images in which the human

limbs are oriented in 3D. In addition, the problem can be extended for the full human body. Also,

an optimization framework can be setup to optimize the geometry of the model simultaneously in

case a video i.e. sequence of images are being processed. This adaptability in the model shape will

result in more plausible pose estimation.

References

1. Sminchisescu C. and Telea A. (2002). Human pose estimation from silhouettes a consis-

tent approach using distance level sets. In International conference on computer graphics,

visualization and computer vision (WSCG).

2. Guan P., Weiss A., Balan A. and Black M.: Estimating human shape and pose from a single

image. In: ICCV (2009).

3. Delp S.L., Anderson F.C., Arnold,A.S., Loan P., Habib A., John C.T., Guendelman E. and

Thelen D.G., OpenSim: open-source software to create and analyze dynamic simulations of

movement, IEEE Trans. Biomed. Eng. 54 (2007), pp. 19401950.

4. Anderson F.C. and Pandy M.G., Dynamic optimization of human walking. Journal of Biome-chanical Engineering, 2001.

5. KTH dataset, Available: http://www.nada.kth.se/cvap/actions/

15


17/38

A MATLAB Code for overall system.

%MATLAB Code for the overall system

c l o s e a l l

c l c

5 i f( e x i s t (w) & isopen(w))c l o s e (w);

d e l e t e (w);

en d

10 c le a r a l l

epsilon_x = 0.1;

i=185;

F = [];

15 X = [];

X1 = [];

X2 = [];

SCORE = [];

SCORE1 = [];

20 SCORE2 = [];

bx = 3;

by = 4;

bz = 0;

rhf = 30.70623249* p i /180;

25 rkf = -55.15546834* p i /180;

lhf = -30.23302178* p i /180;

lkf = -7.28018554* p i /180;

d i r = [cd \KTH\frame];

30

Ireal = imread([ d i r n u m2 st r(i) .jpg]);S = s i z e (Ireal);

% figure

% fig = subplot(2,3,1,Position,[600 600 S(2) S(1)]);

f i g u r e (Position,[400 400 S(2) S(1)]);

35

w = vrworld(lowerbodyu2);

open(w);

% c = vr.canvas(w, gcf);

% subplot(2,3,1)

40 c = vr.canvas(w, g c f);

body = vrnode(w,bodycf);

rthigh = vrnode(w,rthigh);

rleg = vrnode(w,rleg);lthigh = vrnode(w,lthigh);

45 lleg = vrnode(w,lleg);

nodes = [body rthigh rleg lthigh lleg];

x0 = [bx; by; bz; rhf; rkf; lhf; lkf]; %bodyx, bodyy, bodyz,

50 %right hip flexion, right knee flexion, left hip flexion, left knee flexion

16


18/38

f i g u r e ;

f o r i=194:234

55 x = x0;

Ireal = imread([ d i r n u m2 st r(i) .jpg]);

% figure(6);

s u b p l o t (2,3,2);imshow(Ireal);

60 % imwrite(Ireal,[Iorg num2str(i-184) .jpg]);

Ireal = im2bw(Ireal,0.2);

% figure(7);

s u b p l o t (2,3,3);

imshow(Ireal);

65 % imwrite(Ireal,[Ibin num2str(i-184) .jpg]);

Stats = regionprops(Ireal,Area,BoundingBox,Centroid);

human = f i n d ([Stats(:).Area]>500);

Ireal(1:uint8(Stats(human).Centroid(2)),:) = 0;

% figure(5);

70 s u b p l o t (2,3,4);

imshow(Ireal);

% imwrite(Ireal,[Ilimb num2str(i-184) .jpg]);

ho l d on;

p l o t (Stats(human).Centroid(1),Stats(human).Centroid(2),r*);

75 Stats = regionprops(Ireal,Area,BoundingBox,Centroid);

human = f i n d ([Stats(:).Area]>250);


px = x;

k = 0 ;

80 w h i l e ((norm(x-px)/norm(x))>epsilon_x | k==0)

px = x;

x = [x0(1:2);x(3:7)];

[x, f l a g ] = ALM1(x,c,nodes,Ireal);

i f( f l a g ==1)

85 d i s p ([Constrained Optimization Failed->Unconstrained Optimization

Failed->Swans Bounding Failed]);

break;

e l s e i f( f l a g ==2)

d i s p ([Constrained Optimization Failed->Unconstrained Optimization

90 Failed->Linear Search Failed]);

break;


d i s p (Constrained Optimization Failed->Powells Method Failed);

break;

95 en d

x = [px(1:2);x(3:7)];

[x, f l a g ] = ALM2(x,c,nodes,Ireal,Stats(human).Centroid(1),...

Stats(human).Centroid(2));

100 i f( f l a g ==1)



break;


17


19/38


Failed->Linear Search Failed]);

break;



110 break;

en d[x,Score1, f l a g ] = ALM3(x,c,nodes,Ireal);

i f( f l a g ==1)


115 Failed->Swans Bounding Failed]);

break;




120 break;



break;

en d125 k = k+1;

en d

X2 = [X2, x];

SCORE1 = [SCORE1 Score1];

bx = x(1);

130 by = x(2);

bz = x(3);

rhf = x(4);

rkf = x(5);

lhf = x(6);

135 lkf = x(7);

body.translation = [bx by bz];

rthigh.rotation = [1 0 0 -rhf];

rleg.rotation = [1 0 0 -rkf];

140 lthigh.rotation = [1 0 0 -lhf];

lleg.rotation = [-1 0 0 lkf];

vrdrawnow;





Ivrml = c a p t u r e (c);

imwrite(Ivrml,[Output1_ num2str(i) .jpg],jpg);

150 x1 = x;

x = [x(1:3);x(6:7); x(4:5)] %Swapping the angles of left and right leg

% pause

k=0;

w h i l e ((norm(x-px)/norm(x))>epsilon_x | k==0)

155 px = x;

x = [x0(1:2);x(3:7)];

[x, f l a g ] = ALM1(x,c,nodes,Ireal);

18


20/38

i f( f l a g ==1)



break;



Failed->Linear Search Failed]);165 break;



break;

en d

170 x = [px(1:2);x(3:7)];

[x, f l a g ] = ALM2(x,c,nodes,Ireal,Stats(human).Centroid(1),...

Stats(human).Centroid(2));

i f( f l a g ==1)



break;

e l s e i f( f l a g ==2)d i s p ([Constrained Optimization Failed->Unconstrained Optimization


180 break;



break;

en d

185 [x,Score2, f l a g ] = ALM3(x,c,nodes,Ireal);

i f( f l a g ==1)



break;

190 e l s e i f( f l a g ==2)



break;


195 d i s p (Constrained Optimization Failed->Powells Method Failed);

break;

en d

k = k+1;

en d

200 X2 = [X2, x];

SCORE = [SCORE Score];

bx = x(1);by = x(2);

bz = x(3);

205 rhf = x(4);

rkf = x(5);

lhf = x(6);

lkf = x(7);

210 body.translation = [bx by bz];

19


21/38



lthigh.rotation = [1 0 0 -lhf];


215 vrdrawnow;


rleg.rotation = [1 0 0 -rkf];lthigh.rotation = [1 0 0 -lhf];


220 Ivrml = c a p t u r e (c);

imwrite(Ivrml,[Output2_ num2str(i) .jpg],jpg);

en d

pa us e;

c l o s e (w);

225 d e l e t e (w);

B MATLAB Code for Augmented Lagrangian Method imple-

mented for Objective 1.

f u n c t i o n [x1, f l a g ] = ALM1(x,c,nodes,Ireal)

% Augmented Lagrangian Method

bzu = 30; %Upper limit on body z coordinate

5 bzl = -30; %Lower limit on body z coordinate

R = 2 ; %ALM Penalty Coefficient

x0 = x; % x = [bx by bz rhf rkf lhf lkf]

sigma0 = 0;

10epsilon_f = 10;

epsilon_x = 0.1;

%Optimization 1

15 f = @(x) Objective1(x,c,nodes,Ireal);

g1 = @(x) bzu-x(3);

g2 = @(x) x(3)-bzl;

g_penalty = @(g,x,sigma) (g(x)+sigma


22/38

PF = [P(x0,sigma)];

F = [f(x0)];

G = [g1(x0) g2(x0)];

35

w h i l e ( abs(pP-P(x1,sigma))>epsilon_f | norm(x1-px)>epsilon_x | i==0 )

i = i+1;

pP = P(x1,sigma);px = x1;

40 [x1, f l a g ] = Powell(@(x) P(AX(px,x),sigma),CX(px),epsilon_f );

x1 = AX(px,x1);

i f( f l a g ==1)



45 break;




break;

50 e l s e i f( f l a g ==3)

d i s p (Constrained Optimization Failed->Powells Method Failed);break;

en d

S = [S; sigma];

55 I = [I; i];

X = [X, x1];

PF = [PF; P(x1,sigma)];

F = [F; f(x1)];

G = [G; g1(x1), g2(x1)];

60 sigma = [((g1(x1)+sigma(1))Swans Bounding Failed]);




75 e l s e i f( f l a g ==3)

d i s p (Constrained Optimization Failed->Powells Method Failed);en d

f p r i n t f (\n\n);

21


23/38

C MATLAB Code for Augmented Lagrangian Method imple-


f u n c t i o n [x1, f l a g ] = ALM2(x,c,nodes,Ireal,xc,yc)


bxu = 45; %Upper limit on body x coordinate

5 bxl = -45; %Lower limit on body x coordinate

byu = 45; %Upper limit on body y coordinate

byl = -45; %Lower limit on body y coordinate

10 R = 2 ; %ALM Penalty Coefficient

x0 = x; % x = [bx by bz rhf rkf lhf lkf]

sigma0 = 0;

epsilon_f = 5;

15 epsilon_x = 0.1;

%Optimization 2

f = @(x) Objective2(x,c,nodes,Ireal,xc,yc);

g1 = @(x) x(1)-bxl;

20 g2 = @(x) bxu-x(1);

g3 = @(x) x(2)-byl;

g4 = @(x) byu-x(2);

g_penalty = @(g,x,sigma) (g(x)+sigmaepsilon_f | norm(x1-px)>epsilon_x | i==0 )

i = i+1;

45 pP = P(x1,sigma);

px = x1;

[x1, f l a g ] = Powell(@(x) P(AX(px,x),sigma),CX(px),epsilon_f );

x1 = AX(px,x1);

i f( f l a g ==1)

22


24/38



break;



55 Failed->Linear Search Failed]);

break;e l s e i f( f l a g ==3)


break;

60 en d

S = [S; sigma];

I = [I; i];

X = [X, x1];


65 F = [F; f(x1)];

G = [G; g1(x1), g2(x1), g3(x1), g4(x1)];

sigma = [((g1(x1)+sigma(1))Linear Search Failed]);


85 d i s p (Constrained Optimization Failed->Powells Method Failed);

en d

f p r i n t f (\n\n);

D MATLAB Code for Augmented Lagrangian Method imple-


f u n c t i o n [x1,Score, f l a g ] = ALM3(x,c,nodes,Ireal)


% close all

% clear all

5 % clc

rhfu = 95* p i /180; %Upper limit on right hip flexion anlge

rhfl = -40* p i /180; %Lower limit on right hip flexion anlge

23


25/38

10 rkfu = 0* p i /180; %Upper limit on right knee flexion anlge

rkfl = -120* p i /180; %Lower limit on right knee flexion anlge

lhfu = 95* p i /180; %Upper limit on left hip flexion anlge

lhfl = -40* p i /180; %Lower limit on left hip flexion anlge

15

lkfu = 0* p i /180; %Upper limit on left knee flexion anlge

lkfl = -120* p i /180; %Lower limit on left knee flexion anlge

R = 100; %ALM Penalty Coefficient

20 x0 = x; % x = [bx by bz rhf rkf lhf lkf]

sigma0 = 0;

epsilon_f = 100;

epsilon_x = 0.2;

25

%Optimization 3

f = @(x) Objective3(x,c,nodes,Ireal);

g1 = @(x) x(4)-rhfl;

g2 = @(x) rhfu-x(4);

30 g3 = @(x) x(5)-rkfl;

g4 = @(x) rkfu-x(5);

g5 = @(x) x(6)-lhfl;

g6 = @(x) lhfu-x(6);

g7 = @(x) x(7)-lkfl;

35 g8 = @(x) lkfu-x(7);

g_penalty = @(g,x,sigma) (g(x)+sigmaepsilon_f | norm(x1-px)>epsilon_x | i==0 )

i = i+1;

pP = P(x1,sigma);

60 px = x1;

[x1, f l a g ] = Powell(@(x) P(AX(px,x),sigma),CX(px),epsilon_f);

24


26/38

x1 = AX(px,x1);

i f( f l a g ==1)



break;


d i s p ([Constrained Optimization Failed->Unconstrained OptimizationFailed->Linear Search Failed]);

70 break;



break;

en d

75 S = [S; sigma];

I = [I; i];

X = [X, x1];


F = [F; f(x1)];

80 G = [G; g1(x1), g2(x1), g3(x1), g4(x1), g5(x1), g6(x1), g7(x1), g8(x1)];

sigma = [((g1(x1)+sigma(1))


27/38

%MATLAB Function to evaluate the Objective Function 1 value

f u n c t i o n [val] = Objective1(x,c,nodes,Ireal)

bx = x(1);

by = x(2);

5 bz = x(3);

rhf = x(4);rkf = x(5);

lhf = x(6);

lkf = x(7);

10

body = nodes(1);

rthigh = nodes(2);

rleg = nodes(3);

lthigh = nodes(4);

15 lleg = nodes(5);



rleg.rotation = [1 0 0 -rkf];20 lthigh.rotation = [1 0 0 -lhf];


vrdrawnow;


25 Ivrml=im2bw( c a p t u r e (c),0.1);

Areal = sum(sum(Ireal));

Avrml = sum(sum(Ivrml));

val = (Areal-Avrml)2;

f i g u r e (2);

30 s u b p l o t (2,3,5);

imshow(Ivrml);

ho l d on;t e x t (60,10,num2str(val),color,white);

F MATLAB Code for evaluation of Objective 2.


f u n c t i o n [val] = Objective2(x,c,nodes,Ireal,xc,yc)

bx = x(1);

by = x(2);

5 bz = x(3);

rhf = x(4);rkf = x(5);

lhf = x(6);

lkf = x(7);

10

body = nodes(1);

rthigh = nodes(2);

rleg = nodes(3);

lthigh = nodes(4);

26


28/38

15 lleg = nodes(5);





lleg.rotation = [-1 0 0 lkf];vrdrawnow;






Ivrml=im2bw( c a p t u r e (c),0.1);

Stats = regionprops(Ivrml,Centroid,Area);

30 human = f i n d ([Stats(:).Area]>250);

f i g u r e (2);

s u b p l o t (2,3,5);

imshow(Ivrml);

35 ho l d on;

i f( i s e m p t y(Stats(human)))


x = Stats(human).Centroid(1);

y = Stats(human).Centroid(2);

40 e l s e

x = 0 ;

y = 0 ;

en d

45 val = (x-xc)2+(y-yc)2;

t e x t (60,10,num2str(val),color,white);

G MATLAB Code for evaluation of Objective 3.


f u n c t i o n [val] = Objective3(x,c,nodes,Ireal)

bx = x(1);

by = x(2);

5 bz = x(3);

rhf = x(4);

rkf = x(5);

lhf = x(6);lkf = x(7);

10

body = nodes(1);

rthigh = nodes(2);

rleg = nodes(3);

lthigh = nodes(4);

15 lleg = nodes(5);

27


29/38






vrdrawnow;

rthigh.rotation = [1 0 0 -rhf];rleg.rotation = [1 0 0 -rkf];




Ivrml=im2bw( c a p t u r e (c),0.1);

Areal = sum(sum(Ireal));

30 Avrml = sum(sum(Ivrml));

Isub = abs(Ivrml-Ireal);

val = sum(sum(Isub));

f i g u r e (2);

s u b p l o t (2,3,5);

35 imshow(Ivrml);

s u b p l o t (2,3,6);imshow(Isub);

ho l d on;

t e x t (60,10,num2str(val),color,white);

H MATLAB Code for Powells Conjugate Direction Method.

f u n c t i o n [x1, f l a g ] = Powell(f,x,epsilon_f)

%Powells Conjugate Direction Method

x1 = x;5 px = x1;

a = 0 ; %intial value for alpha

d = 0.1;%delta for swans method

kMax = 10;

10 S = e y e ( l e n g t h (x1));

d i s p (Results using Powells Conjugate Direction Method);

k =1;

j=1;

px = x1;

15 w h i l e ((k==1)|( abs(f(px)-f(x1))>epsilon_f))

px = x1;

J = [];X = [x1];

F = [f(x1)];

20 Alpha = [a];

i f( s i z e (S,1)==1)

start = 1;

e l s e

start = s i z e (S,1)+1;

25 en d

28


30/38

f o r j=start:-1:1

i f j==1

s = S(:, s i z e (S,1));

e l s e

30 s = S(:,j-1);

en d

x2 = @(a) x1+a*s;[lb,ub, f l a g ] = swans(a,d,f,x2);

i f( f l a g ==1)

35 d i s p (Unconstrained Optimization Failed->Swans Bounding Failed);

break;

en d

[oa, f l a g ] = goldensection(lb,ub,f,x2,epsilon_f);

i f( f l a g ==2)

40 d i s p (Unconstrained Optimization Failed->Linear Search Failed);

break;

en d

J = [J; j];

Alpha = [Alpha; oa];

45 x1 = x2(oa);

X = [X,x1];

F = [F;f(x1)];

en d

i f( s i z e (S,1)==1)

50 S = s ;

e l s e

S = [S(:,2:en d), s];

s = (X(:, s i z e (X,1))-X(:,1))./norm(X(:, s i z e (X,1))-X(:,1));

en d

55 J = [J; 0];

d i s p ([Iteration num2str(k)]);

d i s p ( j a x1->xn f(x));

d i s p ([ f l i p u d (J) Alpha X F])

k=k+1;

60 i f( f l a g ==1)

d i s p (Unconstrained Optimization Failed);

break;


d i s p (Unconstrained Optimization Failed->Linear Search Failed);

65 break;

e l s e i f(k==kMax)

f l a g = 3;

d i s p (Powells Method Failed);

break;

70

en den d

I MATLAB Code for Golden Section Method.

f u n c t i o n [oa, f l a g ] = goldensection(lb,ub,f,x2,epsilon_f)

%Golden Section Method

29


31/38

t = 0.618; %Golden section ratio

5 tola = 0.01; %convergence tolerance on alpha value for Golden Section

ilimit = 1000;

f l a g = 0;

glb = lb;10 gub = ub;

i f(glb==gub)

oa = 0;

r e t u r n

en d

15 delf = 1000;

dela = 1000;

niterate = 1;

ga1 = glb+t*(gub-glb);

20 ga2 = glb+(1-t)*(gub-glb);

gf1 = f(x2(ga1));

gf2 = f(x2(ga2));

25 w h i l e ((delf>epsilon_f | dela>tola) & niterategf2 & ga1>ga2)

gub = ga1;

ga1 = ga2;

30 gf1 = gf2;

ga2 = glb+(1-t)*(gub-glb);

gf2 = f(x2(ga2));

oa = ga2;

delf = abs(gf1-gf2);

35 e l s e i f(gf1ga2)

glb = ga2;

ga2 = ga1;

gf2 = gf1;


40 gf1 = f(x2(ga1));

oa = ga1;


e l s e i f(gf1==gf2 & ga1>ga2)

glb = ga2;

45 gub = ga1;


ga2 = glb+(1-t)*(gub-glb);gf1 = f(x2(ga1));

gf2 = f(x2(ga2));

50 oa = (ga1+ga2)/2;


en d

dela = abs(ga1-ga2);

niterate = niterate+1;

55 % pause;

30


32/38

en d

i f(niterate==ilimit)

oa = 0;

d i s p (Linear Search Failed);

60 f l a g = 2;

en d

J MATLAB Code for Swanns Bounding Method.

f u n c t i o n [lb, ub, f l a g ] = swans(a,d,f,x2)

%Swanns Bounding

f l a g = 0;

lb = a;

5 ub = a;

n = 0 ;

a0 = a-d;

a1 = a;

a2 = a+d;

10 fa0 = f(x2(a0));

fa1 = f(x2(a1));

fa2 = f(x2(a2));

i f(fa2>fa1 & fa0=fa1 & fa0>=fa1)lb = min(a0,a2);

ub = max(a0,a2);

en d

25 w h i l e (fa2


33/38

break

en d

45 en d

K Objective Function Output for different inputs

Figure 7: Objective function values obtained for a sequence of subtracted images, (1)-(4) are shown

in Figure 8.

L The inputs used and output at each stage of the image pro-

cessing

32


34/38

(i) (ii) (iii) (iv)

(v) (vi) (vii) (viii)

(ix) (x) (xi) (xii)

(xiii) (xiv) (xv) (xvi)

(xvii) (xviii) (xix) (xx)

(xxi) (xxii) (xxiii) (xxiv)

Figure 8: Substracted image obtained for different inputs.(a)-(d) Original Image, (e)-(h) human

separated image, (i)-(l) human lower limbs, (m)-(p) VRML model images, (q)-(t) binary model

images, (u)-(x) subtracted images

33


35/38

M The detailed output obtained for images in KTH dataset

(i) (ii) (iii)

(iv) (v) (vi)

(vii) (viii) (ix)

(x) (xi) (xii)


images, the second and the third column show the output model images for the estimated pose.

34


36/38

(xiii) (xiv) (xv)

(xvi) (xvii) (xviii)

(xix) (xx) (xxi)

(xxii) (xxiii) (xxiv)

(xxv) (xxvi) (xxvii)


images, the second and the third column show the output model images for the estimated pose

(contd.).

35


37/38

(xxviii) (xxix) (xxx)

(xxxi) (xxxii) (xxxiii)

(xxxiv) (xxxv) (xxxvi)

(xxxvii) (xxxviii) (xxxix)

(xl) (xli) (xlii)



(contd.).

36


38/38

(xliii) (xliv) (xlv)

(xlvi) (xlvii) (xlviii)

(xlix) (l) (li)

(lii) (liii) (liv)

(lv) (lvi) (lvii)



(contd.).

priyanshu report

Documents