Forward and Inverse Optimal Control ofBipedal Running

Katja Mombaur, Anne-Helene Olivier, and Armel Cretual

Abstract This paper discusses forward and inverse optimal control problems forbipedal human-like running, with a focus on inverse optimal control. The (forward)optimal control problem looks for the optimal solution for a problem formulation,i.e. given objective function and given dynamic constraints. The inverse optimalcontrol problem is more challenging and consists in determining the objective func-tion and potentially unknown parts in the dynamic model that best reproduce a so-lution that is known from measurements. Periodic running motions are modeled ashybrid dynamic models with multiple phases and discontinuities, based on a three-dimensional multibody system model with 25 degrees of freedom. We investigate arecorded running motion on a treadmill at 10 km/h running speed and identify thebest possible objective function based on some hypotheses for potential contribu-tions to this objective function. For this, we apply a previously developed inverseoptimal control technique which uses a combination of a direct multiple shootingmethod and a derivative-free optimization technique, and we demonstrate here thatit also works for problems of the given complexity.

Katja MombaurInterdisciplinary Center for Scientific Computing (IWR), Heidelberg University, INF 368, 69120Heidelberg, Germany, Associate Researcher at LAAS-CNRS, Toulousee-mail: kmombaur@uni-hd.de, kmombaur@laas.fr,

Anne-Helene OlivierIRISA/INRIA de Rennes, Campus universitaire de Beaulieu, 35042 Rennes, Francee-mail: annehelene.olivier@gmail.com

Armel CretualUniversite de Rennes 2, Campus de la Harpe, 35044 Rennes, Francee-mail: armel.cretual@univ-rennes2.fr

1

2 Katja Mombaur, Anne-Helene Olivier, and Armel Cretual

1 Introduction

It is a common assumption that motions of humans and animals are optimal due toevolution, learning and training [1]. As a logical consequence, from a mathematicalperspective dynamic motion tasks such as walking and running can be formulated asoptimal control problems. In this paper we discuss two ways in which optimal con-trol can be used to explore human locomotion: (a) in the classical forward optimalcontrol sense to generate natural motions without any help by motion capture data;and (b) in the inverse optimal control sense, to identify objectives of human motionfrom experiments. We show in particular the application of these optimal control andinverse optimal control approaches to three-dimensional human running motions.

Fig. 1 Two different types of problems: The classical (forward) optimal control problem consistsin finding the optimal solution for a given objective function and given dynamic constraints. Theinverse optimal control problem consists in determining the objective function (and potentiallyunknown parts in the dynamic model) that best reproduce a solution that is (partly) known frommeasurements

Human and human-like running motions are investigated in various fields of re-search such as humanoid robotics, computer graphics, biomechanics, sports scienceor orthopedics. In robotics, achieving real running motions on a humanoid robot sys-tem is a benchmark that various companies and research labs have tried to achieve[2, 3]. In computer graphics, the goal is to generate realistic human-like runningmotions on an avatar (e.g. [4]). For both tasks, forward optimal control can be ef-ficiently applied to generate feasible and realistic running motions. We have donethis in previous research, aiming e.g. at the generation of natural looking runningmotions [5], or of the most stable running in the open-loop sense of an anthropo-morphic system [6].

Forward and Inverse Optimal Control of Bipedal Running 3

When looking at biological running systems, the inverse question becomes moreinteresting. In biomechanics, orthopedics or sports sciences the goal is to fully un-derstand mechanics and control of normal and pathological gaits in humans, and touse this knowledge to improve motions after an accident or illness in the orthope-dic case, or to increase performance to the highest possible level in sports. In thiscase the question for the optimization criterion underlying each of these situationsbecomes interesting.

In this paper, we investigate running motions on a treadmill at a slow pace (10km/h). We show how a previously developed method for inverse optimal control [7]can be applied to identify objective functions underlying this type of motions.

The paper is organized as follows: Section 2 describes the multibody systemmodel of three-dimensional human running used in this study. Section 3 outlines theformulation and solution of (forward) optimal control problems for the generationof optimal running motions. Section 4 discusses the formulation and solution of theinverse optimal control problem and gives results for a particular running example.Section 5 gives some conclusions and outlines future research directions.

2 Modeling human running motions

To describe the human body we use a multibody system model in three dimensionswith 12 bodies and 25 degrees of freedom (DOF) in flight - 6 global DOF associatedwith the positon and orientation of the pelvis and 19 internal DOF related to internaljoint angles. The system is equipped with torque actuators for each of the 19 internalDOF describing the action of the human muscles. For geometry and inertia param-eters, we use the anthropometric data given by de Leva [8]. The model describeshuman-like forefoot running, i.e. there is no flat foot ground contact but only point-like contact with the ball of foot. Detailed muscle dynamics are not included in themodel yet, but we plan to do so at a later stage of this research. For more details onthe current model, see the 3D model in [5]. With respect to the model described inthis paper, we have removed the spring-damper elements for the present study.

Running motions consist of a sequence of alternating flight phases and single-leg contact phases. The running motion considered in this study is assumed to beperiodic and symmetric, i.e. right and left steps are assumed to be identical. Eachphase of the motion (flight phase and single-leg contact phase) is described by itsown set of ordinary differential or differential-algebraic equations. Between phases,there may be discontinuities in the velocities, e.g. at touchdown of the foot on thefloor which are assumed to be fully inelastic. The running model therefore takes theform of a periodic hybrid dynamical system. The above assumptions allow us toreduce the model of the periodic running motion to the model of a single step with asubsequent leg shift and periodicity constraints (see figure 1). The total time of thestep T as well as the individual phase times are free variables of the model.

We describe the motion during flight phase by a set of ordinary differential equa-tions of the following form:

4 Katja Mombaur, Anne-Helene Olivier, and Armel Cretual

Fig. 2 The model of an (infinite) sequence of periodic running steps can be reduced to the modelof one step with leg shift and periodicity constraints

M(q, p)q+N(q, q, p)q = F(q, q, p,M ), (1)

with mass matrix M and the vector N combining all nonlinear effects. F is the vectorof all external forces (such as gravity, muscle torques M , drag etc. We have usedthe automatic model generator of HuMAnS by Wieber [9] to generate the terms Mand N of the 3D running model (see [5] for details).

During single leg contact phase, we keep the same number of coordinates eventhough the number of DOF is reduced by the fact that one ball of foot is fixed to theground by a constraint of the form g(q) = 0. This results in redundant coordinatesand a system of differential algebraic equations (DAE) for the equations of motion:

q = v (2)v = a (3)(

M GT

G 0

)(aλ

)=

(−N +F

γ

)(4)

gpos = g(q(t), p) = 0 (5)gvel = G(q(t), p) · q(t) = 0. (6)

with acceleration a := q and Lagrange multipliers λ . G denotes the Jacobianof the position constraints G = (∂g/∂q), and γ the corresponding Hessian γ =((∂G/∂q) q) q. Eqns. (5) and (6) describe the invariant manifolds on position andvelocity level that the solution must satsify. In the optimization, we take into accountthe unilateral nature of the ground contact constraint (i.e. the ground can not pull butonly push against the foot) by formulating an inequality constraint on the Lagrangemultiplier associated with the normal contact force. Phase changes between flightphase and contact phase are not explicitly time dependent but state dependent:

s(q(τs),v(τs), p) = 0. (7)

Touch-down occurs when a the foot gets down to the height of the ground, and lift-off takes place when the vertical contact force (represented by the negative of therespective Lagrange multiplier in eqn. (4)) becomes zero.

Forward and Inverse Optimal Control of Bipedal Running 5

The discontinuities of the velocities at touchdown (resulting from the fact thatthe velocity of the foot contact point is instantly set to zero at inelastic contact andthat this shock wave propagates through the whole body) can be computed as:(

M GT

G 0

)(v+Λ

)=

(Mv−

0

). (8)

v− and v+ are the velocities immediately before and after impact, respectively. Ma-trices M and N are the same as in eqn. (4) above.

Periodicity constraints are imposed in the model on all velocity variables v and areduced set of position variables qred eliminating the coordinate describing the for-ward running direction of the robot, after formulating the leg shift (compare fig.2).

3 Generation of natural human running by means of forwardoptimal control

In humanoid robotics or computer graphics one often faces the task to generatea natural motion for a system with given geometry and inertia characteristics. Toachieve this task without any information at all from motion capture or any otherhuman measurements, optimal control techniques can be efficiently used. Optimiza-tion has a clear advantage over approaches based on pure simulation. Simulationalways requires to fix important quantities in advance: if it is performed on a for-ward dynamics model the input forces and torques have to be pre-specified to obtainthe resulting motion, and if performed on an inverse dynamics model the positionand velocity histories have to be fixed to be able to calculate the required drivingtorques and forces. However, typically none of the quantities is exactly known a pri-ori. Optimization-based simulation allows to leave forces, torques and the motionfree and to determine them all simultaneously according to some desired optimiza-tion criterion.

In order to generate optimal running motions, the following optimal control prob-lem must be solved:

minx(·),u(·),T

∫ T

0φ(x(t),u(t)) dt + Φ(T,x(T )) (9)

s. t. x(t) = f j(t,x(t),u(t)) for t ∈ [τ j−1,τ j],

j = 1, ...,nph, τ0 = 0,τnph = T (10)

x(τ+j ) = x(τ−j ))+ J(τ−j ) for j = 1, ...,nph (11)g j(t,x(t),u(t))≥ 0 for t ∈ [τ j−1,τ j] (12)req(x(0), ..,x(T )) = 0 (13)rineq(x(0), ..,x(T ))≥ 0 (14)

6 Katja Mombaur, Anne-Helene Olivier, and Armel Cretual

In this formulation, x(t) denotes the vector of state variables, summarizing positionand velocity variables of the runner, and u(t) is the vector of control variables of thesystem, in this case the muscle torques Mi. τ is the vector of phase switching timeswith total step time T = τnph .

Eqn. (9) describes the objective function in a general form and is further dis-cussed below. Eqns. (10) and (11) are placeholders for the hybrid dynamic model ofthe running motion discussed in the previous section. In addition, there are contin-uous inequality constraints of form (12), including lower and upper bounds on allvariables, but also more complex relations between several variables, and coupledand decoupled pointwise equality (13) and inequality constraints (14), such as startand end point constraints, phase switching conditions or periodicity constraints.

Fig. 3 Qualitative comparison of optimized running motion and a photographed sequence by Muy-bridge in the late 19th century[15]

For the solution of these multi-phase optimal control problems we use the pow-erful optimal control code MUSCOD which has been developed at IWR Heidelberg[10, 11]. This code can be applied to mechanical DAEs of the above form, as de-scribed in [12]. MUSCOD uses a direct method (also called first-discretize-then-optimize approach) for control discretization, and the multiple shooting techniquefor state parameterization. The result of these two discretization steps for whichidentical grids are chosen is a nonlinear programming problem (NLP) of large di-mension which is solved by a specially tailored sequential quadratic programming(SQP) method.

Our experience has shown that objective functions φ j(t,x(t),u(t), p) minimiz-ing a sum of weighted torques squared, and possible additionally minimizing thevariation of torques, lead to very natural running motions [5], as a qualitative com-parison with human motion shows (see figure 3). Since the focus of this paper isthe inverse optimal control problem for running motions, we will not discuss thisforward optimal control problem further, and refer to [5] for details instead.

Forward and Inverse Optimal Control of Bipedal Running 7

4 Identification of human objectives during locomotion bymeans of inverse optimal control

This investigation goes beyond the qualitative reasoning of the previous section,since for a thorough biomechanical analysis one is interested in the true objectivefunction applied by the human. The specific optimization criterion for locomotion indifferent situations is generally unknown, and it can be expected that in most casesa combination of multiple criteria is used. The optimal behavior during a particu-lar locomotion task can be observed and measured by motion capture, force plateor EMG measurements etc. The inverse optimal control problem consists in deter-mining, from a solution that is (partly) known from measurements, the optimizationcriterion that has produced this solution.

4.1 Formulation and solution of inverse optimal control problem

To solve the inverse optimal control problem, we make the assumption that a setof reasonable independent base functions Ψi(t) for the objective function can beestablished. The relative contribution of these base functions Ψi(t) expressed by aweight factor αi remains to be determined by the algorithm. The inverse optimalcontrol problem can be formulated as:

minα

m

∑j=1||z∗(t j;α)− zM(t j)||2 (15)

where z∗(t;α) is determined by the solution of

minx,u,T

∫ T

0

[n

∑i=1

αiΨi(x(t),u(t))

]dt (16)

s. t. constraint eqns. (10) - (14)

Inverse optimal control problems are difficult since they require the solution of anoptimal control problem and a parameter identification problem at the same time.The vector z describes the set of variables which are used for the comparison ofmodel and measurement in the parameter identification problem which can be eithera subset of the state and control variables or other quantities that can be directlyor indirectly measured and determined from the computed solution. For a moredetailed discussion, see below.

We have developed a bilevel optimization technique (compare fig. 4) capable tosolve this type of problems based on two powerful existing techniques. The upperlevel in this method is responsible for the iterations over the unknown weight factorsα such that the fit between measurements and optimal control problem solution isimproved. In each upper level function evaluation a forward optimal control prob-lem has to be solved for the current set of αi. We apply a derivative-free optimiza-

8 Katja Mombaur, Anne-Helene Olivier, and Armel Cretual

Fig. 4 Solution of inverse optimal control problem as bilevel optimization problem

tion technique (i.e. it only requires function evaluations and no gradient informa-tion) which is always favorable if function evaluations are expensive and noisy andderivative information can therefore not be generated reliably. We use the derivative-free optimization code BOBYQA (Bound Optimization BY Quadratic Approxima-tion) by Powell [13] which is also capable to handle bounds on the weight parame-ters.

In the lower level, the task is to efficiently solve the forward optimal controlproblem which arrises in each iteration of the upper level. For this we apply thedirect boundary value problem approach MUSCOD that was already described inthe previous section.

This bilevel inverse optimal control method has already been applied to identifythe cost functions that produce locomotion paths in point to point motions in freespace [7] as well as in interaction scenarios. The purpose of this paper is to showthat it can equally be applied to solve the multi-phase problems based on complexmultibody system models that we are facing in the present study.

4.2 Objective functions of human running identified frommeasurements

An important prerequisite for any inverse optimal control computation is the estab-lishment of a basis of potential objective functions which then form the base func-tions Ψi(x(t),u(t)) in equation (16). These functions are context specific and shouldreflect the current expert guesses on the problem under investigation. It is obviousthat their choice is crucial, since any inverse optimal control problem solution canonly be as good as choice of base functions permits. We recall that the goal of theinverse optimal control task is to find an objective function that gives a meaning-ful explanation in terms of characteristic physical properties of the system, such asenergy, velocities, muscle activations etc. It is therefore not helpful to perform in-

Forward and Inverse Optimal Control of Bipedal Running 9

verse optimal control using a mathematical basis of functions such as Fourier seriessince the resulting weights of the Fourier terms would not give the desired insightsinto the system’s physical behavior. But for a choice of base functions formulated interms of physical quantities it must be guaranteed that they are independent - in thesense that there should not be different (combinations of) base objective functionproducing the same solution (a simple example for this would be a minimizationof time or a minimization of distance for a motion at constant speed, which wouldhave exactly the same solution). In such a redundant situation it would obviouslybe impossible for inverse optimal control to make a selection between any linearcombination of these two functions.

For the investigation of running motions, we have to distinguish two differentsituations (and we claim that there is a policy change between these two):

1. Running at full speed (sprinting) reaching the physical limits:In this case, we formulate the following hypotheses for potential contributions tothe objective function

• maximization of running speed• maximization of frequency /minimization of step time• minimization of duty factor (relative contact time, see [14])• ...

2. Running at controlled slow to medium speed (jogging) at which runners typicallyare capable to set their speed to the desired value and then adjust their runningstyle. For this mode, we formulate the following hypotheses:

• minimization of joint torques (squared)• minimization of joint accelerations (squared)• minimization of head motions (e.g. velocities squared)• maximization of stride length• ...

In this paper, we focus on the second case, i.e. running at controlled low tomedium jogging speed, and we investigate a particular experiment with runningspeed 10km/h = 2.78m/s. This average speed is imposed as a constraint in the opti-mal control problem. We do not investigate the objective functions underlying run-ning at maximum speed, since this might require detailed muscle models in order tomore precisely describe the existing physical limits.

Another issue to be solved before starting the computations is the choice of quan-tities z that are used to determine the match between computational model with 25DOF and measurements of the 43 marker positions see figure 5 in the least squaressense in eqn. (15). It is still an ongoing discussion in biomechanics and also in hu-manoid robotics on which level a fit between human measurements on the one sideand the mathematical model or the humanoid robot on the other side should best beachieved. There are different possibilities, e.g.

10 Katja Mombaur, Anne-Helene Olivier, and Armel Cretual

Fig. 5 Recorded marker positions are used to reconstruct movements of joint centers and othercharacteristic points: the deviation between corresponding points in the model and the experimentsis minimized by inverse optimal control

• matching joint angles and optionally also position and orientation of the cen-tral body (need to be derived from measured marker positions, but are directlyavailable as state variables in the model);

• matching marker positions (are directly available from measurements, but haveto be computed as functions of the state variables in the model);

• matching positions of characteristic points such as joint centers or dominantbones (typically require computations on the measurement side based on mea-sured markers as as well on the model side based on state variables).

Also combinations of the choices above are of course possible and might be reason-able. We have for the time being chosen the last option and fit measurements andmodel at the following points in the 3D runner: right and left hip points, knees, an-kles, balls of the feet, shoulders, elbows and wrists as well as head center, xyphoid,suprasternale and root point in the pelvis. Using 3D position information for these18 points, this leads to a matching vector z of dimension 54 in eqn. (15).

Based on the reasoning above, we have formulated the following parameterizedobjective function which combines base functions for the minimization of all 19joint torques squared, a minimization of the head motion (which has to be expressedin terms of the angular velocities of the trunk since in our model the head is rigidlyconnected to the trunk) and a maximization of stride length (i.e. a minimization of

Forward and Inverse Optimal Control of Bipedal Running 11

the negative stride length):

min

{∫ T

0(

19

∑i=1

α jM2i )dt +

∫ T

0(β1ω

2x,tr +β2ω

2y,tr)dt− γ1∆ypelvis

}(17)

We have not added the terms related to the accelerations of the limbs since wefeel that there may be some redundancy with the torque criterion. We assume herethat the weight factors respect body symmetries, i.e. are identical for correspondingterms in the left and right body half. This means that the 19 weight factors for thetorque terms can be reduced to 10 unknown parameters. Overall, we then have 13weight factors to be determined by inverse optimal control.

Our computations have resulted in the following best set of weight factors for theobjective function proposed above:

Table 1 Weight factors of objective function contributions determined by inverse optimal control

αhip αknee αankle αtrunk αshoulder αelbow β γ

0.56 3.7 4.0 9.1 9.1 5.8 435 33710.7 3.0 3000.0 0.0

Figures 6 and 7 show the resulting 19 torque and 25 position histories of thesolution with the above objective function. Velocity variables are not shown forreasons of space. This solution represents the best possible match between the mea-surements and the computations with the chosen model and the selected basis ofobjective function. The fit is quite good but there are still some differences whichcan only be further reduced if some additional adjustment in the model and theproblem formulation are made:

• The model of the three-dimensional runner (section 2) needs to be refined andseveral DOF have to be added since it can be shown that even with a a free choiceof torque inputs (solving a pure fitting problem and leaving the inverse optimalcontrol question apart) no perfect fit can be achieved. In particular the upperbody and the head need additional DOFs. Kinematic and dynamic data used forthis study was taken from anthropometric tables and scaled to the subject’s sizeand weight. However, for a more detailed fit, the data should ideally be furtherpersonalized by taking into account the subject’s individual proportions . Alsothe foot contact modeling with a point contact of the ball which is more realisticfor very fast running might be revised for this slower running speed.

• Additional terms might be required in the objective function (17). These canbe new functions expressed in variables of the current model such as positions,velocities, accelerations, torques etc. Other examples are terms related to muscleactivity or fatigue or to the effect of passive tissue. These terms would require anadditional enhancement of the runner model by adequate muscle models.

12 Katja Mombaur, Anne-Helene Olivier, and Armel Cretual

Fig. 6 Torques of the computational solution that best reproduces the measurements

5 Conclusion & Perspectives

In this paper, we have discussed the use of optimal control for the investigation ofthree-dimensional human running motions. We have summarized previous resultsfor the generation of natural running motions by forward optimal control. The focusof this paper was on inverse optimal control, and we have shown first results for arunning motion on a treadmill at 10 km/h. The best possible combination of weightfactors for an objective function of terms related to torque minimization, head mo-tion minimization and stride maximization was determined. The inverse optimalcontrol computations have been preformed using a previously developed inverseoptimal control technique which uses a combination of a direct multiple shootingmethod and a derivative-free optimization technique. We have shown here for the

Forward and Inverse Optimal Control of Bipedal Running 13

Fig. 7 25 position variables of the solution of inverse optimal control

first time, that it can also be applied to multi-phase problems with discontinuitiesand complex multi-body system models.

While these first results are already quite promising, there still remains somework to do until we have achived a reliable explanation of the true objectives un-derlying human running. As discussed above, we are currently working on somerefinements of the model in terms of additional DOF, personalized parameters, andpossibly also the addition of muscle models. In addition, other base terms will beadded in the objective functions based on biomechanical literature. Later, we willalso validate the objective function by performing inverse optimal control compu-tations at different speeds (experiments for 12, 14, 16, and 18 km/h have alreadybeen performed but not yet been analyzed) and by extending the study from one toseveral subjects. With this, it would be possible to answer e.g. if objective functionsare individual or can be generalized, and if and at which speeds there is the policychange between running at controlled speed and running at high speed discussed insection 4.

Acknowledgements This work was performed when Katja Mombaur was working at LAAS-CNRS Toulouse and the three authors were cooperating within the Locanthrope project financedby the ANR (French National Research Agency). We thank M.J.D. Powell, Cambridge, for mak-ing available his optimization code BOBYQA and the Simulation and Optimization group ofH. G. Bock at the University of Heidelberg for providing the optimal control code MUSCOD.

14 Katja Mombaur, Anne-Helene Olivier, and Armel Cretual

References

[1] R. McN. Alexander: “Optima for Animals”, Princeton University Press, 1196[2] Toyota partner robot website: http://www.toyota.co.jp/en/special/robot/, run-

ning video on youtube: http://www.youtube.com/watch?v=yBUcfFbndVY[3] Honda Asimo Website: http://asimo.honda.com/[4] J. K. Hodgins, “Simulation of human running,” in Proceedings of IEEE Inter-

national Conference on Robotics and Automation, vol. 2, May 1994, pp. 1320– 1325.

[5] G. Schultz and K. Mombaur: “Modeling and Optimal Control of Human-like Running”, IEEE/ASME Transactions on Mechatronics, 15 (5), 783–792(2010).

[6] K. Mombaur, Using optimization to create self-stable human-like running,Robotica, Vol. 27, 2009, p. 321-330, published online June 2008

[7] K. Mombaur, A. Truong and J.P.Laumond: From human to humanoid loco-motion - an inverse optimal control approach. Autonomous Robots 28 (3),369–383 (2010)

[8] P. de Leva: Adjustments to Zatsiorsky-Seluyanov’s segment inertia parame-ters. Journal of Biomechanics 29 (9), 1223–1230 (1996)

[9] P. B. Wieber.: Humans toolbox, http://www.inrialpes.fr/bipop/software/humans/,July 2007.

[10] D. B. Leineweber, I. Bauer, H. G. Bock, J. P. Schloder: “An efficient multipleshooting based reduced SQP strategy for large-scale dynamic process opti-mization”, Comput. Chem. Engng, 27, 2003.

[11] H.G. Bock, and K.J. Plitt: A multiple shooting algorithm for direct solutionof optimal control problems. Proceedings of the 9th IFAC World CongressBudapest, 243–247 (1984).

[12] K. D. Mombaur, H. G. Bock, J. P. Schloder and R. W. Longman Open-loopstable solution of periodic optimal control problems, ZAMM (Journal of Ap-plied Mathematics and Mechanics), Vol. 85, No. 7, July 2005

[13] M.J.D. Powell: ”The BOBYQA algorithm for bound constrained optimizationwithout derivatives”, Report No. DAMTP 2009/NA06, Centre for Mathemati-cal Sciences, University of Cambridge, UK.

[14] A.E. Minetti, R. McN. Alexander, “A Theory of Metabolic Costs for BipedalGaits”. J Theor. Biol, 186 (4), 467-476, 1997. process optimization - part I:theoretical aspects. Comput. Chem. Engng. 27, 157–166 (2003)

[15] Muybridge, E.: The Human Figure in Motion. Dover Press (1957)