Temporally Coherent Full 3D Mesh Human Pose Recovery from

Monocular Video

Jian Liu, Naveed Akhtar and Ajmal Mian

Abstract

Advances in Deep Learning have recently made it possibleto recover full 3D meshes of human poses from individualimages. However, extension of this notion to videos forrecovering temporally coherent poses still remains unex-plored. A major challenge in this regard is the lack ofappropriately annotated video data for learning the de-sired deep models. Existing human pose datasets onlyprovide 2D or 3D skeleton joint annotations, whereas thedatasets are also recorded in constrained environments.We first contribute a technique to synthesize monocularaction videos with rich 3D annotations that are suitablefor learning computational models for full mesh 3D hu-man pose recovery. Compared to the existing methodswhich simply “texture-map” clothes onto the 3D humanpose models, our approach incorporates Physics based re-alistic cloth deformations with the human body move-ments. The generated videos cover a large variety ofhuman actions, poses, and visual appearances, whereasthe annotations record accurate human pose dynamicsand human body surface information. Our second majorcontribution is an end-to-end trainable Recurrent Neu-ral Network for full pose mesh recovery from monocularvideo. Using the proposed video data and LSTM basedrecurrent structure, our network explicitly learns to modelthe temporal coherence in videos and imposes geometricconsistency over the recovered meshes. We establish theeffectiveness of the proposed model with quantitative andqualitative analysis using the proposed and benchmarkdatasets.Keywords: Human Pose Recovery, 3D Human Recon-struction, Full Mesh Recovery, Data Synthesis.

1 Introduction

Recovering human poses from monocular (as op-posed to multiview) images is an efficient approach sinceit does not require cumbersome calibration or high costequipment. It has many applications in pose transfer,human movement analysis and action recognition. Un-til recently, the techniques used for human pose recoveryaimed at predicting skeletal joint configurations from im-ages [1–5]. However, recent findings [6, 7] ascertain that,using deep learning, it is possible to reconstruct full 3Dhuman meshes from monocular images with the help ofparameterized body and shape configurations [8]. Full3D mesh recovery has clear advantages over the sparseskeleton recovery of human poses, as the former capturesthe inner pose dynamics as well as the outer 3D human

Figure 1: (Top) Illustration of realism in our data due toPhysics based cloth movement. (Bottom) Compared toconventional frame-based 3D pose recovery [6], the pro-posed method better models the temporal variation andenforces geometric consistency of poses. The highlightedframes show frames where the difference between the twotechniques is significant.

bodies. These advantages multiply when, instead of in-dividual images, human meshes can be recovered for fullvideos while incorporating the temporal dynamics of thebody movement.

Video based full mesh recovery of human poses has fur-ther applications in precision modeling of human actions,virtual try-on, automatic animation, human-computer in-teraction and so on. However, progress in this research di-rection is currently hampered by the unavailability of ap-propriately annotated data for learning the desired com-putational models. Due to the constraints of sensor spec-ifications, data modality and data sample size; existingdatasets for human pose recovery, e.g. [2, 9, 10] are notparticularly helpful in recovering full mesh poses fromvideos. On the other side, the requirement of panopticstudios [11] or full body scanners [12,13] to annotate datafor this task restrict researchers in generating specializeddata for their specific problems in this area.

This work first addresses the problem of generatingappropriately annotated training data to learn compu-tational models that can recover full 3D meshes of hu-man poses from monocular videos. To that end, we in-

1

arX

iv:1

906.

0016

1v2

[cs

.CV

] 1

Jul

201

9

troduce a video generation technique that provides rich3D annotations for realistic monocular videos. The gen-erated videos are able to easily incorporate a wide varietyof human actions and other visual variations related toe.g. cloth-body and cloth-gravity interactions, cloth tex-tures, lighting conditions, camera viewpoints, and scenebackgrounds. The annotations recorded for these videosinclude parameters of 3D human avatar (shape), 3D skele-ton (pose) and its 2D image projection, and even thevertices of full human meshes (pose) in the videos. Toaccurately capture the movements of clothes and theirinteraction with human bodies, we employ a Physics en-gine that endows the generated videos with fine-grainedrealistic effects. To the best of our knowledge, this is thefirst of its kind ability of a data generation technique inthe broader research direction of video/image based hu-man action analysis. The videos/frames resulting fromour technique are much more realistic as compared tothose generated by the approaches that simulate clothesas textures on human models, see Fig. 1-top. The sourcecode of our technique and the resulting data will be madepublic for the broader research community.

The second major contribution of this work is an end-to-end trainable Recurrent Neural Network that recoversfull 3D human pose meshes from monocular video. Theproposed network embeds a foundational building blockthat processes an individual video frame in a recurrentstructure. We incorporate attention mechanism in therecurrent structure that provides contextual conditioningbased on low-level visual features. The network preciselymodels the spatio-temporal dynamics of human move-ments in videos and imposes geometric consistency overthe recovered mesh sequences by incorporating a bodyshape smoothness loss in the training process.

We analyze our technique using the proposed dataand the benchmark Human3.6M [10] and UCF101 [14]datasets. Our experiments demonstrate that the pro-posed method is able to achieve very promising fullmesh recovery for pose estimation from monocular video,e.g. Fig. 1-bottom. Due to the nascency of this researchdirection, this article also makes a minor contribution inintroducing three new evaluation metrics to more appro-priately analyze the mesh pose recovery from videos ascompared to joint based pose recovery metrics used forindividual video frames.

2 Related Work

Traditional techniques of human pose recovery usually ap-ply pictorial structure models [15–19] to optimize bodypart configurations. However, due to the rapid advance-ments in deep learning, recovering articulated humanposes using Convolutional Neural Network (CNN) mod-els has become increasingly popular. For instance, Weiet al. [1] proposed Convolutional Pose Machines (CPM)to estimate 2D pose keypoints by learning a multi-stageCNN. The authors used a sequential convolutional struc-ture to capitalize on the spatial context and iterativelyupdated the belief maps. In their method, the recep-tive field of neurons is carefully designed at each stage

to allow the learning of complex and long-range correla-tions between the body parts. Similarly, Newell et al. [20]proposed a Stacked Hourglass method to process visualfeatures across different scales. Their results are consol-idated to better capture various spatial relationships as-sociated with human body. Nevertheless, the CPM andStacked Hourglass only work for static frame/images anddo not account for any geometric consistency between dif-ferent video frames.

To estimate 2D human poses in video, Pfister et al. [21]exploited the temporal context in video by combining in-formation across multiple frames with optical flow. Theyused the resulting information to align heatmap predic-tions from the neighbouring frames. As a more recent at-tempt in modeling temporal information for human poserecovery, Luo et al. [5] re-modelled CPM as a RecurrentNeural Network to replace the multiple stages of CPMwith sequential LSTM cells. The concept of using hand-crafted optical flow or recurrent structure to model tem-poral information in pose recovery task is beneficial. Nev-ertheless, both of these methods remain limited to recov-ering 2D keypoints only.

It is challenging to extend 2D keypoints recovery meth-ods to recover 3D skeletons, as the latter demands so-phisticated solutions. For example, Camillo [22] had toenforce additional constraints on the relative lengths ofhuman limbs and the body joint kinematics to select validlimb configuration. Ramakrishna et al. [23] proposed anactivity-independent technique to recover 3D joint con-figurations using 2D locations of anatomical landmarks.They also leveraged a large motion capture corpus as aproxy to infer the plausible 3D configurations. A fewcontributions in this direction have also formulated 3Dskeleton recovery as a supervised learning problem. Forinstance, Pavlakos et al. [3] proposed a volumetric tech-nique to estimate 3D human poses from a single image.Their volumetric representation converts the 3D coordi-nate regression problem to a more manageable predictiontask in a discretized space. With this representation, theyconcatenated multiple fully convolutional network com-ponents to implement a iterative coarse-to-fine learningprocess.

Mehta et al. [24] enhanced CNN supervision with in-termediate heat-maps, and used transfer learning fromin-the-wild 2D pose data to improve the generalization toin-the-wild images for 3D pose recovery task. As a typ-ical per-frame pose estimation technique, their methodexhibits temporal jitters in video sequences. Anothertechnique, VNect [2] formulates the 3D skeleton recoveryproblem as a CNN pose regression task that follows anoptimization process, termed kinematic skeleton fitting.It is specifically designed to improve temporal stability ofthe recovered poses. Wang et al. [4] proposed a two-steptechnique for 3D pose estimation, named DRPose3D. Inthe first step, it uses a Pairwise Ranking CNN to extractdepth rankings of human joints from images. In the sec-ond step, it uses this information to regress the 3D poses.This method depends on a 2D pose estimator for com-puting the initial joint heat maps. Consequently, it cannot be treated as an end-to-end trainable technique.

2

Due to its attractive applications, full 3D mesh poserecovery from images is an emerging direction. Many re-cent methods adopt the parametric human model [8] inthis regard. For instance, Alldieck et al. [25–27] inferred3D shape of person with details including hair and clothusing parametric model. Yu et al. [28,29] also used depthsensor to reconstruct 3D body shapes in cloth. They usedcloth simulation for single view human performance cap-ture. Kanazawa et al. [6] adopted the parametric humanmodel for 3D mesh pose recovery and predicted its poseand shape parameters from monocular images in an end-to-end manner. Such a learning-based method requirestraining data with 3D annotations [30], a requirement ful-filled by very few datasets [10, 31]. The lack of trainingdata has also driven research to exploit generative adver-sarial networks for 3D pose learning. Kanazawa et al. [6]used unpaired 3D annotations to create a factorized ad-versarial prior. Similarly, Yang et al. [7] learned a dis-criminator to enforce the 3D pose estimator to generatethe plausible poses. However, a major limitation of suchmethods is that their complexity increases significantlywhen moving from frame to video modeling.

The challenges in manual annotation of large-scale datahas also led researchers to synthesize annotated data. Forinstance, Lassner et al. [32] proposed a generative modelto create people and manipulate their clothes. However,their method is frame-based which compromises the dy-namic details of the clothes. Varol et al. [33] proposedSURREAL to synthesize human images for the tasksof body segmentation and depth estimation. AlthoughSURREAL is able to generate RGB human images and3D joints annotations it has multiple shortcomings. Forinstance, the variations of human actions are limited inSURREAL. Moreover, the images in the dataset remainunrealistic in terms of interaction between human modelsand their clothes. This happens because the data gen-eration method simply wraps 2D cloth textures onto hu-man models. Our literature survey reveals that end-to-end learning for 3D pose mesh recovery is a promisingdirection but remains largely unexplored due to the un-availability of large-scale data with rich 3D annotations.Moreover, due to single frame based benchmark datasets,existing methods are also limited to process individualframes. We address these issues by proposing a data gen-eration method and a temporally coherent technique forfull 3D mesh human pose recovery from videos.

3 Data Generation

As the first major contribution of this work, below weintroduce our method of computationally generating hu-man action videos with rich 3D annotations for training.

3.1 Human Pose and Shape Model

To represent human avatars, we use the Skinned Multi-Person Linear (SMPL) model [8] as parametric represen-tation of human avatars in our data generation pipeline.SMPL provides a skinned vertex-based representationthat can encode a wide variety of human body shapes in

natural poses. The fact that SMPL model is created sta-tistically using a large number of real humans also makesit suitable for the pose recovery task. An avatar in SMPLrepresentation is given as a tupleM(β,θ), where β and θrespectively encode the body shape and pose. The bodyshape is parameterized by the first 10 coefficients of shapePCA space, hence β ∈ R10. For θ, selected bones in hu-man skeleton are represented in a hierarchical tree, whereeach bone/node is connected to its parent, and the wholeskeleton is anchored to a root node. At each body joint j,an axis-angle rotation vector ρj ∈ R3 controls the rota-tion of a child bone relative to its parent bone. The ori-entation of whole body is controlled by the root rotationvector ρ0 ∈ R3. All of the above bone kinematics infor-mation is summarized by the pose parameter θ ∈ R3K ,where K = 23 for the 23 body joints chosen by the SMPLrepresentation.

In terms of SMPL representation, an avatar’s pose Pi ∈R3×N is a mapping of a tuple Mi(βi,θi) to N = 6890vertices describing the surface of an avatar. It is also pos-sible to extract 2D keypoints and 3D skeletons using thetechnique of [8]. Our method uses the SMPL representa-tion to record an avatar’s pose that is later rendered to avideo frame using the graphics pipe-line discussed below.

3.2 Model Pose Variations

We exploit the CMU MoCap database (http://mocap.cs.cmu.edu) to bequeath the SMPL avatars with a largevariety of natural poses and motions recorded using realhumans. The CMU MoCap dataset covers more than2500 different action sequences that capture the dynamicsof 3D skeleton joints. We use the MoSh technique [34] tomap CMU joint locations to SMPL model parameter -resulting in human avatars. For a given CMU MoCapsequence, MoSh estimates the SMPL parameter θ thatbest explains the body joint rotations corresponding tothe CMU skeleton data. Multiple SMPL β parameterscan then be chosen that animate the same action underdifferent body shapes. We call the thus generated SMPLsequences as “MoShed” sequences.

When using the MoShed CMU sequences, the pose vari-ations are upper bounded by the total CMU pose types.To enhance the pose variations, we employ a pose inter-polation technique to create new MoShed sequences thatconsist of novel poses. While interpolating between twoposes, we choose widely contrasting poses as the startingand ending pose. See the real poses A and B in Fig. 2as representative examples. This choice results in the in-terpolated poses that are significantly different from theavailable real pose sequences in the CMU dataset, im-proving the pose variety in our dataset.

The selection of contrasting real poses for the interpo-lation is made as follows. Consider two MoShed CMUMoCap sequences X = Ximi=1 and Y = Yjnj=1 thatrespectively contain m and n frames. Each frame inthese sequences contains an independent human posethat is represented by SMPL pose parameter θ. Wedefine the distance between a pair of human poses asdisti,j = ||Xi(θ) − Yj(θ)||2, and compute a distancematrix D ∈ Rm×n for all the pose pairs for action se-

3

Figure 2: Illustration of pose interpolation: The first andlast pose in each row are original poses, denoted as Aand B. All the remaining poses in a row are generated byinterpolating between their real pose A and B.

quences X and Y using this distance. We select thetwo poses for interpolation by getting the frame indicesi, j = argmax(D), and record disti,j = max(D). Thesepairs are used as the starting and ending frames for thecreation of new pose sequence through interpolation.

As mentioned in Section 3.1, θ ∈ R3K represents axis-angle rotation for each body joint relative to its parentbone. Compared to Quaternion rotation, Axis-angle ro-tation normalizes the rotation axis and multiplies it withthe rotation magnitude. As θ represents relative rota-tion and it works independently on each body joint, itis convenient to perform linear pose interpolation usingthe θ parameters of the two poses. The number of in-terpolated frames is decided by the distance between thetwo frames. It can be observed in Fig. 2 that under thisstrategy, the transition between the two original poses re-mains smooth, whereas the interpolated poses appear asa person performing atypical actions. All the original andinterpolated action sequences are used to render RGB ac-tion videos in our data generation scheme.

3.3 RGB Actions with Realistic Clothes

The avatars resulting from SMPL representation can onlyrepresent humans with minimal or tightly-fitted clothes.Consequently, previous attempts of using this representa-tion to generate data in the broader domain of human ac-

Figure 3: First row: Types of garments designed inthis work to apply to SMPL models. Second and thirdrow: Physics based application of garments to sequenceof poses performing dance moves.

tions, e.g. [35] extracted (unwrapped) texture maps fromhuman scans, and fitted those onto the avatars. On onehand, the texture variations become limited under thisstrategy. On the other, this scheme does not correctlyimitate clothes in the real-world. For instance, simpletexture on an avatar does not behave in a ‘cloth-like’ man-ner, even for slightly loose clothes. Moreover, it does notcause any occlusion to the body shape, which is oftenthere for the loose clothes. Not to mention, using onlythe texture also deprives the generated data of importanttemporal cues that cloth dynamics provide in the real-world actions.

We address this problem by modeling a number of gar-ments for the SMPL avatars, and using those for datageneration. In this work, the garment simulation is per-formed with a soft-body Physics engine. Such enginesare used by fashion designing software, e.g. Marvelous-Designer7 (MD7) to achieve realistic effects. We adaptthe engine from MD7 that supports cloth pattern cut-ting, sewing and its physical simulation. In the first rowof Fig. 3, we illustrate different types of garment designedin this work. The second and third rows of the figure il-lustrate the results of applying two of these garments to aSMPL model under different poses of the avatar with thePhysics engine based simulation. The shown six poses aresampled from a sequence (left to right) of a female avatarperforming dance moves. Notice the realistic effects interms of wrinkles, draping and the overall cloth move-ment in our data. We show a single texture design for allthe avatars for better visualization. It is apparent that,applying static textures on avatars simply can not providethe fine details and the realistic interactions between clothand human body that is provided by our technique. Wealso provide a qualitative comparison of the color framesconstructed with our data generation technique with apopular existing method that uses texture-based clothing

4

Figure 4: Comparison of an action sequence rendered byour method and SURREAL [35]. Our method enables finecloth details and realistic cloth-body interactions, whileSURREAL fits unwrapped textures onto the avatars, re-sulting in a less realistic “body-painting” effect. See thetop of female affixed to legs in the bottom row.

in Fig. 4.We simulate cloth movements with the help of time-

varying partial differential equations, which are solvedas ordinary differential equations employing discritiza-tion [36]. In our case, a cloth is modelled as a set ofparticles [mi, xi] with interconnecting springs, where mi

and xi are mass and geometric state vector of the i-thparticle. The dynamics of the cloth is governed by theNewton‘s Equation:

mixi = Fint + Fext, (1)

where Fint and Fext respectively denote the collective in-ternal and external forces acting on the cloth, and xi isthe second derivative of xi with respect to the time. Iden-tifying the internal forces in cloth deformation is a hardproblem. However, it can be reduced to the problem ofdifferentiating the potential energy E of the cloth parti-cles spatially. This simplification leads to the followingrelationship:

− ∂E

∂x= Fint. (2)

The equation governing internal forces on a particle xihence becomes,

mixi = −[∂E

∂x]x=xi

. (3)

Putting the cloth particles into vectors, and accountingfor the overall external forces, we can re-write our equa-tion as:

Mx = −∂E∂x

+ Fext, (4)

where M is a diagonal matrix formed with mi, represent-ing mass distribution of the cloth, and Fext models theextra external forces acting on the cloth as air-drag, con-tact and constraint forces, internal damping, etc. Theoverall external force on a cloth is also a function of xas well as x. Taking the time into account, the aboveequation can be written as

x(t) = M−1(−∂E∂x

+ F )(t). (5)

Our Physics engine solves the above equation numeri-cally [37] to get the time derivatives of the cloth particles.Using those with the particle locations results in realis-tic cloth deformations with its movements. The enginecomputes the Energy E while accounting for the defor-mations of ‘stretch’, ‘shear’ and ‘bending’ for the realisticeffects. Employing a Physics engine to simulate realisticcloth movements in 3D human videos is a unique contri-bution of this work. Our implementation and resultingdata will be made public for the broader research com-munity.

3.4 Scene Variations in Videos

We generate action videos that are rich in scene varia-tions. We apply different backgrounds, cloth textures,illuminations, and viewpoints to render SMPL avatars toactual RGB frames. Below, we describe these variationsand our method to incorporate them in more detail.

Background and cloth texture variations: We useover 400 spherical images for the environmental back-grounds in our videos. These images are collected fromonline resources, such as Google Images. Our video gen-eration pipeline additionally employs conventional 2Dimages for background during the rendering process.Towards that end, we exploit the Places365-Standarddataset [38] and use its test split to generate differentbackgrounds. We set the scales and rotations of the back-ground scenes at random during the rendering process.To vary the cloth textures, we use the DTD [39] and Fab-rics [40] datasets and randomly choose a texture for eachvideo.

Lighting variations: We setup four surface lightspointing towards the SMPL avatars and randomly changetheir strengths during rendering to simulate illuminationvariations encountered in the real-world scenes.

Viewpoint variations: We render the moving SMPLavatar from four camera viewpoints setup in the East,West, South, and North of the avatar. To keep the avatarsin the frame center, we set the cameras to track the Pelvisjoint of the SMPL model. The camera sensor size and thefocal length are set to 32mm and 180mm respectively,and the output resolution is fixed to 250 × 250. Sincewe use a telephoto camera, we carefully scale the back-ground during the image rendering to better blend therendered human models and their backgrounds. Differ-ent hyper-parameters of the setup discussed above areoptimized empirically to achieve the best realistic visualappearances in the resulting videos.

5

Figure 5: Approach schematics: (A) Basic building block to process individual frames. A frame I is encoded anda Regressor is trained using Iterative Error Feedback (IEF). Rich 3D annotations provide ground truth for SMPLparameters β,θ and 2D, 3D skeletons. (B) The block is used in a recurrent setup with as many LSTM cellsas the clip length that share Encoder and Regressor weights. An additional collective loss is defined over βt forgeometric consistency across the frames. The LSTM cells implicitly model temporal dynamics of a video. (C) Showsthe attention mechanism that is incorporated to encode contextual information from low-level visual features tocondition refreshing of the recurrent states. (D) Legend for the figure.

3.5 Synthesizing the Videos

Our data generation pipeline has three main steps toget from the MoShed CMU action sequences to real-istic videos. In the first step, we import the SMPLavatar and its respective MoShed CMU sequences intoBlender - an open source 3D rendering software (https://www.blender.org/) - and 3D render them. We achieve3D avatar animation in this step with bodies that do nothave clothes. In the second step, we apply clothes andperform Physics based simulation of the clothed avatarsand record their mesh information. The physical prop-erties of the clothes are preset to emulate realistic cloth-body and cloth-gravity interactions in this phase. In thethird step, we again use the Blender with clothed meshesand synthesize videos by varying different scene attributesas discussed in §3.4.

Our cloth simulation requires human models in anima-tion to start from a standard “T-shape” or “A-shape”pose, which are not guaranteed for many actions in CMUMoCap. To address this issue, we interpolate extra Nframes prior to the target MoCap action sequence, to en-sure a smooth transition from a standard “T-shape”poseto the real poses of interest. In this work, we rendervideos for more than 5, 000 sequences using four differ-ent viewpoints. In total, we generate more than 3 millionvideo frames in our dataset that can find applications intraining deep models for various tasks, e.g. human actionrecognition, pose estimation.

4 Full Mesh Pose Recovery

The proposed data generation technique enables effectivefull mesh human pose sequence recovery directly from avideo. To achieve this, we design a recovery mechanismfor individual video frames and apply it in a recurrentstructure for videos, as illustrated in Fig. 5.

4.1 Mesh Recovery from a Frame

Our data generation technique provides realistic videoframes for which we also have rich 3D annotations. Weuse this data to learn a neural network model that canpredict a 3D body mesh with accurate body shape andpose in individual frames. Concretely, this model predictsa vector ϕ = [θ,β,R, t, s], where θ and β are SMPL pa-rameters (see §3.1), R ∈ R3×3 describes the global rota-tion, t ∈ R2 records the translation within the frame ands ∈ R is the scale of the 3D mesh. As shown in Fig. 5(A),there are two main components of the model, an ‘En-coder’ and a ‘Regressor’. We implement the Encoder asa CNN and the Regressor as an MLP. We defer the ex-act implementation details of these networks to §4.3 forcontinuity.

For an input video frame I, the Encoder computes afeature vector E(I) ∈ RD, which is regressed by the Re-gressor to estimate ϕ = R(E(I)). To keep our modelcompact, we adopt Iterative Error Feedback (IEF) totrain the Regressor. In IEF, the ith iteration computes

6

ϕi+1 = ϕi + ∆ϕi, where ∆ϕ is the increment. This pro-cedure is initialized with the ϕ resulting from the meanpose in our dataset. The predicted vector ϕ can be usedto generate a 3D pose mesh P ∈ R3×N under the SMPLrepresentation. Moreover, this mesh can be regressed toachieve 3D skeleton [8] that we denote by 3DJ ∈ R3×Q,where Q is the number of body joints. We further projectthe 3D joints to 2D key points as follows

2DJ = sΨ(R 3DJ) + t, (6)

where Ψ(.) is the orthographic projection operator. Weuse the 2D skeleton joints to define a ‘projection loss’

Lproj =∑i

||χi(2DJi −2D Ji)||1, (7)

where χ(.) is a masking operator that turns those joints tozero that are not visible in the ground truth 2D skeleton,and ||.||1 computes the `1-norm. On similar lines, we alsodefine a 3D joint loss as follows

L3Djoint =∑i

||3DJi −3D Ji||22. (8)

Moreover, we also define a SMPL parameter loss as

Lsmpl =∑i

||[βi,θi]− [βi, θi]||22. (9)

Finally, the overall loss for our network is defined as acombination of the above described losses, given as

L = Lproj + δ(L3Djoint + Lsmpl). (10)

In this formulation, the parameter δ is set to 1 when theground truth 3D annotation is available and 0 otherwise.This parameter is useful when a training batch includessamples with only 2D keypoints annotations. Humanpose recovery by Kanazawa et al. [6] can be related toour technique for frame based mesh recovery. However,our full mesh recovery method goes beyond individualframes to videos, as described below.

4.2 Mesh Recovery from a Video

Our data allows full mesh based supervised learning ofdeep models directly from videos, which was previouslynot possible. To recover meshes from videos, we treatour mechanism for frame processing (§ 4.1) as a basicbuilding block and use it in a recurrent structure. We alsoincorporate attention layer in this recurrent model. Theattention layer aims to encode low-level visual features ofinput images, and provide contextual conditions on therecurrent operations. The resulting technique explicitlymodels the temporal dynamics of action sequences and isable to enforce geometric consistency on the reconstructedposes across the frames. This makes the recovered 3Dposes appear more natural and realistic, as illustrated inFig. 1.

The proposed network for full mesh pose recovery is il-lustrated in Fig. 5 (B). We embed the frame-based build-ing block in a recurrent structure by sharing the weights

of the Encoder and Regressor. Given an input videoclip It, s.t. t ∈ 1, 2, ..., T; where T is the numberof frames in the clip, the same Encoder is used to ex-tract visual features from each frame. In addition to thefeature vector E(It), we probe the low-level feature mapE(It)low ∈ RL×D, which can be represented as a set ofannotation vectors at = a1t , ..., aLt , ait ∈ RD. With anattention layer fatt which is implemented as a multi-layerperceptron (MLP) [41], the context vector zt of frame Itis encoded with the annotation vectors at.

eit = fatt(ait), (11)

αit =

exp(eit)∑Lk=1 exp(ekt )

. (12)

Here, αit represents the weights of each annotation vector

ait and∑L

i=1 αit = 1. Intuitively, it represents the impor-

tance of particular visual feature elements, and, therefore,indicates “where” and “how much” the network shouldpay attention to. With the calculated weights αi

t, weadopt a deterministic “soft” attention mechanism [42] tocompute the context vector zt as follows

zt =

L∑i=1

αita

it. (13)

For the recurrent structure, we employ LSTM [43] forits gate and memory design that makes its training moreeffective. Following the implementation in [44], the over-all behaviour of LSTM is represented mathematically by

ct = ft ct−1 + it gt, (14)

ht = ot tanh(ct). (15)

In the above equations, it, ft, ct, ot, ht are the input, for-get, memory, output and hidden state of the LSTM attime step t, and denotes the Hadamard product. Basedon the above equations, we denote the LSTM cell functionas ht = Ω(it) which controls the inflow and outflow of in-formation through the LSTM at tth frame. The internalstates of LSTM are controlled by the affine transforma-tions with trainable parameters. The total time step ofLSTM is set to T .

To incorporate the attention mechanism into the recur-rent network, we make the context vector zt conditionedon the previous hidden state of LSTM, i.e. ht−1. Intu-itively, as LSTM advances in a frame sequence, how thenetwork pays its attention is conditioned on the informa-tion it has processed in the previous time steps. Hence,we can rewrite Eq. (11) as

eit = fatt(ait, ht−1). (16)

Using the above configuration, the network computes thedesired vector ϕt for the tth frame as

ϕt = R(Ω(E(It)⊗ zt ⊗ϕt−1)); t = 1, 2, ..., T, (17)

where ⊗ denotes vector concatenation. Note that, theinput to an LSTM cell at each time step corresponds to

7

the frame’s encoded feature augmented with the contex-tual information derived from low-level visual features,and the prediction from the last time step. At time 0,we assign the prediction ϕ0 as the mean value of SMPLparameters for our data. As for the cell state ct and theoutput state ht, we initialize them as (c0, h0) = I(E(I1)),where I(.) is implemented with an MLP with two hiddenlayers that maps an encoded feature vector to the initialstates. This initialization strategy helps in faster trainingof the overall recurrent network.

To enforce geometric consistency along a predicted posesequence, we propose an additional shape smoothness lossLshape, for the recurrent network as

Lshape =

T−1∑t=1

||βt+1 − βt||22, (18)

where βt is obtained from the Regressor at the tth timestamp. In our recurrent structure, this loss inherentlyaccounts for all the frames in a video clip. Accordingto [1], intermediate supervision helps in mitigating thevanishing gradient problem in the recurrent networks, andalso helps in better conditioning of the learning process.Hence, we eventually define our overall loss function as

L =

T∑t=1

λ((Lproj)t + δ(L3D)t) + Lshape, (19)

where L3D=L3Djoint+Lsmpl, and λ is a hyper-parameter.

4.3 Implementation Details

To implement the Encoder (E), we use the ResNet-50model [46] pre-trained on ImageNet [47]. We considerthe convolution activation prior to Softmax as an imagefeature, and the activation values of the layer “resnet-v2-50/block4” as the low-level feature map. We realize theRegressor (R) as a Multiple Layer Perceptron (MLP) withtwo fully-connected hidden layers, and an output layerthat has the same dimension as vector ϕ. Both E and Rare shared by every time step for the LSTM, whereas weempirically set the width of the hidden unit for the LSTMto 2048, and use T = 4. Our model for videos is trained intwo main steps. First, we train a frame-based model andthen use it in the second step for video based training. Wecopy the Encoder and Regressor weights for the recurrentmodel for initialization, and then train the model withstochastic gradient descent using Adam optimizer [48] inan end-to-end manner. We set the learning rate to 10−5,and batch size to 16.

To train a model, we also add samples from LSP, LSP-extended [49] MPII [50], MS COCO [9], Human3.6M [51]and MPIINF-3DHP [2] to our data, in addition to theproposed data. These datasets have been filtered to re-move images that are too small or have less that 6 visiblejoints. The standard train/test split is used. Datasetssuch as LSP, COCO and MPII consist of independentframes, for which we replicate the individual frames toform training video clips.

5 Evaluation

5.1 Quantitative Evaluation

Metrics Evaluation of a full 3D pose recovery methodis not straightforward. It requires computing point-to-point errors between predicted and the ground truth‘mesh’ points. For that, point-to-point mesh registrationis required which is often not possible with the existingdatasets. Consequently, recent works mostly resort to re-porting Mean Per Joint Position Error (MPJPE) for 3Dskeletons. For benchmarking, we also adopt MPJPE asone of our evaluation metrics. Nevertheless, since ourdata provides the possibility of point-to-point registra-tion and mesh recovery directly from videos, we furtherintroduce the following new metrics to more appropri-ately evaluate a model for video-based full mesh pose re-covery: (a) Mean Per Vertex Position Error (MPVPE),(b) Mean Running Vertex Position Variation (MRVPV)and (c) Mean Running Shape Variation (MRSV).

We define the MPVPE as

MPVPE =1

M

M∑j=1

(N∑i=1

||vi − vi||2

), (20)

where M and N respectively denote the frame length andthe vertices in a mesh, and vi; and vi are the ground-truthand predicted vertex locations, respectively. This metriccan be considered a mesh variant of MPJPE. To explicitlyaccount for the temporal dimension of a video, we defineMRVPV as

MRVPV =1

M

M−1∑j=1

(N∑i=1

||vj+1,i − vj,i||p

), (21)

where ||.||p denotes the `p-norm. We consider both `1 and`2 norms in this work, denoting the resulting variants byMRVPV1 and MRVPV2. We also define MRSV as

MRSV =1

M

M−1∑i=1

(||βi+1 − βi||p

), (22)

denoting its `1 and `2 norm variants by MRSV1 andMSRV2, respectively. By definition, this metric gives anestimate of how well the shape of the computed avataris maintained between consecutive video frames. A lowervalue ensures better geometric consistency in terms of theavatar shape.

3D Joints Evaluation For 3D joint evaluation, we usethe standard MPJPE metric. We also follow [45] to adjustthe global misalignment for the reconstructed 3D joints byapplying a similarity transform via the Procrustes Anal-ysis (PA). The adjusted error is then reported as PA-MPJPE. We use both Human3.6M dataset and the pro-posed data for evaluation. Table 1 and Table 2 summarizeour results, where our method is coined as Mesh VIdeoPosE Recovey (MVIPER). In the reported results, HMR†is our enhancement of HMR [6], which is achieved by finetuning it on our dataset.

8

Table 1: 3D joint evaluation results on Human3.6M datasets: Evaluation metric PA-MPJPE, units mm. Theproposed method is abbreviated as Mesh VIdeo PosE Recovery (MVIPER). Per action and mean error is reportedfor two evaluation protocols. See text for protocol description.

Protocol-1 Direc. Discu. Eat Greet Phone Photo Pose Purchase Sit SitD. Smoke Wait WalkD. Walk WalkT. Mean

HMR [6] 52.3 54.7 54.3 57.1 60.9 70.4 51.6 49.9 65.7 76.0 58.6 52.5 60.2 45.2 53.6 57.5HMR† 51.2 52.4 53.8 56.9 59.9 65.0 50.4 49.2 66.3 73.1 59.2 52.6 60.0 46.6 53.9 56.7MVIPER (ours) 48.1 48.8 49.6 55.3 53.8 63.4 49.4 48.0 58.5 67.4 54.4 52.2 59.3 47.3 54.3 54.0

Protocol-2 Direc. Discu. Eat Greet Phone Photo Pose Purchase Sit SitD. Smoke Wait WalkD. Walk WalkT. Mean

SMPLify [45] 62.0 60.2 67.8 76.5 92.1 77.0 73.0 75.3 100.3 137.3 83.4 77.3 79.7 86.8 81.7 82.3HMR [6] 53.2 56.8 50.4 62.4 54.0 72.9 49.4 51.4 57.8 73.7 54.4 50.0 62.6 47.1 55.0 56.7HMR† 52.1 53.9 51.4 61.1 54.4 66.1 49.6 48.7 58.3 69.9 54.6 50.0 60.6 49.3 55.5 55.7MVIPER (ours) 48.0 46.0 46.0 57.1 48.6 61.3 47.7 46.8 54.1 67.1 48.9 50.1 59.1 47.8 56.1 52.3

Table 2: 3D Joint evaluation results on the proposed data.

Method MPJPE PA-MPJPE

SMPLify [45] 152.1 109.3HMR [6] 133.2 81.3HMR† 125.6 77.6MVIPER (ours) 93.2 60.5

To achieve the results in Table 1, we follow the stan-dard practice of using 5 subjects (ID: S1, S5, S6, S7, S8)for training and 2 subjects (ID: S9, S11) for testing. Weemploy two standard protocols. Protocol-1: uses sam-ples from all four provided viewpoints for testing, andProtocol-2: uses only the frontal viewpoint samples. Toevaluate MVIPER under frame based protocols, we repli-cate a frame multiple times to form a clip. In the Tables,our method is able to outperform the existing methodsconsistently. It is also notable that HMR† is able to per-form better than HMR demonstrating the effectivenessof the proposed dataset for the task of 3D pose recoveryin general. We note that this work focuses on recoveringfull 3D meshes, hence we include only those pose recoverymethods in our comparisons that have this ability.

3D Mesh Evaluation We evaluate the performanceof our method for 3D pose mesh recovery fromvideos/images, and compare it with the existing SMPL-based methods that have the mesh recovery ability. Ta-ble 3 summarizes the results of our experiments using theevaluation metrics discussed above. As can be seen, theproposed method consistently outperforms the state-of-the-art 3D human pose mesh recovery methods. Again,the gain of HMR† over HMR demonstrates the effective-ness of the proposed dataset.

5.2 Ablation Study

The dataset proposed in this work enables learning 3Dpose recovery models with full supervision in terms of2D keypoints, 3D skeleton, and SMPL pose and shapeparameters. Our model fully exploits these supervisionlabels for the pose recovery. In this section, we studythe contribution of each of these supervision labels to theoverall performance of our technique by re-training the

Table 3: 3D mesh evaluation on proposed dataset.

Method MPVPEMRSV1MRSV2MRVPV1MRVPV2

SMPLify [45] 1426.9 0.85 0.41 257.9 4.64HMR [6] 1056.5 0.82 0.36 194.0 4.31HMR† 923.7 0.76 0.32 191.2 4.25MVIPER (ours) 692.7 0.51 0.29 178.7 4.02

Table 4: Ablation study for training MVIPER with dif-ferent loss combinations. For each row, MVIPER is re-trained with the designated losses and then evaluated onHuman3.6M dataset. Definition of losses can be found inSection 4.1.

Protocol-1 Protocol-2

Loss Combinations MPJPEPA-MPJPEMPJPEPA-MPJPE

Lproj 139.0 72.6 117.3 67.2Lproj+L3Djoint 82.6 56.4 83.7 55.7Lproj+Lsmpl 110.7 58.2 105.3 57.6L3Djoint+Lsmpl 83.8 58.0 88.9 58.3Lproj+L3Djoint+Lsmpl 82.2 54.0 81.5 52.3

MVIPER model with different loss combinations that aredesigned to capitalize on different supervision labels.

The first column of Table 4 shows the loss combinationsused for this ablation study. For each combination, we re-train the MVIPER model and report quantitative resultson Human3.6M dataset under Protocol-1 and Protocol-2.Both MPJPE and the adjusted error PA-MPJPE are re-ported. The results clearly demonstrate the importanceof direct 3D supervision for the pose recovery task. With2D objective Lproj , the model learning suffers from se-rious depth ambiguity. With 3D supervision, in bothcases of L3Djoint or Lsmpl, the accuracy improvementis remarkable. Considering rows 2 and 3 of the table,Lproj+L3Djoint achieves higher accuracy in MPJPE thanLproj+Lsmpl, while both combinations have comparableperformance with global alignment (PA-MPJPE). Thisis intuitive because both L3Djoint and Lsmpl reflect full3D joints information, while L3Djoint emphasizes more onthe absolute joint locations, which benefits the 3D jointsevaluation. The last row of Table 4 demonstrates thatMVIPER achieves the best performance when we com-bine all the loss terms.

9

Figure 6: Qualitative comparison for HMR and our ap-proach on Human3.6M. For each method, the recoveredmesh and its 60 rotation are displayed. The rotatedmeshes are shown for better visualization.

5.3 Qualitative Evaluation

Representative examples for qualitative performancecomparison between our method and the competingframe-based technique, i.e. HMR is shown in Fig. 6 andFig. 7. The examples are selected at random from Hu-man3.6M dataset. Note that, Human3.6M dataset doesnot provide ground truth for 3D meshes, hence we canonly show the qualitative results. In Fig. 6, when theperformer’s left arm is occluded by body, HMR loses itsspatial cues and fails to reconstruct it appropriately, whileour approach is able to construct the left arm properlyby accounting for the temporal cues. The 60 rotation ofmesh is provided for better visualization. Similarly, forthe complex motions, as illustrated in Fig. 7, HMR alsofails to recover accurate and consistent poses. On theother hand, the proposed method performs a much moreaccurate pose recovery.

In addition to Human3.6M, we further conducted qual-itative analysis on UCF101 [14] - an “in-the-wild” actiondataset unseen by our method and the benchmark HMRmethod during training. From the UCF101 dataset, weselected action sequences in which body motions repre-sent the major contents of video frames. We show rep-resentative qualitative results on this dataset in Fig. 8.and Fig. 9. For each sequence evaluated by MVIPERand HMR, we highlight the frames which show significantquality difference. Our method demonstrates consistent

Figure 7: Qualitative comparison for HMR and our ap-proach on Human3.6M. Our approach recovers more ac-curate and coherent pose sequence.

advantage over frame-based HMR. In Fig. 8, HMR failsto recover the uncommon “handstand” poses, while ourmethod recovers it properly. We attribute this differenceto our synthetic training data which includes more posevariations, and hence improves the performance of the re-sulting model. As our method models temporal variationsand imposes geometric consistency, it is able to generatesmooth transition for recovered pose sequences. On theother hand, frame-based methods like HMR suffer fromtemporal jitters and sudden orientation error as visible inFig. 9.

5.4 Video-based Motion Transfer

This is the first work to explore video-based full meshpose recovery. This problem can find many interestingapplications in e.g. animations. As a possible applicationof our approach we explore video-based motion transfer.In this application, we input a real-world video of a hu-man performing an action and transfer that motion toan avatar in virtual world. The proposed MVIPER usesthe input video to compute 3D meshes for each frame,which are subsequently rendered with different cloths andbackgrounds to create virtual actions. The whole pro-cess transfers a real-life human motion to a virtual worldavatar. We illustrate a representative example from thisexperiment in Fig. 10. As can be seen, the proposed tech-nique is able to transfer the motion with good fidelity due

10

Figure 8: Qualitative comparison. Challenging “hand-stand” poses are accurately recovered by our method be-cause it benefits from a large amount of pose variationsthat are present in the proposed dataset.

to accurate human mesh recovery.

6 Conclusion

We proposed a data generation technique and an end-to-end trainable RNN to recover full 3D meshes of hu-man poses directly from monocular videos. Our datageneration method exploits Computer Graphics to gen-erate realistic action videos with a large amount of scenevariations in backgrounds, cloth textures, illuminationsand viewpoints. To further enhance action and pose vari-ations in the generated data, pose interpolation is em-ployed to create novel pose sequences between largelyvaried pose pairs. Moreover, we embed a Physics enginein data generation to produces vivid cloth deformationsand cloth-body interactions. This is the first successfulapplication of a Physics engine in contemporary humandata synthesis technique for learning deep models. By us-ing the proposed action video dataset and a parameter-ized human model, we also developed a neural networkfor video-based full mesh pose recovery. Our networkembeds a basic building block in a recurrent structureand explicitly encodes temporal variations of input videoframes, and imposes geometric consistency over the recov-ered meshes across the video frames. We evaluated the

Figure 9: Qualitative comparison. HMR suffers from asudden body orientation error, as it does not consider thetemporal context from neighbouring frames.

proposed method on our dataset and Human3.6M, usingconventional per-joint error metrics, as well as more ad-vanced per-vertex error metrics introduced in this work.Qualitative comparison is provided on Human3.6M andUCF101 action datasets. Both quantitative and qualita-tive results demonstrate that our method achieves verypromising results for full mesh pose recovery from videos.

Acknowledgment

This research was sponsored by the Australian ResearchCouncil (ARC) grant DP160101458 and partially sup-ported by ARC grant DP190102443. The Tesla K-40GPU used for this research was donated by the NVIDIACorporation.

References

[1] S.-E. Wei, V. Ramakrishna, T. Kanade, andY. Sheikh, “Convolutional pose machines,” inCVPR, 2016, pp. 4724–4732.

[2] D. Mehta, S. Sridhar, O. Sotnychenko, H. Rhodin,M. Shafiei, H.-P. Seidel, W. Xu, D. Casas, andC. Theobalt, “Vnect: Real-time 3d human pose es-timation with a single rgb camera,” ACM Transac-tions on Graphics, vol. 36, no. 4, p. 44, 2017.

11

Figure 10: Example of motion transfer performed usingthe proposed MVIPER. Motions are transferred from realto virtual world with new cloth and background in thevirtual world.

[3] G. Pavlakos, X. Zhou, K. G. Derpanis, andK. Daniilidis, “Coarse-to-fine volumetric predictionfor single-image 3d human pose,” in CVPR. IEEE,2017, pp. 1263–1272.

[4] M. Wang, X. Chen, W. Liu, C. Qian, L. Lin, andL. Ma, “Drpose3d: Depth ranking in 3d human poseestimation,” IJCAI, 2018.

[5] Y. Luo, J. Ren, Z. Wang, W. Sun, J. Pan, J. Liu,J. Pang, and L. Lin, “Lstm pose machines,” inCVPR. IEEE, 2018, pp. 5207–5215.

[6] A. Kanazawa, M. J. Black, D. W. Jacobs, and J. Ma-lik, “End-to-end recovery of human shape and pose,”in CVPR, 2018, pp. 7122–7131.

[7] W. Yang, W. Ouyang, X. Wang, J. Ren, H. Li, andX. Wang, “3d human pose estimation in the wild byadversarial learning,” in CVPR, vol. 1, 2018.

[8] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll,and M. J. Black, “Smpl: A skinned multi-person lin-ear model,” ACM Transactions on Graphics, vol. 34,no. 6, p. 248, 2015.

[9] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Per-ona, D. Ramanan, P. Dollar, and C. L. Zitnick, “Mi-crosoft coco: Common objects in context,” in ECCV.Springer, 2014, pp. 740–755.

[10] C. Ionescu, D. Papava, V. Olaru, and C. Sminchis-escu, “Human3.6m: Large scale datasets and predic-tive methods for 3d human sensing in natural envi-ronments,” PAMI, vol. 36, no. 7, pp. 1325–1339, jul2014.

[11] H. Joo, H. Liu, L. Tan, L. Gui, B. Nabbe,I. Matthews, T. Kanade, S. Nobuhara, andY. Sheikh, “Panoptic studio: A massively multiviewsystem for social motion capture,” in Proceedings ofthe IEEE International Conference on Computer Vi-sion, 2015, pp. 3334–3342.

[12] G. Pons-Moll, S. Pujades, S. Hu, and M. J. Black,“Clothcap: Seamless 4d clothing capture and re-targeting,” ACM Transactions on Graphics (TOG),vol. 36, no. 4, p. 73, 2017.

[13] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black, “Amass: Archive of mo-tion capture as surface shapes,” arXiv preprintarXiv:1904.03278, 2019.

[14] K. Soomro, A. R. Zamir, and M. Shah, “Ucf101: Adataset of 101 human actions classes from videos inthe wild,” arXiv preprint arXiv:1212.0402, 2012.

[15] M. Andriluka, S. Roth, and B. Schiele, “Pictorialstructures revisited: People detection and articu-lated pose estimation,” in CVPR. IEEE, 2009, pp.1014–1021.

[16] P. F. Felzenszwalb and D. P. Huttenlocher, “Picto-rial structures for object recognition,” Internationaljournal of computer vision, vol. 61, no. 1, pp. 55–79,2005.

[17] B. Sapp, D. Weiss, and B. Taskar, “Parsing humanmotion with stretchable models,” in CVPR 2011.IEEE, 2011, pp. 1281–1288.

[18] Y. Yang and D. Ramanan, “Articulated human de-tection with flexible mixtures of parts,” IEEE trans-actions on pattern analysis and machine intelligence,vol. 35, no. 12, pp. 2878–2890, 2013.

[19] M. Andriluka, S. Roth, and B. Schiele, “Monocular3d pose estimation and tracking by detection,” in2010 IEEE Computer Society Conference on Com-puter Vision and Pattern Recognition. IEEE, 2010,pp. 623–630.

[20] A. Newell, K. Yang, and J. Deng, “Stacked hourglassnetworks for human pose estimation,” in EuropeanConference on Computer Vision. Springer, 2016,pp. 483–499.

[21] T. Pfister, J. Charles, and A. Zisserman, “Flowingconvnets for human pose estimation in videos,” inProceedings of the IEEE International Conference onComputer Vision, 2015, pp. 1913–1921.

[22] C. J. Taylor, “Reconstruction of articulated objectsfrom point correspondences in a single uncalibratedimage,” CVIU, vol. 80, no. 3, pp. 349–363, 2000.

[23] V. Ramakrishna, T. Kanade, and Y. Sheikh, “Recon-structing 3d human pose from 2d image landmarks,”in ECCV. Springer, 2012, pp. 573–586.

12

[24] D. Mehta, H. Rhodin, D. Casas, P. Fua, O. Sotny-chenko, W. Xu, and C. Theobalt, “Monocular 3dhuman pose estimation in the wild using improvedcnn supervision,” in 3D Vision (3DV). IEEE, 2017,pp. 506–516.

[25] T. Alldieck, M. Magnor, W. Xu, C. Theobalt, andG. Pons-Moll, “Video based reconstruction of 3d peo-ple models,” in Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition, 2018,pp. 8387–8397.

[26] ——, “Detailed human avatars from monocularvideo,” in 2018 International Conference on 3D Vi-sion (3DV). IEEE, 2018, pp. 98–109.

[27] T. Alldieck, M. Magnor, B. L. Bhatnagar,C. Theobalt, and G. Pons-Moll, “Learning to recon-struct people in clothing from a single rgb camera,”arXiv preprint arXiv:1903.05885, 2019.

[28] T. Yu, Z. Zheng, K. Guo, J. Zhao, Q. Dai, H. Li,G. Pons-Moll, and Y. Liu, “Doublefusion: Real-time capture of human performances with inner bodyshapes from a single depth sensor,” in Proceedings ofthe IEEE Conference on Computer Vision and Pat-tern Recognition, 2018, pp. 7287–7296.

[29] T. Yu, Z. Zheng, Y. Zhong, J. Zhao, Q. Dai, G. Pons-Moll, and Y. Liu, “Simulcap: Single-view humanperformance capture with cloth simulation,” arXivpreprint arXiv:1903.06323, 2019.

[30] M. Omran, C. Lassner, G. Pons-Moll, P. Gehler,and B. Schiele, “Neural body fitting: Unifying deeplearning and model based human pose and shape es-timation,” in 2018 International Conference on 3DVision (3DV). IEEE, 2018, pp. 484–494.

[31] L. Sigal, A. O. Balan, and M. J. Black, “Humaneva:Synchronized video and motion capture dataset andbaseline algorithm for evaluation of articulated hu-man motion,” IJCV, vol. 87, no. 1-2, p. 4, 2010.

[32] C. Lassner, G. Pons-Moll, and P. V. Gehler, “A gen-erative model of people in clothing,” in Proceedingsof the IEEE International Conference on ComputerVision, 2017, pp. 853–862.

[33] G. Varol, J. Romero, X. Martin, N. Mahmood, M. J.Black, I. Laptev, and C. Schmid, “Learning fromSynthetic Humans,” in CVPR, 2017.

[34] M. Loper, N. Mahmood, and M. J. Black, “Mosh:Motion and shape capture from sparse markers,”ACM Transactions on Graphics, vol. 33, no. 6, p.220, 2014.

[35] G. Varol, J. Romero, X. Martin, N. Mahmood, M. J.Black, I. Laptev, and C. Schmid, “Learning fromsynthetic humans,” in CVPR, 2017, pp. 109–117.

[36] D. Baraff and A. Witkin, “Large steps in cloth sim-ulation,” in Computer graphics and interactive tech-niques. ACM, 1998, pp. 43–54.

[37] M. Hauth, “Numerical techniques for cloth simula-tion,” system (figure 2 (a), vol. 15, p. 3, 2005.

[38] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, andA. Torralba, “Places: A 10 million image databasefor scene recognition,” PAMI, vol. 40, no. 6, pp.1452–1464, 2018.

[39] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, ,and A. Vedaldi, “Describing textures in the wild,”in CVPR, 2014.

[40] C. Kampouris, S. Zafeiriou, A. Ghosh, andS. Malassiotis, “Fine-grained material classificationusing micro-geometry and reflectance,” in ECCV.Springer, 2016, pp. 778–792.

[41] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville,R. Salakhudinov, R. Zemel, and Y. Bengio, “Show,attend and tell: Neural image caption generationwith visual attention,” in International conferenceon machine learning, 2015, pp. 2048–2057.

[42] D. Bahdanau, K. Cho, and Y. Bengio, “Neural ma-chine translation by jointly learning to align andtranslate,” arXiv preprint arXiv:1409.0473, 2014.

[43] S. Hochreiter and J. Schmidhuber, “Long short-termmemory,” Neural computation, vol. 9, no. 8, pp.1735–1780, 1997.

[44] W. Zaremba, I. Sutskever, and O. Vinyals, “Recur-rent neural network regularization,” arXiv preprintarXiv:1409.2329, 2014.

[45] F. Bogo, A. Kanazawa, C. Lassner, P. Gehler,J. Romero, and M. J. Black, “Keep it smpl: Auto-matic estimation of 3d human pose and shape froma single image,” in ECCV. Springer, 2016, pp. 561–578.

[46] K. He, X. Zhang, S. Ren, and J. Sun, “Identitymappings in deep residual networks,” in ECCV.Springer, 2016, pp. 630–645.

[47] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, andL. Fei-Fei, “Imagenet: A large-scale hierarchical im-age database,” in CVPR. Ieee, 2009, pp. 248–255.

[48] D. P. Kingma and J. Ba, “Adam: A methodfor stochastic optimization,” arXiv preprintarXiv:1412.6980, 2014.

[49] S. Johnson and M. Everingham, “Learning effectivehuman pose estimation from inaccurate annotation,”in CVPR, 2011.

[50] M. Andriluka, L. Pishchulin, P. Gehler, andB. Schiele, “2d human pose estimation: New bench-mark and state of the art analysis,” in CVPR, 2014,pp. 3686–3693.

[51] C. Ionescu, D. Papava, V. Olaru, and C. Sminchis-escu, “Human3. 6m: Large scale datasets and pre-dictive methods for 3d human sensing in natural en-vironments,” PAMI, vol. 36, no. 7, pp. 1325–1339,2014.

13