Real-Time Curve-Skeleton Extraction of Human-Scanned Point CloudsApplication in Upright Human Pose Estimation

Frederic Garcia and Bjorn OtterstenInterdisciplinary Centre for Security Reliability and Trust (SnT), University of Luxembourg, Luxembourg

{frederic.garcia, bjorn.ottersten}@uni.lu

Keywords: curve-skeleton, skeletonization, human pose estimation, object representation, point cloud, real-time

Abstract: This paper presents a practical and robust approach for upright human curve-skeleton extraction. Curve-skeletons are object descriptors that represent a simplified version of the geometry and topology of a 3-Dobject. The curve-skeleton of a human-scanned point set enables the approximation of the underlying skeletalstructure and thus, to estimate the body configuration (human pose). In contrast to most curve-skeleton extrac-tion methodologies from the literature, we herein propose a real-time curve-skeleton extraction approach thatapplies to scanned point clouds, independently of the object’s complexity and/or the amount of noise withinthe depth measurements. The experimental results show the ability of the algorithm to extract a centeredcurve-skeleton within the 3-D object, with the same topology, and with unit thickness. The proposed approachis intended for real world applications and hence, it handles large portions of data missing due to occlusions,acquisition hindrances or registration inaccuracies.

1 INTRODUCTION

Human pose estimation not only is one of the funda-mental research topics in computer vision, but a nec-essary step in active research topics such as scene un-derstanding, human-computer interaction and actionor gesture recognition; a side-effect of the recent ad-vances in 3-D sensing technologies.

In this paper, we address the problem of humanpose estimation in the context of 3-D scenes scannedby multiple consumer-accessible depth cameras suchas the Kinect or the Xtion Pro Live. To that end, wepropose to use curve-skeletons, a compressed repre-sentation of the 3-D object. Curve-skeletons are ex-tremely useful in computer graphics and increasinglybeing used in computer vision for their valuable aidto address many visualization tasks including shapeanalysis, animation, morphing and/or shape retrieval.Indeed, curve-skeletons represent a simplified versionof the 3-D object with a major advantage of preserv-ing both its geometry and topology information. This,in turn, allows to estimate the object configuration orobject pose after approximating its skeletal structure.

The remainder of the paper is organized as fol-lows: Section 2 covers the literature review on themost recent curve-skeleton extraction approaches forpoint cloud datasets. Section 3 introduces our real-time curve-skeleton extraction approach. In Section 4

we evaluate and analyze the resulting curve-skeletonsfrom multiple scanned 3-D models. To do so, bothreal and synthetic data have been considered. Finally,concluding remarks are given in Section 5.

2 Related Work

Curve-skeleton makes decisive contribution forhuman pose estimation as it enables to estimate thebody configuration by fitting the underlying skele-tal structure. Consequently, an extensive researchcan be found in the literature, with many approachesstrongly dependent on the requirements of their ap-plications. In the following we only review the mostrepresentative methods for curve-skeleton extractionfrom point cloud datasets. For a complete review,we refer the reader to the comprehensive survey ofCornea et al. (Cornea et al., 2007).

Skeletonization algorithms can be divided in fourdifferent classes: thinning and boundary propagation,distance field-based, geometric, and general/fieldfunctions (Cornea et al., 2007). However, recent ap-proaches are providing excellent results by combiningdifferent techniques from different classes (Cao et al.,2010) (Sam et al., 2012) (Tagliasacchi et al., 2009).Au et al. (Au et al., 2008) proposed a curve-skeletonextraction approach by mesh contraction using Lapla-

cian smoothing. They used connectivity surgery topreserve the original topology and an iterative pro-cess as an energy minimization problem with con-traction and attraction terms. Though special atten-tion must be paid to the contraction parameters, themethod fails in the case of very coarse models. Wa-tertight meshes are required due to the mesh connec-tivity constraint from the Laplacian smoothing. Caoet al. (Cao et al., 2010) extended their work to pointcloud datasets. To do so, a local one-ring connec-tivity of point neighborhood must be done for theLaplacian operation, which significantly increases theprocessing time. An alternative approach proposedby Tagliasacchi et al. (Tagliasacchi et al., 2009) usesrecursive planar cuts and local rotational symmetricaxis (ROSA) to extract the curve-skeleton from in-complete point clouds. However, their approach re-quires special attention within object joints and it isnot generalizable to all shapes. Similarly, Sam etal. (Sam et al., 2012) use the cutting plane idea butonly two anchors points must be computed. Theyguarantee centeredness by relocating skeletal nodesusing ellipse fitting. However, although the result-ing curve-skeletons from this work preserve most ofthe required properties cited by Cornea et al. (Corneaet al., 2007), they are impractical when real-time isrequired. We herein overcome this limitation by com-puting the skeletal candidates in the 2-D space.

3 Proposed Approach

In the following we introduce our approach to ex-tract the curve-skeleton from upright human-scannedpoint clouds. Our major contribution, in which we ad-dress the real-time constraint, is based on the extrac-tion of the skeletal candidates in the 2-D space. Wehave been inspired by image processing techniquesused in silhouette-based human pose estimation (Liet al., 2009). The final 3-D curve-skeleton resultsfrom back projecting these 2-D skeletal candidates totheir right location in the 3-D space.

Let us consider a scanned point cloud P repre-sented in a three-dimensional Euclidean space E3 ≡{p(x,y,z)|1 ≤ x ≤ X ,1 ≤ y ≤ Y,1 ≤ z ≤ Z}, and de-scribing a set of 3-D points pi representing the under-lying external surface of an upright human body. Thefirst step concerns the voxelization of the given pointcloud to account for the point redundancy resultingfrom registering multi-view depth data. To that end,we have considered the surface voxelization approachpresented in (Garcia and Ottersten, 2014). The result-ing point cloud P ′ presents both a uniform point den-sity and a significantly reduction of the points to be

further processed. Our goal is to build a 2-D imagerepresenting the front view of the 3-D body in orderto extract its curve-skeleton using 2-D image process-ing techniques. To do so, we define a cutting-plane π

being parallel to x and y-axis of E3 and intersectingP ′ at p, the mean point or centroid of the point set P ′,i.e., p = 1

k ·∑ki=1 pi. From our experimental results,

π crossing p provides good body orientations in thecase of upright body postures, e.g., walking, runningor working. Alternative body configurations are dis-cussed in Section 4.1. We determine the body orienta-tion by fitting a 2-D ellipse onto the set of 3-D pointslying on π, i.e., ∀pi|pi(z) = p(z). We note that in prac-tice and due to point density variations, 3-D pointsare not necessarily lying on π. Therefore, we con-sider those 3-D points in P ′ with pi(z) ∈ [p(z)± λ],and λ being related to the point density of P ′, asdepicted in Fig. 1a. We note that λ has to be cho-sen large enough to ensure a good description of thebody contour. In turn, the higher the point densitythe smaller the λ value should be. Fig. 1b illus-trates the resulting 2-D ellipse using the Fitzgibbonet al. (Fitzgibbon and Fisher, 1995) approach, imple-mented in OpenCV (Bradski and Kaehler, 2008). rand s are unitary vectors along the major and minor el-lipse axes, respectively. c is the ellipse centroid. Oncethe body orientation is known, we build the 2-D imageI representing the front view of the human body. Todo so, we can project P ′ to the 2-D plane π′ definedby the ellipse major axis r and with normal vectors. Image dimensions are given by the projections ofthe minimum pmin = (xmin,ymin,zmin) and maximumpmax = (xmax,ymax,zmax) 3-D point coordinates fromP ′. An alternative solution is to translate P ′ fromthe coordinate frame C1 = (r, s, r× s), and originat pmax, depicted in Fig. 2a, to the coordinate frame

(a) (b)

Figure 1: (a) Selected 3-D points (in green color) forellipse fitting. (b) Fitted ellipse using OpenCV (Brad-ski and Kaehler, 2008).

(a) P ′ (b) P ′′

Figure 2: (a) P ′ at C1. (b) P ′′ at C2.

C2 = (−r,−(r× s),−s), and origin at (0,0,0), de-picted in Fig. 2b, i.e., P ′′ = TC2

C1·P ′. By doing so, the

2-D plane π′ coincides with the two-dimensional Eu-clidean space with x and y axes coincident to E3, i.e.,E2 ≡ {I(m,n)|1≤ m≤M,1≤ n≤ N}. Therefore,

I(m,n) ={

255 if pi ∈ P ′′ ∀i0 otherwise, (1)

with m = δ · round(pi(x)

)and n = δ · round

(pi(y)

),

and δ the conversion factor between metric units andpixels (herein we set δ = 100, i.e., 1 mm = 1 pixel).Fig. 3a shows the resulting image I after projectingP ′′ onto E2.

In 2-D image processing, distance field or dis-tance transform (DT) is commonly used to build im-age maps D from binary images. Pixel values indicatethe minimum distance between the selected pixel andits nearest boundary pixel, i.e., the closest zero pixel.That is, D(u) = min{d(u,v)|I(v) = 0}, with d(u,v)being the Euclidean, Manhattan or Chessboard dis-tance metric. We note that the resulting image mapis highly sensitive to zeros pixels, and thus holes inthe image will significantly alter the resulting imagemap. A valid approach to fill the body silhouette inthe case of a constant density of points and withoutomission of data is the use of morphological operatorssuch as dilation and erosion operators. However, fill-ing the body silhouette becomes more intricate whentreating incomplete point clouds with large portionsof data missing, e.g., around the pelvis in Fig. 3a. Wetherefore propose to extract the contours of I and fillthe area bounded by the most external contour, i.e.,the body silhouette, which results in a dense body sil-houette, as shown in Fig. 3c. To do so, we use thecontour following algorithm proposed by Suzuki etal. (Suzuki and Abe, 1985) (see Fig. 3b). Among the

(a) I. (b) (c)

(d) D. (e) ∆D. (f) J.

Figure 3: (a) I from (1). (b) Contours of I. (c) Bodysilhouette. (d) Euclidean DT D. (e) Laplace operator∆D. (f) J from (2).

aforementioned distance metrics, we herein have con-sidered the Euclidean DT described in (Felzenszwalband Huttenlocher, 2004). From the resulting imagemap D in Fig. 3d, we realize that higher-valued mappixels correspond to centered pixels within the sil-houette boundary and thus, skeletal candidates. In-deed, skeletal candidates coincide with low-valuededges on the result of the Laplace operator on D,i.e., ∆D = ∂2D/∂2m+ ∂2D/∂2n, as can be observedin Fig. 3e. The final 2-D coordinates of the skeletalcandidates J are given by an adaptive thresholding onthe low-valued edges in ∆D, i.e.,

J(u) ={

255 if ∆D(u)< τ

0 otherwise, (2)

being τ the adaptive threshold value. A valid τ

value can be automatically obtained using Otsu’smethod (Sezgin and Sankur, 2004). The pixel posi-tions of the skeletal candidates from (2) correspond tothe x and y coordinates of the 3-D points qi that willconstitute the final curve-skeleton J , as depicted inFig 4, i.e., qi(x) = m and qi(y) = n ∀J(m,n) = 255.We determine the missing qi(z) coordinate for eachskeletal candidate J(ui) by defining the line li withunit vector the z-axis of E3 and intersecting at the cor-

Figure 4: (a) Skeletal candidates J (in blue color) andthe location of a skeletal candidate J(ui) in the 3-Dspace, i.e., qi(z).

responding skeletal candidate J(ui). The missing co-ordinate corresponds to the middle point between thelower p j and upper pk intersected points on the exter-nal contour of the selected slice, also computed usingthe contour following algorithm in (Suzuki and Abe,1985), i.e., qi(z) = p j(z)+

(pk(z)−p j(z)

)/2. The fi-

nal 3-D curve-skeleton J results from projecting backthe 3-D points qi from C2 to C1, i.e., J = TC1

C2· J =

(TC2C1)−1 · J .

4 Experimental results

In the following we evaluate the proposed curve-skeleton extraction approach on both real and syn-thetic data. In addition, we present a visualcomparison against the curve-skeleton extractionvia Laplacian-based contraction approach presentedin (Cao et al., 2010), and the runtime analysis. Allreported results have been obtained using a MobileIntel R© QM67 Express Chipset with an integratedgraphic card Intel R© HD Graphics 3000. The pro-posed approach has been implemented in C++ lan-guage using the OpenCV (Bradski and Kaehler, 2008)and PCL (PCL, 2014) libraries.

Four different 3-D models have been consid-ered to evaluate the proposed curve-skeleton extrac-tion approach, i.e., the standing Bill from the V-Rep repository (VRE, 2014), and the Nilin Combat,Iron Man and Minions, from the publicly availableTF3DMTM repository (TF3, 2014). The vast majorityof available 3-D models are built as textured polyg-onal models, being flexible and easily to be renderedby computers. However, the number of polygons isfar below the number of 3-D points of a real scannedobject; mainly within regions with uniform geome-try. Consequently, we have transformed the represen-tation of the considered 3-D models from polygonalmeshes to real 3-D point sets. To do so, we havesimulated a multi-view sensing system composed of4 RGB-D cameras using V-Rep (VRE, 2014), a very

Figure 5: Experimental environment using V-Rep.

versatile robot simulator tool in which the user canreplicate real scenarios, as shown in Fig. 5. Syntheticdata results from the registration of the depth data ac-quired by each virtual RGB-D camera.

Fig. 6 shows the scanned 3-D models (in purplecolor), after data registration and surface voxeliza-tion, and their respective extracted curve-skeletons.We note that the resulting curve-skeletons present thesame topology and geometry as the 3-D object; beingunit wide and centrally placed within the underlyingscanned surface.

We show, in Fig. 7, a visual comparison againstCao et al. approach (Cao et al., 2010). One of thelimitations of Cao et al. approach is the need of1-ring neighborhood to perform the Laplacian-basedcontraction. As a result, their approach fails in thepresence of large portions of data missing, as occursaround the pelvis in Fig. 7c and Fig. 7f. Furthermore,contraction and attraction weights must be manu-ally tunned in order to obtain a unit thickness curve-skeleton. Herein, we have considered the default set-tings provided by the authors. We also note that theconsumption time to extract each curve-skeleton isabove 1 minute.

Real data has been generated from a multi-viewsensing system composed of 2 consumer-accessibleRGB-D cameras, i.e., the Asus Xtion Pro Live cam-era, with opposed field-of-views, i.e., with no dataoverlapping. The relationship between the two cam-eras was determined using the stereo calibrationimplementation available in OpenCV (Bradski andKaehler, 2008) and a transparent checkerboard (blacksquares are visible from both sides). Better regis-tration approaches based on ICP, bundle adjustmentor the combination of both can also be considered.However, it is shown in Fig. 8 that the current ap-proach perfectly extracts the curve-skeleton on such acoarse registered point clouds, handling large portionsof missing data as well as registration inaccuracies.Fig. 8 presents 3 coarse registered point clouds of twomans and one woman. Note that despite the large por-

(a) (b) (c) (d)

(e) (f) (g) (h)

(i) (j) (k) (l)

(m) (n) (o) (p)

Figure 6: Curve-skeleton extraction from synthetic object-scanned point clouds. 1st row, Standing Bill(23650 points). 2nd row, Nilin Combat (18730 points). 3rd row, Iron Man (33358 points). 4rd row, Minions(18426 points).

tions of data missing due to self occlusions and inac-curate registration of both views, our approach is ableto extract an accurate curve-skeleton that preservesboth topology and geometry of the scanned model.

Table 1 reports the running time to extract thecurve-skeleton of the 3-D models presented in Fig. 6and Fig. 8. We note that we have reported CPU-based values without using data-parallel algorithms orgraphics hardware (GPU). Most of consumption timeis invested on determining the missing 3-D coordi-nate of each skeletal candidate whereas 2-D process-ing time is almost negligible, as it was expected.

4.1 Limitations and future work

The resulting curve-skeleton depends strongly on theDT computed from the front view of the human body.Therefore, occluded body parts due to crossing armsor legs will not be considered and can provide wrong

skeletal candidates. Furthermore, having one or botharms too close to the torso will make DT to considerthem as part of the torso. Similarly, if both legs are tooclose, they can be considered as a single one. We planto solve these body configurations by using both tem-poral and RGB information. Indeed, we herein pro-pose a very fast approach to estimate a coarse curve-skeleton that will be the basis to fit a human modelskeleton. We plan to use a progressive fitting of theskeleton model starting from the torso limb. In gen-eral, the torso limb corresponds to the lowest-valuededges from the Laplace operation output.

5 Concluding Remarks

A real-time approach to extract the curve-skeletonof an upright human-scanned point cloud has been de-scribed. Our main contribution is in estimating the

(a) (b) (c) (d) (e) (f)

(g) (h) (i) (j) (k) (l)

(m) (n) (o) (p) (q) (r)

Figure 7: Visual comparison against curve-skeleton extraction via Laplacian-based contraction (Cao et al., 2010).1st row, Standing Bill. 2nd row, Nilin Combat. 3rd row, Iron Man. 1st and 4th col., Input dataset. 2nd and 5th col.Resulting curve-skeleton using our approach. 3nd and 6th col. Resulting curve-skeleton using (Cao et al., 2010).

2-D skeleton candidates using image processing tech-niques. By doing so, we address the real-time con-straint. The final curve-skeleton results from locatingthe previous computed 2-D skeleton candidates in the3-D space. The resulting curve-skeleton preserves thecenteredness property as well as the same topologyand geometry as the 3-D model. Future work includesthe treatment of alternative body configurations thanupright. The use of temporal and RGB informationwill be also investigated in order to increase the ro-bustness of the approach.

Acknowledgements

This work was supported by the National Re-search Fund, Luxembourg, under the CORE project

C11/BM/1204105/FAVE/Ottersten.

REFERENCES

(2014). Point Cloud Library (PCL). http://pointclouds.org/.(2014). TF3DMTM. http://tf3dm.com/.(2014). Virtual robot experimentation platform (v-rep).

http://www.coppeliarobotics.com/.Au, O. K.-C., Tai, C.-L., Chu, H.-K., Cohen-Or, D., and

Lee, T.-Y. (2008). Skeleton extraction by mesh con-traction. In ACM SIGGRAPH 2008 Papers, pages44:1–44:10. ACM.

Bradski, G. and Kaehler, A. (2008). Learning OpenCV:Computer Vision with the OpenCV Library. O’ReillyMedia, 1st edition.

Cao, J., Tagliasacchi, A., Olson, M., Zhang, H., and Su, Z.(2010). Point Cloud Skeletons via Laplacian Based

(a) (b) (c) (d)

(e) (f) (g) (h)

(i) (j) (k) (l)

Figure 8: Curve-skeleton extraction from upright human-scanned point clouds. 1st row, Man 1 (18053 points).2nd row, Man 2 (21263 points). 3rd row, Woman 1 (12986 points).

Table 1: Running time to extract the curve-skeleton ofthe 3-D models presented in Fig. 6 (units are in ms).

3-D ModelRunning time

Orientation Silhouette D ∆D pi(z) Total

Standing Bill2.5 0.2 0.3 0.0 193.0 252.2

(23650 points)

Nilin Combat1.6 0.0 0.1 0.1 159.6 206.0

(18730 points)

Iron Man2.7 0.1 0.4 0.2 390.6 473.0

(33358 points)

Minions1.5 0.3 0.1 0.2 105.4 150.8

(18426 points)

Man 11.3 0.7 0.3 0.1 232.7 279.5

(18053 points)

Man 20.9 0.4 0.4 0.1 287.5 341.6

(21263 points)

Woman 10.4 0.1 0.3 0.1 109.3 142.1

(12986 points)

Contraction. In Shape Modeling International Con-ference (SMI), pages 187–197.

Cornea, N., Silver, D., and Min, P. (2007). Curve-skeleton

properties, applications, and algorithms. IEEE Trans-actions on Visualization and Computer Graphics,13(3):530–548.

Felzenszwalb, P. F. and Huttenlocher, D. P. (2004). Dis-tance transforms of sampled functions. Technical re-port, Cornell Computing and Information Science.

Fitzgibbon, A. and Fisher, R. B. (1995). A buyer’s guide toconic fitting. In In British Machine Vision Conference,pages 513–522.

Garcia, F. and Ottersten, B. (2014). CPU-Based Real-TimeSurface and Solid Voxelization for Incomplete PointCloud. In IEEE International Conference on PatternRecognition (ICPR), pages 2757–2762.

Li, M., Yang, T., Xi, R., and Lin, Z. (2009). Silhouette-based 2d human pose estimation. In InternationalConference on Image and Graphics (ICIG), pages143–148.

Sam, V., Kawata, H., and Kanai, T. (2012). A robustand centered curve skeleton extraction from 3d pointcloud. Computer-Aided Design and Applications,9(6):969–879.

Sezgin, M. and Sankur, B. (2004). Survey over imagethresholding techniques and quantitative performanceevaluation. Journal of Electronic Imaging, 13(1):146–168.

Suzuki, S. and Abe, K. (1985). Topological structural anal-ysis of digitized binary images by border following.

Computer Vision, Graphics, and Image Processing,30(1):32–46.

Tagliasacchi, A., Zhang, H., and Cohen-Or, D. (2009).Curve skeleton extraction from incomplete pointcloud. In ACM SIGGRAPH 2009 Papers, SIGGRAPH’09, pages 71:1–71:9. ACM.