![Page 1: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/1.jpg)
Skeltrack - Open Source Skeleton Tracking
Joaquim Rocha, IgaliaLinuxTag 2012 - Wunderbare Berlin
![Page 2: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/2.jpg)
Guten Tag!
✩ I am a developer at Igalia✩ I like doing innovative stuff like OCRFeeder and SeriesFinale✩ and today I am presenting my latest project: Skeltrack
![Page 3: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/3.jpg)
The Kinect
![Page 4: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/4.jpg)
Microsoft's Kinect was the first camerawith a price affordable to the public
![Page 5: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/5.jpg)
The USB connection is open and thus hackable
![Page 6: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/6.jpg)
This originated Open Source projects like the libfreenect,a library to control the Kinect device and get its information
![Page 7: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/7.jpg)
We created a GLib wrapper for libfreenect called GFreenect
![Page 8: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/8.jpg)
GFreenect offers asynchronous functions (and some synchronous aswell) and makes it easy to use with other GNOME technologies
![Page 9: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/9.jpg)
GObject Introspection = free bindings (Python, Javascript, Vala)
![Page 10: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/10.jpg)
Kinect has a structured light camera which gives depth information
![Page 11: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/11.jpg)
But that's raw information... values from 0-2048
![Page 12: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/12.jpg)
libfreenect/GFreenect can give those values in mm
![Page 13: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/13.jpg)
![Page 14: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/14.jpg)
Still...
![Page 15: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/15.jpg)
It does NOT tell you there is a person in the picture
![Page 16: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/16.jpg)
Or a cow
![Page 17: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/17.jpg)
Or an ampelmann
![Page 18: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/18.jpg)
Let alone a skeleton and where its joints are
![Page 19: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/19.jpg)
For this you need a skeleton tracking solution
![Page 20: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/20.jpg)
Three proprietary/closed solutions exist:
![Page 21: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/21.jpg)
Microsoft Kinect SDK: non-commercial only
![Page 22: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/22.jpg)
OpenNI: commercial compatible
![Page 23: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/23.jpg)
Kinect for Windows: commercial use allowedbut incompatible with the XBox's Kinect
![Page 24: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/24.jpg)
![Page 25: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/25.jpg)
Conclusion: There were no Free solutions toperform skeleton tracking... :(
![Page 26: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/26.jpg)
So Igalia built one!
![Page 27: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/27.jpg)
Enter Skeltrack
![Page 28: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/28.jpg)
What we wanted:✩ A shared library, no fancy SDK✩ Device independent✩ No pattern matching, no databases✩ Easy to use (everybody wants that!)
![Page 29: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/29.jpg)
Not as easy as it sounds!
![Page 30: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/30.jpg)
After some investigation we found Andreas Baak'spaper "A Data-Driven Approach for Real-Time FullBody Pose Reconstruction from a Depth Camera"
![Page 31: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/31.jpg)
However this paper uses a database ofposes to get what the user is doing
![Page 32: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/32.jpg)
So we based only part of our work on it
![Page 33: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/33.jpg)
How does it work?
![Page 34: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/34.jpg)
First we need to find the extremas
![Page 35: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/35.jpg)
Make a graph whose nodes are the depth pixels
![Page 36: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/36.jpg)
Connect two nodes if the distance is less than acertain value
![Page 37: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/37.jpg)
Connect the different graph's components by usingconnected-component labeling
![Page 38: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/38.jpg)
Choose a starting point and calculate Dijkstra toeach point of the graph; choose the furthest point.There you got your extrema!
![Page 39: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/39.jpg)
Then create an edge between the starting pointand the current extrema point with 0 cost andrepeat the same process now using the currentextrema as a starting point.
![Page 40: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/40.jpg)
This comes from Baak's paper and the differencestarts here: choosing the starting point
![Page 41: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/41.jpg)
Baak chooses a centroid as the starting point
We choose the bottom-most point starting from thecentroid (this showed better results for the upperbody extremas)
![Page 42: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/42.jpg)
So we got ourselves some extremas!What to do with them?
![Page 43: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/43.jpg)
What extrema is a hand, a head, a shoulder?
![Page 44: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/44.jpg)
For that we use educated guesses...
![Page 45: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/45.jpg)
We calculate 3 extremas
![Page 46: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/46.jpg)
Then we check each of them hoping they are the head
![Page 47: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/47.jpg)
How?
![Page 48: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/48.jpg)
For each extrema we look for the points in placeswhere the shoulders should be, checking their distancesbetween the extrema and between each other.
![Page 49: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/49.jpg)
If they obey those rules then we assume they arethe head'n'shoulders (tm)
![Page 50: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/50.jpg)
With the remaining 2 extremas, we will try to see ifthey are elbows or hands
![Page 51: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/51.jpg)
How to do it?
![Page 52: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/52.jpg)
Calculate Dijkstra from the shoulders to each extrema
![Page 53: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/53.jpg)
The closest extrema to any of the shoulders is either ahand of an elbow of that shoulder
![Page 54: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/54.jpg)
How to check if it's a hand or an elbow?
![Page 55: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/55.jpg)
If the distance between the extrema and the shoulder isless than a predefined value, then it is an elbow. Otherwiseit is a hand.
![Page 56: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/56.jpg)
If it is a hand, we find the elbow by choosing the first point(in the path we created with Dijkstra before) whose distanceexceeds the elbow distance mentioned before
![Page 57: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/57.jpg)
![Page 58: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/58.jpg)
There is still some things missing...
![Page 59: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/59.jpg)
Future work
![Page 60: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/60.jpg)
Hands from elbows: If one of the extremas is an elbow, weneed to infer where the hand is
![Page 61: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/61.jpg)
Smoothing: Smooth the jittering of the joints
![Page 62: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/62.jpg)
Robustness: Use restrictions to ignore objects that are notthe user
![Page 63: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/63.jpg)
Multi-user: Track more than one person at a time
![Page 64: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/64.jpg)
And of course, get the rest of the joints: hips, knees, etc.
![Page 65: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/65.jpg)
How to use it?
![Page 66: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/66.jpg)
Asynchronous API
![Page 67: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/67.jpg)
SkeltrackSkeleton *skeleton = SKELTRACK_SKELETON (skeltrack_skeleton_new ());skeltrack_skeleton_track_joints (skeleton, depth_buffer, buffer_width, buffer_height, NULL, on_track_joints, NULL);
![Page 68: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/68.jpg)
![Page 69: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/69.jpg)
Synchronous API
![Page 70: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/70.jpg)
SkeltrackJointList list;list = skeltrack_skeleton_track_joints_sync (skeleton, depth_buffer, buffer_width, buffer_height, NULL, NULL);
![Page 71: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/71.jpg)
Skeleton Joint:
ID: HEAD, LEFT_ELBOW, RIGHT_HAND, ...x: X coordinate in real world (in mm)y: Y coordinate in real world (in mm)screen_x: X coordinate in the screen (in pixels)screen_y: Y coordinate in the screen (in pixels)
![Page 72: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/72.jpg)
Code/Bugs: https://github.com/joaquimrocha/Skeltrack
![Page 73: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/73.jpg)
Nifty Tools for Development:
GFreenect: https://github.com/elima/GFreenect
GFreenect Utils: https://github.com/joaquimrocha/gfreenect-utils
![Page 74: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/74.jpg)
GFreenect Python Example
![Page 75: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/75.jpg)
Tool: record-depth-file
![Page 76: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/76.jpg)
Tool: depth-file-viewer
![Page 77: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/77.jpg)
Questions?
![Page 78: Skeltrack: A Free Software library for skeleton tracking (LinuxTag 2012)](https://reader035.vdocuments.mx/reader035/viewer/2022081400/555837e9d8b42ac6078b498e/html5/thumbnails/78.jpg)
Creative Commons pictures from flickr:Kinect: Auxo.co.krAmpelmann: echiner1Kid Playing: Rob WelshSkeleton: Dark Botxy