motionhub: middleware for unification of multiple body
TRANSCRIPT
MotionHub: Middleware for Unification of Multiple BodyTracking Systems
Philipp LadwigMixed Reality and Visualization
Group (MIREVI)University of Applied Sciences
Düsseldorf, [email protected]
Kester EversMixed Reality and Visualization
Group (MIREVI)University of Applied Sciences
Düsseldorf, [email protected]
Eric J. JansenMixed Reality and Visualization
Group (MIREVI)University of Applied Sciences
Düsseldorf, [email protected]
Ben FischerMixed Reality and Visualization
Group (MIREVI)University of Applied Sciences
Düsseldorf, [email protected]
David NowottnikMixed Reality and Visualization
Group (MIREVI)University of Applied Sciences
Düsseldorf, [email protected]
Christian GeigerMixed Reality and Visualization
Group (MIREVI)University of Applied Sciences
Düsseldorf, [email protected]
Figure 1: MotionHub is an open-sourcemiddleware that offers interfaces tomultiple body tracking systems. a) Two users werecaptured and tracked by a Microsoft Azure Kinect. b) The graphical user interface of MotionHub. The green cubes representan OptiTrack recording while the yellow ones represent an Azure Kinect live capture. c) MotionHub streams unified skeletalrepresentation in real time to clients such as the Unity game engine via a plug-in.
ABSTRACTThere is a substantial number of body tracking systems (BTS), whichcover a wide variety of different technology, quality and price rangefor character animation, dancing or gaming. To the disadvantageof developers and artists, almost every BTS streams out differentprotocols and tracking data. Not only do they vary in terms of scaleand offset, but also their skeletal data differs in rotational offsetsbetween joints and in the overall number of bones. Due to thiscircumstance, BTSs are not effortlessly interchangeable. Usually,software that makes use of a BTS is rigidly bound to it, and a change
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from [email protected] ’20, July 15–17, 2020, Jersey City/ Virtual, NJ, USA© 2020 Copyright held by the owner/author(s). Publication rights licensed to theAssociation for Computing Machinery.ACM ISBN 978-1-4503-7505-4/20/07. . . $15.00https://doi.org/10.1145/3401956.3404185
to another system can be a complex procedure. In this paper, wepresent our middleware solutionMotionHub, which can receive andprocess data of different BTS technologies. It converts the spatialas well as the skeletal tracking data into a standardized format inreal time and streams it to a client (e.g. a game engine). That way,MotionHub ensures that a client always receives the same skeletal-data structure, irrespective of the used BTS. As a simple interfaceenabling the user to easily change, set up, calibrate, operate andbenchmark different tracking systems, the software targets artistsand technicians. MotionHub is open source, and other developersare welcome to contribute to this project.
CCS CONCEPTS• Computing methodologies → Motion processing; • Infor-mation systems→ Multimedia streaming;
KEYWORDSBody tracking, middleware, skeletal data, motion capture, AzureKinect, OptiTrack
MOCO ’20, July 15–17, 2020, Jersey City/ Virtual, NJ, USA Ladwig et al.
ACM Reference Format:Philipp Ladwig, Kester Evers, Eric J. Jansen, Ben Fischer, David Nowottnik,and Christian Geiger. 2020. MotionHub: Middleware for Unification of Mul-tiple Body Tracking Systems. In 7th International Conference on Movementand Computing (MOCO ’20), July 15–17, 2020, Jersey City/ Virtual, NJ, USA.ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3401956.3404185
1 INTRODUCTIONReal-time body tracking is an indispensable part of many interac-tive art performances, mixed-reality applications and games. Dueto the progress in research, higher computing power and advancesin sensor technology, a large number of BTSs that are suitable forsuch applications have been developed in recent years. Develop-ers are spoilt for choice: They have to select a system and mustconsider various advantages and disadvantages concerning price,accuracy and size of the tracking area. However, in many casesnot all requirements are known at the beginning of a project andwill only be developed over time. This can pose a challenge as asubsequent change to another tracking system can be costly. Insuch a scenario, a middleware that allows for an effortless switchbetween different BTSs would be useful.
The term ’middleware’ was probably first used in 1968 [35]. Sincethen, various sub types have been described, but the basic meaninghas never changed: A middleware receives, processes and transmitsinformation between at least two peers. Middlewares are often ableto understand multiple information representations and to translatethem into a standardized or unified one.
The contribution of this paper is the open-source middleware’MotionHub’ and an open-source game engine plugin, which canbe seen in Fig. 1b and c. The intention behind our work is to unifythe output data of the wide variety of available BTSs. We believethat a unification will not only lead to an easier switch betweenBTSs during production but that it will also reduce the preparationtime for developments that involve BTSs. Integrating a BTS into anexisting application requires a manual effort. Then, if developerswant to integrate another BTS or exchange the previous one withit, an additional effort must be considered. In some cases, an updateto a newer version of a BTS is desired, which produces an evenhigher manual effort. Our intention is to integrate a generic BTSprotocol into applications once in order to reduce the overall timespent for maintenance, the setup and the switch between differentBTSs.
Beyond this, more benefits can be mentioned. If a middlewareunderstands different BTSs, it is possible to prioritize and to switchbetween them automatically to exploit the advantages of each sys-tem. A possible scenario could be the use of a BTS that is accuratebut limited in tracking space and a different one that has a largertracking space but is less accurate. A middleware could prioritizethe more accurate BTS as long as the tracked person stays in the’accurate’ tracking space and automatically switches to the ’lessaccurate’ space whenever he or she leaves it.
Despite the advantages, the concept of the MotionHub also has amajor drawback: By nature, a middleware induces a delay becauseit receives, converts and resends the data between two ends, whichrequires a certain processing time. Therefore, this aspect receivesspecial attention in this work and we evaluate it at the end of thispaper.
A demo video with a summary of MotionHub’s capabilities canbe watched here: https://youtu.be/7caURXod-ag.
The Git repository can be found here: https://github.com/Mirevi/MotionHub.
2 RELATEDWORKBody tracking is a wide field with many years of research anddevelopment, and it has produced a large number of different soft-ware and hardware approaches as well as standardizations and fileformats. In this section, we only mention the most common andimportant ones for MotionHub.
2.1 Standards and File FormatsThe probably best known and oldest de facto standard for dataexchange in virtual reality is the Virtual-Reality Peripheral Network(VRPN) by Taylor et al. [42]. It offers simple and unified interfacesto a broad range of devices of different manufacturers. Many ofthese devices share common functionalities such as 6-DOF trackingor button input while the way of accessing these functions differsbetween manufacturers. VRPN unifies functions across differentdevices as generic classes such as vrpn_Tracker or vrpn_Button.Therefore, it can be seen as both a standard and a middleware. Theapproach of MotionHub is similar, but it focuses on body tracking.
The first official international standard for humanoid animationis the H-Anim, which was created within the scope of the Extensible3D (X3D) standard and is a successor of the Virtual Reality Model-ing Language (VRML) [13, 14]. H-Anim was published in 2006 [15],updated in 2019 [16, 17] and is one of the only efforts yet to createan official open standard for humanoid avatar motion and dataexchange.
COLLADA and FBX are interchange file formats for 3D appli-cations and are widely used today. While humanoid animation isnot the focus of COLLADA, its open and versatile structure enablesdevelopers to save body tracking data. Compared to COLLADA, theproprietary FBX format mainly focuses on motion data but lacks aclear documentation, which has led to incompatible versions.
While COLLADA and FBX can also be used for writing andreading 3D geometry, the Biovision Hierarchy (BVH) file formatwas developed exclusively for handling skeletal motion data andis therefore simpler in structure. It is supported by many bodytracking applications and, because of its simplicity and less over-head compared to other file formats, it is often used for real-timetransmission of humanoid motion data. A deeper and more com-prehensive overview of further tracking file formats is given byMeredith and Maddock [30].
2.2 Software and HardwareMicrosoft started shipping the Kinect in 2010 and thereby allowedthe body tracking community to grow substantially since the Kinectwas an affordable and capable sensor. During this time, PrimeSensemadeOpenNI [39] andNITE [39, p.15] publicly available. OpenNI of-fers low-level access to the Microsoft Kinect and other PrimeSensesensors, while NITE was a middleware that enables the user toperform higher-level operations such as body tracking and gesturerecognition. PrimeSense stopped the distribution of OpenNI andNITE after having been acquired by Apple Inc. Although OpenNi
2
MotionHub: Body Tracking Middleware for Unification of... MOCO ’20, July 15–17, 2020, Jersey City/ Virtual, NJ, USA
is still available on other web pages [39], there is no active develop-ment anymore.
Based on the success of the Kinect and other PrimeSense sen-sors, many proprietary BTSs such as iPi Mocap Studio [27], BrekelBody [2] or Nuitrack [21], which are based on RGB-D streams, havebeen developed. Examples for BTSs that use IMU-based tracking areXsens MVN [47], Perception Neuron [37] or Rokoko Smartsuit [12].Optical solutions that are based on passive or active (infrared)markers include OptiTrack Motive:Body [24], Qualisys Track Man-ager [1], ART-Human [18], Vicon Tracker [28] and Motion AnalysisCortex [23].
The concept ofOpenVR [29], which is implemented in SteamVR [6],and OpenXR [26] is similar to the idea behind MotionHub. It unifiesthe interfaces of a large number of different types of mixed-realityinput and output devices to a single library. The fact that it isopen source is probably an important reason for its success. WhileOpenVR and OpenXR do not focus on body tracking, there are ef-forts to extend their functionalities. For example, IKenima [20] hasdeveloped an application named Orion, which is based on inversekinematics and utilizes HTC Vive trackers, which are attached tothe hip and the feet. IKinema was acquired by Apple Inc. in 2019,and they stopped the distribution of Orion.
MiddleVR [33] is a proprietary middleware that supports variouscommercial input and output devices and offers a generic interfaceto the Unity game engine [43]. Body tracking is supported, but itis not the focus of the software. The commercial software mostclosely related to MotionHub is Reallusion iClone with its plug-inMotion LIVE [25]. It focuses on real-time body tracking and supportsseveral different tracking systems for face, hand and body capture.
2.3 ResearchThe research that is most closely related to our work is Open-Tracker [40] and Ubiquitous Tracking (UbiTrack) [36, 45]. Both sys-tems are generic data flow network libraries for different trackingsystems. Unlike MotionHub, they do not focus solely on body track-ing. They offer a generic interface to object tracking systems formixed-reality applications similar to the idea behind VRPN. Al-though OpenTracker and UbiTrack have a different focus thanMotionHub and the research was conducted more than 16 yearsago, the concept of unification is similar and reusable for our work.
Suma et al. [41] have developed the FAAST (Flexible Action andArticulated Skeleton Toolkit), which is based on OpenNI and NITE.It provides a VRPN server for streaming the user’s skeleton jointsover a network. However, the development is discontinued. Damas-ceno et al. [8, 9] present a middleware for multiple low-cost motioncapture devices that is applied to virtual rehabilitation. A similarsystem is suggested by Eckert et al. [10] for the use of playingexergames.
OpenPTrack [34] is one of the most recent systems. It is not de-scribed as a middleware itself but rather as a BTS. However, sinceOpenPTrack supports the processing of multiple RGB-D camerastreams of various manufacturers and uses different tracking al-gorithms (such as its own or OpenPose [4]), it rather acts as amiddleware. Similar to MotionHub, OpenPTrack is open source andfocuses mainly on body tracking. The difference between the twosystems is that OpenPTrack solely concentrates on working with
RGB-D streams while MotionHub is based on the intention of in-cluding different tracking technologies such as optical, IMU-basedor magnetic tracking in order to exploit the specific advantagesof each individual technology. Therefore, our approach requires amore generic approach to be able to fuse, calibrate and merge theheterogeneous data of different BTSs.
To the authors’ knowledge, there is no middleware availabletoday that supports high-cost and low-cost BTSs as well as recentsystems and different technologies (not only RGB-D streams) and isopen source. Although the concept of a body tracking middleware isnot new, MotionHub meets the mentioned aspects and is therefore,to our knowledge, a unique system with a valuable contribution tothe community.
3 SYSTEMMotionHub receives raw skeletal data from different BTSs andprocesses it to create a unified skeleton in real time. The raw datais generated by the respective software development kits (SDKs)of each BTS. Each system uses different transmission methodsand protocols, hierarchy structures, units, coordinate systems anddifferent numbers of joints as well as rotation offsets between joints.To perform a correct transformation to the unified skeleton ofthe MotionHub, each input type requires a specific procedure forreceiving, transforming and mapping data. As soon as the data isavailable in its proper format, it is transmitted to the MotionHubclient via the Open Sound Control (OSC) [46] protocol.
3.1 Unified SkeletonIn order to unify the output data of different BTSs, we transform theheterogeneous skeletal-data structures into a generic skeleton rep-resentation consisting of 21 joints, as shown in Fig. 2. The structureof the joints is based on the Unity humanoid avatar skeleton [44], issimilar to the H-Anim LAO-1 standard [3, p.20] and is also used by
HEAD
TOE_R
FOOT_R
LEG_R
UPLEG_R
TOE_L
FOOT_L
LEG_L
UPLEG_L
HAND_RARM_R FOREARM_R
SHOULDER_RHAND_L
ARM_LFOREARM_L
SHOULDER_LNECK
CHEST
SPINE
HIPS
Figure 2: The unified skeleton structure streamed byMotion-Hub.
3
MOCO ’20, July 15–17, 2020, Jersey City/ Virtual, NJ, USA Ladwig et al.
BTSs such as Azure Kinect. This standard of joint names and jointindices allows for a structured implementation of the transformedskeletal data for avatar animation in external applications. Eachjoint is represented by a global position (Vector3), a global rotation(Quaternion) and a confidence value (Enum [none, low, mediumand high]) in a right-handed coordinate system. All raw skeletaldata is processed based on this unification, which will be discussedin detail later on.
Some BTSs like Azure Kinect provide joint confidence valuesbased on distance and visibility. For joints of BTSs that do notprovide confidence values, for example OptiTrack, we use ’high’ asdefault value.
3.2 Subsystem ArchitectureEach BTS has its own native capture frequency and update rate.In order to receive and process incoming data as fast as possible,the data processing code needs to be executed independently fromthe main program loop. As a solution to this problem, we incorpo-rated autonomous threads for each BTS within the MotionHub. Atracking thread receives raw skeletal data from the respective SDK,processes it and immediately sends it to the (game engine) client.While using threads, it is important to protect memory areas fromsimultaneous access with appropriate ’atomic’ structures. Manyconsumers such as tracker threads, the UI thread, the render threador the network sender thread access protected areas. We experi-enced a better performance and a lower latency of processing andsending data when protected memory was copied to pre-allocatedareas first, with a subsequent processing on the copy instead oflocking critical sections during the time of processing.
To stream skeletal data to the game engine client, we use theOSC protocol [46] because of its simplicity. The structure of ourprotocol is similar to the Biovision Hierarchy (BVH) file formatbut is extended with further MotionHub-specific control messages,which are necessary for the communication between MotionHuband the client side. Furthermore, we created a skeletal-data rep-resentation that is more compact than BVH or VRPN-based datastreams in order to further reduce the network latency. As ourlower-level protocol we prioritize UDP over TCP because we prefera fast connection and low latency over packet loss recovery. EachOSC package consists of translation (three float values) and rota-tion (four float values) data for each joint as well as the skeletonID (integer). Furthermore, a render window is integrated into theUI module to visualize incoming as well as transformed skeletonrepresentations. Skeleton joints are rendered in different colorsdepending on their confidence values, as shown in Fig. 3.
3.3 Dependencies and Build SystemMotionHub is written in C++ because of two reasons: First, C++ isconsidered as one of the fastest programming languages [19], whichis an important aspect for a real-time body tracking middleware.The second reason is that building an interface to numerous SDKsis only possible in C++, because a significant number of BTSs onlyprovide a SDK that is based on this programming language.
Because Microsoft’s body tracking SDK [31, 32] is based on neu-ral networks, it requires the NVIDIA CUDA®Deep Neural Networklibrary (cuDNN) [38]. For handling matrix, vector and quaternion
Figure 3: The OpenGL render window in a) indicates thecurrent tracking confidence of joints by the aid of color.Some BTSs, such as Azure Kinect, provide such values. Yel-low joints have a ’medium’ confidence valuewhile red jointshave ’low’ confidence, because they are occluded, as can beseen in b). For BTSs that do not deliver a confidence value,a default value can be selected manually. In this figure, therecording of OptiTrack’s value is ’high’ by default and ren-dered in green.
calculations, the MotionHub uses Eigen [11]. The user interface iscreated with the Qt 5 framework [5].
MotionHub aims to be as open as possible, and it automaticallydownloads and configures several software dependencies whenbuilding with CMake [22]. This fact significantly reduces the re-quired amount of work for developers and allows for an easierdevelopment in the future. Binaries are also available. We havechosen Windows as our target operating system, because manySDKs are only available for this system.
3.4 User InterfaceMotionHub’s main UI can be seen in teaser Fig. 1b. A detailed viewof the right side of the UI is shown in Fig. 4. The white numbers 1and 2 indicate the buttons for adding and removing BTSs. Number3 is a toggle button, which starts or stops the global tracking loop.All BTSs can be started or stopped together or individually.
As shown in Fig. 4 below number 4, selecting a tracker in thelist (orange box) will display its properties in the Tracker PropertyInspector. Different tracking systems have different coordinate sys-tems. For example, OptiTrack has its center in the middle of thetracking area on the floor while the origin of the Azure Kinect liesinside of the camera. When multiple BTSs are combined in one
4
MotionHub: Body Tracking Middleware for Unification of... MOCO ’20, July 15–17, 2020, Jersey City/ Virtual, NJ, USA
Figure 4: MotionHub user interface: Tracker Control Panel(1–3) and Tracker Property Inspector window (4).
tracking area, their origins must be spatially merged. Therefore,the user can offset the coordinate system origin of different BTSsin the Tracker Property Inspector.
3.5 Conversation MatricesProcessing joint position data to the MotionHub’s unified coor-dinate space is done in multiple steps: applying translation, rota-tion and scale offsets to merge tracking spaces and, if necessary,mirroring the correct axes. Rotations of joints, however, are mostcomplicated and differ between BTSs. A list of specific rotations ofthe implemented BTSs can be found online in MotionHub’s docu-mentation: https://github.com/Mirevi/MotionHub. In addition tothe coordinate system, we had to consider the skeletal structure.In some BTSs the skeleton is structured in a hierarchy so that thejoint rotations are in local coordinate spaces. These local values aretransformed to global rotations by the MotionHub before they aretransmitted to the receiver side. For example, each joint rotation ofthe Azure Kinect system is offset by different values.
𝑅𝑖 = 𝐼𝑖𝑂−1𝑖 𝑇
The output rotation quaternion R for all joints i of a tracker is theproduct of the tracker-specific global coordinate system transfor-mation T, the inverse global offset orientation O, and the raw inputrotation I.
Figure 5: The data flow (image template kindly provided byDaemen et al. [7]).
5
MOCO ’20, July 15–17, 2020, Jersey City/ Virtual, NJ, USA Ladwig et al.
Figure 6: TheUnity game engine plug-in: a) shows the avatarview in the Unity Renderer. b) shows the debug view display-ing the position and rotation of all joint axes.
3.6 Game engine clientTo be able to receive skeletal data in a game engine, we developed areceiver package for the Unity engine. It contains code that createsa character for each received skeleton and animates it with givenrotation values shown in Fig. 6. Character animation is solved inthe plug-in code by the multiplication of all inverse joint-rotationvalues of the character in a hierarchy order and the joint rotationin T-pose.
𝑅𝑖 (𝑐𝑙 𝑖𝑒𝑛𝑡 ) = 𝐼𝑖𝑇𝑖
𝑛∏𝑘=𝑗 (𝑖)+1
𝑟−1𝑓 (𝑖,𝑘)
For all joints i, the transmitted rotation quaternion I is multipliedby the character’s joint rotation in T-pose T and the product of allinverse joint rotations r in the skeleton hierarchy above the currentjoint. While f(i,k) returns the joint which is k nodes above i in thehierarchy, j(i) shows on which hierarchy level the joint is located.The process iterates through the joint hierarchy upwardly, startingwith the parent of the joints and ending with the root joint, withn representing the number of iterations. Afterwards, the productquaternion R is applied to the character’s local rotation.
During the development of a plug-in and the integration of aBTS into MotionHub, it is critical to preview the processed data inorder to identify and debug rotation offsets on different axes. Tofacilitate this process in Unity, our plug-in is able to visualize debugoptions, as can be seen in Fig. 6b. These options include the togglefor the display of the skeleton ID, avatar position, joint axes, jointnames and avatar meshes. The skeleton ID is equal to the internalone of the MotionHub and is transferred to the client by OSC datapackages. Moreover, the plug-in is designed to be avatar mesh
Figure 7: Playing ’Human Tetris’ with the MotionHub. a)shows the physical set-up with OptiTrack and Azure Kinect.b) shows our evaluation game created in Unity 3D.
independent. This means that a switch between different skeletonsis possible without code changes as long as Unity recognizes themas humanoid.
4 EVALUATIONIn order to verify the practical benefit of the MotionHub during thedevelopment and production of an application, we created an inter-active game called ’Human Tetris’. It incorporates the movementsof two users. The procedure is as follows: Player #1 starts with arandom body pose. The shadow of this pose is orthogonally pro-jected onto a wall and is freezed when Player #1 verbally confirmsthat he or she is ready. Next, player #2 needs to prepare and tryto copy the body posture and shadow of player #1 while the wallvisually approaches the players. As soon as the wall reaches player#2, the game calculates and displays a score depending on how wellthe posture was imitated. The game is shown in Fig. 7, and a videoof the game can be watched here: https://youtu.be/O_5hiweZQhE
4.1 ProceduresWe conducted three tests: 1.) playing Human Tetris, 2.) measuringthe time for switching between BTSs and 3.) measuring the delayinduced by MotionHub. For the first test, we played two rounds ofHuman Tetris and incorporated two Azure Kinects as well as anOptiTrack motion tracking system with 24 cameras (12 x Prime 13and 12 x Prime 17W) and a suit with 50 passive markers. We ran oneinstance of each the game, OptiTrack Motive and the MotionHub,
6
MotionHub: Body Tracking Middleware for Unification of... MOCO ’20, July 15–17, 2020, Jersey City/ Virtual, NJ, USA
Table 1: Delays between physicalmotion and recognizedmo-tion by two specific BTSs (MH stands for MotionHub).
System @30Hz With MH Without MH Induced delay
OptiTrack 127ms 114ms 13msAzure Kinect 222ms 151ms 71ms
which receives the data of three BTSs simultaneously. Both AzureKinects and OptiTrack Motive ran on the same PC (Intel i7 6700K,Nvidia GTX 1080). The communication between the BTSs and Mo-tionHub was conducted via localhost. First, we played a round withonly one Azure Kinect and OptiTrack while we covered the secondKinect in order to prove and illustrate for the demo video that thissensor is deactivated. At the beginning of the second round, the Op-tiTrack player left the tracking area, and another player uncoveredthe Kinect and joined the game. After the first test, we gatheredqualitative data from the players in a non-structured interview.
The second test encompassed measuring the time required forswitching between two BTSs inMotionHub.We utilized our HumanTetris game to conduct the test and switched between OptiTrackand Kinect.
In the last test, we recorded the visual response times with ahigh-speed camera to be able to measure the induced delay. There-fore, we recorded the time between a physical movement and therecognition of this movement for each BTS. The test was conductedwithout MotionHub (visual responses in Motive and Kinect BodyTracking Viewer) and with MotionHub (visual response in the Mo-tionHub render window and in the Unity game engine renderer).MotionHub and the Unity game engine were executed on the samePC (Intel i7 6700K, Nvidia RTX 2080) and were connected by alocalhost connection via UDP. We used a computer monitor with arefresh rate of 144Hz and the camera captured with 1000 framesper second (Sony DSC-RX10M4). OptiTrack and Kinect workedwith an update rate of 30Hz (33.3ms each). Any higher updaterates of the OptiTrack system results in a significant decrease of theKinect’s tracking quality due to infrared light interference with theOptiTrack system. We used Motive:Body v2.0.2 and Azure KinectBody Tracking SDK v1.0.1.
4.2 ResultsThe first test showed that the concept of MotionHub is applicable.The players were familiar with both BTSs and reported no unex-pected behavior of the tracking systems except for a slightly higherdelay of the Azure Kinect.
In the second test, the measured time for switching between twodifferent BTSs (from OptiTrack to Azure Kinect) was 8 seconds.
For the third test, the results of the measurements of the induceddelay, which we conducted with the high-speed camera, can be seenin Table 1. Previous tests had showed significantly higher delaysdue to coupling the frequency of sending out data packets withthe main application loop of the MotionHub. Shorter delays wereachieved by independent tracker threads that send out new dataas soon as it is available (without waiting for the main applicationloop, which also draws the user interface and processes user input).This solution is described in more detail in section 3.2.
We assume that the difference of the induced delay by Motion-Hub between OptiTrack (13ms) and Azure Kinect (71ms) is causedby different refresh rates of the underlying SDKs. OptiTrack Motiveand the thread of NatNetSDK within MotionHub exchange datawith more than 240Hz (although the cameras only capture with30Hz) while the Azure Kinect Body Tracking SDKs exchange datawith 30Hz.
We did not mention the delay between the renderers of Motion-Hub and Unity in Table 1 because we were not able to visuallyidentify a difference on the camera images. Both renderers showedsynchronized output and therefore the same delay. However, inorder tomake a statement about the network delay betweenMotion-Hub and the game engine, we used the timestamps of the networkpackets for measuring the time required for sending and receivingan UDP packet on the same PC (localhost), and it constantly stayedbelow 1 ms even when MotionHub was sending data.
5 POSSIBILITIES AND FUTUREWORKThe current status and the concept of MotionHub can be widelyextended. For example, a node-based network structure would bepossible, in which multiple MotionHub instances with differentBTSs could work together simultaneously. Each MotionHub nodecould stream tracking data to a master node that merges the track-ing data into a common coordinate space and performs sensorfusion. Possible applications of such a structure could be improvingthe tracking quality when tracking a single person or realizing thetracking of a large number of people at the same time. In order toaccomplish this, a reliable method of calibrating and spatially merg-ing the coordinate systems of different BTSs must be found. Whilemerging different RGB(-D) cameras is a well-studied problem, itremains an open question how to successfully merge BTSs with dif-ferent technologies such as IMU-based tracking and optical-markertracking. Future research is required in this regard.
Another feature of the concept of MotionHub could be to addfurther tracking modules such as face or hand tracking. Manymanufacturers and research publications focus on either body, faceor hand tracking. With some minor alterations, MotionHub couldbe extended to include the tracking of these body areas.Furthermore, the acquisition of data for machine learning couldalso be realized because (unified) data of different sensors couldbe acquired simultaneously within one application. Moreover, thisfunctionality could also be used for benchmarking different systemssimultaneously.
6 CONCLUSIONSWe have presented and evaluated our open-source system Motion-Hub. It allows for merging tracking data of different body trackingsystems into a standardized skeleton, which can be streamed toclients such as a game engine. The MotionHub induces an addi-tional delay of 13ms for a marker-based optical tracking system(OptiTrack) and 71ms for a markerless optical tracking system(Azure Kinect). Our system enables its users to change the trackingsystem on the fly and without further configurations within thereceiver side. MotionHub has pointed out the feasibility of promis-ing features to extend and combine the possibilities of availableBTSs. In the future, we intend to further develop those features. We
7
MOCO ’20, July 15–17, 2020, Jersey City/ Virtual, NJ, USA Ladwig et al.
plan to add a semi-automatic calibration procedure for matchingdifferent coordinate systems between BTSs and extend MotionHubwith the support of more BTSs such as OpenPTrack [34], Open-Pose [4] and others. We hope to contribute a valuable solution forthe community.
ACKNOWLEDGMENTSWe thank the MIREVI Group and the ’Promotionszentrum Ange-wandte Informatik’ (PZAI) in Hessen, especially Ralf Dörner. Thisproject is sponsored by: German Federal Ministry of Educationand Research (BMBF) under the project numbers 16SV8182 and13FH022IX6. Project names: HIVE-Lab (Health Immersive VirtualEnvironment Lab) and Interactive body-near production technol-ogy 4.0 (german: ’Interaktive körpernahe Produktionstechnik 4.0’(iKPT4.0)).
REFERENCES[1] Qualisys AB. 2020. Qualisys Track Manager (QTM). (2020). Retrieved February
24, 2020 from https://www.qualisys.com/software/qualisys-track-manager/[2] Brekel. 2020. Affordable tools for Motion Capture & Volumetric Video. (2020).
Retrieved February 24, 2020 from https://brekel.com/[3] Don Brutzman. 2006. Humanoid Animation (H-Anim). (2006). Retrieved Febru-
ary 29, 2020 from https://x3dgraphics.com/slidesets/X3dForAdvancedModeling/HumanoidAnimation.pdf
[4] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2018.OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields.CoRR abs/1812.08008 (2018). arXiv:1812.08008 http://arxiv.org/abs/1812.08008
[5] The QT Company. 2020. Qt 5. (2020). Retrieved February 12, 2020 fromhttps://www.qt.io/
[6] Valve Corporation. 2020. SteamVR on Steam. (2020). Retrieved February 26,2020 from https://store.steampowered.com/app/250820/SteamVR
[7] Jeff Daemen, Jens Herder, Cornelius Koch, Philipp Ladwig, RomanWiche, and KaiWilgen. 2016. Semi-Automatic Camera and Switcher Control for Live Broadcast.In Proceedings of the ACM International Conference on Interactive Experiences forTV and Online Video (TVX ’16). Association for Computing Machinery, New York,NY, USA, 129–134. https://doi.org/10.1145/2932206.2933559
[8] Eduardo Damasceno. 2014. A Motion Capture Middleware for Exergames Incresethe precision with Neural Nets. (06 2014).
[9] E. F. Damasceno, A. Cardoso, and E. A. Lamounier. 2013. An middleware formotion capture devices applied to virtual rehab. In 2013 IEEE Virtual Reality (VR).171–172. https://doi.org/10.1109/VR.2013.6549417
[10] M. Eckert, I. Gómez-Martinho, J. Meneses, and J. F. M. Ortega. 2016. A modularmiddleware approach for exergaming. In 2016 IEEE 6th International Conferenceon Consumer Electronics - Berlin (ICCE-Berlin). 169–173. https://doi.org/10.1109/ICCE-Berlin.2016.7684747
[11] Eigen. 2020. C++ template library for linear algebra. (2020). Retrieved February21, 2020 from http://eigen.tuxfamily.org
[12] Rokoko Electronics. 2020. Smartsuit Pro - Motion Capture Suit. (2020). RetrievedFebruary 24, 2020 from https://www.rokoko.com/en/products/smartsuit-pro
[13] International Organization for Standardization. 1997. ISO/IEC 14772-1:1997.https://www.iso.org/standard/25508.html. (1997).
[14] International Organization for Standardization. 2003. ISO/IEC 14772-1:1997/AMD1:2003. https://www.iso.org/standard/30998.html. (2003).
[15] International Organization for Standardization. 2006. ISO/IEC 19774:2006.https://www.iso.org/standard/33912.html. (2006).
[16] International Organization for Standardization. 2019. ISO/IEC 19774-1:2019.https://www.iso.org/standard/64788.html. (2019).
[17] International Organization for Standardization. 2019. ISO/IEC 19774-2:2019.https://www.iso.org/standard/64791.html. (2019).
[18] Advanced Realtime Tracking GmbH. 2020. ART Human. (2020). RetrievedFebruary 24, 2020 from https://ar-tracking.com/products/software/art-human/
[19] Isaac Gouy. 2020. The Computer Language Benchmarks Game. (2020).Retrieved February 27, 2020 from http://benchmarksgame.wildervanck.eu/which-programs-are-fastest.html
[20] IKinema. 2020. IKinema Orion. (2020). Retrieved February 25, 2020 fromhttps://youtu.be/Khoer5DpQkE
[21] 3DiVi Inc. 2020. Nuitrack Full Body Skeletal Tracking Software. (2020). RetrievedFebruary 24, 2020 from https://nuitrack.com/
[22] Kitware Inc. 2020. CMake. (2020). Retrieved February 27, 2020 from https://cmake.org/
[23] Motion Analysis Inc. 2020. Cortex Software. (2020). Retrieved February 24, 2020from https://www.motionanalysis.com/software/cortex-software/
[24] NaturalPoint Inc. 2020. OptiTrack Motive:Body. (2020). Retrieved February 24,2020 from https://optitrack.com/products/motive/body/
[25] Reallusion Inc. 2020. iClone Motion LIVE. (2020). Retrieved June 07, 2020 fromhttps://mocap.reallusion.com/iclone-motion-live-mocap/default.html
[26] The Khronos Group Inc. 2020. OpenXR. (2020). Retrieved February 25, 2020from https://www.khronos.org/openxr/
[27] iPi Soft LLC. 2020. iPi Soft- Markerless Motion Capture. (2020). RetrievedFebruary 24, 2020 from http://ipisoft.com/
[28] Vicon Motion Systems Ltd. 2020. Tracker - Delivery Precise Real-World Data| Motion Capture Software. (2020). Retrieved February 24, 2020 from https://www.vicon.com/software/tracker/
[29] Joe Ludwig. 2020. OpenVR. (2020). Retrieved February 25, 2020 from https://github.com/ValveSoftware/openvr
[30] Michael Meredith and Steve Maddock. 2001. Motion Capture File Formats Ex-plained. Production (01 2001).
[31] Microsoft. 2020. Azure Kinect Body Tracking SDK. (2020). RetrievedFebruary 12, 2020 from https://docs.microsoft.com/en-us/azure/kinect-dk/body-sdk-download
[32] Microsoft. 2020. Azure Kinect Sensor SDK. (2020). Retrieved February 12, 2020from https://github.com/microsoft/Azure-Kinect-Sensor-SDK
[33] MiddleVR. 2020. MiddleVR SDK. (2020). Retrieved February 10, 2020 fromhttps://www.middlevr.com/middlevr-sdk/
[34] Matteo Munaro, Filippo Basso, and Emanuele Menegatti. 2016. OpenPTrack.Robot. Auton. Syst. 75, PB (Jan. 2016), 525–538. https://doi.org/10.1016/j.robot.2015.10.004
[35] Peter Naur and Brian Randell. 1969. Software Engineering: Report of a ConferenceSponsored by the NATO Science Committee, Garmisch, Germany, 7-11 Oct. 1968,Brussels, Scientific Affairs Division, NATO.
[36] J. Newman, M. Wagner, M. Bauer, A. MacWilliams, T. Pintaric, D. Beyer, D.Pustka, F. Strasser, D. Schmalstieg, and G. Klinker. 2004. Ubiquitous tracking foraugmented reality. In Third IEEE and ACM International Symposium on Mixedand Augmented Reality. 192–201. https://doi.org/10.1109/ISMAR.2004.62
[37] Noitom. 2020. Perception Neuron Motion Capture. (2020). Retrieved February24, 2020 from https://neuronmocap.com/
[38] NVIDIA. 2019. NVIDIA CUDA® Deep Neural Network library. (2019). RetrievedFebruary 12, 2020 from https://developer.nvidia.com/cudnn
[39] Inc. Occipital. 2020. OpenNI 2 SDK. (2020). Retrieved February 10, 2020 fromhttps://structure.io/openni
[40] Gerhard Reitmayr and Dieter Schmalstieg. 2005. OpenTracker: A Flexible Soft-ware Design for Three-Dimensional Interaction. Virtual Real. 9, 1 (Dec. 2005),79–92.
[41] E. A. Suma, B. Lange, A. S. Rizzo, D. M. Krum, and M. Bolas. 2011. FAAST: TheFlexible Action and Articulated Skeleton Toolkit. In 2011 IEEE Virtual RealityConference. 247–248. https://doi.org/10.1109/VR.2011.5759491
[42] Russell M. Taylor, Thomas C. Hudson, Adam Seeger, Hans Weber, Jeffrey Juliano,and Aron T. Helser. 2001. VRPN: A Device-Independent, Network-TransparentVR Peripheral System. In Proceedings of the ACM Symposium on Virtual RealitySoftware and Technology (VRST ’01). Association for Computing Machinery, NewYork, NY, USA, 55–61. https://doi.org/10.1145/505008.505019
[43] Unity Technologies. 2019. Preparing Humanoid Assets for export.(2019). Retrieved February 7, 2020 from https://docs.unity3d.com/Manual/UsingHumanoidChars.html
[44] Unity Technologies. 2020. Unity - Manual: Humanoid Avatars. (2020).Retrieved February 29, 2020 from https://docs.unity3d.com/Manual/AvatarCreationandSetup.html
[45] M. Wagner, A. MacWilliams, M. Bauer, G. Klinker, J. Newman, T. Pintaric, and D.Schmalstieg. 2004. Fundamentals of Ubiquitous Tracking. In IN ADVANCES INPERVASIVE COMPUTING. Austrian Computer Society, 285–290.
[46] MatthewWright. 2005. Open Sound Control: An Enabling Technology forMusicalNetworking. Org. Sound 10, 3 (Dec. 2005), 193–200. https://doi.org/10.1017/S1355771805000932
[47] XSens. 2020. MVN Animate. (2020). Retrieved February 24, 2020 from https://www.xsens.com/products/mvn-animate
8