anim-actor: understanding interaction with digital puppetry using
TRANSCRIPT
Anim-Actor: Understanding Interaction with Digital Puppetry using Low-Cost Motion Capture
Luís Leite FEUP - Faculdade de Engenharia da Universidade do
Porto
Veronica Orvalho IT, DCC/FCUP – Instituto de Telecomunicações
ABSTRACT Character animation has traditionally relied on complex and time-consuming key frame animation or expensive motion capture techniques. These methods are out of reach for low budget animation productions. We present a low-cost performance-driven technique that allows real-time interactive control of puppets for performance or film animation. In this paper we study how users can interpret simple actions like, walking, with different puppets. The system was deployed for the Xbox Kinect and shows how low-cost equipment’s can provide a new mean for motion capture representation. Last, we performed a pilot experiment animating silhouettes and 3D puppets in real-time, based on body movements interpreted by non-expert artists, representing different Olympic Sports that vary on visual representation for other participants to identify. As a result, interaction with 2D puppets needs more interpretation than with 3D puppets.
Categories and Subject Descriptors C.2.0 [OPENframework].
General Terms Performance, Experimentation, Human Factors.
Keywords Real-Time Animation, Digital Puppetry, Virtual Marionette, Motion Capture, Performance Animation, Digital Puppeteer.
1. INTRODUCTION Animating a character is a complex process that requires artistic skills. But imagine that you want to create a very simple character animation and you don’t have those special skills! Digital puppetry can help to solve this problem, turning character animation as “simple” as controlling a marionette, but instead of frame-based animation we are doing performance animation, which involves more interaction or acting.
Digital Puppetry differs from computer conventional key frame character animation as it involves performing characters, and differs from conventional puppetry techniques because the virtual puppets are not tangible. Digital Puppetry can be used for educational, or entertainment purposes, like virtual puppet shows, live TV shows, among others. Building a digital puppet is some how analogous to creating a traditional marionette; it is necessary to define the interaction model, the body rigging and properties that give the unique expression to the puppet, like physical
properties or freedom of movement.
In this article we present, Anim-Actor, a performance-driven full body motion capture interaction technique to manipulate 3D and 2D characters. Anim-Actor allows to control digital characters in real-time by mapping the movement of the human body to the character. We described how this method was implemented for Xbox Kinect. We also perform a pilot study that compares the interaction between 2D and 3D puppets using body movement. The goal was to have the audience identifying actions, like lifting weights, made by performers using the motion capture system, to understand if digital puppetry is easier to use as an animation method in contrast to traditional animation techniques.
1.1 BACKGROUND The most common approaches to create animations using digital puppetry include: strings to control the puppet with a digital glove [1]; computer vision for tracking color marks in a object that controls the marionette movements [3]; using a multi-touch surface for direct manipulation of bi-dimensional shape puppets [4]; a multi-modal interface to simulate different marionette techniques in a virtual puppet show [2]; a motion capture system for performance-driven with gesture recognition to trigger behavior animation for virtual theater [5]. These researches contributed to a greater knowledge on digital puppetry, but some are complex to use or to implement, and others use expensive equipment. We propose easy to use affordable systems for non-expert artists.
2. SYSTEM DESCRIPTION We implemented a low-cost optical motion capture system to study the potential for real-time animation. We use a Microsoft Kinect device depth camera based on PrimeSense technology that computes a skeletal model of a character. This is an affordable and easy to use interface presenting a fast calibration process, without the need of body markers for motion tracking.
The system we implemented comprises four parts: Motion capture; skeletal model; communication; animation / rendering.
1. For capturing the actor we used the Microsoft Kinect device;
2. The OpenNI framework as API with sensorKinect based on the code of the PrimeSense/Sensor driver for skeletal model;
3. We use OSC (Open Sound Control) as a communication message protocol via network through OSCeleton to make skeleton joint information available to several applications at same time increasing system flexibility;
4. For skeleton mapping and rendering the silhouette we use a 2D real-time animation software “Animata” for the 3D puppet we use the game engine “Unity”. Figure 1 shows the workflow for the capture and skeletal model, and 2D / 3D rendering environment.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Late Breaking Result, ACE'2011 - Lisbon, Portugal. Copyright 2011 ACM 978-1-4503-0827-4/11/11…$10.00.
Figure 1. (left) motion tracking and skeletal model; (center) silhouette figure in Animata; (right) 3D puppet in Unity.
3. EXPERIMENT We prepare a survey based on a guessing game like Pictionary. 8 Olympic Sports were chosen for the experiment, to be interpreted by the "users" and identified by the audience. The sports were weightlifting, boxing, karate, basketball, shooting, arch, tennis, and hockey.
Three different comparison tests were made for the experiment with different surveys, one for performance measurement and two for the guessing game. The first test was made for comparing user positions (front and side) interacting with the profile silhouette, the second for comparing the silhouette with the 3D puppet and the third for measuring the performance of both puppets.
Figure 2. (left) Kinect capturing the performer; (right) Observers identifying sports.
For this experiment we used a non-probability method with a convenience sampling. We choose a focus group with 54 volunteers (19 male and 35 female) with an average age of 19, ranging from 16 to 30. All of the participants were students, non-expert artists, non-sportsman; none of them had used motion capture systems, and played with Microsoft Kinect before.
4. RESULTS AND CONCLUSIONS The interaction with the silhouette brings some dimensional constraints forcing the participants to seek the best body positions to perform the proposed actions. On the other hand the interaction with the 3D puppet shows no major constraints with a straightforward interaction.
We expected more silhouette positive identifications from a side capture position of the actors, which corresponded to the silhouette profile position, but instead, the results showed more positive identifications in a front position. Weightlifting from front position was the most identified sport with 100% because of the arms position and the lifting of the body, followed by the karate in front position with 94% because it was the only sport
with intense leg movement as shown in the chart from figure 3. The actors instead of trying to interpret the natural position of the silhouette to follow, they searched new ways for interacting with puppet. We believe that the interaction with a two-dimensional puppet with motion capture resembles to the control of a real marionette, but instead of using rods, strings or gloves, we use our body as the controller. And so, it requires a good interpretation of the action for the acting. The actor becomes the puppeteer.
Rather, the three-dimensional animation of the puppet is presented as a direct representation of body movement like in motion-driven games or classical motion capture animations. This explains the better positive results in the identification, with a total of 84% positive identifications for the 3D puppet against 72% for the silhouette based on the chart from figure 4.
In this exploratory experiment we found that the suggested affordable animation solution was well accepted by the participants, demonstrating the minimum needed performance to be used in real-time puppet animation. We believe that affordable motion capture solutions can be used as an easy and intuitive way for non-expert artists to animate simple characters for entertainment purposes like story telling.
5. ACKNOWLEDGMENTS This work is partially funded by Instituto de Telecomunicações and the projects LIFEisGAME (Funded by FCT) and VERE (funded by EU). Thanks to Luís Silva, Rui Espirito, Helena Santos.
6. REFERENCES [1] Bar-Lev, A., Bruckstein, A., Elber, G.: Virtual marionettes: a
system and paradigm for real-time 3D animation. The Visual Computer, Vol. 21, pp. 488--501 (2005)
[2] Leite, L.: Marionetas Virtuais – Animação interactiva em tempo real, Master Thesis in Multimedia Communication Technologies, Engineering Faculty of the University of Porto (FEUP), Porto, Portugal (2006)
[3] Sirota, A., Sheinker, D., Yossef, O.: Controlling a Virtual Marionette using a Web Camera, Technion - Israel Institute of Technology, Israel (2003)
[4] Takeo, I., Yuki, I.: Implementing As-Rigid-As-Possible Shape Manipulation and Surface Flattening. Journal of Graphics, GPU, and Game Tools, A.K.Peters, Volume 14, Number 1, pp. 17--30 (2009)
[5] Wu, Q.,Boulanger, P, Kazakevich, M., Taylor, R.: A real-time performance system for virtual theater. In: Proceedings of the 2010 ACM workshop on surreal media and virtual cloning (SMVC '10). pp. 3--8. ACM, New York, USA. (2010)
Weightlifting Boxing Karate Tenis Arch Basketball Hockey Shooting
2D 100% 79% 90% 50% 70% 30% 0% 100%
3D 85% 100% 70% 60% 100% 90% 100% 60%
Iden
tifica
tion
2D vs 3D
Figure 3. Silhouette chart results for positive identification from side and front positions.
Weightlifting Boxing Karate Tenis Arch Basketball Hockey Shooting
Front 100% 89% 94% 50% 89% 11% 88%
Side 66% 69% 50% 89% 58% 58%
Iden
tific
atio
n
Silhouette - capture positions
Figure 4. 2D/3D puppet identification comparison chart.