Gaming controllers for research robots: controlling a humanoid

robot using a Wiimote

Andrej Gams1 and Pierre-Andre Mudry2

1”Jozef Stefan” InstituteDepartment of Automation, Biocybernetics and Robotics

Jamova cesta 39, 1000 Ljubljana, Slovenia

2Ecole Polytechnique Federale de Lausanne (EPFL)Biologically Inspired Robotics Group (BIRG)Station 14, CH - 1015 Lausanne, Switzerland

E–mail: andrej.gams@ijs.si, pierre-andre.mudry@epfl.ch

Abstract

In this article we describe the use of a standardgame console joystick, namely the Nintendo Wi-imote, for controlling a HOAP-2 humanoid robot.We give a short overview on the use of tangible userinterfaces, followed by the description of the usedgame controller and the measurements of some ofits characteristics. We show the ease of applicabil-ity of inexpensive and robust standard game con-trollers for direct translation between the user’s in-tent and the robot actions on a two-handed robotdrumming task.

1 Introduction and motiva-tions

Recent achievement in the Micro Electronic Me-chanical Systems (MEMS) field enabled several de-velopments in today’s consumer electronic devices.For example, force plates, devices that were costingseveral thousands of euros only a few years ago, cannow be replaced by inexpensive micro-engineeredpressure sensors and be used in game consoles toexercise yoga or train aerobics.

In addition to this, recent advances in MEMStechnology also permitted the development of pre-cise, low-cost accelerometers that were rapidlyadopted in a wide range of devices ranging frommobile phone to computers. Of particular interest

is the appearance of such devices in mass-marketproducts such as games consoles where accelerom-eters are used to extract users’ hand gestures orarm movements. Thanks to their low price, wideavailability and robustness, these haptic interfacespossess several key elements that make them inter-esting for controlling robots.

In this article, we will demonstrate the ease ofapplying standard game controllers for the task ofcontrolling anthropomorphic robots. We proceedas follows: in the next section, a brief overview ofexisting human-robot interaction methods is pre-sented. After that, focus is put on the Wiimotecontroller itself, shown in Fig. 1, presenting itsspecifications and characteristics, and the means ofinterfacing with a standard PC in section 3.2. Fi-nally, applying the Wiimote to a drumming taskin the real world is examined before concluding.

2 Related work on tangibleuser interfaces

The need to capture arm and hand motion is ofgreat interest when interacting with anthropomor-phic robots because this method can provide a di-rect translation between the user intent and therobot action. Thus, the usage of gestures for HRIis not new: several systems that use gloves [7], vi-sion [4] or motion-capture techniques [6] have beensuccessfully used to control robots in the past. To

1

Figure 1: The Wiimote controller

recognize motions and be able to control robots,accelerometers-based systems have also been builtand discussed in several publications such as [5].

When considering game controllers in particular,several research projects have been recently usingthem in various scenarios such as interface for digi-tal painting [2] or even HRI, [3] being a very inter-esting study of this kind of interface.

3 System design and imple-mentation

3.1 Modern game controllers

The two main providers for accelerometer-basedgame controllers are Nintendo with its Wiimotecontroller and Sony with the SixAxis controller.When used as haptic interfaces, both possess sev-eral qualities that make them interesting to beused. Notably, due to their mass product ori-gin, they are inexpensive, both controllers exam-ined here costing around 50 ¿ . Moreover, they arewidely available and, because they are meant to beused by children, relatively solid.

However, if they show many similarities, the Wi-imote only possess the capability of precise move-ment tracking. Thanks to an embedded one mil-lion pixels infrared (IR) camera placed in front thedevice, the controller can track several IR LEDsplaced in a small plastic casing (the sensor bar) thatshould normally be placed under or above the TV.The presence of this capability allows precise de-tection of where the user points and how far away

from the LEDs the device is placed, the distancebetween the LEDs being known. Thus, it becomespossible to get pitch, yaw and roll information forthe device in addition to the three-axis accelerome-ters information. In addition to that, the controlleralso possess several buttons, a small speaker and amotor that can make it rumble.

Interfacing the Wiimote to a computer is rela-tively straightforward in the sense that its Blue-tooth implementation (a BCM2042 chip fromBroadcom) complies with the HID standard. Assuch, a Bluetooth interface and some widely-available open-source programs (that exist for dif-ferent operating systems) suffice for the controllerto be recognized on a standard PC. After the con-nection has been established, the Wiimote can beused as a mouse emulation or interfaced directly toget the various sensor values.

The accelerometers themselves are comprisedin a standard ADXL330 chip from Analog De-vices. The chip provides 8 bits of resolution ona ±3g m s−2 range on each axis for a sample rateof about 100 Hz. Using these figures, it is possi-ble to estimate the resolution limit of the device tobe around 0.23 m s−2 per bit of information, witha 10% sensitivity. Because accelerations are mea-sured by the forces exerted on a small moving masssuspended by polysilicon springs, it is not possi-ble to statically measure the tilt of the Wiimote.However, it is still possible to get this informationusing the IR camera by measuring the deviationfrom horizontal angle between the LEDs.

3.2 Interfacing the Wiimote and theHOAP-2 robot

The HOAP-2 is a 25-DOF humanoid robot devel-oped by Fujitsu and can be controlled using a stan-dard PC running the Real-time Linux environment.In our work, we separate the tasks of robot con-trol and data acquisition/processing, devoting eachtask to a dedicated computer as depicted in Fig. 2.This way, we can effectively control the robot usingthe operating system/environment of our choosing.

As is presented in Fig. 2, the computer run-ning Matlab/Simulink environment connects to theWiimote using Bluetooth. The GNU Wiiuse li-brary provides built-in functions for connecting tothe Wiimote and was interfaced in Simulink S-functions for the task. Thus, the data received from

BluetoothInterface

Comm

Comm RobotPositionControl

LAN

Matlab SimulinkUser space RT Module

RT Linux

Computer1 Computer2

Robot

FIFO

Figure 2: Control scheme for controlling theHOAP-2 robot with a Wiimote

the Wiimote are first processed locally and thensent via the network to the computer controllingthe HOAP-2 robot using the UDP protocol.

When the second computer receives the data,they are copied to a FIFO structure, an operationthat then triggers the transfer of data to the Real-time module, a component that controls the servosof the robot. The data acquisition and communi-cation between the computers run at 100 Hz, whilethe real-time position control of the robot runs at1000 Hz. Referential values for position betweentwo consecutive packets from the control computerare generated by interpolation.

3.3 Data from the Wiimote

In this subsection we show the correlation of thedata of some of the DOFs of the Wiimote. Fig. 3shows the results of changing the orientation of thedevice around the axis along its length, namely theroll.

0 2 4 6 8 10 12 14 16 18 20−200

−100

0

100

200

roll

[°]

0 2 4 6 8 10 12 14 16 18 20−200

−100

0

100

200

t [s]

pitc

h [°

]

Figure 3: Correlation of roll and pitch when chang-ing roll of the device. The Wiimote itself can out-put both filtered and raw data. Only the filtereddata is presented.

To demonstrate the correlation of the recorded

orientations we changed several times the roll of thedevice by 90◦, while trying to maintain all otherorientations unchanged. As it can be seen in thefigure, when the device is rotated by 90◦, the outputof the pitch values changes as well. The reasonfor such a behavior is the well-known singularity ofthe Roll-Pitch-Yaw orientation. The same happenswhen we change only the pitch as shown in Fig. 4.

0 2 4 6 8 10 12 14 16−200

−100

0

100

200

roll

[°]

0 2 4 6 8 10 12 14 16−200

−100

0

100

200

t [s]

pitc

h [°

]

Figure 4: Correlation of roll and pitch when chang-ing pitch of the device. Only the filtered data ispresented.

To measure yaw, the Wiimote has to be pointedin the direction of the IR LEDs, as it utilizes thechange of the LEDs’ location to calculate its yaw.This can be clearly seen in Fig. 5, where we showthe coupling of the position and the orientation.

0 2 4 6 8 10 12 14 16 18 200

500

1000

1500

x [p

ixel

]

0 2 4 6 8 10 12 14 16 18 20−40

−20

0

20

40

yaw

[°]

yaw

Figure 5: Correlation of yaw and x DOF of thedevice.

4 Drumming task

Different ways of controlling robots using the Wi-imote can be implemented. One way is to use pre-recorded moves and execute them when recognized,i.e. when performing a certain move, the measureddata are compared with a set of recorded movesthat form a trajectory and, when a match in thedatabase is found, the corresponding pre-recordedmove is executed. Another possibility is to couplethe motion of the Wiimote and the robot directly.

Figure 6: Two handed drumming of the HOAP-2robot.

This approach requires some filtering of the datathat would be otherwise too noisy.

We used the latter approach but we extended itto teaching the robot the moves. This was donewith a system for learning and imitating trajec-tories of unknown frequency and waveform. Thesystem, described in [1], also acts as a sort of filterand with it we could successfully teach the robotthe desired drumming trajectories. The experimen-tal setup was based on two Wiimotes, one to con-trol each arm of the HOAP-2, managing 4 DOFper arm. The control itself was done my mappingthe motion of the two Wiimotes to the tips of thedrumsticks and, thanks to an inverse kinematics al-gorithm, to achieve motion in task space. Fig. 6shows the HOAP-2 robot drumming.

5 Conclusions and futurework

We showed in this article how standard consumerelectronics components can be used to provide arobust and widely-accessible way to explore inno-vative HRI methods. Notably, we showed that tightspatial mapping between user and robot translatesto a very effective control method. Using the Nin-tendo Wiimote, even though it has some draw-backs, is an effective way of intuitive control ofrobots. We expect that off-the-shelf electronic de-vices will prove even more useful in future HRI

applications, one such step is also the announcedadd-on for the Wiimote, codenamed MotionPlus,that will provide a 1:1 tracking of arm movementsthanks to an integrated micro-gyroscope.

References

[1] A. Gams, S. Degallier, A. J. Ijspeert, and L. J.Dynamical system for learning the waveformand frequency of periodic signals - applicationto drumming. In Proc. of the Int. Conf. RAAD2008, to appear.

[2] J. Grubert, M. Hanckock, S. Carpendale,E. Tse, and T. Isenberg. Interacting withStroke-Based Rendering on a Wall Display.Technical Report 2007-882-34, Departmentof Computer Science, University of Calgary,Canada, October 2007.

[3] C. Guo and E. Sharlin. Exploring the use oftangible user interfaces for human-robot inter-action: a comparative study. In CHI ’08: Proc.of the 26th SIGCHI Conf. on Human factors incomputing systems, pages 121–130, New York,NY, USA, 2008. ACM.

[4] M. Hasanuzzaman, T. Zhang, V. Am-pornaramveth, P. Kiatisevi, Y. Shirai, andH. Ueno. Gesture based human-robot inter-action using a frame based software platform.Systems, Man and Cybernetics, 2004 IEEE In-ternational Conference on, 3:2883–2888 vol.3,Oct. 2004.

[5] F. G. Hofmann, P. Heyer, and G. Hommel.Velocity profile based recognition of dynamicgestures with discrete hidden markov models.In Proc. of the Int. Workshop on Gesture andSign Language in Human-Computer Interac-tion, pages 81–95, London, UK, 1998. Springer-Verlag.

[6] S. Iba, J. Weghe, C. Paredis, and P. Khosla.An architecture for gesture-based control of mo-bile robots. Proc. of the Int. Conf. on Intelli-gent Robots and Systems (IROS’99), 2:851–857vol.2, 1999.

[7] D. Sturman and D. Zeltzer. A survey of glove-based input. Computer Graphics and Applica-tions, IEEE, 14(1):30–39, Jan 1994.