Avoiding Conflicts of Operators in Multi-user Teleoperation Systems

Takahiro Kanno∗

Kyoto University

Yasuyoshi Yokokohji†

Kobe University

ABSTRACT

In this paper, control methods to avoid the conflict of operators’commands in multi-user teleoperation system is considered. Theproposed methods insert virtual nonlinear springs between the mas-ter arms and the end of wave transmission line of the multilateralcontroller. The nonlinear springs becomes softer when the inputforce from the operators are above a certain threshold and limit theinput force. Experiments are conducted in order to compare the ef-fectiveness of the methods. The experimental results show that theproposed two-step gain improves the operation performance.

1 INTRODUCTION

Bilateral teleoperation enables a human operator to intuitively com-mand the remote robot arm feeling the reaction force from the envi-ronment. It can be applied to manipulations in hazardous environ-ments such as nuclear plants and outer space, or scaled manipula-tions such as micro surgery.

Conventionally, position and force information are transmittedbetween a single master arm and a single slave arm. On the otherhand, multilateral teleoperation systems in which information aretransmitted among three or more master and slave arms are beingdeveloped. Several configurations are possible for multilateral tele-operators such as robot operation training using multiple mastersand a single slave [11] and commanding group of robots using onemaster arm [3]. Multi-master systems can also contribute to safetyof the teleoperation due to the redundancy of operators. Such sys-tems keep stable even if one of the operators drops from the op-eration due to illness, failure of master arms, or communicationblackouts.

The authors already proposed a control method of multilateralteleoperation systems for various applications mentioned abovebased on wave-variables, which guarantees stability under constantcommunication delay [5].

As for multilateral control, various methods are already pro-posed. A wave-variable-based control of a dual-master-single-slavesystem is proposed by Carignan et al. [2]. Nudehi et al. proposedH∞ -based control for surgical training system [11]. They proposedto use the weighted sum of master inputs when generating the slavemotion in order to adjust the authority between the trainer and thetrainee. Katsura et al. introduced disturbance observer for multilat-eral control [6], which realizes the law of action and reaction amongseveral manipulators, but communication time-delay is not consid-ered. Four-channel architecture is extended to multi-user teleoper-ation by Khademian et al. [7]. They introduced a method to ad-just authority like in [11]. Morris et al. developed a haptic trainingsystem for bone surgery [9]. In this system, force and position ofenvironment are transmitted to trainee. Fotoohi et al. analyzed thecentralized and distributed controller of multi-user haptic systemfor short time-delay of LAN [4]. Moghimi et al. introduced a con-trol method which enables nonlinear position and force mappings

∗e-mail: tak.kanno@gmail.com†e-mail: yokokohji@mech.kobe-u.ac.jp

[8]. However, their method is abstract and mapping functions them-selves are not discussed in detail.

In conventional bilateral (single-master-single-slave) teleopera-tors, control objectives are simple: tracking of position and forcebetween the master and the slave. For multilateral teleoperation, onthe other hand, there are mainly two types of control objectives areused currently. One set of control objective is that reference posi-tion of the slave is determined by the weighted sum of the positionsof the masters and the force reference of each master is equal tothat of the slave arm, while the other is that positions of the all mas-ter/slave arms are the same and sum of the forces applied to all armsis zero, i.e., the force equilibrium. The latter shows more intuitivebehavior to the operators because it mimics the dynamics that mul-tiple operators are manipulating the remote environment through abranched bar.

However, in force equilibrium approaches, a problem occurs be-cause inputs from all operators are treated equally. In cooperativeteleoperation by two users, for example, the operator may have toinput extremely large counter force when the other operator inputsa wrong force or trajectory. In multilateral control methods usingweighted sum approach such as in [11] and [7], authority can beadjusted by choosing the appropriate weights. However, authorityis not introduced yet for wave-variables-based multilateral controlby the authors, where inputs from operators are treated equally. Itis necessary to introduce additional control methods in order to setthe input priority on each master or slave arm.

In this paper, a method to adjust a priority of each operator inwave-based multilateral teleoperator is proposed. The proposedmethods introduce virtual nonlinear-spring behaviors between themaster arm and the controller. Although this paper focuses onwave-variable-based control, it is possible to extend the proposedmethods to other control architectures due to the simplicity of themethods. Experiments are conducted using dual-user system tocompare the proposed methods.

2 MULTILATERAL CONTROL USING WAVE VARIABLES

In this section, wave-variables-based multilateral control method bythe authors is shortly explained. Please refer to [5] for details.

2.1 Wave variables

Anderson and Spong proposed a teleoperation control based onscattering transformation to guarantee the passivity of the systemagainst the time delay between the master and the slave [1]. Themethod is later reformulated by Niemeyer and Slotine using theterm “wave variables” [10].

Instead of transmitting position or force variables, wave vari-ables are transmitted each other so that the transmission line mim-ics the dynamics of distributed mass-spring system shown in Fig.1.The wave variables are defined as follows:

uuum(t) =bxxxmd(t)+ fff m(t)√

2b(1)

vvvs(t) =bxxxsd(t)− fff s(t)√

2b(2)

where xxxmd(t) and xxxsd(t) denote the velocities at the endpoint oftransmission line of the master side and the slave side, respectively.

401

IEEE World Haptics Conference 201314-18 April, Daejeon, Korea978-1-4799-0088-6/13/$31.00 ©2013 IEEE

Figure 1: Physical interpretation of wave-based control

The force from the master arm to the mass-spring system is de-fined by fff m(t), while the one from the endpoint to the slave armis defined by fff s(t). The parameter b > 0 is called “characteristicimpedance”, which affects the amount of virtual mass and stiffnessof virtual spring. Wave variables at the receiving side is definedsimilarly as the sending side:

uuus(t) =bxxxsd(t)+ fff s(t)√

2b= uuum(t −T1) (3)

vvvm(t) =bxxxmd(t)− fff m(t)√

2b= vvvs(t −T2) (4)

where T1 denotes the delay from the master to the slave and T2 fromthe slave to the master.

2.2 Position control of manipulators

In this paper, proportional-derivative (PD) control is used to con-nect the master or slave arm and the endpoint of distributed mass-spring. Control laws are as follows:

− fff m = Kp(xxxmd − xxxmr)+Kd(xxxmd − xxxmr) (5)

fff s = Kp(xxxsd − xxxsr)+Kd(xxxsd − xxxsr) (6)

where xxxmd and xxxsd are the reference positions which are obtainedby inverse-transformation from wave-variables, while xxxmr and xxxsr

denote the real positions of master and slave manipulators. Theconstant Kp and Kd are the proportional and derivative gains, re-spectively. Force variables fff m and fff s are the input forces from theactuators.

2.3 Multilateral control

In the architecture proposed by the authors, multilateral control isrealized by connecting multiple distributed mass-spring system atthe node as shown in Fig.2. Each master or slave arm is locatedat the terminal, and each terminal and the node is connected usingconventional wave-variables transmission line. The system behavesas if multiple operators are directly operate the remote slave armvia a branched flexible rod. Note that, similar to section 2.2, theendpoint of wave transmission line and the master/slave arm arecoupled by PD control.

First, consider that n distributed constant systems from the ter-minals (master or slave) are connected at the node. Define xxxTi andfff Ti as terminal-side end-point position and force of i-th transmis-sion line, and xxxNi and fff Ni as node-side position and force. Similarto (1), (2), (3), and (4), wave variables at each transmission line isdefined as follows:

uuuTi(t) =bxxxTi(t)+ fff Ti(t)√

2b, vvvTi(t) =

bxxxTi(t)− fff Ti(t)√2b

(7)

uuuNi(t) =bxxxNi(t)+ fff Ni(t)√

2b, vvvNi(t) =

bxxxNi(t)− fff Ni(t)√2b

(8)

Since distributed mass-spring are connected at the node, positionconstraint condition and force equilibrium condition is written asfollows:

xxxN1 = xxxN2 = · · ·= xxxNn (9)

∑i

fff Ni = 0 (10)

Node

Terminal

Figure 2: Wave-variable-based multi-user teleoperator

Figure 3: Conflict of operators’ input motions

Finally, using these conditions and definition of wave variables, thewave variable which should be returned from the node to each ter-minal is calculated as follows:

vvvNi =−uuuNi +2

n∑

i

uuuNi (11)

When the time delay is not constant, wave-integral-error feed-back [12] should be applied to each transmission line to compen-sate for the position drift caused by time-varying delay, though it isnot present in this paper due to space limitation.

3 CONTROL METHOD TO AVOID THE CONFLICT OF OPERA-TORS

3.1 Proposed method

In wave-based multilateral teleoperation, which is based on forceequilibrium, operators’ command forces are simply summed. Whenthe operators input different trajectory, the slave arm does not moveappropriately as shown in Fig.3.

In this section, a method to set the priority to each master arm isproposed so that the operation is smoothly continued even if thereis a conflict of command of the operators.

The proposed method replaces the PD control described in Sec-tion 2.2 with the following control to restrict the input force of theoperator:

fff Ti = Kp(xxxTi − xxxTir)+Kd(xxxTi − xxxTir) (12)

where xxxTir denotes the real position of i-th manipulator. The map-ping function Kp(eee) characterizes the nonlinear spring between the

402

(a)PD control (b)Input satulation (c)Two-step gain

Figure 4: Conflict avoiding controllers

Figure 5: Nonlinear spring added to the master controller

master arm and the transmission line, where eee△= xxxTi − xxxTir. In this

paper, following two functions are selected as mapping function,while many other functions can be considered as candidates.

• Input saturation

Kp(eee) =

{

Kpeee (‖eee‖< fth/Kp)ftheee/‖eee‖ (otherwise)

(13)

• Two-step gain

Kp(eee) =

Kpeee (‖eee‖< fth/Kp)Kp2eee+ fth(1−Kp2/Kp)eee/‖eee‖

(otherwise)(14)

where fth denotes the threshold where the spring becomes softerand Kp2 < Kp is the gain when the input force exceeds the thresh-old. These mapping functions are shown in Fig.4. When introduc-ing the proposed method, a nonlinear spring gets softer with largeroperator input than threshold as shown in Fig.5. Input saturationdisables the operator to input larger force than the threshold, whiletwo-step gain gives the operator the possibility to apply large forcestemporarily in special cases that operators need to apply large forcesto the environment cooperatively or the prior operator without non-linear spring inputs wrong commands.

The proposed method looks similar to variable impedance, but isdifferent in that it realizes nonlinear and time-invariant dynamics,while variable impedance methods realizes time-varying dynamics.Variation of system dynamics during operations may have negativeeffects on operator perception and system stability, but the dynam-ics is constant in the proposed method and stability is guaranteed asshown in the next section. Change of authority during the operationis not considered in this paper and remains as the future work.

Note that there are possible methods other than the proposedmethods such as:

• Force scaling

• Energy monitoring

The force scaling is a method that force at the slave side is mag-nified and then displayed as follows:

fff Ti = εTi( fff Ni) (15)

where εTi is the force scaling parameter assigned to i-th terminal.This method certainly suppress the force applied to the environ-ment, but the environment impedance displayed to the operator isdifferent from the original one. Moreover, position drift betweenthe master and the slave is generated when εTi is nonlinear or time-varying.

Energy monitoring is a method that measures an input energyfrom the operator and stops the data transmission or reduce theamount of wave variables when the energy exceeds the certainthreshold. This method has an advantage that system passivity isguaranteed without any other compensation method, but it is toodifficult to choose appropriate energy threshold. Energy monitor-ing also causes position drift.

Not only the force scaling but also the proposed nonlinear springalso changes the displayed environment impedance. Here a fewanalyses are conducted to compare them. The displayed impedanceusing each method is analyzed here. For simplicity, it is assumedthat there are no time delay, the mass and friction of master andslave arm is negligible, and the system is 1-DOF single-master-single-slave. If the environment is a mass-spring-damper system,the slave-side dynamics is written as follows:

Menvxs = Kp(xm − xs)+Kd(xm − xs)−Kenvxs −Denvxs, (16)

where Menv, Denv, and Kenv denote the mass, damping, and springcoefficients, respectively, and xm and xs denote the position of themaster and slave arms. The master-side dynamics, after introducingforce scaling ε , is written as follows:

fm = ε{Kp(xs − xm)+Kd(xs − xm)}. (17)

By substituting xs in (16) to (17), the displayed environment dy-namics fm/xm is derived as follows:

fm

xm= ε

{

−(Kds+Kp)+(Kds+Kp)

2

Menvs2 +(Kd +Denv)s+(Kp +Kenv)

}

.

(18)Figure 6 shows the displayed environment dynamics, where Menv =1[kg], Denv = 1[Ns/m], and Kenv = 100[N/m]. The gains are set toKp = 100, Kd = 1, and ε = 1 for normal PD control, while Kp = 20,Kd = 1, and ε = 1 for PD control with low proportional gain (cor-responding to two-step gain), and Kp = 100, Kd = 1, and ε = 2 forPD control with force scaling. It is shown that the proposed methodmakes the apparent environment impedance softer, while force scal-ing makes it harder and heavier. Energy monitoring, which stopsthe wave output, is not analyzed here because it is obvious that theoperator can no longer feel the environment.

3.2 Passivity of the nonlinear spring

This section shows that the proposed nonlinear spring preserves thesystem passivity, though it is intuitively obvious. Since a systemconsisting of passive subsystems is also passive, only the passivityof nonlinear position control is shown.

First, the passivity condition is defined as:

EPC(t) =−∫ t

0

(

xxxmi(τ)T fff mi(τ)+ xxxTi(τ)

T fff Ti(τ))

dτ ≥ EPC(0),

(19)

403

Figure 6: Apparent environment impedance displayed to the operator

where xxxmi and fff mi denote the position and motor input of the i-th master arm, respectively, and EPC is the energy stored in theposition controller. The following control law is assumed:

fff mi =− fff Ti = fff (eee)+Kd eee, (20)

where fff (eee) is the mapping function and eee△= (xxxTi − xxxmi). Then the

energy EPC can be rewritten as:

EPC =

∫ t

0

[

eee(τ)T { fff (eee(τ))+Kd eee(τ)}]

dτ

=∫ t

0Kd |eee(τ)|2dτ +

∫ t

0eee(τ)T fff (eee(τ))dτ, (21)

and

∫ t

0eee(τ)T fff (eee(τ))dτ =

∫ t

0

deeeT

dτfff (eee(τ))dτ (22)

=∫ eee(t)

000fff (εεε)T dεεε (23)

=F(eee(t)). (24)

Thus, the position controller is passive if the potential functionF(eee(t)) is non-negative and F(000) = 0. In the case of two-step gain,the potential function is represented as follows:

∫ eee(t)

000fff (εεε)T dεεε

=

1

2Kp|eee|2 (|eee|< fth/Kp)

1

2Kp2|eee|2 + fth

(

1− Kp2

Kp

)

|eee|+ c (otherwise),

(25)

where the constant of integration c△= 1

2 (Kp2−Kp

K2p

) f 2th is determined

so that the potential function becomes continuous. In this case∫ eee(t)

000fff (εεε)T dεεε ≥ 0 and passivity is ensured. Passivity of input satu-

ration is also ensured by substituting Kp2 = 0 to eq. (25).

4 EXPERIMENTS

Experiments are conducted to compare characteristics of the twoproposed controllers and the original controller.

4.1 Experimental setup

A dual-master-single-slave experimental system is constructed as-suming a situation that two operators cooperatively command a re-mote robot arm.

Figure 7: Experimental system

Figure 8: Virtual environment

Two SensAble PHANTOM Omni devices are used as the masterarms. They are controlled in a single computer with Windows 7Professional OS and sampling time of the controllers is 1 [msec].As for the slave, a virtual environment shown in Fig.7 is introduced.It includes six virtual push buttons, which can be pushed vertically.Dynamic simulation of the virtual environment is run at the sam-pling time 0.1 [msec] in another computer with Windows 7 Profes-sional, while the control sampling time of the virtual slave manipu-lator is 1 [msec]. Virtual environment is rendered using OpenGL at30 [Hz] in parallel with the dynamic simulation. The virtual slave isa mass point, whose mass is 3 [kg] and has 1 [Ns/m] of viscous fric-tion. The spring constant of virtual push buttons is set to 1 [N/m].

The node of the multilateral controller is implemented in thesame PC as the virtual environment. Characteristic impedance ofthe wave variables is set to b = 30 [Ns/m]. As for network commu-nications among the masters, the slave, and the node, UDP (UserDatagram Protocol) is used and the sampling time of data trans-mission is 2 [ms]. Since all of the systems in this experiment arein local area network and there are almost no delays, virtual timedelay of 100 [ms] is introduced between the two computers. Theimage of the virtual environment is captured and transmitted to themasters using video transmission software “LiveCapture2”.

Note that the proposed method can be implemented to delay-freeteleoperation. It is true that delay-free experiment can compare thepure performance of the proposed nonlinear control, but it is betterto introduce realistic delay in order to confirm the effectiveness ofthe proposed method in real manipulation tasks.

4.2 Tasks

In the experiments, a series of numbers from ‘1’ to ‘6’ is displayedto the two operators as shown in Fig.9. The operators are asked topush the virtual buttons corresponding to the displayed numbers.

404

Figure 9: Task display

Table 1: Confidence and correctness sets

Operator 1 Operator 2Set Conf. Corr. Conf. Corr.

1 ++ X ++ X

2 ++ X + X

3 ++ X +4 ++ X -5 ++ X ++6 + X ++ X

7 + ++ X

++ : Certain (Displayed in master console in black),

+ : Uncertain (Displayed in master console in blue),

- : Unknown (Hidden), X: Correct

Each displayed number is colored according to the reliability: inblack if it is reliable, in blue if it is wrong with the probability 50%, or hidden.

Assuming that Operator 1 takes the role of a more accomplishedoperator than Operator 2, the seven sets of displayed reliability andcorrectness are introduced as shown in Table 1. In Sets 1, 2, and6, both operators are shown the correct number and the operationwill be smoothly done anyway. However, in Sets 3, 4, 5, and 7, thedisplayed numbers to the operators are different, which make theconflicts between the operators. Operator 2 is not confident in Set 3and has no information in Set 4. In such situations, Operator 2 willfollow the motion of Operator 1, though the operation performancein Set 3 is lower than in Set 4 because Operator 2 will first tryto push the displayed number. Wrong numbers are displayed inblack when ‘Set 5’ in the table is used, which assume that Operator2 is misunderstanding the task. The performance will extremelydegrade since Operator 2 strongly move the master arm to wrongdirection. Set 7 is the only case where Operator 1 is wrong.

In the experiment, numbers in each set is used twice and in total14 numbers are displayed after shuffled.

PD control is used for master arm of Operator 1, while controllaws is randomly selected from PD control, eq. (13), and eq. (14).Parameters of the control system is shown in Table 2.

Two males in their 20’s participated in the experiment. Each ofthem took the role of Operator 1 for 12 times and Operator 2 for 12times. Participants are not allowed to talk about the displayed num-bers or to orally command the other operator, though it is allowedto talk about something which are not related to the experiment.

4.3 Experimental results

The rate that the operators pushed the correct buttons for each con-troller and confidence/correctness set is shown in Table 3.

Almost all trials are successful for Set 1, 2, 3, 4, and 6. However,failure rates are higher for Set 5 and 7, which are discussed in detailin this section.

In Set 5, success rate is higher using nonlinear spring controlthan using PD control. Thus it is achieved that Operator 1 getsthe higher priority than Operator 2 by restricting the force input ofOperator 2.

In Set 7, on the other hand, success rate is lower using the inputsaturation, one of the proposed method. When the input saturationis introduced, Operator 2 cannot correct the mistakes of Operator

Table 2: Control parameters

Name Variable Value

Master P gain Kp 500 N/mMaster D gain Kd 2 Ns/mMaster 2nd gain Kp2 20N/mSlave P gain Kp 100 N/mSlave D gain Kd 10 Ns/mForce threshold fth 1 N

Table 3: Success rate of the task (%)

Set PD control Input saturation Two-step gain

1 100 100 942 100 100 1003 94 100 944 94 100 1005 81 94 1006 88 100 1007 56 31 63

Avg. 88 89 93

1 because of the limitation of input force. By using two-step gain,on the other hand, Operator 2 can input enough force to informOperator 1 of the correct motions.

5 CONCLUSION

In this paper, methods to avoid the conflict of operators in mul-tilateral teleoperation systems are proposed. The proposed meth-ods insert virtual nonlinear spring between the master arm and themultilateral controllers in order to restrict the input force from theoperators.

Experiments are conducted to compare the two proposed meth-ods and conventional PD control by two participants with differentlevel of skill. The results show the effectiveness of the proposedtwo-step gain method.

The future works are to conduct experiments by more partici-pants to compare the methods statistically, and to develop a soft-ware which enables the construction of multilateral teleoperatorseasily.

REFERENCES

[1] R. J. Anderson and M. W. Spong,  Bilateral Control of Teleoperators

with Time Delay,¡ IEEE Transactions on Automatic Control, vol. 34,

no. 5, pp. 494–501, 1989.

[2] C. R. Carignan and P. A. Olsson,  Cooperative control of virtual ob-

jects over the internet using force-reflecting master arms, ¡ in Pro-

ceedings of the 2004 IEEE International Conference on Robotics &

Automation, pp. 1221–1226, 2004.

[3] A. Franchi, P. R. Giordano, C. Secchi, H. I. Son, and H. H. Bulthoff,

  A Passivity-Based Decentralized Approach for the Bilateral Teleop-

eration of a Group of UAVs with Switching Topology,¡ in 2011 IEEE

International Conference on Robotics and Automation, pp. 898–905,

2011.

[4] M. Fotoohi, S. Sirouspour, and D. Capson,  Stability and Performance

Analysis of Centralized and Distributed Multi-rate Control Architec-

tures for Multi-user Haptic Interaction, ¡ The International Journal of

Robotics Research, vol. 26, no. 9, pp. 977–994, Sep. 2007.

[5] T. Kanno and Y. Yokokohji,  Multilateral teleoperation control over

time-delayed computer networks using wave variables,¡ in 2012 IEEE

Haptics Symposium (HAPTICS), pp. 125–131, 2012.

405

[6] S. Katsura, Y. Matsumoto, and K. Onishi,   Realization of Law of

Action and Reaction by Multilateral Control, ¡ IEEE Transactions on

Industrial Electronics, vol. 52, no. 5, pp. 1196–1205, 2005.

[7] B. Khademian and K. Hashtrudi-Zaad,  Shared control architectures

for haptic training: Performance and coupled stability analysis, ¡ The

International Journal of Robotics Research, vol. 30, no. 13, pp. 1627–

1642, Mar. 2011.

[8] S. Moghimi, S. Sirouspour, and P. Malysz,  Haptic-enabled Collab-

orative Training with Generalized Force and Position Mappings, ¡ in

2008 Symposium on Haptic Interfaces for Virtual Environment and

Teleoperator Systems, 2008, pp. 287–294.

[9] D. Morris, C. Sewell, F. Barbagli, K. Salisbury, N. Blevins, and S.

Girod,   Visuohaptic Simulation of Bone Surgery for Training and

Evaluation, ¡ IEEE Computer Graphics and Applications, vol. 26, no.

6, pp. 48–57, Nov. 2006.

[10] G. Niemeyer and J. J. . Slotine,   Stable adaptive teleoperation, ¡

Oceanic Engineering, IEEE Journal of, vol. 16, no. 1, pp. 152–162,

1991.

[11] S. S. Nudehi, R. Mukherjee, and M. Ghodoussi,  A Shared-Control

Approach to Haptic Interface Design for Minimally Invasive Telesur-

gical Training, ¡ IEEE Trans. on Control Systems Technology, vol.

13, no. 4, pp. 588–592, 2005.

[12] Y. Yokokohji, T. Tsujioka, and T. Yoshikawa,  Bilateral Control with

Time-Varying Delay including Communication Blackout, ¡ in Proc.

the 10th Symposium on Haptic Interfaces for Virtual Environment and

Teleoperator Systems, pp. 285–292, 2002.

406