Download - Teleoperation with Force Feedback - ULisboa
Teleoperation with Force Feedback:Easing Unmanned Vehicles Operation in Unknown Scenarios
André Casqueiro da Costa Ferreira
Thesis to obtain the Master of Science Degree in
Mechanical Engineering
Supervisors: Prof. Alexandra Bento MoutinhoProf. Jorge Manuel Mateus Martins
Examination Committee
Chairperson: Prof. João Rogério Caldas PintoSupervisor: Prof. Alexandra Bento Moutinho
Member of the Committee: Prof. Pedro Vieira Gamboa
November 2015
Acknowledgments
I would like to express my sincere gratitude to my advisors Prof. Alexandra Moutinho and Prof. Jorge
Martins for their guidance and encouragement and, most of all, for accepting me in this great adventure.
I thank my fellow labmates, Afonso Ferreira, Antonio Campos, Diogo Ruivo, David Salvador, Nuno
Chaveiro and Renato Severiano, for the stimulating discussions, for always willing to give advice and be
guinea pigs for my experiments, and for all the fun we have had in these last months. I could not have
imagined having a better team. I am especially thankful to Rodrigo Coelho for the support in decoding
the laser scanner and Diogo Ruivo for providing the quadcopter model used in the development of a
simulator. Mr. Raposeiro and Eng. Camilo Christo for the time and technical support.
A thank you also extends to everyone who volunteered to participate in the experiments.
My old friend Jose Miguel Cerdeira for proofreading this thesis so quickly and offering suggestions
for its improvement.
Finally, a couple of personal notes. A heartfelt thank you to my family for their unconditional support
and for believing in me. Last but not the least, I would like to thank Joana Coelho, for always being there,
for her patience and for helping me maintain healthy levels of insanity.
v
Resumo
Com o avanco da tecnologia, a presenca de veıculos nao-tripulados esta a aumentar e a utilizacao
destes em tarefas complexas que requerem interacao com o ambiente esta a crescer. No entanto, os
radios-comando tipicamente utilizados nao sao intuitivos e nao proporcionam uma percecao do pro-
cesso adequada.
Tendo isto em conta, o principal objetivo desta dissertacao e o desenvolvimento de um sistema
generico de teleoperacao haptica que possa ser aplicado em veıculos nao-tripulados aquaticos, ter-
restres e aereos. O sistema proposto inclui um novo feedback haptico baseado em vibracao e um novo
sistema anti-colisao que tem em consideracao tanto as indicacoes do operador como as limitacoes do
veıculo no ambiente que o rodeia. Como resultado de uma experiencia com um veıculo terrestre nao
tripulado, verificou-se uma diminuicao no tempo de prova e na carga total de trabalho do operador. O
feedback baseado em vibracao foi comparado com outros feedbacks hapticos numa experiencia em
que os utilizadores tiveram que identificar a direcao de um obstaculo virtual. O feedback proposto
superou os restantes tanto em tempo de resposta como em precisao.
Numa segunda fase, a eficiencia do sistema proposto foi avaliada atraves de uma experiencia em
que os utilizadores operaram um quadricoptero num ambiente virtual. Os resultados obtidos confirmam
o esperado aumento de transparencia durante a operacao, assim como um aumento de conforto do
operador e seguranca do veıculo com um tempo de prova reduzido.
Palavras-chave: Teleoperacao, Controlo Haptico, Feedback Haptico, Sistema Anti Colisao,
Veıculos Nao Tripulados
vii
Abstract
With the emerging technologies, the presence of unmanned vehicles is increasing and their relevance in
complex tasks that require interaction with the environment is growing. However, typical radio controllers
are not intuitive and do not provide the user with an appropriate situation awareness.
Taking this into account, the main goal of this thesis is the development of a generic haptic teleoper-
ation scheme with increased transparency for unmanned water, ground or aerial vehicles. The proposed
scheme includes a novel haptic feedback based on vibration and a new anti-collision method that takes
into account both vehicle and environment constraints. User experiments with an unmanned ground
vehicle using the anti-collision method show a reduction in both task completion time and human op-
erator workload. The vibration feedback was compared with other haptic feedbacks in an experiment
where users had to identify the direction of a single obstacle. The existing feedback solutions were
outperformed both in preciseness and celerity.
In a second experiment subjects evaluated the effectiveness of the vibration feedback in a 3D sce-
nario, where a quadcopter was maneuvered through a maze in a simulator. Results confirm the ex-
pected increase in obstacle awareness and task easiness. Furthermore, the use of a haptic feedback
also increased the operator’s comfort while executing the task, it decreased the number of collisions and
reduced the task completion time.
Keywords: Teleoperation, Haptic Control, Force Feedback, Obstacle Avoidance, Unmanned
Vehicle
ix
Contents
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
Resumo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
Nomenclature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Problem definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Objectives and Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.5 Structure of the Dissertation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Methods 7
2.1 Haptic interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2 Master-Slave Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3 Obstacles Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3.1 State of the Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3.2 Newly Developed Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.4 Anti-Collision System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.4.1 State of the Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.4.2 Newly Developed Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.5 Haptic Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.5.1 State of the Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.5.2 Vibration Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.5.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.5.4 Methods Comparison Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3 2D Experiment 31
3.1 Platform OMNI-ANT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
xi
3.2 Goals and Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.3 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.4 User Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.5 Subjects and Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.6 Maze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.7 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.8 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4 3D Experiment 39
4.1 Quadcopter Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.2 Goals and Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.3 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.4 User Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.5 Subjects and Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.6 Maze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.7 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.8 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5 Conclusions 49
5.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Bibliography 53
xii
List of Tables
3.1 LIDAR scanner main characteristics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.2 Comparison of questionnaire results between anti-collision systems. . . . . . . . . . . . . 36
3.3 Comparison of questionnaire results between different haptic systems. . . . . . . . . . . . 37
xiii
List of Figures
1.1 Generic achitecture of bilateral teleoperation . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1 Haptic interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Local Force diagram. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3 BRF contour plot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.4 PRF contour plot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.5 Examples of resulting obstacle direction vector for different scenarios. . . . . . . . . . . . 15
2.6 Anti-collision methods diagram. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.7 Relation between velocity and distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.8 Wall approach example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.9 Corrected velocity for the vehicle approaching a wall from south, using avoidance system. 19
2.10 Corrected velocity for the vehicle approaching a wall from south, using deconfliction system. 20
2.11 Initial velocity space. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.12 Resulting allowed velocity space for an obstacle approaching the vehicle from north-east. 21
2.13 Avoidance scenarios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.14 Deconfliction scenarios. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.15 Force and Stiffness feedback representation . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.16 Vibration feedback example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.17 Vibration feedback period and amplitude along distance to obstacle. . . . . . . . . . . . . 25
2.18 Mater device width response evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.19 Mater device frequency response evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.20 Haptic experiment visual interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.21 Main results of haptic experiment for each type of feedback: force, stiffness and vibration
feedback. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.1 OMNI-ANT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.2 Master system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.3 Experiment visual interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.4 Top view of the maze used for experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.5 Comparison results between avoidance system (AS) and deconfliction system (DS). . . . 36
3.6 Comparison results between no haptic feedback, force, stiffness and vibration feedbacks. 37
xv
3.7 Feedback type preference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.1 Examples of user visual interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.2 Simulation maze 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.3 Simulation maze 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.4 Simulation maze 3. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.5 Results comparison between different cases. . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.6 Questionnaire results comparison for different cases. . . . . . . . . . . . . . . . . . . . . . 46
xvi
Nomenclature
Acronyms
AFF Artificial Force Field
BRF Basic Risk Field
DOF Degree of Freedom
GPF Generalized Potential Field
LIDAR Light Detection And Ranging
PRF Parametric Risk Field
PTZ pan-Tilt-Zoom
UAV Unmanned Aerial Vehicle
UGV Unmanned Ground Vehicle
VTOL Vertical Take-Off and Landing
Greek symbols
θobs Principal direction of obstacles
(φ, θ, ψ) Euler angles (roll, pitch and yaw)
(θ, φ, r) Spherical coordinates (azimuth, elevation and radial distance)
Roman symbols
Cm Master device damping coefficient
fa Applied force
fl Local force
Km Master device spring coefficient
q0 Master end effector center position
qm Master end effector position
xvii
qs Slave position
u Unitary vector
vs Slave velocity
Rc Critical radius
Rd Danger radius
Ro Observation radius
Rw Warning radius
d Radial distance from slave to obstacle
dobs Principal direction of obstacles risk magnitude
t Time
Vmax Maximum speed
Subscripts
c Corrected
m Master
r Reference
s Slave
xviii
Chapter 1
Introduction
With the increasing development of technology, robots are becoming more and more present in our lives.
In particular, unmanned vehicles are substituting the presence of humans in operations that imply difficult
access and health risks. However, typical radio controllers do not provide the necessary precision
and easiness for these operations. In this thesis, we present an intuitive scheme for haptic bilateral
teleoperation of unmanned vehicles with obstacle avoidance system.
1.1 Motivation
Since the beginning of robotics studies, teleoperation has been one of the main issues of investiga-
tion. Teleoperation or telerobotics refers to the operation of a machine at a distance, implying a barrier
between the operator and the environment. This barrier, besides distance, may also be imposed by
hazardous environment or a large difference in scaling. Whereas this separation can be small enough
to have the human operator and the robot occupying the same room.
The first case of a teleoperated system dates back to the 1940s by Raymond C. Goertz [1]. Using
electrical and mechanical components, he created manipulators for humans to handle radioactive ma-
terial from behind shielded walls. However, without a feeling of touch, these manipulators were slow
and awkward to operate. In order to overcome this constraint, it became clear that the operator should
be able to feel the environment through the robot. This concept known as bilaterality, refers to the
case where, during teleoperation, information flows in both ways. Not only does the operator control
the robot, as the robot transmits information related to its environment. Using sensors, the robot per-
ceives its surroundings and may then reflect back to the operator the states of the task being performed.
This information is perceived by the operator through his visual, tactile and hearing senses, either using
images of the operation, force feedback or other signals.
Nowadays, classical bilateral teleoperated manipulators are mostly used in the field of surgery. In
2001 teleoperation was used in surgery for the first time when a surgeon in New York (USA) operated
a patient in Strasbourg (France) [2]. Meanwhile, in order to overcome the human desire to explore
unknown and dangerous environments, mobile robots were developed. In opposition to classical ma-
1
nipulators, mobile robots do not have a limited workspace, they are not physically bounded to a certain
position of space. In particular, teleoperation was implemented in the control of unmanned ground vehi-
cles (UGV). Initially they were used mainly for image extraction, reconnaissance, surveillance and target
acquisition [3, 4]. However, with the industrial growth they became an important presence in field appli-
cations, such as crop harvesting, land demining and indoor transportation [5]. In spite of their extended
maneuverability, UGVs are restricted to the surface on which they move.
With the beginning of flight history, the importance of aerial robots was accentuated due to the
number of causalities associated with early flights. Since then, aerial robotics have accompanied flight
evolution. During World War II, the first fixed-wing teleoperated unmanned aerial vehicles (UAVs), com-
monly named drones, appeared. Their initial purpose was for anti-aircraft training and posterior for
surveillance, reconnaissance and other war affairs [6, 7].
During the turn of the twenty-first century, the increase of processors power and the appearing of
more reliable and cheaper sensors with smaller dimensions has caused a growing interest in UAVs. The
first commercial quadrotors appeared and their usage took a shift to more everyday life activities. A
quadrotor or quadcopter, is an aircraft propelled by four rotors. While typical fixed-wing aerial robots,
such as airplanes, require runways to gain speed for lift-off, quadrotors are capable of vertical take-
off and landing (VTOL), as well as hovering in constant position. For this reason, these vehicles are
preferable for indoor operations and when space is a constraint. They also present a few advantages
over helicopters. First, the use of four rotors allows for a relatively smaller diameter, also meaning less
kinetic energy. Second, quadrotors are easier to control and provide a less aggressive flight, which
allows an increased maneuverability in indoor operations. Finally, quadrotors are usually more crash-
resistant and less expensive to maintain and repair.
Recently, the quadrotors presence is growing stronger, being used for personal use and for activities
that require interaction with the environment. Some of the most relevant applications of quadrotors are:
inspection of structures with difficult or dangerous access [8, 9] (for example, inspection of high voltage
power cables [10]), search and rescue missions [11], journalism [12], investigation [13, 14] and for film
making. The use of quadrotors in these applications prevents the endangering of human lives while
reducing costs. However, for these activities the human operator needs to easily navigate the vehicle
while focusing on its mission. Typical controllers (RC transmiter) are not intuitive, beginners are usually
confused by its mechanisms and take a long time to master it. Husak [15] verified that using a haptic
controller allows a more intuitive flight and also provides the user a higher level of abstraction. From
the greek haptesthai, meaning ”to contact” or ”to touch”, haptic refers to the use of tactile sensation
in human interaction with computers. Furthermore, in Husak experiments, beginners using a haptic
controller were able to complete an obstacle course with reduced number of collisions.
Besides the physical controller, teleoperation is usually accompanied by a video transmission from
the vehicle on-board camera. However, seeing only what is in front of the unmanned vehicle, the human
controller is not able to perceive its surrounding. Some solutions may include the use of more than
one camera or a PTZ (pan-tilt-zoom) camera. However, these solutions require an extra weight on the
unmanned vehicle, being an issue for aerial vehicles for example. Aside from not being able to precisely
2
recognize proximity to visible obstacles, there is no awareness of obstacles in other directions.
Having this in mind, in this thesis we complement the process of teleoperation by using the sense
of touch. In other words, we implement and evaluate a teleoperation scheme based on force feedback.
Through on-board sensors, the unmanned vehicle is able to detect obstacles in its surroundings. Be-
sides preventing any collision, due to vehicle’s anti-collision system, the vehicle control system transmits
the obstacles information to the master system. The master system then uses actuators to haptically
inform the operator of the presence of obstacles in the vehicle’s proximity.
Until now we have seen a brief summary of the origins and evolution of teleoperation. As we will see
in the next section, the distance between the operator and the vehicle raises various challenges in the
use of teleoperated vehicles. In the following, the organization of a teleoperation system is explained,
along with the main difficulties and goals of teleoperation.
1.2 Problem definition
A generic unmanned vehicle bilateral teleoperation system can be represented by the diagram in Fig.1.1.
The user interacts with the teleoperation system through the master device. The master device is con-
sidered the set of devices used to receive and transmit information to the operator, for example: RC
transmitters, joysticks, computer mouse, keyboards, screens, sound columns and others. In haptics, the
master device consists of at least a haptic controller capable of transmitting forces and motions between
the user and the master controller. The master controller is responsible for mapping the user inputs to
the respective motion of the slave, as well as calculating the appropriate force feedback (e.g., inversely
proportional to the distance to existing obstacles).
The slave controller is composed of a high-level controller and a low-level controller. The high-
level controller trades information with the master controller through the communication channel and,
if necessary, adapts the master output according to the slave state and surrounding, e.g., to prevent
a collision. The low-level controller is in charge of regulating the dynamics of the slave (usually aerial
vehicles are unstable and require a control system to maintain flight). The slave device refers to the
physical vehicle that is being operated. This schema can be applied to any aerial, ground or underwater
vehicle.
Human operator
Master Device
Master controller
Communication channel
Slave controller
Slave device
Environment
Human operator
Master Device
Master controller
Communication channel
Slave controller
Slave device
Environment
Teleoperation system
Figure 1.1: Generic architecture of bilateral teleoperation
3
The main challenges in classical robotic teleoperation are caused by network induced imperfections
such as time delays and packet losses, which are primary sources of instability and lack of transparency.
Besides these, the haptic teleoperation of unmanned vehicles adds some additional challenges. First,
there is a dissimilarity between the master and the slave workspaces. While the master is bounded by
the device mechanism, the slave vehicle has usually no limits to its movement. Second, there is no
direct interaction between the vehicle and the environment, i.e., the haptic feedback does not originate
from physical contact with the environment, but from artificial fields built around obstacles. Third, the
states of the vehicle are difficult to measure and most of the times they have to be estimated.
In teleoperation there are two main goals, stability and transparency [16]. The teleoperation is stable
if the stability of the closed-loop system is established irrespective of the behavior of the operator or the
environment. The stability of the system is usually verified through the passivity of the system. Assuming
that the environment and human operator are passive, and if the teleoperation passivity is verified, then
we can guarantee passivity of the closed-loop [17].
Slawinski et al. [18] introduce the concept of absolute transparency, which measures the trans-
parency of bilateral teleoperation through three criteria: local, remote and instantaneous transparency.
The first two quantify the similarity between the state of the slave and how it is felt by the operator, and
between the state of operator and how it is seen by the slave, respectively. The third criterion measures
the delay in communication and how fast the operator feels the slave and vice-versa.
These goals are generally conflicting. However, satisfying these requirements assures the function-
ality of the teleoperation system.
1.3 Related work
Before 2009, most of the works regarding the use of haptic methods for teleoperation of single UAVs has
been made by Lam et al. [19–21] and Brandt and Colton [22]. These first works focus mainly in the use
of haptic feedback to assist the user and provide a collision free teleoperation. Moreover, in [23], Lam et
al. reduce the effects of communication time delays through introduction of wave variables. Mahony et
al. [24] investigated the use of optical flow to provide force feedback to an operator’s joystick to facilitate
collision free teleoperation.
More recent works include a series of studies by Mersha et al. [25–28]. In these studies, an approach
based on network theory and port-Hamiltonian systems is used to formulate the proposed teleoperation
scheme and evaluate its passivity. In [29], Stramigioli et al. introduce the concept of virtual slave which
is later extended by Mersha et al. [30] to a multidimensional and underactuated case. Other relevant
works include the application of an admittance framework for teleoperation of UAVs by Mahony and
Hou [31–34].
All aforementioned works share a similar conclusion: the teleoperation of unmanned vehicles can be
improved through the use of a haptic feedback transmitting information regarding the state of the vehicle
and its surroundings to the remote operator.
4
1.4 Objectives and Contributions
The main goal of this work is the development of a haptic teleoperation scheme that enhances the
performance of the human operator during complex tasks that require interaction of unmanned vehicles
with the environment. The developed scheme should help operators to perform their teleoperation task
safely and efficiently, while reducing the workload.
With respect to the stated goals, the main specific contributions of this thesis are summarized as
follows:
1. Design of a new collision avoidance method (section 2.4.2), and performance comparison with
existing solutions. The solution comprehends two steps: i) the determination of the obstacles
principal direction vector; and ii) the agreement of the operator requested velocity vector with the
previously obtained obstacles vector. Two systems may be applied for this agreement, avoidance
or deconfliction systems. The former, slows down the vehicle to avoid collision. The latter, adapts
the vehicle velocity so it is both safe and similar to the desired velocity. This method is implemented
directly in the vehicle system, providing a safe operation even if communication with the master
fails.
2. Design of a novel type of force feedback based on vibration (section 2.5.2). By taking advantage
of the sensibility to vibration, users should get a better perception of the vehicle surroundings
from the vibration produced by the haptic controller. This new solution increases the teleoperation
transparency and safety level while maintaining a low task workload.
3. Validation and verification of the feasibility and efficiency of the proposed control architecture
through experimental tests in a 2D scenario (Chapter 3). The proposed vibration feedback is com-
pared with other haptic feedbacks in an experiment where subjects had to teleoperate a ground
omnidirectional vehicle through an unseen and unknown maze. Furthermore, the proposed anti-
collision methods are evaluated during this experiment.
4. Design of a 3D simulator where a quadcopter could be virtually teleoperated through different
indoor scenarios (Chapter 4). This interface was afterwards used to evaluate the effectiveness of
the proposed vibration feedback in a 3D environment. In this second experiment, subjects had to
teleoperate a virtual quadcopter over three different 3D mazes.
Part of the work described in items 1 - 3 is accepted to be presented at ROBOT’2015: Second Iberian
Robotics Conference, being published in Springer’s Advances in Advances in Intelligent Systems and
Computing Series [35].
1.5 Structure of the Dissertation
The organization of this dissertation is briefly described as follows. In Chapter 2 the main components
of a generic haptic teleoperation scheme are divided into five sections, for each section the state of the
5
art is presented and if applicable it is followed up by the developed theory. Furthermore for each section,
implementation issues are discussed.
A first experiment where subjects had to teleoperate a UGV through an unseen maze is fully de-
scribed in Chapter 3. Afterwards, a second experiment using the quadcopter simulator is presented in
Chapter 4.
Finally, in Chapter 5, this work main conclusions are presented along with recommendations for
future investigation opportunities.
6
Chapter 2
Methods
In this chapter, the main theoretical aspects of teleoperation are discussed. The most common tech-
niques used in teleoperation of unmanned robots are presented along with newly developed techniques
which, afterwards, are compared.
This chapter is organized as follows. Section 2.1 presents the different possible interfaces between
the user and the haptic master device. Section 2.2 studies different mapping techniques for teleopera-
tion, that is, how the signal from the master device can be mapped as a reference for the slave vehicle.
Afterwards, section 2.3 focuses on different obstacle perception methods, in which the vehicle surround-
ings is evaluated and represented by variables that can later be used for the following two sections.
Section 2.4 uses the previously obtained parameters to adapt the vehicle reference and prevent any col-
lision with the environment. Finally, in section 2.5, the different methods that can be used to haptically
inform the user of the vehicle surroundings (haptic feedbacks) are discussed.
In each of these sections, a brief overview of the state of the art is presented and, when applicable,
followed up by the developed theory. Afterwards, implementation issues are discussed and, if necessary,
validated. In the final section, an experiment where different haptic feedbacks were compared is also
presented.
2.1 Haptic interfaces
In haptics, there are two types of interface between the user and the master device: impedance and
admittance. Impedance refers to the case where the user applies a displacement to the master device,
and the device responds with a force (Fig. 2.1(a)). On the other side, in admittance control, the user
applies a force to the master device, and the device responds with a proper displacement (Fig. 2.1(b)).
User Master Slave
d
f
vr
fr
(a) Impedance interface.
User Master Slave
f
d
vr
dr
(b) Admittance interface.
Figure 2.1: Haptic interfaces.
7
Impedance is the most common configuration in haptic devices [27, 29, 36]. Typical impedance con-
figured devices are low inertia, low friction and backdrivable devices that can measure the position of the
end-effector and exert force and/or torque to the user [37]. The most commonly used impedance devices
in haptic teleoperation are joysticks with force feedback, the Novint Falcon, the Geomagic Phantom, and
the Force Dimension Omega series.
Admittance devices are generally heavy, stiff and not easily backdrivable [37]. These devices are
capable of measuring the input force applied to its end effector, and respond with an appropriate dis-
placement. Most of the work in teleoperation with unmanned aerial vehicles using admittance interface
was done by Hou and Mahony [31–34, 38]. In [38], Schill et al. applied the first admittance framework
for hovering vehicles. The results obtained are promising, however the device used (Novint Falcon)
does not provide the necessary rigidity for admittance control, causing a faulty feedback perception.
Using a modified Novint Falcon, in 2013, Hou controled a quadrotor by scaling the force applied on the
master device to the speed of the vehicle. The speed of the vehicle was then mapped to the position
of the master end effector [31]. Afterwards, Hou and Mahony realized a series of studies using admit-
tance interfaces [32–34]. During these studies, two admittance devices were developed, including an
admittance trackball capable of force feedback. In these works, the authors defend that an admittance
interface provides the pilot with a better feel of the vehicle dynamics.
In [39], Soares compares the use of impedance and admittance interfaces in the teleoperation of
aerial vehicles. Results show that during free flight, the admittance framework provides the user with
better haptic cues regarding the state of the vehicle. However, in the presence of obstacles and when
environment interactivity is desired, the use of an impedance interface is preferred. The main difference
verified in these interfaces is their bandwidths. Admittance acts as a low pass filter, which is adequate for
slave pose tracking. On the other hand, impedance acts as a high pass filter which is more appropriate
for fast changing cues, e.g., a force feedback regarding the presence of an obstacle.
2.1.1 Implementation
This thesis presents a teleoperation scheme that allows the control of unmanned vehicles in obstacle-
laden environments. Accordingly to the results previously referred, an impedance interface should allow
a more appropriate control for this type of scenarios. Furthermore, the master device used in this work,
the Novint Falcon, has already been ruled out for for admittance control. As mentioned before, Hou and
Mahony [31, 38] have already applied an admittance framework with a Novint Falcon and arrived at the
conclusion that this device is not appropriate for admittance control. For these reasons, we opted to use
an impedance interface for the developed teleoperation scheme.
One of the functions of the master controller is to ensure a stable control of the master device end
effector. Typically, in impedance, when released, the end effector of the master device should move
to the center of operation, which corresponds to a zero velocity reference for the slave. While some
devices achieve this through physical springs and dampers (for example, most type of joysticks), haptic
manipulators require this control to be implemented through a virtual spring and damper system.
8
The implemented spring and damper system block diagram is represented in Fig. 2.2. The user
applies a force, fa, on the end effector, which causes the end effector to move. The end effector dis-
placement, qm, is then used to calculate the spring and damper force, which we denominate as local
force, fl. During teleoperation, the user does not control the applied force, but rather the position of the
end effector, which is eventually used to map the slave reference. Consequently, during teleoperation,
instead of forcing a displacement of the end effector, the local force should simply increase the user
perception of the actual position of the end effector. As a reminder, if an admittance interface was used,
the force applied by the user would be the system input, and the position of the end effector would be
solely controlled by the master controller.
Master Device end effector
Local Force Renderer
fa
fl
qm +
+
Figure 2.2: Local Force diagram.
The local force, fl, can be represented by the following expression:
fl = −Km(qm − qo)−Cmqm (2.1)
where Km and Cm correspond to spring and damper matrix coefficients, vectors qm and qm are the
position and velocity of the master device, respectively and qo is the zero position of the master device.
The local force parameters (Km and Cm) have to be tuned taking into account the following aspects:
teleoperation transparency, teleoperation comfort, and when released, stability of the manipulator. Re-
garding the spring coefficient, Km, as it increases, the workspace center position and frame axes be-
come more accentuated, therefore, increasing the user perception of the workspace. For example, if the
user is pushing the manipulator forward and desires to move slightly to the side, a well-defined spring
force would allow the user to easily understand if the position of the manipulator is coincident with the
respective axis or located near to the axis. On the other hand, if Km is too low, the user might be unable
to perceive it, and consequently loose understanding of the workspace. However, a high spring coeffi-
cient also increases the required force to move the manipulator, resulting in an increased task workload
as well as a reduction of the process comfort. Furthermore, a strong spring force may override other
forces, such as a force feedback.
A solution to improve the spring coefficient tuning is to limit the spring force through saturation. Limit-
ing the spring force allows to use a higher spring coefficient while avoiding its disadvantages. The values
used as boundaries for the spring force should take into account the previously mentioned objectives as
well as the haptic device being used and its characteristics.
The damping coefficient, Cm, has an important role in diminishing the oscillation of the manipulator
when released. However, if this value is too high, the user will experience a feeling of drag in the end ef-
9
fector that will contribute to increase the sensitive overload of the user, hence reducing the transparency
of the teleoperation. In order to increase the teleoperation comfort and transparency, the manipulator is
allowed to have small oscillations when released, as long as its stability is assured.
2.2 Master-Slave Mapping
In order to overcome the difference in the workspaces of the master and the slave devices, a car driving
metaphor [40] (rate control) is usually used [27, 41, 42]. In opposition to pose control, where the pose
of the master is mapped to the pose of the slave, in rate control, the pose of the master is mapped to
the velocity reference of the slave. This method allows the human operator to manipulate an unbounded
robot through a bounded device.
When using the impedance framework, the mapping of the master device position is described as
qsr = λq qm (2.2)
for pose control and as
vsr = λv qm (2.3)
for rate control, where λq and λv are scaling matrices and qsr and vsr represent the desired reference
position and velocity for the slave, respectively. It is common in impedance control to feedback to the
master device the error between the desired position/velocity and the real slave position/velocity. This
feedback allows the user to feel the dynamics of the teleoperated vehicle.
When an admittance framework is used, the mapping is described as:
qsr = λq fa (2.4)
vsr = λv fa (2.5)
respectively for pose and rate control, and where fa represents the force applied by the user on the
master haptic device. The feedback is done through a displacement of the master device, that represents
the actual movement of the slave for pose mode. For rate mode, the master control moves the master
device to a position that represents the real velocity of the slave.
Some authors consider that rate control does not provide the user with the precision required for
potential application areas such as inspection and object manipulation. To address this issue, Mersha
et al. present a kinetic scrolling-based position mapping [25]. This algorithm is inspired by the kinetic
scrolling of smart phones. The mapping strategy has two modes, direct and sliding mode. In direct
mode, the master’s position is mapped to the slave’s reference position. In sliding mode, the vehicle
slides accordingly to the speed of the vehicle at the moment of switch. To realize the two modes, a
virtual mass, mv, is coupled with the master device through a spring-damper, during direct mode. In
sliding mode, the virtual mass is decoupled from the master and a viscous damper is introduced. The
10
position of this virtual mass is then scaled to a reference position of the slave.
When covering fairly large areas, the scrolling-based technique requires a constant switching be-
tween the two modes. For this reason, the same authors present another solution, a switching-based
mapping [26, 28]. The switching-based mapping combines the rate and the pose based control strate-
gies to maneuver the unmanned vehicle, guaranteeing control in unbounded environments and in-
creased precision. In this solution, the operator is allowed to switch the current mode (pose or rate)
by pressing a button on the haptic device. However, switching at a nonzero position of the master re-
quires alternative interpretations. According to the interpretations, two submodes are identified for each
mode, namely the normal and reset submodes.
When switching from rate to pose mode at a nonzero master position, two situations may occur. The
current position of the slave is interpreted as an offset between the master’s zero position and the slave.
This corresponds to resetting the center of the workspace to its original position. The second case is to
define the current position of the master as the new center of operation. The two cases represent the
reset and normal submodes, respectively.
Following the same reasoning, when switching from pose to rate mode at a nonzero master position,
the current position of the master can be interpreted as either a nonzero velocity reference or a zero
velocity reference. These two situations are referred to as reset and normal submodes, respectively.
In both pose and rate modes, the normal submode results in a smoother transition. However, the
normal submode may result in an asymmetrical mapping of the master’s position. The submode should
be chosen accordingly to the situation for a better performance.
Omari et al. [43] also introduced a new mapping capable of handling the issue of precision versus
unlimited workspace. In this method, based on the bubble technique [44], a virtual sphere is created
around the zero position of the haptic device. Inside this sphere, a pose mapping is applied, i.e., the
position of the haptic device is mapped to the position of the slave. Outside the sphere, rate mode is
applied.
When the master device is inside the sphere, the mapping directly relates the position of the master
with the slave position, around the center of operation. In this case, the center of operation remains
constant. When the operator moves outside the position sphere, the center of operation starts moving in
the direction of the movement. The speed at which the center of operation moves, and consequently the
slave, is determined by the position of the end effector. When the operator reenters the position sphere,
the center of operation stops moving and the slave can be operated around this new position.
The authors referred above defend that rate mode may not provide the necessary precision for teleop-
eration of unnamed vehicles. However, initial teleoperation tests using an UAV support that the precision
obtained using a rate control is satisfactory and suitable for this thesis objectives. Consequently, in this
thesis the master-slave mapping is implemented using a rate control. The remaining mapping methods
are presented as alternatives for future work.
11
2.3 Obstacles Perception
2.3.1 State of the Art
The first step to generate a haptic feedback is to examine the slave surroundings and calculate a suitable
intensity and direction for the force feedback. The most common method is to use artificial force fields
(AFF), also referred to as potential fields. Originally, AFFs purpose is to map obstacles as repulsive
forces acting on the vehicle and targets as attractive forces. Summing the repulsive forces of each ob-
stacle with the attractive force of the target destination, results in a final vector containing the magnitude
and direction for a collision free path towards the destination.
In teleoperation, the human operator decides where to go in real time. Consequently, there are no
target destinations or attractive forces. However, the repulsive forces can be mapped to feedback forces
that will assist the operator during the teleoperation process.
The artificial force field proposed in [45] was based on the distance between a robot arm and an
object. However, since the algorithm only takes into account the distance to objects, repulsive forces
would always be present, even when moving away from the obstacle. Applying this method in haptic
teleoperation would require the operator to constantly counteract the repulsive forces, increasing the
operation workload.
In [46], Krogh presents the generalized potential field (GPF), which takes into account the vehicle
deceleration limits and the speed in relation to an obstacle. The GPF is calculated as the inverse of the
difference between the maximum (tmax) and minimum avoidance time (tmin). The minimum avoidance
time is the time required to bring the speed towards the obstacle to zero, using the vehicle maximum
deceleration amax. The maximum avoidance time is defined as the time required to brake the vehicle
just before collision, using a constant deceleration (smaller than the maximum deceleration).
The GPF still has some limitations if used for haptic feedback. As the difference between tmax
and tmin tends to zero, the potential field tends to infinity which would be impractical due to hardware
limitations. Furthermore, the GPF only maps obstacles in the direction of movement, which could result
in collision with lateral obstacles.
In [21], Lam extends the concept of GPF by including a saturation of the potential field and using
distance as stop criteria instead of time. The modified potential field is referred to as basic risk field
(BRF). Instead of using the difference between the maximum and minimum avoidance times, a reserve
avoidance distance is defined as the difference between the braking distance, d, when decelerating with
less than the maximum deceleration and the minimum distance, dstop, required to brake at maximum
deceleration. Figure 2.3 shows an example of the resulting GPF in two situations, vehicle is static and
moving forward.
The BRF is normalized between zero and one, respectively for minimum risk and maximum risk
situation. This field has the advantage of calculating force for small negative velocities at small distances
from obstacles. The main disadvantage appointed by the author [21] is the rather large size of the field
for high velocities, which may result in a difficult control through narrow passages.
In [21], Lam also presents another potential field that allows more control over the size and shape
12
Figure 2.3: Two-dimensional contour plots of the BRF, adapted from [21]. (a) vs = 0 m/s. (b) vs = 4 m/s.
of the risk field: the parametric risk field (PRF). The PRF consists of two boundaries: a maximum risk
boundary which limits the critical region with a risk of one, and an outside boundary from which the risk
becomes zero. The critical zone is defined by a vehicle protection radius plus a braking distance, dstop,
similar to the one used in BRF. The outer boundary is defined by a distance, dmin, from the critical region
and a complementary distance which depends on the direction of motion and compensates the operator
reaction time. Between boundaries, the risk value is defined by the ratio between the distance to the
critical zone and the distance between boundaries in that section.
The contour of the PRF is exemplified in figure 2.4 for the vehicle in a static position and in a moving
forward case. As the figure shows, this AFF produces a more restricted contour in comparison to BRF.
After obtaining an AFF, a final avoidance vector has to be calculated. Assuming that obstacles are
detected by angular sensors, the AFF can be used to calculate the risk magnitude for each sensor radial
line. Afterwards, the individual risk vectors obtained have to be integrated into a final vector. There are
several ways to join the individual vectors: sum, mean and max/min, i.e. the sum of the largest positive
and negative risk vectors. In all cases, the final vector is limited to one.
In [32], Hou proposes a new approach for obstacle perception, the dynamic kinesthetic boundary
(DKB). It is constructed as a boundary for the vehicles velocity based on the distance to the obstacle.
This boundary takes into account two parameters, a safety distance and a threshold distance. Safety
distance is used to limit how close the vehicle will approach obstacles while the threshold distance
defines at which distance the field starts acting. Besides using the value obtained to send a signal to
the user, this value is also used to limit the velocity of the vehicle. In case of perfect velocity tracking,
the dynamic kinesthetic boundary ensures that the vehicle never collides with the environment.
13
Figure 2.4: Two-dimensional contour plots of the PRF, adapted from [21]. (a) vs = 0 m/s. (b) vs = 4 m/s.
2.3.2 Newly Developed Methods
This thesis presents two newly developed obstacle perception methods. The first method consists of
obtaining a principal obstacle vector that represents the major direction and proximity of obstacles sur-
roundings the vehicle. The second method, developed by Ruivo [35], uses the information regarding the
slave environment to define the allowed velocity space, similarly to the DKB.
In the first method, we first need to calculate the principal direction of obstacles surrounding the ve-
hicle. Consider spherical coordinates in the body fixed frame of the slave represented by radial distance
r, azimuth θ and elevation φ. Let d (θ, φ, t) denote the radial distance from the origin of the body fixed
frame of the slave to the first obstacle along direction (θ,φ) at time t. For each point inside an observation
zone, defined by the observation radius Ro, consider a unitary vector pointing from the obstacle to the
vehicle frame origin, described by:
u (θ, φ, t) = − cos θ cosφ i− sin θ cosφ j− sinφk (2.6)
The principal direction of obstacles, θobs, is computed by taking a weighted average of all unitary
vectors within the observation zone (d (θ, φ, t) < Ro):
θobs(t) =
θd(t)
‖θd(t)‖, if ‖θd(t)‖ 6= 0.
0, otherwise.(2.7)
θd(t) =∑θ,φ
u(θ, φ, t) f(Ro − d(θ, φ, t)) (2.8)
14
where f is a monotonic crescent function, with f(0) = 0. For obstacles near the observation radius, f
has a close to zero value. On the other hand, for obstacles closer to the vehicle, f presents a higher
weighting value. In other words, since the directions of obstacles are weighted accordingly to their
proximity, closer obstacles will have a higher impact in the final direction vector. Changing the function
f allows to define the relative importance amongst close and distant obstacles. This results in a final
unitary vector that points away from the main direction of obstacles.
Figure 2.5 presents some 2D examples of the resultant unitary obstacle direction vector for three
different scenarios. The values used for these examples were extracted from a LIDAR planar scanner
during teleoperation of a UGV. The vehicle is represented by a cross in the center of the figure, the
thick line represents the obstacles detected by the sensor and the small vectors represent the weighted
unitary vectors of each obstacle point detected. In the first case, the vehicle is close to a wall, and
the resultant obstacle direction vector points perpendicular to the wall. In the second case, two small
obstacles are represented. In this case each obstacle pushes the vehicle away and the final vector
pushes the vehicle away from both obstacles. In the third scenario, the vehicle is close to a corner.
Since the vehicle is equidistant from both walls, the final direction vector points away from the corner.
If the vehicle was closer to one of the walls, the vectors originated from that wall would have a higher
weight and the final vector would consequently become slightly inclined away from that wall.
x [m]
y[m
]
Ro3obs
(a) Wall.
x [m]
y[m
]
Ro3obs
(b) Two small obstacles.
x [m]
y[m
]
Ro3obs
(c) Corner.
Figure 2.5: Examples of resulting obstacle direction vector for different scenarios.
The magnitude of the risk is calculated using the minimum value of d (θ, φ, t) inside the observation
zone:
dobs(t) = min (d (θ, φ, t)) (2.9)
Instead of calculating a value between zero and one for the risk value, the distance to obstacles is
used directly. Smaller distances correspond to higher risk values. In conclusion, this method allows to
describe the slave environment using a single direction vector, θobs, and a risk magnitude, dobs.
The second method used was developed by Ruivo [35]. As in the previous method, this method
starts by selecting all points inside the observation zone, defined by the observation radius, Ro. Af-
terwards, obstacles are identified by clustering consecutive points and subsequently these clusters are
15
simplified through a line simplification method. This results in each obstacle being defined by a set of
line segments. The final step is to calculate the closest point to the vehicle frame for each line segment.
Unlike the first method that resulted in a single vector, this method results in a group of feature vectors,
each defined by a distance and a direction. The number of resulting feature vectors depends on the
slave surroundings.
The main difference between the obstacle perception methods present in the bibliography and the
proposed methods is how the risk magnitude is calculated. Typically, the risk magnitude is obtained as
a relation between the current speed of the vehicle and the distance to obstacles (e.g. time to impact
and reserve avoidance distance). This procedure makes sense since high risk situations will correspond
to stronger feedbacks that will force the user to take countermeasures. However, when an autonomous
anti-collision system is implemented, this notion of risk loses its significance, since the anti-collision
system will stop the vehicle from colliding. On the other hand, the transparency of the process becomes
more relevant. For example, using a time to impact procedure, the resulting magnitude of the feedback
when moving the slave against an obstacle can be the same for both a large distance at high slave
speed, against a small distance at low speed. This causes the user to be unable to understand how close
the slave is to the obstacle. By calculating the risk magnitude solely on the distance to obstacles, the
proposed methods provide a more transparent teleoperation, while the anti-collision method guarantees
the process safety.
2.4 Anti-Collision System
Besides sending a haptic feedback to the user, the results of the methods presented in the previous sec-
tion can also be used for collision avoidance. In this case the risk vector may be adapted in accordance
to the collision avoidance system.
2.4.1 State of the Art
Most authors consider obstacle avoidance in teleoperation through the haptic device [21, 41, 47]. After
evaluating the slave surroundings, a corresponding feedback is sent to the master device, which in turn,
either directly or through the user reaction, causes a displacement of the end effector and consequently
guides the vehicle away from the obstacle.
In [32], Hou uses the dynamic kinesthetic boundary to simultaneously drive a signal for haptic feed-
back and to limit the velocity of the slave along the direction to the obstacle. As the vehicle approaches
an obstacle, the maximum velocity in that direction is reduced until it becomes null at a predetermined
safety distance from the obstacle. Similarly, in [48], a distance to obstacles function is used to modify the
reference velocity of the slave. However, in this case, repulsive velocities are calculated for all obstacles
and summed into a final repulsive velocity which is added to the slave reference velocity.
16
2.4.2 Newly Developed Methods
Two obstacle perception methods were applied for collision avoidance. The first method (main direction
method) uses the main direction and distance to obstacles, the second method (clipping method) uses
the obstacles feature vectors. For both methods, two systems were implemented, an avoidance system
and a deconfliction system. The former ignores the user’s command if it means getting closer than
allowed to an obstacle. The latter, besides preventing the vehicle from getting too close to an obstacle,
also adapts the vehicle velocity to be as similar as possible to the velocity reference. Figure 2.6 shows
a diagram of the anti-collision methods used.
Main direction
method
Avoidance
Deconfliction
Avoidance
Deconfliction
Clipping
method
Anti-collision
methods
Figure 2.6: Anti-collision methods diagram.
Although some authors use the haptic feedback to ensure a collision free teleoperation, the delays
present in communication between master and slave will affect the efficiency of the anti-collision system.
Furthermore, unsuccessful communication between master and slave may jeopardize the system safety
and result in collision. By implementing the anti-collision directly in the vehicle, we can assure that even
if communications are cut down, the vehicle will not hit an obstacle.
Distance to Obstacle [m]
0 Rc
Rd
Rw
Veh
icle
vel
oci
ty[m
/s]
-Vmax
0
Vmax Critical
ZoneDangerZone
WarningZone Safe Zone
Figure 2.7: Relation between velocity and distance to obstacle.
The approach used for anti-collision is represented in figure 2.7 for the one dimension case. Accord-
ing to the distance to obstacle, the speed reference towards the obstacle is limited. For an obstacle
in the safe zone, the speed towards the obstacle is only restricted by the maximum physically allowed
speed of the vehicle, Vmax. Below the warning radius, Rw, the speed is reduced until it becomes zero
at distance Rd, the danger radius. If the obstacle enters the danger zone, the maximum allowed speed
becomes negative, i.e. the vehicle is forced to move away from the obstacle. In normal conditions the
obstacle should never enter the critical zone. If it does happen then the vehicle moves away from the
obstacle at maximum speed. The critical zone is defined by the critical radius, Rc, which consists of the
17
vehicle radius plus an extra safety distance. The critical radius also guarantees safety during rotational
movements, even for non-circular vehicles.
Proceeding to a real case scenario with two or three dimensions, the problem becomes more com-
plex. For example, when pushing the slave against a wall, the anti-collision system should limit the
speed towards the wall to avoid a crash. However, the user should still be able to freely move the vehicle
parallel to the wall. In figure 2.8, the two approaches used (avoidance and deconfliction system) are
represented for the wall example. With the avoidance system, if there is any obstacle along the direc-
tion of the velocity reference, then the vehicle velocity is reduced according to the relations previously
shown. This results in the vehicle being slowed down until it stops at a radial distance Rd from the
obstacle. On the other hand, if the deconfliction system is used, then only the component of the velocity
towards the obstacle is limited and any tangential component is kept unchanged. For the wall example,
the deconfliction system results in an intuitive smooth displacement along the wall.
x position
y po
sitio
n
Rd
avoidancedeconflictionwallspeed reference
Figure 2.8: Vehicle approaching wall with constant velocity reference. Difference between avoidancesystem and deconfliction system.
The first anti-collision method is implemented through comparison of the velocity reference vector
with the main obstacle vector. To verify if the desired velocity reference has a component directed
towards an obstacle, we just need to verify if the dot product between the velocity reference vector and
the main obstacle vector is negative:
θobs(t) · vsr(t) < 0 (2.10)
If this condition is verified, then the vehicle reference velocity has to be adjusted accordingly to the
distance to obstacle. The distance to obstacle, or risk value, is evaluated using the variable dobs. For
the plain avoidance system, if the obstacle is inside the warning zone (Rd < dobs < Rw) then the speed
reference is proportionally reduced as the distance to Rd reduces:
vsc = vsr
dobs −Rd
Rw −Rd(2.11)
where vsc represents the slave corrected reference velocity. If the obstacle gets closer than Rd, then the
user command is ignored and the vehicle is pushed away from the obstacle with the same direction as
18
θobs. The new velocity reference is given by:
vsc = Vmax θobsdobs −Rc
Rd −Rc(2.12)
Inside the critical zone, the vehicle is always pushed away from the obstacle at maximum speed:
vsc = Vmax θobs (2.13)
Remember that θobs is an unitary vector that points away from obstacles.
Figure 2.9 presents examples of the resulting corrected velocity in the presence of a wall at different
distances. In the first case, the wall is inside the warning zone so the corrected velocity corresponds
to a proportional reduction of the original reference velocity. In the second case, at distance equal to
the danger radius, the corrected velocity is null. Finally, in the third case, inside the danger zone, the
reference velocity is ignored and the corrected velocity gets the direction of θobs.
x [m]
y [m
] vref
3obs
vc
(a) Wall inside the Warning Zone.
x [m]
y [m
]
vref
3obs
(b) Wall in the Danger Radius.
x [m]
y [m
]
vref
3obs
vc
(c) Wall inside the Danger Zone.
Figure 2.9: Corrected velocity, vc, for the vehicle approaching a wall from south, using avoidance system.The reference velocity is constant for the three cases.
For the deconfliction system, these equations become slightly different. For an obstacle inside the
warning zone, only the component of the reference velocity towards the obstacle is reduced. This
component is obtained using the dot product:
vsc = vsr − θobs (θobs · vsr)Rw − dobsRw −Rd
(2.14)
Notice that the term θobs (θobs · vsr) represents the component of the reference velocity towards the
obstacle. The termRw − dobsRw −Rd
varies from zero to one as the distance to obstacle gets closer to Rd. So
as the vehicle gets closer to an obstacle, the component of the velocity towards the obstacle is gradually
reduced from its velocity reference. If the obstacle enters the danger zone, then the component of the
velocity towards the obstacle is completely removed and starts increasing in the opposite direction:
vsc = vsr − θobs (θobs · vsr) + θobs VmaxRd − dobsRd −Rc
(2.15)
If the obstacle gets closer than Rc, the vehicle is repelled from the obstacle at maximum speed:
19
vsc = vsr − θobs (θobs · vsr) + θobs Vmax (2.16)
The examples previously presented using the avoidance system are now presented in Fig. 2.9 for
the deconfliction system. In the first case, inside the warning zone, the velocity reference is once again
reduced, however, when using the deconfliction method, only the vertical component of the velocity
(which is parallel to θobs) is reduced. In the second case, at distance equal to the danger radius,
the vertical component becomes null, being left only the horizontal component, which is parallel to
the wall. In the final case, inside the danger zone, the vertical component of the velocity reference
changes direction while the horizontal component is unchanged. It can easily be seen that when the
avoidance system was used, the vehicle was forced to stop when directed at the wall. However, using
the deconfliction system, the vehicle maintains a velocity parallel to the wall, while the velocity towards
the wall is adapted to maintain a safety distance.
x [m]
y [m
] vref
3obs
vc
(a) Wall inside the Warning Zone.
x [m]
y [m
]
vref
3obs
vc
(b) Wall in the Danger Radius.
x [m]
y [m
]vref
3obs
vc
(c) Wall inside the Danger Zone.
Figure 2.10: Corrected velocity, vc, for the vehicle approaching a wall from south, using deconflictionsystem. The reference velocity is constant for the three cases.
In conclusion, the developed avoidance anti-collision system can be described by the following set
of equations:
vsc =
vsr
dobs −Rd
Rw −Rd, if Rd ≤ dobs < Rw and (θobs · vsr) < 0.
Vmax θobsdobs −Rc
Rd −Rc, if Rc ≤ dobs < Rd.
Vmax θobs, if dobs < Rc.
vsr , otherwise.
(2.17)
while the deconfliction anti-collision system is described by the equations:
20
vsc =
vsr − θobs (θobs · vsr)Rw − dobsRw −Rd
, if Rd ≤ dobs < Rw and (θobs · vsr) < 0.
vsr − θobs (θobs · vsr) + θobs VmaxRd − dobsRd −Rc
, if Rc ≤ dobs < Rd and (θobs · vsr) < 0.
vsr − θobs (θobs · vsr) + θobs Vmax, if dobs < Rc and (θobs · vsr) < 0.
vsr , otherwise.
(2.18)
The second anti-collision approach used in this thesis was developed by Ruivo [35]. This method
starts by drawing the allowed velocity space of the slave. For example, for a planar omnidirectional
vehicle, the initial allowed velocity space can be described by a circle of radius Vmax, the maximum
speed allowed for the vehicle (Fig. 2.11).
vx [m/s]
-Vmax 0 V
max
v y[m/s]
-Vmax
0
Vmax
Figure 2.11: Initial velocity space.
Afterwards, the feature vectors obtained during the obstacle detection process are used to clip the
allowed velocity space. The clipping is done in accordance to the distance to obstacle relation previously
presented (Fig 2.7). Figure 2.12 shows an example of the clipping done by a single object approaching
the vehicle from north-east.
vx [m/s]
-Vmax 0 V
max
v y[m/s]
-Vmax
0
Vmax
(a) Obstacle inside the Warning Zone.
vx [m/s]
-Vmax 0 V
max
v y[m/s]
-Vmax
0
Vmax
(b) Obstacle in the Danger Radius.
vx [m/s]
-Vmax 0 V
max
v y[m/s]
-Vmax
0
Vmax
(c) Obstacle inside the Danger Zone.
Figure 2.12: Resulting allowed velocity space for an obstacle approaching the vehicle from north-east.
Finally, after doing the clipping for all obstacle feature vectors, the reference velocity is adapted
accordingly to the new allowed space velocity. Figure 2.13 represents the different possible scenarios
21
for the avoidance system. If the slave reference velocity belongs to the allowed space velocity, then it
is unchanged. If the reference velocity is outside the allowed space, two situations may occur. If the
desired velocity crosses the allowed velocity space, its magnitude is reduced until it is inside the allowed
space (2.13(b)). If it does not cross the allowed space, it is ignored and substituted by the closest to the
origin point (no velocity) that belongs to the allowed space (2.13(c)).
vx [m/s]
-Vmax 0 V
max
v y[m/s]
-Vmax
0
Vmax
(a) Desired velocity inside allowed veloc-ity space.
vx [m/s]
-Vmax 0 V
max
v y[m/s]
-Vmax
0
Vmax
(b) Desired velocity crossing allowed al-lowed velocity space.
vx [m/s]
-Vmax 0 V
max
v y[m/s]
-Vmax
0
Vmax
(c) Desired velocity outside allowed ve-locity space.
Figure 2.13: Avoidance scenarios. Circle defines velocity space. Shaded area: allowed velocity space.Black arrow: desired velocity. Red arrow: corrected velocity.
The same scenarios shown for the avoidance system are presented in figure 2.14 for the decon-
fliction system. The deconfliction system always takes into account the desired velocity. If the slave
reference velocity belongs to the allowed space velocity, then, as in the avoidance case, it is unchanged.
If the reference velocity is outside the velocity allowed space, then it is always substituted by the clos-
est velocity that belongs to the allowed space. This method allows a collision-free teleoperation while
maintaining a displacement as similar to the desired as possible.
vx [m/s]
-Vmax 0 V
max
v y[m/s]
-Vmax
0
Vmax
(a) Desired velocity inside allowed veloc-ity space.
vx [m/s]
-Vmax 0 V
max
v y[m/s]
-Vmax
0
Vmax
(b) Desired velocity crossing allowed al-lowed velocity space.
vx [m/s]
-Vmax 0 V
max
v y[m/s]
-Vmax
0
Vmax
(c) Desired velocity outside allowed ve-locity space.
Figure 2.14: Deconfliction scenarios. Circle defines velovity space. Shaded area: allowed velocityspace. Black arrow: desired velocity. Red arrow: corrected velocity.
Although providing similar results, in the presence of multiple obstacles the clipping method provides
a more robust solution for both avoidance and deconfliction systems. However, the signal used for the
main direction method can simultaneously be used as a haptic feedback reference. Furthermore, the
22
main direction method requires less processing time. The method used should be chosen according to
the available processing time. If, for example due to sensors low sampling rates or low communication
rates, the anti-collision system is not the limiting factor, then the clipping method should be used. Oth-
erwise, if the anti-collision system is limiting the cycle rate, then the main direction method should be
used.
2.5 Haptic Feedback
In the preceding sections we saw some of the most recent methods used to analyze the vehicle sur-
roundings and provide an obstacle related single final vector of risk. Besides being used for anti-collision
systems, this vector can also be used to hapticly transmit relevant information to the user. In this section
we present some solutions regarding how this trade of information can be made. Furthermore, a novel
haptic feedback based on vibration is also presented.
2.5.1 State of the Art
Assuming we have a vector containing information concerning the vehicles surrounding obstacles, e.g.
the final risk vector obtained using AFFs, we now want to use the master device to transmit a simple
signal to the user hand so that the user becomes aware of the present risk without increasing the task
workload. The master device is considered to have a local spring-damper system that pushes the end
effector to its center of operation. The first idea that comes to mind is to directly map the risk vector as
a force to be exerted on the master end effector. In [19], Lam presents this method as force feedback.
This force points away from the obstacle and its intensity is proportional to the magnitude of the risk, i.e.
the force increases with proximity to an obstacle.
The force feedback can be interpreted as a force offset (fig. 2.15), shifting the neutral position of the
master device. When moving in the direction of an obstacle, the force exerted by the operator in the end
effector has to increase to maintain it in the same position. If the operator releases the end effector, the
slave vehicle will move away from the obstacle until it reaches a safety distance, since the end effector
is pushed to a non-neutral position. In the presence of an obstacle, this method requires the operator
to counter the force feedback to maintain a neutral position, therefore, increasing the task workload.
Furthermore, when navigating through a narrow corridor or close to multiple closely-spaced obstacles,
the force offsets may vary continuously, causing the operator to be unable to accurately comprehend the
state of the vehicle. A simple solution is to reduce the force feedback gain. However, reducing the force
feedback gain may result in forces too small for the operator to feel.
In [19], Lam also presents a second feedback method, the stiffness feedback. The stiffness feed-
back is interpreted as an extra spring to the local spring-damper system, which stiffness increases with
proximity to an obstacle. As in the preceding case, the operator needs to increase the applied force to
the end effector in order to maintain the reference velocity when heading to an obstacle. In this case, no
haptic feedback is given to the operator when the end effector is in a neutral position (fig. 2.15) or when
23
displacement [m]
0
forc
e[N
]
0
localforcestifness
Figure 2.15: Representation of local force, force feedback and stiffness feedback along stick displace-ment. Adapted from [19].
moving away from an obstacle. If released, the end effector moves to the neutral position.
Stiffness feedback increases situation awareness with reduced task workload. However, stiffness
feedback presents some drawbacks. This method requires that the user moves the end effector to be
able to feel the feedback and for small displacements the feedback might be unperceivable, since small
displacements result in relatively small repulsive forces. In comparison with force feedback, stiffness
feedback has the disadvantage of not autonomously moving away from obstacles, since the end effector
is pushed to a neutral position.
Posteriorly, a series of studies [23, 49, 50] were done using a combination of force and stiffness
feedback and finally in [20] the force-stiffness feedback is presented. Combining force and stiffness
feedback allows preservation of the force feedback autonomous obstacle deviation. The stiffness com-
ponent permits a reduction in force feedback gain without loss of transparency, which therefore results
in improved stability and less control signal oscillations when close to obstacles. Experimental results
using force-stiffness feedback show that this method significantly increases the safety of teleoperation,
however it did not result in a lower workload compared to force feedback alone [50].
Results using the previously mentioned haptic feedback methods indicate that the use of haptic
feedback increases teleoperation safety at the cost of increased workload. Therefore, a trade-off must be
made between safety and workload. Lam points out that it might be more relevant to aim for high levels
of safety with same workload, than to reduce workload while maintaining the same level of safety [50].
2.5.2 Vibration Feedback
During this thesis, a new type of feedback was developed, the vibration feedback. The vibration feedback
is designed as a pulsated force feedback which frequency increases with proximity to obstacles. Some
of the methods mentioned before require the user to counteract the feedback forces, possibly increasing
the workload of the task. As verified in [51], we believe that by taking advantage of the vibrotactile
sensing, it is possible to intuitively transmit information of the slave surroundings without demanding
high physical efforts from the operator. Moreover, small vibrations seem easier to perceive than small
24
increases in force, therefore improving the human perception of obstacles through the master device.
time [s]
am
plitu
de
[N]
0
A
width
period
Figure 2.16: Vibration feedback example.
Figure 2.16 provides an example of vibration feedback and respective pulse parameters to be ad-
justed: width, period and amplitude. The duration of the pulse (width) should be chosen according to the
capacities of the master device. In case a too small value is used, the user might be unable to perceive
the vibration feedback. On the other side, higher width values will make the user feel small kicks instead
of a simple vibration, causing discomfort.
Both amplitude and frequency (inverse of the period) of the signal should increase with proximity to
obstacles. Humans are capable of tactile feeling vibrations with frequencies ranging from 0.4 to 1000 Hz
with optimal sensitivity achieved at frequencies between 150 and 300 Hz [52]. However, common haptic
manipulators are not capable of exerting such frequencies. Consequently, the bandwidth used should be
as close as possible to the optimal range. To further increase the available bandwidth for feedback, the
amplitude and period of the signal should vary as represented in Fig. 2.17. First, the period is reduced
with proximity to the obstacle, while the amplitude of the force is kept constant. After reaching minimum
desired period, the amplitude starts increasing. In the safe zone, there is no haptic feedback so the
amplitude of vibration is set to zero. If the obstacle is closer than the danger radius, then the frequency
and amplitude variables are kept at maximum value.
peri
od [
s]
0
pf
pi
Distance to obstacle[m]
0 Rd
Rw
ampl
itude
[N
]
0
ai
af
Figure 2.17: Vibration feedback period and amplitude along distance to obstacle.
25
Besides extending the usable range of operation, the proposed combination of period and amplitude
also ensures that the feedback starts by providing a light vibration, and only when closer to an obstacle
the intensity of vibration is increased.
Since the vibration is designed as a pulsated force, besides the width, amplitude and period, the
direction of application can also be adjusted. When using the main direction of obstacles, we want the
vibration feedback to always point along the main direction of obstacles. However, the feedback should
only be given if the user is driving the vehicle towards the obstacle, i.e., if the end effector is being
pushed in the direction of the obstacle. This can be done through comparison of the angle between the
end effector position and the obstacle main direction vector. If the end effector is at the neutral position
or pointing along the main obstacle direction (the main obstacle direction vector points away from the
obstacle), there will be no feedback. As the user starts pushing the end effector close to the obstacle,
the feedback sensation should increase and reach its maximum when the desired direction is coincident
with the obstacle. Let qm be a vector that represents the end effector position, with origin at the center of
operation. Then the amplitude of vibration can be multiplied by a factor, Af , which depends on the angle
between desired direction and the obstacle direction. This factor is given by the following expression:
Af =
(−2(
qm
‖qm‖· θobs
)− 1
)2
, ifqm
‖qm‖· θobs < 0.5
0, otherwise(2.19)
with a value between zero and one. If the angle between the obstacle direction and the desired direction
is bigger than 60◦, then the amplitude is set to zero, which corresponds to having no feedback. Other-
wise, if the angle is smaller than 60◦, Af has an increasing value and reaches a value of one for an angle
equal to 0◦.
2.5.3 Implementation
Before implementing the haptic feedback system, the haptic device should first be tested in order to find
out the right parameters to be used. Even if using the same type of device as the one used in this thesis
(Falcon from Novint), these verifications should be done, since the results depend on the state of the
device.
To find the lower limit for the width of the vibration signal, the device end effector was held in place
while a pulsated signal with constant frequency was sent to the device (this pulsated signal controls
the desired force to be applied on the manipulator, not the desired position). Afterwards, the recorded
position of the end effector was compared with the pulsated signal to verify if the device was able to
track the pulsated signal.
Figure 2.18 shows the results obtained for three different values of width using a frequency of 10 Hz.
As can be seen, when the width of the signal is smaller than 0.02 s the device becomes unable to track
the signal and starts skipping some of the pulses. While holding the manipulator, this inability to follow
the pulses can actually be felt as a reduction in frequency and degrades the signal interpretability.
To measure the lower limit of the period of the signal, a similar approach was used. While holding the
26
(a)
positionpulse(b)
time [s]0 0.5 1 1.5 2
(c)
Figure 2.18: Master device response to pulsated signal with different width values and constant fre-quency equal to 10 Hz. (a) width = 0.02 s. (b) width = 0.015 s. (c) width = 0.010 s.
end effector, a pulsated signal was sent to the master controller with constant width and which frequency
was changed between tests. Afterwards, the recorded position of the end effector was compared with
the desired signal. The results obtained for different values of period are presented in Fig. 2.19. As can
be observed, the device started to fail tracking the signal for periods with value smaller than 0.04 s which
corresponds to a frequency of 25 Hz. Once again, the lower limit of the period corresponds to the lowest
value of period where the device is still capable of tracking the desired pulsated signal.
(a)
positionpulse(b)
time [s]0 0.2 0.4 0.6 0.8 1
(c)
Figure 2.19: Master device response to pulsated signal with different period values and constant widthequal to 0.02 s. (a) period = 0.04 s. (b) period = 0.035 s. (c) period = 0.030 s.
The previous results indicate that, using the Novint Falcon, the smallest values that can be used for
width and period for the vibration feedback are 0.02 s and 0.04 s respectively. Since the obtained values
are above the optimal sensitive bandwidth, 150 and 300 Hz, they can be used for implementation. The
pulse width of the vibration signal was set as constant and equal to 0.02 s. The value of 0.04 s was
used as the lower boundary of the signal period. For the upper boundary of the signal period, a value
of 0.1 s was used. The value 0.1 corresponds to the highest value for which the signal was still easily
27
and quickly perceived as a warning signal. The resulting feedback has a frequency that varies from 10
to 25 Hz, which is outside and below the optimal bandwidth.
2.5.4 Methods Comparison Results
With the objective of comparing the effectiveness of the developed vibration haptic feedback with feed-
backs normally used in teleoperation, an experiment was conducted. In this experiment, subjects had
to identify the direction of a virtual obstacle through a haptic feedback transmitted by the master device.
Setup
Three haptic feedbacks were tested during this experimentation: force feedback, stiffness feedback and
vibration feedback. Remembering, the force feedback acts as constant force that points away from the
obstacle and which intensity increases with proximity to the obstacle; the stiffness feedback increases
the stiffness of the master device when the end effector is moved towards the obstacle, stiffness increase
is proportional to the proximity to obstacle; finally, the vibration feedback causes the end effector to
vibrate when moved towards an obstacle, vibration frequency increases with proximity to obstacle.
During this experiment, each subject had to complete eighteen trials, six trials for each type of feed-
back. In each trial, a virtual obstacle would appear in one of eight possible directions in the horizontal
plane. As time passes during a round, the distance to the obstacle reduces at a steady rate, increasing
the force feedback intensity, and consequently, making the identification of the obstacle easier. At the
beginning of each trial, the distance to the obstacle is equal to the warning radius, corresponding to
minimum feedback activity. After twenty seconds it reaches maximum feedback activity. A round ends
when the user makes a guess, independently of being right or wrong. To find the obstacle direction,
subjects had to move the master end effector and try to feel the haptic feedback. Figure 2.20 presents
the visual interface of this experiment. After identifying the location of the obstacle, subjects could move
the haptic device to directly point a direction and select it by pressing a button on the device. The order
of the eighteen rounds and the direction of obstacles was randomly generated for each subject.
Figure 2.20: Visual interface, one button for each direction.
28
Twenty eight subjects with little to no experience using haptic devices participated in this experiment.
The subjects were informed of the objectives of the task and about the presence of different haptic
feedbacks. However, the subjects were not aware of how many different feedbacks were used, or how
they worked. The master device used was the 3 DOF fully-actuated haptic joystick Novint Falcon. The
performance evaluation of each haptic feedback was based on the number of correct guesses and on
the time taken to make a decision.
Results and discussion
The main results of the haptic experiment are shown in Fig. 2.21. A full-factorial ANOVA [53] test was
applied. In average, subjects correctly answered 67.2 % of the times using vibration feedback, against
37.5 % and 45.8 % for force and stiffness feedbacks respectively - see Fig. 2.21(a). The respective
average response times for vibration, force and stiffness feedbacks are 13.48 s, 16.98 s and 18.33 s -
Fig. 2.21(b).
force sti, vib
corr
ect
answ
ers
[%]
0
20
40
60
80
100
(a) Percentage of correct answers for each hapticfeedback. ANOVA test (p<0.001).
force sti, vib
tim
e[s]
0
10
20
30
40
(b) Average responde time for each haptic feed-back. ANOVA test (p<0.01).
Figure 2.21: Main results of haptic experiment for each type of feedback: force, stiffness and vibrationfeedback. Central mark = median; Box = 25th and 75th percentiles; whiskers = min and max values.
The obtained results show that the vibration feedback outperformed the two other feedbacks in both
response time and perception transmission efficiency. As expected, even with low intensity, vibration
allows the user to quickly gain awareness of an obstacle presence. By moving close to an obstacle,
stiffness and vibration feedback intensity would increase making it easier to select a direction. On the
other side, subjects had more difficulty choosing between adjacent directions when using the force
feedback.
29
Chapter 3
2D Experiment
With the purpose of comparing different haptic feedbacks and two collision avoidance systems in a
real case scenario, we conducted an experiment which consists of solving an unseen maze. During
this experiment, subjects had to teleoperate a custom made omnidirectional ground vehicle through an
unseen and unknown obstacle-laden course.
3.1 Platform OMNI-ANT
The vehicle being teleoperated during this experiment is the OMNI-ANT shown in figure 3.1. This vehicle
was designed as an omnidirectional vehicle to obtain a movement similar to the 2D motion of a typical
multirotor with altitude regulation. The holonomic movement is achieved with three omni wheels present
in the vehicle that allow the vehicle to move and rotate in any desired direction in the plane.
Figure 3.1: Slave vehicle, OMNI-ANT.
The central processing unit of the vehicle is an Arduino Mega ADK and the motors are controlled via
a MD25 motor controller board. The communication with the master is done wirelessly through an XBee
31
radio module. To perceive its surroundings, the vehicle is equipped with a Hokuyo URG-04LX-UG01, a
planar LIDAR scanner. The sensor main characteristics are presented in table 3.1.
Table 3.1: LIDAR scanner main characteristics.
Model URG-04LX-UG01
Output digitalCommunication USBRange 4 mAccuracy ± 3 % of measurementSample rate 10 HzScan angle 240◦
Angular resolution 0.36◦
3.2 Goals and Objectives
In this experiment, the three haptic feedbacks were implemented and compared: force feedback, stiff-
ness feedback and vibration feedback. An extra case was implemented where no haptic feedback was
used. This case should allow a better evaluation of the haptic feedbacks effectiveness. Furthermore,
the two different anti-collision methods (avoidance and deconfliction) were tested, both using the allowed
velocity space system.
The evaluation of each method was based on the following variables: task completion time, NASA
TLX workload index [54] and the results of questionnaires made to the user after each task. The task
completion time is the total time required to complete the maze. After each run, subjects were asked to
rate their workload using the NASA TLX rating scale, which assumes that the user workload is composed
of six sources: mental demand, physical demand, temporal demand, performance, effort and frustration
level. For each source, subjects use a scale to provide a classification from low to high. Besides
completing the NASA TLX, subjects also answered the following questionnaire:
• Was it easy to be aware of obstacles?
• Was it confortable to operate the vehicle?
• Was it easy to execute the task?
• Was the force felt on the manipulator helpful?
For each question, subjects selected a value from one to five, where one corresponds to ‘No’ and five to
‘Yes’.
3.3 Procedure
During this experiment, each subject had to complete a task that consists of teleoperating the slave
vehicle, OMNI-ANT, through an unseen maze. This task was to be completed eight times by each
32
subject, four times using the avoidance anti-collision method and four times using the deconfliction anti-
collision method. For each anti-collision method, four different haptic feedbacks were used (one haptic
feedback for each trial): force feedback, stiffness feedback, vibration feedback and no feedback. In an
attempt to reduce the learning curve effect and to obtain unbiased results, the order of the eight trials
was randomly generated for each subject.
3.4 User Interface
The user interface was composed of a screen, a haptic device and a keyboard and mouse used to
answer the questionnaires. As can be seen in Fig. 3.2(a), the view of the user was blocked, making it
impossible to see the vehicle and the maze.
The haptic device used during this experiment is the Falcon from Novint (Fig. 3.2(b)). This device is
connected to a desktop computer running Matlab, which controls the master device and connects with
the slave wirelessly though a XBee. The linear velocity of the vehicle is controlled through translation of
the master device end effector in a horizontal plane (vertical movement is not used in this experiment).
The rotation of the vehicle is obtained by pressing one of two buttons present in the grip (one button
for clockwise rotation and other for anti-clockwise rotation). If released, a spring and damper system
pushes the end effector to the center of operation causing the vehicle to stop.
(a) User interface. (b) Novint Falcon.
Figure 3.2: Master system.
During operation, a top view of the slave surroundings was displayed on the screen (Fig. 3.3). This
image is created using the distance values obtained by the LIDAR scanner. The vehicle is drawn in the
middle of the display and the detected obstacles appear around it, resulting in a view similar to a radar
image. After each trial, a questionnaire would appear on the screen so subjects could promptly fill them
online. Furthermore, this allowed to immediately save all results in a format suitable for post analysis.
33
Figure 3.3: visual interface. LIDAR scanner top view.
3.5 Subjects and Instructions
Altogether, twenty eight subjects between ages 20 and 49 years of age with little to no experience using
haptic devices participated in this experiment. Before beginning the experiment, subjects were told
about the presence of different types of haptic feedbacks and to be aware of them. However, they were
not told how many different feedbacks existed, neither the dissimilarities between them. Furthermore,
subjects did not know which feedback was being used at which round or the existence of a round without
feedback. As a main goal, subjects were challenged to find the exit of the maze as fast as possible, while
maintaining a special attention to the haptic feedbacks.
With the purpose of obtaining some insight of how the vehicle moves and to gain some sensibility
with the haptic device, before starting the tests, subjects were allowed to pilot the vehicle while seeing it.
Afterwards, the vehicle was placed inside the unseen and unknown maze and subjects were required to
teleoperate the vehicle towards the exit of the maze. The trial ended when the vehicle exited the maze.
During tests the protocol was not to give any instructions regarding the maze or the vehicle control. After
completing each trial, subjects had to fill the NASA TLX rating sheet, which promptly appeared on the
screen. Upon completion of the first questionnaire, the second questionnaire (previously mentioned)
appeared. At the end of the experiment, subjects were asked to talk about how they felt during this
experience, what they liked the most and what they did not like. This allowed to obtain, in a more
informal way, information that could not be obtained through specific questionnaires.
3.6 Maze
The maze used during this experiment is shown in figure 3.4. The maze is shaped like a ‘U’ with some
characteristics intended to decrease proximity to obstacles and consequently increase haptic feedback
activity. Halfway, an obstacle placed in the center of the 180 deg turn creates two possible paths. While
34
the exterior path is quite wide, allowing an easy passage, the inner path width is just slightly wider than
twice the danger radius. As a result, the user has to be a little insistent to pass through this choke point.
If this path was even tighter, the obstacle avoidance system would not allow the vehicle to go through.
At the end of the maze, an obstacle forces the user to move slightly to the right, once again to increase
haptic feedback activity.
The obstacle avoidance parameters used during this experiment were: 0.5 m, 0.25 m and 0.2 m,
respectively for the warning radius, danger radius and critical radius.
Figure 3.4: Top view of the maze used for experiments.
3.7 Results
This section presents the main results of this 2D experiment. Before examining the results, it was verified
that the first trial done by each subject was affected mostly by the subjects capacities, rather than the
influence of the methods used. Consequently, the first trial, and for a few cases the second trial, of each
subject, were not included in the results analysis.
A first analysis compares the overall results using the avoidance system and the deconfliction system.
A significant decrease in the duration of the task when the deconfliction system is used is confirmed
in the results of Fig. 3.5(a), with a reduction in the average completion time from 46.94 s to 34.30 s.
The NASA TLX scores (Fig 3.5(b)) indicate that the deconfliction system resulted in less workload, as
expected.
The mean and standard deviation of the questionnaires results between avoidance and deconfliction
systems can be observed in table 3.2. Results indicate that subjects considered that the deconfliction
system provides a more comfortable teleoperation and made the task easier, one way ANOVA results
confirm this difference (p<0.001). On the other hand, the influence of the anti-collision system in obsta-
cle awareness was statistically insignificant. The effect of the haptic feedback was also unaffected by
the anti-collision system used.
Regarding the different haptic feedbacks used, figure 3.6(b) shows a slight increase in workload for
35
AS DS
tim
e[s]
0
20
40
60
(a) Task completion time. ANOVA test (p<0.0001).
AS DS
work
load
0
20
40
60
(b) NASA-TLX scores. ANOVA test (p<0.001).
Figure 3.5: Comparison results between avoidance system (AS) and deconfliction system (DS). Centralmark = median; Box = 25th and 75th percentiles; whiskers = min and max values.
Table 3.2: Comparison of questionnaire results between anti-collisionsystems. Answers were scaled from one to five, being one ’No’ and five’Yes’.
Avoidance Deconfliction
meanstandarddeviation
meanstandarddeviation
Was it easy to beaware of obstacles?
4.54 0.68 4.74 0.60
Was it confortable tooperate the vehicle?
4.37 0.71 4.68 0.67
Was it easy toexecute the task?
4.39 0.63 4.68 0.60
Was the force felt onthe manipulator helpful?
3.78 1.20 3.69 1.49
the force feedback. However, the difference in workload between vibration feedback, stiffness feedback
and no feedback was statistically insignificant. Similarly, task completion time (see Fig. 3.6(a)), was not
affected by the type of feedback used, except for the force feedback in which an increase was noticed.
The increase in task completion time verified with force feedback may be due to the force feedback
ability to move the master device end effector to a non-neutral position, which may cause the vehicle to
do extra movements and consequently increase the time necessary to complete the task. Results also
show that subjects were slightly faster when using no haptic feedback. Since there were no collisions
or penalties for getting too close to obstacles, subjects could have a carefree operation. However, upon
receiving a feedback warning, subjects took more time to increase the distance to obstacle, resulting in
an increased operation safety at the cost of an increase in task completion time.
The mean and standard deviation of the values obtained from the questionnaires for each haptic
feedback is shown in table 3.3. Accordingly to the increase in workload when using the force feedback,
users also classified this feedback as the most uncomfortable. As expected, subjects considered the
case when no haptic feedback was used, to be the least helpful case. Nevertheless, using no haptic
36
no force sti, vib
tim
e[s]
0
20
40
60
(a) Task completion time. ANOVA test (p<0.05).
no force sti, vib
work
load
0
20
40
60
(b) NASA-TLX scores. ANOVA test (p<0.05).
Figure 3.6: Comparison results between no haptic feedback, force, stiffness and vibration feedbacks.Central mark = median; Box = 25th and 75th percentiles; whiskers = min and max values.
feedback case was still classified as relatively helpful in comparison to the remaining feedbacks. This
high classification may be due to some misperception by the user during teleoperation, for example
misinterpreting the local force as a haptic feedback. As can be observed, the difference for the rest of
the results between each type of haptic feedback was not significant.
Table 3.3: Comparison of questionnaire results between different haptic systems. Answers were scaledfrom one to five, being one ’No’ and five ’Yes’.
No feedback Force Stiffness Vibration
meanstandarddeviation
meanstandarddeviation
meanstandarddeviation
meanstandarddeviation
Was it easy to beaware of obstacles?
4.54 0.81 4.55 0.69 4.75 0.51 4.71 0.53
Was it confortable tooperate the vehicle?
4.68 0.51 4.25 0.92 4.59 0.65 4.59 0.63
Was it easy toexecute the task?
4.64 0.62 4.25 0.77 4.66 0.48 4.59 0.56
Was the force felt onthe manipulator helpful?
3.39 1.51 3.66 1.31 4.00 1.19 3.88 1.33
In the end of the experiments, subjects were asked to try to identify the different kinds of haptic
feedback used and differentiate their favorite (Fig 3.7). Although having a lower score, the stiffness
feedback was placed in second most of the times, whereas the vibration and force feedback switched
between first and third favorite choices. Some subjects either liked all solutions with haptic feedback or
were unable to identify the different feedback solutions.
3.8 Discussion
Results show that, as expected, the deconfliction system outperformed the avoidance system both in
task duration reduction and in task workload reduction. Regarding the use of haptic feedback, results
37
none force sti, vibpre
fere
nce
[%]
0
10
20
30
40
50
4
8
4
12
Figure 3.7: Feedback type preference.
indicate that haptic feedback can indeed increase obstacle awareness. It was also verified that the
task workload was the lowest when no haptic feedback was used, this is expectable and, as mentioned
before, an increase in workload is acceptable if it means increasing the teleoperation safety.
Comparison between effectiveness of the different types of haptic feedbacks was not significant.
However, some causes can be pointed out. First, the visual interface had a strong presence during the
teleoperation and could sometimes override the haptic feedback, causing the user to focus on the screen
and ignore the haptic feedback. Besides, through the visual feedback, some subjects were able to
maintain a safety distance from obstacles, hence, reducing the haptic feedback activity. Second, during
questionnaires, many subjects would give answers close to the limits of the scale, and in subsequent
trials could not give higher (or lower) grades. Albeit inconclusive, results show that independently of the
methods used, subjects considered the teleoperation to be easy and confortable, and to have a relatively
low workload, which indicates that the proposed teleoperation scheme delivers a satisfactory solution
for a real case scenario.
Analysis of subjects opinions, concerning the type of feedback used, indicate that the used feed-
back should be adapted to each user, rather than choosing the haptic feedback that shows the highest
performance value. On the other side, although being successful in making subjects move the vehicle
away from obstacles, haptic feedback can be misinterpreted as a failure signal. During this experiment,
subjects were not penalized for getting close to obstacles. Introducing a penalty for getting too close to
obstacles may facilitate the comparison between haptic feedbacks effectiveness, and reduce the nega-
tive impact of successful haptic feedbacks.
38
Chapter 4
3D Experiment
In the previous experiment, subjects compared different haptic feedbacks through the teleoperation of an
unmanned ground vehicle. Even though results were satisfactory, the UGV represents a 2D scenario and
does not allow to classify the efficiency of haptic feedbacks in 3D scenarios, e.g., during teleoperation
of a quadcopter. In order to evaluate the proposed teleoperation scheme and the efficiency of the haptic
feedback in a 3D scenario, a second experiment was conducted. In this second experiment, subjects
had to control a simulated quadcopter through three unknown virtual mazes.
4.1 Quadcopter Model
This section presents the model and the control system of the virtual quadcopter used in this experiment.
The model of the vehicle and respective dynamics controller were provided by Ruivo [55], therefore only
a brief description is presented.
The stabilized attitude dynamics of the vehicle was designed as second order decoupled systems for
roll and pitch angles, and as a first order system for yaw angular velocity, as described in the following
equation:
φ
φ
θ
θ
ψ
ψ
=
0 1 0 0 0 0
−ωφn2
2ζφωφn 0 0 0 0
0 0 0 1 0 0
0 0 −ωθn2
2ζθωθn 0 0
0 0 0 0 0 1
0 0 0 0 0 − 4
Tψs
φ
φ
θ
θ
ψ
ψ
+
0 0 0
−ωφn2
0 0
0 0 0
0 −ωθn2
0
0 0 0
0 0 4
Tψs
φr
θr
ψr
(4.1)
where the quadcopter attitude is defined by the three Euler angles φ, θ and ψ, respectively, roll, pitch
and yaw, defined in the body fixed frame of reference, ωn is the natural frequency, ζ is the damping
coefficient and Ts is the settling time of the respective angle.
Afterwards, the position dynamics of the vehicle, as seen from the global frame of reference, is
39
modeled as that of a particle subjected to a force vector f , as follows,
x
x
y
y
z
z
=
0 1 0 0 0 0
0 0 0 0 0 0
0 0 0 1 0 0
0 0 0 0 0 0
0 0 0 0 0 1
0 0 0 0 0 0
x
x
y
y
z
z
+
0 0 0
1m 0 0
0 0 0
0 1m 0
0 0 0
0 0 1m
fx
fy
fz
(4.2)
where m is the quadcopter mass and x, y and z represent the position of the quadcopter along the
global frame of reference respective axis. The force applied to this particle, f , is the sum of thrust, t,
weight, w, and aerodynamic drag, d, defined as follows,
t = R (φ, θ, ψ)
0
0
tltmax
, w = m
0
0
−9.81
(4.3)
d = −R (φ, θ, ψ)DcR (φ, θ, ψ)T |v|vT (4.4)
where R (φ, θ, ψ) is the rotation matrix from quadcopter body frame to global frame, tl is the throttle
level which is designed as a first order system and tmax is the maximum throttle force. Dc is the drag
coefficient matrix and v the quadcopter velocity in the global frame.
Finally, a PID controller is implemented to calculate the appropriate attitude angles and throttle level
references according to the user desired speed (or corrected speed if an anti-collision system is active)
and yaw angular speed.
4.2 Goals and Objectives
The main goal of this experiment is to validate the usefulness of the vibration feedback in a 3D real case
scenario, for example for the teleoperation of a subaquatic or aerial vehicle. Taking this objective into
account, three different cases were implemented:
• A0F0 – Without anti-collision system and without haptic feedback.
• A0F1 – Without anti-collision system and with haptic feedback.
• A1F1 – With anti-collision system and with haptic feedback.
When the anti-collision is used, the quadcopter is unable to collide with the environment. The quad-
copter model is assumed to have incorporated sensors that provide the distance from the vehicle to
obstacle in all its planar directions, plus upward and downward direction, e.g., using a combination of a
planar LIDAR, for direction in the horizontal plane, with two sonars, for upward and downward direction.
Using the distance towards obstacles, the proposed anti-collision system is used (Section 2.4.2). For
40
the cases without collision avoidance system, it is possible to collide. In case of collision, a red light
flashes on the screen and the user is unable to move the vehicle for three seconds.
In this experiment only the vibration feedback is implemented. If the haptic feedback is activated,
subjects should feel a vibration on the manipulator when close to obstacles. Otherwise, if the haptic
feedback is not activated, subjects will not receive any type of feedback.
The evaluation of each case was based on the following variables: task completion time, number of
collisions and the results of questionnaires made to the user after each task. The task completion time
is the total time required to complete the maze. The number of collisions is the total number of collisions
during that trial. The questionnaire is composed of the following questions:
• Was it easy to be aware of obstacles?
• Was it confortable to operate the vehicle?
• Was it easy to execute the task?
• How strong was the force felt on the manipulator?
• Was the force felt on the manipulator helpful?
For each question, subjects selected a value from one to ten, where one corresponds to ‘No’ and
ten to ‘Yes’, except for the fourth question where one corresponds to ‘Did not felt it’ and ten to ‘Very
strong’. In the first experiment (Chapter 3) some subjects gave a response close to the maximum on
their first trial. Consequently, in subsequent trials, these subjects could not respond with a higher value.
By increasing the response range to ten, instead of the previously used value of five, users tendency to
choose responses close to the extremes is reduced. Furthermore, using a range between one and ten
allows a better differentiation of responses. Adding to this, since there is an even number of responses,
subjects are compelled to choose a negative or positive response.
4.3 Procedure
During this experiment, each subject had to complete a task that consists of teleoperating a virtual
quadcopter through one of three different 3D mazes. Each subject repeated this task nine times, three
times for each maze. For each maze the three different cases were used: A0F0, A0F1 and A1F1, one
for each trial. In an attempt to reduce the learning curve effect and to obtain unbiased results, the order
of the nine trials was randomly generated for each subject.
At the end of the experiment, subjects were asked if they wanted to repeat the experiment. If they
answered yes, then they would be able to repeat the nine trials (with a new order), resulting in extra
results. A post analysis with these extra results was done taking into account users added experience.
41
4.4 User Interface
As in the first experiment, the user interface was composed of a screen, a haptic device and a keyboard
and mouse used to answer the questionnaires, Fig. 3.2(a).
As before, the haptic device used during this experiment is the Falcon from Novint (Fig. 3.2(b)). This
device is connected to a desktop computer running Matlab, where a second Matlab instance runs the
simulator. The linear velocity of the vehicle is controlled through translation of the master device end
effector, including upward and downward directions. Rotation of the vehicle is obtained by pressing one
of two buttons present on the grip (one button for clockwise rotation and other for anti-clockwise rotation).
If released, a spring and damper system pushes the end effector to the center of operation causing the
vehicle to hover.
During operation, subjects had a first person view of the vehicle on the screen (Fig. 4.1). This view
is similar to when a camera is mounted on the vehicle, facing forward. To achieve this perspective, the
point of view is attached to the vehicle body fixed frame of reference and moves with the same attitude
of the vehicle - see Fig. 4.1(c) where the quadcopter is in a rolled position.
(a) (b) (c)
Figure 4.1: Examples of user visual interface.
4.5 Subjects and Instructions
Altogether, forty six subjects between 20 and 57 years of age with little to no experience using haptic
devices participated in this experiment. Contrary to what was done in the first experiment, before be-
ginning the experiment, subjects were told about the three possible cases, how they work and also how
the vibration feedback functions. They were told about the existence of different mazes, but were not
told how many different mazes they had to complete. Furthermore, subjects did not know which case
or maze was being used at each trial. As a main goal, subjects were challenged to find the exit of the
maze as fast as possible, while maintaining a special attention to the haptic feedbacks.
With the purpose of obtaining some insight on how the vehicle moves and to gain some sensibility
with the haptic device, before starting the tests, subjects were allowed to pilot the vehicle inside a small
square room without anti-collision system and without haptic feedback. Afterwards, subjects had to
complete one of three different mazes. The trial ended when the quacopter exited the maze. After com-
pleting each trial, subjects had to fill the previously mentioned questionnaire, which promptly appeared
42
on the screen, which also allowed to immediately save all results in a format suitable for post analysis.
During tests the protocol was not to give any instructions regarding the maze or the vehicle control. At
the end of the experiment subjects gave small comments regarding the experience.
4.6 Maze
The mazes used during this experiment are shown in figures 4.2, 4.3 and 4.4. These mazes have one
characteristic in common, they were designed with the intent of decreasing proximity to obstacles and
consequently increase haptic feedback activity. Through a combination of different rooms, corridors
and sideways, upward and downward curves, subjects are required to move the haptic manipulator in
all directions in order to complete each maze. The dark area in the first and second mazes (Figs. 4.2
and 4.3) represent a “no light” zone. Upon entering this zone, lights are turned off, causing a significant
decrease in visual perception. After exiting the “no light” zone, light returns to the normal state. The red
area in mazes one and three represents a usual obstacle which subjects had to pass below.
Theses mazes are built using the Matlab patch function. Afterwards, during operation, only the point
of view is modified guaranteeing the necessary speed to run the simulation in real time. The designed
simulator can easily be adapted for new experiments with different mazes and also for experiments of
different nature, for example, for biological learning where one might be interest in understanding how
users behave when confronted with certain obstacles.
The obstacle avoidance parameters used during this experiment were: 1.0 m, 0.3 m and 0.2 m,
respectively for the warning radius, danger radius and critical radius. A collision would occur at a distance
of 0.1 m.
50
40
x [m]
30
20
10
00 5
1015
y [m]
20
5
0
-5
z[m
]
(a) Isometric view.
x[m
]
0
5
10
15
20
25
30
35
40
45
50
y [m]0 5 101520
z [m]-505
(b) Top and side view.
Figure 4.2: Simulation maze 1. Blue dot represents initial position. Black area: “no light” zone. Redarrow: maze exit.
43
35
30
25
x [m]
20
15
10
5
00
5
10
y [m]
0
5
-5
z[m
]
(a) Isometric view.x
[m]
0
5
10
15
20
25
30
35
40
45
50
y [m]0 5 101520
z [m]0510
(b) Top and side view.
Figure 4.3: Simulation maze 2. Blue dot represents initial position. Black area: “no light” zone. Redarrow: maze exit.
4035
30
x [m]
2520
1510
500
510
15
y [m]
2025
-5
0
5
z[m
]
(a) Isometric view.
x[m
]
0
5
10
15
20
25
30
35
40
45
50
y [m]0 5 101520
z [m]0510
(b) Top and side view.
Figure 4.4: Simulation maze 3. Blue dot represents initial position. Red arrow: maze exit.
44
4.7 Results
This section presents the main results obtained. A first analysis compares the task completion time and
number of collisions between the three use cases. Figure 4.5(a) shows a reduction in task completion
time when the haptic feedback is used, with a reduction in the average completion time from 62.8 s
to 58.7 s and a further reduction to 57.7 s if the anti-collision system is used. Using the A1F1 case,
subjects were able to complete the maze almost 10 % faster in comparison with the A0F0 case. Results
present in Fig. 4.5(b) show that the use of haptic feedback had a major impact in reducing the number
of collisions, achieving a mean reduction of almost 50 % from 2.16 to 1.23 collisions for each trial. When
the anti-collision system was used, there were no collisions, so this case is not graphically represented.
A0F0 A0F1 A1F1
tim
e[s]
0
20
40
60
80
100
120
(a) Task completion time. ANOVA test (p<0.05).
A0F0 A0F1
Collisio
ns
0
2
4
6
(b) Number of collisions. ANOVA test (p<0.001).
Figure 4.5: Results comparison between different cases. Central mark = median; Box = 25th and 75thpercentiles; whiskers = min and max values.
The results of the questionnaire are presented in Fig. 4.6 according to the use case. Figure 4.6(a)
shows that, as expected, when using a haptic feedback, subject’s obstacle awareness increased signif-
icantly, with mean response values increasing from 7.70 to 8.56 and 8.67, respectively for the A0F1 and
A1F1 cases. Regarding the operation comfort, Fig. 4.6(b) shows similar results, with mean response
values increasing from 8.13 to 8.75 and 8.83, respectively for the A0F0, A0F1 and A1F1 cases.
Figure 4.6(c) shows an almost linear increase in task easiness for the three cases, with mean re-
sponse values varying between 7.42, 8.25 and 8.58, respectively for the A0F0, A0F1 and A1F1 cases.
Finally, figures 4.6(d) and 4.6(e) show the results of the questions concerning the haptic feedback.
For the A0F0 case, there was no haptic feedback applied, consequently subjects did not feel it and
therefore this case is not represented. Regarding the intensity of the haptic feedback, results indicate
that subjects felt the haptic feedback with slightly more intensity when the anti-collision system was
used. This increase in intensity is justified, since without colliding, subjects could stay more time close
to obstacles, therefore increasing the haptic feedback activity. In some cases, around 8 % of the trials,
users were able to maintain a safe operation without getting close to obstacles. Consequently, although
the haptic feedback was active, subjects did not feel the haptic feedback. For these cases, the response
to the last question loses meaning and therefore was not included in results analyses. Responses
regarding the helpfulness of haptic feedback indicate that the haptic feedback was considered helpful
45
independently of using an anti-collision system or not, with a mean response value of 8.41 and 8.40,
respectively for the A0F1 and A1F1 cases. In overall, the effectiveness of the haptic feedback was not
statistically affected by the use of an anti-collision system.
A0F0 A0F1 A1F1
Answ
er
2
4
6
8
10
(a) Was it easy to be aware of obstacles?ANOVA test (p<0.0001).
A0F0 A0F1 A1F1
Answ
er2
4
6
8
10
(b) Was it confortable to operate the ve-hicle? ANOVA test (p<0.001).
A0F0 A0F1 A1F1
Answ
er
2
4
6
8
10
(c) Was it easy to execute the task?ANOVA test (p<0.0001).
A0F1 A1F1
Answ
er
2
4
6
8
10
(d) How strong was the force felt on themanipulator?
A0F1 A1F1
Answ
er
2
4
6
8
10
(e) Was the force felt on the manipulatorhelpful?
Figure 4.6: Questionnaire results comparison for different cases. Central mark = median; Box = 25thand 75th percentiles; whiskers = min and max values.
4.8 Discussion
Analysis of this experiment results shows that, as expected, the use of haptic feedback not only in-
creases users obstacle perception, as it also significantly reduces task completion time and improves
operation safety through reduction of collisions. The introduction of an anti-collision system provided
a further improve in task completion time and safety. Although the reduction in task completion time
was not as significant as expected when using an anti-collision system, it is important to notice that the
collision penalty used (unable to move for three seconds) was very indulgent in comparison with a real
case scenario.
One possible problem that can be pointed out during haptic teleoperation, is that the haptic feedback
may be considered a source of distraction and, consequently, may difficult users operation and increase
the task workload. Results, however, indicate that the haptic feedback not only improved users perfor-
mance, as it also improved comfort and easiness during operation. Subjects also considered that the
level of intensity of the haptic feedback was adequate. Most subjects considered the haptic feedback to
be strong enough to be felt, but not too strong to cause discomfort.
A second objective of this experiment was to compare the effectiveness of the haptic feedback with
46
and without the use of an anti-collision system. On one side, with an anti-collision system, since there is
no risk of collision (in a real case this risk may still exist but is greatly reduced), users may consider that
the haptic feedback is forcing the user to do unnecessary movements. On the other side, with reduced
visibility of the vehicles surroundings, haptic feedback may help users understand if the desired path is
clear or not. Surprisingly, results show that the effectiveness of the haptic feedback was not affected
by the presence of an anti-collision system. Whether an anti-collision system is used or not, subjects
considered the haptic feedback to be equally helpful.
As a final remark, after concluding the experiment, subjects made a few comments and most of them
share the same conclusion: the haptic feedback was helpful in perceiving the vehicle surroundings and
during the cases where it was not active, subjects wanted it to be activated.
47
Chapter 5
Conclusions
The use of unmanned vehicles in operations that require human intervention is gaining relevance. How-
ever, typically, users have a reduced awareness of the vehicle surroundings and are not provided with
the level of abstraction required for these complex tasks.
Through implementation of the proposed haptic teleoperation scheme, the control of the vehicle
becomes more intuitive, increasing user comfort and decreasing task workload. Furthermore, the use of
a haptic feedback allowed to greatly improve user obstacle awareness, increasing teleoperation safety
and efficiency.
When compared with other methods present in the literature, the proposed obstacle perception
method presents a much simpler solution, requiring less computational time. Furthermore, resulting
in a single vector containing the obstacles main direction, the proposed solution can simultaneously be
used as a reference for both the anti-collision system and the haptic feedback. Since the obstacle per-
ception method is based only on distance to obstacle, the transparency of the teloperation increases in
comparison with other methods based on time to impact.
An anti-collision system that uses the obstacle main distance was also designed. As the obstacle
perception method, this anti-collision system requires a reduced processing time in comparison with
other methods. For the anti-collision method, two sub-modes were implemented, avoidance and decon-
fliction. Comparison of these two sub-modes in an experiment where 28 users had to teleoperate an
UGV through an unseen maze, reveals that, as expected, the deconfliction sub-mode guarantees a sig-
nificant reduction in task completion time as well as in task workload. In opposition to typical solutions,
the anti-collision system was implemented directly in the vehicle, assuring that the vehicle will not collide
even if communication with the operator fails.
A novel haptic feedback based on vibration was introduced. This vibration feedback takes advantage
of the vibrotactile sensing to provide a better perception of the slave environment with reduced task
workload. An experiment where subjects had to point the direction of a single obstacle was conducted.
Results show that the vibration feedback outperformed other haptic feedbacks in both time required to
perceive the obstacle and direction precision. Nonetheless, results regarding the use of different haptic
feedbacks in a real scenario support that there is no optimal type of haptic feedback. Instead of finding a
49
unique type of haptic feedback, results indicate that the haptic feedback should be individually adapted
for each operator.
A simulator of a quadcopter was developed using appropriate dynamics. In this simulator, subjects
can see the front of the quadcopter as if a camera was attached to the vehicle. An experiment was
conducted where 46 subjects had to teleoperate the virtual quadcopter through three different mazes.
Results show that the use of a haptic feedback had a major impact in increasing obstacle awareness,
and also contributed to an increase in operation comfort and easiness. Furthermore, when using a
haptic feedback, the number of collisions reduced significantly, as well as the task completion time. The
impact of the haptic feedback helpfulness was even greater in these 3D scenarios than it was in the
previous 2D experiment. During this experiment, it was also verified that the helpfulness of the haptic
feedback is independent of whether an anti-collision system is used or not.
As final remark, it can be concluded that the proposed haptic teleoperation scheme provides an im-
proved solution for the teleoperation of unmanned vehicles with increased vehicle surroundings aware-
ness and reduction in task workload.
5.1 Future Work
In this thesis, the process of teleoperation was enriched with an anti-collision system and a vibration
haptic feedback. Results indicate that this methodology decreased the risk of the teleoperation and
increased the user awareness of the vehicle surroundings. Nevertheless, regarding the safety and
transparency in teleoperation, there is always room for improvement.
Taking into account this study results, the following recommendation and future investigation oppor-
tunities are given:
• Master device: Explore the use of other haptic manipulators to control the vehicle. The Falcon
worked well. However, this device presented some limitations in terms of force control. Other com-
mercial haptic devices could be used or alternatively a custom manipulator could be constructed.
• Admittance interface: Implementation of an admittance interface and comparison with the pro-
posed scheme. This solution would require a different master device since, as mentioned before,
the Falcon is not appropriate for admittance control.
• Vibration feedback: During implementation of the novel feedback, the vibration bandwidth was
limited by the master device characteristics. The performance of this scheme should improve if an
optimal bandwidth is used (150 to 300 Hz). Once again, a different master device could improve
the obtained results. Another solution is the use of custom made grip with actuators that could
vibrate in local points without using the movement of the end effector. Using this last solution
would allow to mix the vibration feedback (applied only on the grip vibrating components) with
other haptic feedback such as force or stiffness feedback.
• Obstacle perception: Experimentally compare the proposed obstacle perception methods with
50
other methods present in the bibliography that use a time to impact approach rather than a distance
to obstacle approach.
• Experiments: Conduct user studies using a physical UAV in an indoor scenario. The proposed tele-
operation scheme is ready to be implemented on a real UAV. However, an infrastructure equipped
with cameras and nets is necessary to supervise the experiment and guarantee users safety.
• Transparency: This thesis focus on haptic teleoperation. However, the improvement of a teleoper-
ation transparency is not restricted to haptic feedbacks. Some solutions may include other senses
such as vision and hearing. For example, visual cues such as a red light on screen, to warn the
user of close obstacles. Temperature could also be used to inform the user of the state of the
vehicle, e.g., the grip of the master device could become warmer when close to obstacles.
51
Bibliography
[1] T. B. Sheridan. Telerobotics. Automatica, 25(4):487–507, 1989.
[2] J. Marescaux, J. Leroy, F. Rubino, M. Smith, M. Vix, M. Simone, and D. Mutter. Transcontinental
robot-assisted remote telesurgery: feasibility and potential applications. Annals of surgery, 235(4):
487, 2002.
[3] D. W. Gage. Ugv history 101: A brief history of unmanned ground vehicle (ugv) development efforts.
Technical report, DTIC Document, 1995.
[4] T. Fong and C. Thorpe. Vehicle teleoperation interfaces. Autonomous robots, 11(1):9–18, 2001.
[5] B. Siciliano and O. Khatib. Springer handbook of robotics. Springer Science & Business Media,
2008.
[6] K. Dalamagkidis, K. P. Valavanis, and L. A. Piegl. On integrating unmanned aircraft systems into
the national airspace system: issues, challenges, operational restrictions, certification, and recom-
mendations, volume 54. Springer Science & Business Media, 2011.
[7] J. F. Keane and S. S. Carr. A brief history of early unmanned aircraft. Johns Hopkins APL Technical
Digest, 32(3):558–571, 2013.
[8] I. Sa and P. Corke. Close-quarters quadrotor flying for a pole inspection with position based vi-
sual servoing and high-speed vision. In Unmanned Aircraft Systems (ICUAS), 2014 International
Conference on, pages 623–631. IEEE, 2014.
[9] S. Emelianov, A. Bulgakow, and D. Sayfeddine. Aerial laser inspection of buildings facades using
quadrotor. Procedia Engineering, 85:140–146, 2014.
[10] L. F. Luque-Vega, B. Castillo-Toledo, A. Loukianov, and L. E. Gonzalez-Jimenez. Power line in-
spection via an unmanned aerial system based on the quadrotor helicopter. In Mediterranean
Electrotechnical Conference (MELECON), 2014 17th IEEE, pages 393–397. IEEE, 2014.
[11] S. Waharte and N. Trigoni. Supporting search and rescue operations with uavs. In Emerging
Security Technologies (EST), 2010 International Conference on, pages 142–147. IEEE, 2010.
[12] M. Waite. Journalism with flying robots. XRDS: Crossroads, The ACM Magazine for Students, 20
(3):28–31, 2014.
53
[13] S. C. Williams. Studying volcanic eruptions with aerial drones. Proceedings of the National
Academy of Sciences of the United States of America, 110(27):10881, 2013.
[14] W. Zang, J. Lin, Y. Wang, and H. Tao. Investigating small-scale water pollution with uav remote
sensing technology. In World Automation Congress (WAC), 2012, pages 1–4. IEEE, 2012.
[15] B. R. Husak. Quadrotor 3d intuitive flying. Bachelor’s thesis, Faculty of Mathematics and Physics
(MFF), Prague, June 2014. Id:134662.
[16] D. Lawrence et al. Stability and transparency in bilateral teleoperation. Robotics and Automation,
IEEE Transactions on, 9(5):624–637, 1993.
[17] P. F. Hokayem and M. W. Spong. Bilateral teleoperation: An historical survey. Automatica, 42(12):
2035–2057, 2006.
[18] E. Slawinski, V. Mut, P. Fiorini, L. R. Salinas, et al. Quantitative absolute transparency for bilateral
teleoperation of mobile robots. Systems, Man and Cybernetics, Part A: Systems and Humans,
IEEE Transactions on, 42(2):430–442, 2012.
[19] T. Lam, M. Mulder, and M. R. Van Paassen. Haptic feedback for uav tele-operation-force offset
and spring load modification. In Systems, Man and Cybernetics, 2006. SMC’06. IEEE International
Conference on, volume 2, pages 1618–1623. IEEE, 2006.
[20] T. M. Lam, M. Mulder, and M. M. van Paassen. Haptic interface in uav tele-operation using force-
stiffness feedback. In Systems, Man and Cybernetics, 2009. SMC 2009. IEEE International Con-
ference on, pages 835–840. IEEE, 2009.
[21] T. M. Lam, H. W. Boschloo, M. Mulder, and M. M. Van Paassen. Artificial force field for haptic
feedback in uav teleoperation. Systems, Man and Cybernetics, Part A: Systems and Humans,
IEEE Transactions on, 39(6):1316–1330, 2009.
[22] A. M. Brandt and M. B. Colton. Haptic collision avoidance for a remotely operated quadrotor uav in
indoor environments. In Systems Man and Cybernetics (SMC), 2010 IEEE International Conference
on, pages 2724–2731. IEEE, 2010.
[23] T. Lam, M. Mulder, and M. Van Paassen. Haptic feedback in uninhabited aerial vehicle teleoperation
with time delay. Journal of guidance, control, and dynamics, 31(6):1728–1739, 2008.
[24] R. Mahony, F. Schill, P. Corke, and Y. S. Oh. A new framework for force feedback teleoperation of
robotic vehicles based on optical flow. In Robotics and Automation, 2009. ICRA’09. IEEE Interna-
tional Conference on, pages 1079–1085. IEEE, 2009.
[25] A. Ruesch, A. Y. Mersha, S. Stramigioli, and R. Carloni. Kinetic scrolling-based position mapping
for haptic teleoperation of unmanned aerial vehicles. In ICRA, pages 3116–3121, 2012.
54
[26] A. Y. Mersha, S. Stramigioli, and R. Carloni. Switching-based mapping and control for haptic tele-
operation of aerial robots. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International
Conference on, pages 2629–2634. IEEE, 2012.
[27] A. Y. Mersha, X. Hou, R. Mahony, S. Stramigioli, P. Corke, and R. Carloni. Intercontinental haptic
teleoperation of a flying vehicle: A step towards real-time applications. In Intelligent Robots and
Systems (IROS), 2013 IEEE/RSJ International Conference on, pages 4951–4957. IEEE, 2013.
[28] A. Y. Mersha, S. Stramigioli, and R. Carloni. On bilateral teleoperation of aerial robots. Robotics,
IEEE Transactions on, 30(1):258–274, 2014.
[29] S. Stramigioli, R. Mahony, and P. Corke. A novel approach to haptic tele-operation of aerial robot
vehicles. In Robotics and Automation (ICRA), 2010 IEEE International Conference on, pages 5302–
5308. IEEE, 2010.
[30] A. Y. Mersha, S. Stramigioli, and R. Carloni. Bilateral teleoperation of underactuated unmanned
aerial vehicles: The virtual slave concept. In Robotics and Automation (ICRA), 2012 IEEE Interna-
tional Conference on, pages 4614–4620. IEEE, 2012.
[31] X. Hou, R. Mahony, and F. Schill. Representation of vehicle dynamics in haptic teleoperation of
aerial robots. In Robotics and Automation (ICRA), 2013 IEEE International Conference on, pages
1485–1491. IEEE, 2013.
[32] X. Hou and R. Mahony. Dynamic kinesthetic boundary for haptic teleoperation of aerial robotic
vehicles. In Intelligent Robots and Systems (IROS), 2013 IEEE/RSJ International Conference on,
pages 4549–4950. IEEE, 2013.
[33] X. Hou and R. Mahony. An intuitive multimodal haptic interface for teleoperation of aerial robots. In
Robotics and Automation (ICRA), 2014 IEEE International Conference on, pages 838–845. IEEE,
2014.
[34] X. Hou, C. Yu, F. Liang, and Z. Lin. Energy based set point modulation for obstacle avoidance in
haptic teleoperation of aerial robots. In World Congress, volume 19, pages 11030–11035, 2014.
[35] A. Casqueiro, D. Ruivo, A. Moutinho, and J. Martins. Improving teleoperation with vibration force
feedback and anti-collision methods. In ROBOT 2015, 2nd Iberian Robotics Conference, Advances
in Intelligent Systems and Computing Series. Springer, 2015.
[36] A. Franchi, P. Stegagno, M. Basile, and H. Bulthoff. Rgb-d based haptic teleoperation of uavs with
onboard sensors: Development and preliminary results. In IEEE/RSJ IROS’13 International Work-
shop on Vision-based Closed-Loop Control and Navigation of Micro Helicopters in GPS-denied
Environments, pages 1–4, 2013.
[37] R. Wilson and G. Niemeyer. Motion control of impedance-type haptic devices. In Robotics and
Automation, 2009. ICRA’09. IEEE International Conference on, pages 1092–1097. IEEE, 2009.
55
[38] F. Schill, X. Hou, and R. Mahony. Admittance mode framework for haptic teleoperation of hovering
vehicles with unlimited workspace. In Proc. 2010 Australian Conf. Robot. Autom, 2010.
[39] C. D. Soares. Evaluation of admittance and impedance control in teleoperation of flying vehicle.
Master’s thesis, University of Twente, 2013.
[40] S. Lee, G. Sukhatme, G. J. Kim, and C.-M. Park. Haptic teleoperation of a mobile robot: A user
study. Presence: Teleoperators and Virtual Environments, 14(3):345–365, 2005.
[41] I. Farkhatdinov and J.-H. Ryu. Improving mobile robot bilateral teleoperation by introducing vari-
able force feedback gain. In Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International
Conference on, pages 5812–5817. IEEE, 2010.
[42] A. Hacinecipoglu, E. I. Konukseven, and A. B. Koku. Evaluation of haptic feedback cues on vehicle
teleoperation performance in an obstacle avoidance scenario. In World Haptics Conference (WHC),
2013, pages 689–694. IEEE, 2013.
[43] S. Omari, M.-D. Hua, G. Ducard, and T. Hamel. Bilateral haptic teleoperation of vtol uavs. In
Robotics and Automation (ICRA), 2013 IEEE International Conference on, pages 2393–2399.
IEEE, 2013.
[44] L. Dominjon, A. Lecuyer, J.-M. Burkhardt, G. Andrade-Barroso, and S. Richir. The” bubble” tech-
nique: interacting with large virtual environments using haptic devices with limited workspace. In
Eurohaptics Conference, 2005 and Symposium on Haptic Interfaces for Virtual Environment and
Teleoperator Systems, 2005. World Haptics 2005. First Joint, pages 639–640. IEEE, 2005.
[45] O. Khatib. Real-time obstacle avoidance for manipulators and mobile robots. The international
journal of robotics research, 5(1):90–98, 1986.
[46] B. H. Krogh. A generalized potential field approach to obstacle avoidance control. RI/SME, 1984.
[47] F. Schill, R. Mahony, P. Corke, and L. Cole. Virtual force feedback teleoperation of the insectbot
using optical flow. In Proceedings of the 2008 Australasian Conference on Robotics & Automation.
Australian Robotics and Automation Association Inc., 2008.
[48] S. Omari, M.-D. Hua, G. Ducard, and T. Hamel. Bilateral haptic teleoperation of an industrial
multirotor uav. In Gearing Up and Accelerating Cross-fertilization between Academic and Industrial
Robotics Research in Europe:, pages 301–320. Springer, 2014.
[49] T. M. Lam, M. Mulder, and M. Van Paassen. Haptic interface for uav collision avoidance. The
International Journal of Aviation Psychology, 17(2):167–195, 2007.
[50] T. M. Lam, M. Mulder, M. M. Van Paassen, J. A. Mulder, and F. C. Van Der Helm. Force-stiffness
feedback in uninhabited aerial vehicle teleoperation with time delay. Journal of Guidance, Control,
and Dynamics, 32(3):821–835, 2009.
56
[51] T. Debus, T. Becker, P. Dupont, T.-J. Jang, and R. D. Howe. Multichannel vibrotactile display for
sensory substitution during teleoperation. In Intelligent Systems and Advanced Manufacturing,
pages 42–49. International Society for Optics and Photonics, 2002.
[52] L. A. Jones and N. B. Sarter. Tactile displays: Guidance for their design and application. Human
Factors: The Journal of the Human Factors and Ergonomics Society, 50(1):90–111, 2008.
[53] R. V. Hogg and J. Ledolter. Engineering statistics. Macmillan Pub Co, 1987.
[54] S. G. Hart and L. E. Staveland. Development of nasa-tlx (task load index): Results of empirical and
theoretical research. Advances in psychology, 52:139–183, 1988.
[55] D. Ruivo. Formation of unmanned vehicles with collision avoidance capabilities. Master’s thesis,
Instituto Superior Tecnico, Universidade de Lisboa, 2015.
57