Ground-based Search and Retrieve using Aerial Localisation Daniel Ferguson, Eloise Matheson and Dushyant Rao

School of Aerospace, Mechanical and Mechatronic Engineering,

University of Sydney, Australia

Email: {dfer9299, emat3105, drao7945}@uni.sydney.edu.au

Abstract

Co-operative ground / aerial vehicle pairs can

potentially be utilised for search and retrieve

applications in disaster scenarios. A simplified

setup has been implemented using an iRobot

Create platform, coloured retrieval targets and a

roof-mounted camera for localisation. This paper

outlines the design of the entire system, with

particular emphasis on perception, navigation,

guidance and control subsystems. Preliminary

vehicle test results are presented and discussed,

and potential future improvements are suggested.

1 Introduction

Unmanned vehicles have had a large research focus in

recent years, partly due to their ability to replace manned

missions in high-risk situations. They are ideal for

missions in hazardous environments and inaccessible

terrain, and can therefore be invaluable for search and

rescue missions in disaster scenarios. [1]

Co-operative air and ground surveillance is an area of

research that has emerged in recent years. Unmanned

aerial vehicles (UAVs) have the advantage of rapid

coverage, but often experience difficulties reaching the

target, and can only localise ground targets to a coarse

level of accuracy. Conversely, unmanned ground vehicles

(UGVs) can locate targets to high precision but cannot

see through ground obstacles and move more slowly.

Thus, ground-aerial co-operative surveillance is a

synergistic combination, offering rapid coverage,

localisation and movement to a ground target. [2]

Much of the recent work on co-operative aerial / ground

vehicles focuses on control architectures, frameworks and

algorithms applicable to multiple vehicles. Specific

aspects of co-operative control are examined, with respect

to aerial and ground-based platforms ([1] , [2] ).

In this paper, the approach is more on developing an

overall system to perform search and retrieve tasks. While

time constraints have made it necessary to simplify the

system and operating scenarios, the work performed here

represents a general implementation of the entire system.

The implementation discussed here is a simplified version

of an aerial / ground co-operative pair. A fixed roof-

mounted camera is used to represent a UAV localising

both the target and the UGV, the retrieval targets are

coloured cardboard circles and the iRobot Create platform

is used for the UGV.

Accordingly, this paper will outline the broad architecture

of the system, and examine specific subsystems that have

been implemented. In particular, the perception and

navigation algorithms responsible for localising the

targets and the UGV will be discussed, as well as the

guidance and control algorithms that facilitate target

retrieval. Preliminary testing has also been performed

with the integrated system for a range of simple

scenarios; the key results will be outlined and discussed.

Finally, an extensive list of proposed future work will be

provided to identify the tasks required to more accurately

model a true co-operative air / ground vehicle pair.

2 Background

Much work has been done on co-operative ground/aerial

search and retrieval using a variety of sensors in different

combinations. Perception sensors such as laser scanners,

sonar sensors and colour CCD cameras are commonly

utilised and sensors including ultrasonic transducers for

altitude, Global Positioning Systems for location and

Inertial Measuring Units for orientation can be used for

navigation.

Projects such as the Autonomous Vehicle Tracking and

Retrieval (AVATAR) Helicopter use a CCD camera for

aerial perception and a camera and sonar sensors for

ground perception [3] . The General Robotics,

Automation, Sensing and Perception (GRASP) project

uses a stereo pair of cameras for ground perception and a

high resolution firewire camera for aerial perception [4] .

These projects are generic examples of the current

practise of co-operative combinations of data acquisition

between aerial and ground vehicles.

Given limited time scope within our project we decided to

simplify the system in a manner that we could test and

verify our system to gain significant results. To do this

the perception for our aerial vehicle has been simplified

to a single roof mounted stationary colour web camera

and this data is fused with the navigational data gained

from our robotic platform equipped with wheel speed

encoders.

The use of colour segmentation in perception is

widespread, with many algorithms implemented using

various techniques. Many of the algorithms processing

images in the HSV space as opposed to RGB space as it is

quite insensitive to change in intensities such as variance

in lighting conditions. 0

Colour segmentation algorithms can be loosely

categorized into four classifications: pixel based, region

based, edge detection and physics based. Pixel based

imposes thresholds on the histogram, or clustering using

techniques such as clustering or mean shift. [3] [7] [8]

Region based evaluates uniformity criteria to obtain

homogeneous regions. Edge based uses the edges as a

guide to identifying homogeneous regions. Physics based

rely on modelling the underlying physical reflectance

property of the material. [6]

As the scenario for this robot is quite contrived with

simplistic shapes and colours in a reasonable saturation

invariant environment, a simple HSV space pixel based

colour segmentation algorithm was implemented in our

system.

Navigation for a UGV is typically a fused estimate of

multiple localisation measurements. Given that vehicle

navigation is typically a non-linear problem, it is often

achieved using estimation techniques such as a particle

filter or an Extended Kalman filter, using the evolution of

the vehicle’s state based on past observations to estimate

the future state of the vehicle. Simpler implementations

may employ a low pass filter to estimate the vehicle’s

position, and this is the method that has been employed

here.

Guidance and control of ground vehicles is highly

dependent on the platform involved and the application.

Path planning techniques vary considerably, some

emphasising traversability and others focusing on

obstacle avoidance. Dubins’ curves and cubic Bezier

paths [14] are typically used where vehicle turning

curvature is a constraint, while obstacle avoidance may be

achieved using potential field methods [15] , occupancy

grid based methods, or network-based methods such as

visibility graphs and Voronoi diagrams. Optimal paths

may be obtained by using grid-based search algorithms

such as A* or D* [16] , which can also account for

obstacles.

Path following is again highly platform and mission

dependent. For a non-holonomic vehicle (one that is not

controllable in all of its degrees of freedom), PID

controllers are the most widely used, and the heading

error (between path and vehicle headings) and cross-track

error (perpendicular distance between vehicle and path)

may be used to determine the required control inputs [18]

. The pure pursuit guidance law has also been widely

implemented, using a single look-ahead distance

parameter to specify a point ahead on the path and set the

controls to steer the vehicle to that point [17] . This has

the key advantage of easier tuning, with only one

parameter determining how closely the path is followed

and how erratically the vehicle steers.

The current system is operating in an obstacle field, and

continuous paths are desirable for efficient path

following, so the potential fields approach has been used

for path planning, and a pure pursuit guidance algorithm

implemented for path following.

3 Design Overview

Given that this experiment aimed to develop many

aspects of co-operative ground/aerial search and retrieval,

the design of the complete system needed to be simple

and easy to navigate for a user, so as to aid future

expansions to the system. To do this a modular

framework of subsystems and a state transition method

was adopted for the software design and previously

documented and tested hardware was used for the mobile

platform, namely an iRobot.

3.1 Software

The software system is a state based transition

architecture where certain modular sub-systems are called

when necessary. Using a modular approach to the design

means that the algorithms and methods adopted can easily

be manipulated to suit a particular situation, giving a

higher degree of flexibility to the overall system. The

modular subsystems that we have used are Perception,

Navigation, Guidance, Control and Communication. In

more depth, single or multiple sub-systems may be called

by the states Get Target, Collect, Go Bin and Complete

which encompass all the actions our system can be

commanded to complete. This assumes that the system

has been correctly initialised; which calibrates the camera

and sets up communication with the mobile platform.

Figure 1: State Transition Diagram

A brief overview of each system is stated below, though

further details of each subsystem are given in Section 4.

The perception subsystem seeks to establish the

localisation of both the mobile platform and the objects in

the specified arena. Furthermore, it differentiates between

different coloured and sized obstacles and targets, using

HSV colour space and blob extraction algorithms. The

system returns centre points and radii of all targets and

obstacles, as well as the position and heading of the

mobile platform. The OpenCV source libraries were used

extensively in the perception module, including those for

image capture, colour segmentation and blob extraction.

Navigation is responsible for outputting a more refined

estimate of the mobile platforms localisation by fusing

data gained from the vision subsystem as well as that

directly from the onboard wheel encoders of the robot.

The guidance subsystem takes in inputs of what is in the

arena area and duly plans a path for the mobile platform

to follow, given the desired target position and pose. The

path is planned based on obstacle avoidance whilst

producing an achievable path for the robot to follow using

potential field theory.

The control system follows the guidance system in that

once a path plan has been established the correct

movement commands are sent to the robot. The

communications system ascertains whether these

commands are sent via Bluetooth (not currently

implemented in this stage of development) or through a

serial connection.

Feedback is present in the software design from the

perception subsystem, which can check whether each

state transition has successfully occurred.

3.2 Hardware

The iRobot Create Programmable Robot was chosen as

our mobile robotic platform. This development package

provides an established working system which can be

modified to suit various robotic applications. This

versatility made it a good choice in our experiment, as it

was able to provide the dependability and flexibility in

hardware that was required.

This mobile platform has a diameter of 0.35 m and has

onboard wheel encoders which provided constant

information on the robot’s speed and direction. The

accuracy of this data was dependent on the speed at which

the robot was operating, but still was still able to provide

useful data which could be fused in the navigation

subsystem.

The iRobot provided a physical platform which we could

perform our searches with, but was not able to activate

any retrieval methods. To do this, we constructed and

added a novel onboard electromagnetic retrieval system.

This consisted of an electromagnet made from a modified

transformer which was powered by a bank of six 9 volt

batteries, supplying a total capacity of 3.5 AH. The

electromagnet was activated by a 0-5V digital signal from

the iRobot, which enable the retrieval system to be turned

on or off using a N-fet switching circuit. Thin strips of

iron were attached to the targets, ensuring that the targets

could be collected by the iRobot without impinging on

the mobility of the platform.

As the system was designed to perform cooperative

ground/air based localisation, a camera was needed to act

as the aerial perception unit. The Logitech E2500

QuickCam was used for this application. When placed at

average roof height of a room (approximately 2.5 m) it

had a field of view of 1.5x2.5 m which was of adequate

size to perform applications of our system considering the

small scale of our platform. The sizes of our targets were

designed to be smaller than the robot itself whilst bases

were larger. This enabled the perception subsystem to

produce reliable and accurate results.

To communicate with the robot two options were tested,

one using a direct RS232 serial connection and the other

using an AVR microprocessing board with an onboard

Bluetooth module which was able to send serial

commands wirelessly to the robot.

4 Implementation

4.1 Perception

This module seeks to localise the robot and detect all the

obstacles, targets and home bases from an image. To do

this colour segmentation was done on an image of the

arena, filtering applied and finally blob extraction

algorithms used. However before this could be achieved,

the camera needed to be calibrated for our purposes and

setup.

In general, vision based range measurements rely on

calibrating the camera using known intrinsic and extrinsic

properties of the camera. A widely used calibration

technique is to solve a set on linear equations with 12

unknowns. [11] This technique is valid for unknown

scenes as long as the intrinsic properties of the camera do

not change, however calibration often needs to be

repeated when using modern CCD camera for changing

scenes. [12]

In our scenario the situation has been simplified as the

roof mounted camera is stationary, and so is at a constant

distance from the scene picture at a known angle. As long

as the camera does not move from its localised spot,

simple calibration only needs to occur once to establish

the limits of range that the camera perceives. To perform

calibration, the physical size of the image plane needs to

be known, adopting a similar approach used in [13] .

Depth perception is not of consequence in our setup, as

we have simulated a two dimensional arena for the robot.

A BGR image is captured from the webcam, such as

shown in Figure 2 and converted to a HSV representation.

Figure 2: Webcam Sample Image.

From this HSV image, the saturation channel is isolated

as shown in Figure 3.

Figure 3: Saturation Channel

The pixels in this channel corresponding to values lower

then a threshold (which was set such to remove the floor)

are ‘zeroed’ out in the image to give Figure 4.

Figure 4: Saturation Channel with Threshold Applied

This filtered saturation image is then used as the basis for

filtering the hue image, shown in Figure 5.

Figure 5: Hue Channel

The ‘zero value’ saturation pixels are ‘removed’ from the

hue image, leaving only the hue values of the targets.

Figure 6: Hue Channel after Saturation Filtering

From the filtered Hue image, Figure 6 , the image is

segmented based on the value of the hue to give three

distinct RGB segmented images as shown in Figure 7.

As seen in the blue segmented image of Figure 7 there is

some noise generated in the segmentation. To remove this

noise the image is eroded (black pixels enlarged) and then

dilated (the white pixels enlarged).

A blob detection algorithm is then applied to these

segmented images to find the properties of the targets.

This openCV source algorithm [10] has advantages over

other blob extraction techniques as it maps component

contours in linear time in one image pass. This

outperforms other techniques which can take more than

one image pass to label every component. Furthermore

this algorithm does not need to relabel components as

contour mapping is able to connect internal and external

components at the same time.

The algorithm is applied on a binary image of one colour

channel. Doing this makes it easy to analyse one set of

target colours, as well as finding obstacles and localising

the robot. Blob features are extracted and organised

according to their size, and the position and radius of each

recorded. For the targets or obstacles the largest target

found is set to be the base or home plate, and the smaller

ones the targets and/or obstacles depending on the state of

the program. The robot is localised according to two

green blobs placed in such a manner on the robot that the

line between the two centre points on the blobs can be

used to find the pose of the robot.

4.2 Navigation

The navigation solution for the system is obtained by

fusing the perception pose observation �����, ����� with a

prediction from the vehicle wheel encoders��� , ��� .

Each predicted pose estimate is obtained by polling the

vehicle encoders, and the encoders output the change in

forward position ��� and the change in angular position ��� since the previous reading.

The pose prediction can then be determined by:

������ = ����� − 1� + Δ�

������ = ����� − 1� + Δ���� ������ − 1��

������ = ����� − 1� + Δ���� ������ − 1�� (1)

To obtain the vehicle pose, these estimates are combined

with a low pass filter, using:

������ = ���������� + 1 − ������� � �� = !���� + 1 − ! ��� (2)

Here, the α coefficient specifies the linear weightings of

the predicted and observed pose estimates, and has been

set based on test results presented in the following section

of this paper.

When vision-based observations are not available, α is set

to zero, such that the pose estimate is purely based on that

predicted by the vehicle encoders. This enables the

vehicle pose to be estimated even when the vehicle is out

of visual range of the camera or the camera malfunctions.

4.3 Guidance and Control

A number of sub-states have been defined within the

guidance subsystem to facilitate guidance and control

operations. The mission of the robot is subdivided into a

number of smaller tasks, each consisting of a start

position ��� , ��� and goal position ��$ , �$�. At the start

point, the vehicle is in the rotate state, rotating on the spot

to face the goal point.

Top:

Left: Hue Channel

Middle: Saturation Channel

Right: Value Channel

Middle:

Left: Red Threshold Image

Middle: Green Threshold Image

Right: Blue Threshold Image

Bottom:

Left: Red Segmented Image

Middle: Green Segmented Image

Right: Blue Segmented Image

Figure 7: Colour Segmentation Sample

This is achieved by rotating the vehicle at a constant

velocity of 150 mm/sec based on the heading error of the

vehicle pose:

���� = tan() *+,(+-.,(.-/ − �� ���� 0 0, direction = 89:; ���� < 0, direction = =�>?; (3)

Once the heading error has a magnitude of less than A/90 radians, the guidance state is set to goto. The path

planning and path following operations for this state are

outlined here.

4.3.1 Path Planning and Obstacle Avoidance

At any point in the goto state, all objects apart from the

current target / bin pair are considered as obstacles. The

vehicle has a radius of ~175mm and the bins have a

radius of ~75mm. Thus, for a point model of the vehicle

in configuration space, the obstacles can be modelled as

circles with radii of 250mm.

Safe trajectories are generated for the vehicle through the

obstacle field using a heuristic potential field path

planning approach. A path is planned each time the

vehicle enters a goto state, and as such, the system

assumes obstacles are stationary for the duration of

current state.

A positive parabolic potential well is generated at the goal

point, and the obstacles are modelled by positive

Gaussian potentials. The parabolic potential is

advantageous for the goal since it propels the path

towards the goal at any distance, while Gaussians are

useful for obstacles since they can be scaled and

normalised to negate their effect at certain distances from

the obstacle. This means that the potential function can be

a continuous analytical expression, rather than having to

manually set the field to zero outside the distance of

influence of an obstacle, as in [15] . The magnitudes of

the goal potential and obstacle potentials are scaled to

modify their effect on the generated path. The final

potential field is parameterised as follows:

D��, �� = GOALIJKLM �� − �$ N + � − �$ N�

+ OBSIJKLM ∑ exp T(*�.(.UV�WX�+(+UV�W/YZI[\]^ _` (4)

The scale factors here determine the relative magnitudes

of the goal and obstacles potentials, while the

normalisation factor determines the width of the potential

(and consequently its effect over a distance).

From each path point, the next path point is determined

by the gradient descent method, propagating the path

along the direction of the negative 2D gradient:

∆� = − cd�.,+�e. , ∆� = − ef�.,+�

e+ (5)

Figure 8: Potential field path surfaces (top) and contours (bottom) for two obstacles (left), with a local minimum (centre) and

with local minima avoidance (right)

The problem of local minima is a well-documented

downfall of the potential field approach to path planning.

Numerous methods have been prescribed to resolve this

issue, including mathematical techniques to regenerate a

field without local minima, or escape algorithms to propel

the path away from local minima.

Local minima avoidance has been implemented but has

not been integrated with the current system. The method

implemented here employs a virtual obstacle at the

robot’s position when it finds itself in local minima [19] .

The virtual obstacle field is another Gaussian, but has

lower scale and normalisation factors than the obstacles,

such that it represents a flat, broader peak. This is

necessary; if the virtual obstacle is narrow and has a

larger magnitude, there will be situations where the path

will be propelled directly over the obstacle peak [19] .

The path planning implementation is shown in Figure 8.

With GOALSCALE, OBSSCALE and OBSNORM set at 0.4, 1

and 0.05 respectively the generated path safely avoids all

obstacles in configuration space (indicated in red).

4.3.2 Path Following

A pure pursuit guidance law [17] is applied to follow the

generated path. This method utilises a constant look-

ahead distance 8 from the vehicle to a point on the path

ahead (look-ahead point), and sets the vehicle curvature

to turn onto this point.

Figure 9: Geometry of the Pure Pursuit algorithm [17]

With a path point (x,y) relative to the vehicle frame, and

the vehicle position as shown in Figure 9, it is possible to

determine the radius of the turn required to move the

vehicle onto the look-ahead point:

�N + �N = 8N, �N + hN = =N, = = � + h

= = iWN. (6)

For a curved path, it is easier to work with the error angle:

���k��� = .i

= = iNlmn opqq (7)

This error angle is the heading angle to the look-ahead

point in the vehicle frame. In other words, it is the

difference between the vehicle heading and the look-

ahead heading:

k��� = �� − ;s�() �+(+t.(.t� (8)

Thus, at each point in time, the generated path is searched

to find the look-ahead point ��, ��, based on distance 8, and the required turning radius is obtained from the above

equations. With a constant look-ahead distance, the

vehicle turns in towards the path and then follows the

path smoothly. The look-ahead distance can be tuned to

directly affect the path following characteristics of the

vehicle. With a lower look-ahead distance, the vehicle

follows the path more closely, but this requires more

aggressive turning and greater control effort. Conversely,

a larger look-ahead distance yields a smoother vehicle

trajectory, but the path is not accurately followed.

If the goal point is within this fixed look-ahead distance,

then the goal point is used as the look-ahead point, to

ensure the vehicle passes over the goal.

The vehicle is controlled by commanding a drive velocity

and a turning radius, which is directly mapped to the

vehicle’s differential drive system.

By employing the pure pursuit algorithm, the turning

radius is directly determined from the look-ahead point,

and this simplifies the control process. The vehicle

velocity is set constant at 250 mm/s.

4.4 Communications

As previously stated a direct serial RS232 connection was

used to communicate from a laptop to the iRobot. This

two way communication was able to relay the required

serial control commands to the robot and receive data

from the robot.

Vehicle commands are in the form of 8-bit words sent

over a serial RS232 connection. A generic initialisation

sequence is sent to the vehicle to enable user control. The

motion commands are specified with a 5-byte sequence:

v�wxyz, yz{_}, yz{_{ , w~�_}, w~�_{�

The first byte specifies the drive command, the next two

bytes denote the velocity in mm/sec, and the final two

bytes represent the radius in mm. The velocity and radius

are signed, allowing for forward and backward velocities,

and both left and right turns.

θerr

Vehicle encoder readings are obtained sy

sending one byte at regular intervals in the program flow.

Once this byte is sent, triggering encoder readings

distance and angle measurements are read in as two byte

words from the serial connection.

A Bluetooth module has also been

integration with the system has not been completed at

stage. The Bluetooth module on the AVR board

found to have interfacing problems where the wrong

commands are sent to the robot or incorrect signals

received from the encoders. With time this

corrected however for the aim and development stage of

this experiment a direct line of communication

adequate.

5 Results

The key test results for each subsystem are detailed

below.

5.1 Perception

Using proven openCV colour segmentation and blob

extraction algorithms helped to minimise problems or

errors in the code for the perception module, however the

manner in which we combined techniques did need some

analyse to check the results gained were accura

reliable.

Using dilation and erosion algorithms changed the shape

of the detected features from their true representation.

The amount of change can be seen in Table

Table 1: Effect of Erosion and Dilation on Parameters

Area

x

position

Average

Delta 66.00 2.00

Average

Value 358.13

Percentage

Difference 18.43%

Once the colour channels had been filtered and changed

to a binary representation the blob extraction algorithm

was applied. According to the inputted image the blob

extraction could detect false positives and miss positive

readings. This error was minimised by experimenting

with the threshold value, which represented the smallest

number of pixels that the algorithm wou

feature. The performance of the blob extraction algorithm

and the detection as a whole can be seen in

Vehicle encoder readings are obtained synchronously by

sending one byte at regular intervals in the program flow.

, triggering encoder readings, the

distance and angle measurements are read in as two byte

been implemented but

not been completed at this

he Bluetooth module on the AVR board has been

interfacing problems where the wrong

re sent to the robot or incorrect signals are

received from the encoders. With time this will need to be

corrected however for the aim and development stage of

this experiment a direct line of communication is

The key test results for each subsystem are detailed

g proven openCV colour segmentation and blob

extraction algorithms helped to minimise problems or

errors in the code for the perception module, however the

manner in which we combined techniques did need some

analyse to check the results gained were accurate and

Using dilation and erosion algorithms changed the shape

of the detected features from their true representation.

Table 1.

d Dilation on Parameters

y

position

1.00

Once the colour channels had been filtered and changed

to a binary representation the blob extraction algorithm

applied. According to the inputted image the blob

extraction could detect false positives and miss positive

. This error was minimised by experimenting

with the threshold value, which represented the smallest

number of pixels that the algorithm would classify as a

feature. The performance of the blob extraction algorithm

and the detection as a whole can be seen in Figure 10.

Figure 10: Pie chart on the reliability and accuracy of the

blob detections when searching for the two blue features.

5.2 Navigation, Guidance and Control

The navigation, navigation and control results are

combined, because a different navigation solution directly

affects the path following characteristics of the vehicle,

and its trajectory changes.

Testing for these subsystems was performed with

constant target, bin and home positions for consistency.

The α value used in the navigation low

was altered to observe its effect on the vehicle trajectory.

The generated

First, the path planning results have been showcased in

Figure 11. These have been superimposed over a snapshot

of the layout to indicate the actual

positions.

Figure 11 Planned paths for the four successiv

2 Blobs

82%

3 Blobs

3%

Blob Detection Algorithm

Performance

: Pie chart on the reliability and accuracy of the

when searching for the two blue features.

Guidance and Control

The navigation, navigation and control results are

combined, because a different navigation solution directly

affects the path following characteristics of the vehicle,

Testing for these subsystems was performed with

constant target, bin and home positions for consistency.

value used in the navigation low-pass algorithm

was altered to observe its effect on the vehicle trajectory.

ing results have been showcased in

These have been superimposed over a snapshot

the actual target, bin and home

Planned paths for the four successive trials

1 Blob

15%

3 Blobs

Blob Detection Algorithm

Performance

The results obtained showed inconsistencies, and the

generated path varied dramatically for two of the trial

runs (α = 0.5 and α = 0.9). While the path planning itself

is independent of α, it is believed that other factors

(discussed in the next section) were responsible for these

anomalies.

All four trials were successful. On every occasion, the

vehicle successfully lifted the targets and deposited them

onto the corresponding ‘bins’ before proceeding to the

vehicle home. However, during the second two trials, the

vehicle trajectories were adversely affected by the

incorrect planned paths and oscillated significantly about

the planned path. The resulting trajectories for the four

trials are shown in Figure 12.

Also shown in Figure 12 are the navigation position

estimates obtained, based on the vehicle encoder

measurements (predicted, shown in blue), vision

measurements (observed, shown in green) and fused pose

estimate (updated, shown by the red line).

The corresponding heading estimates are presented in

Figure 13. The estimates have been made continuous to

eliminate oscillations between ± 180°, enabling the results

to be interpreted more clearly.

Figure 13: Navigation heading estimates for α = 0.1

(above) and α = 0.2 (below)

Figure 12: Vehicle trajectory for α = 0.1(bottom left), α = 0.2 (bottom right), α = 0.5 (top left), α = 0.9 (top right)

6 Discussion

Using vision as our perception method was an adequate

tool for our experimental setup, and colour segmentation

and blob extraction was found to be a versatile and

sensible way to extract features from our arena.

From a single original BGR image, certain key features

were extracted, and this involved funnelling the

information through various control filters to leave the

desired information behind. Doing this means some

information could be lost, but the following results show

that the information lost was inconsequential in that the

final result was adequate for the navigation, guidance and

control subsystems to function correctly.

By filtering the HSV image and applying erosion and

dilation algorithms information about the blobs was

altered, as shown in Table 1.

The effect on the Cartesian coordinates of the targets was

minimal (less than 2 pixels, as indicated in Table 1) and

thus dilation and erosion has minimal effect on the

performance of the detection of the targets position. The

change in area was much more significant, however

beyond differentiating the biggest from smallest, this

information was not poignant, and assuming this

difference was roughly the same for different sized

objects, it would not alter their size rankings. The change

in area is however important in calibration procedure

however the percentage change was minimised by using a

large calibration area.

The blob extraction was a verifiable way to extract the

features. False positive readings were detected 3% of the

time, and this occurred when a noisy signal was large

enough to be considered as a blob. This caused

inconsistencies in the path planning section if the noisy

blob signal was bigger than the real target blob, as this

false signal would have been labelled as the next target.

The 15% of the time where only 1 blob was detected was

expected, as this represented the stages where the robot

had collected the target, effectively blocking it from the

camera’s view, and once the target had been dropped off

on top of the same coloured bin.

With respect to the vehicle trajectories, the path planning

system yielded mixed results. The generated paths were

successfully able to manoeuvre around the blue bin in all

four trials. However, the two latter trials produced some

erroneous path segments. Upon closer inspection of the

path points generated, it is evident that the sharp

deviations along the path segments are due to outlier

points, but the entire path segments are not fully

complete.

Figure 14: Erroneous path planning segments

This anomaly could be the result of:

• Software errors in the path planning system

• False positives in the vision observation suggesting a

target is present at a different point. This could easily

be caused by interference from the vehicle tether.

The local minima avoidance method applied here also

needs further work before it can be applied to a real

vehicle system. Realistically, more vigorous testing is

required with different obstacle types before it can be

ascertained to work for all situations.

The navigation solution appears reasonable, with both

predicted (encoder-based) and observed (vision-based)

pose estimates yielding good results. However, the

software system is not currently threaded, which means

that the same sampling rate is the same for both encoder

and vision measurements. Realistically, multiple encoder

measurements should be available for each observation,

so the predicted path would be propagated forward at

each update point. With only one prediction for each

observation, the navigation solution is degraded.

Additionally, from Figure 12 we can observe that the

measurements are fairly spaced, suggesting that the

sampling rates for navigation are quite poor.

Incorporating multi-threading into the system would

enhance its performance in both aspects here. It would

enable faster prediction measurements, since these could

be obtained in parallel with the time-consuming vision

measurements.

With regards to data fusion, the navigation filter is far

more effective at low values of α. With large α values, the

vision observation, which is usually further away from

the planned path, is more heavily weighted. With the pure

pursuit algorithm, this makes the vehicle steer more

sharply, and the vehicle overshoots the path to a much

greater extent.

The guidance and control aspects are generally strong,

with the vehicle successfully navigating the waypoints

specified. In particular, by rotating the vehicle on the spot

at each path segment, the vehicle can change direction

sharply, and there is less path overshoot than if it were to

move continuously onto the next path segment. This is

clearly evident in the α = 0.1 and α = 0.2 plots, but less so

for the other two trials, where a lack of consistency is

apparent.

The path following algorithm is sufficient, but produces

some unnecessary oscillation about the path as seen in

Figure 12, and represented by the oscillations in Figure

13. This sort of behaviour is uncharacteristic for the pure

pursuit algorithm. In cases where the lookahead-distance

is too low, the vehicle turns sharply onto the path and

may oscillate slightly, but these oscillations occur over a

relatively large distance and are unusual.

It is likely that this issue is a result of the poor sampling

rate for the system. The magnitude of vision processing

required means the navigation and guidance aspects are

slower than desirable, and the system cannot rapidly be

controlled. Multi-threading the system would again have

an impact here.

7 Conclusion and Future Work

In this simplified implementation, the system was

successful in locating and retrieving the colour targets

across a number of trials. However, on some trials it still

exhibited some inconsistent behaviour, and while the

system was still successful, this contributed to less

desirable results. Thus, the system needs to be further

developed for consistency, in a number of proposed areas:

• The errors in the path planning system need to

be rectified.

• Multi-threading needs to be incorporated to

improve the navigation estimate and the

controllability of the vehicle.

• More analysis needs to be performed on the

guidance aspects, to determine the optimal look-

ahead distance for this application.

• Bluetooth communications needs to be

implemented to eliminate dependence on the

vehicle tether, which is unrealistic and can

occasionally interfere with vision observations.

Once the simplified system is complete and reliable,

further steps will be needed in developing a system that

better models an aerial / ground co-operative pair. These

include:

• Mounting the camera on a moving gimbal to

simulate UAV motion.

• Implementing dynamic obstacles avoidance as

would be required for moving obstacles in the

environment.

• Employing the algorithms on a larger UGV that

could be realistically used in a disaster scenario.

• Implementing a local perception system for the

UGV to supplement the coarse data provided by

the UAV.

• Applying a more robust navigation filter such as

an Extended Kalman Filter, as the vision

observations would be less reliable from a UAV

than from a stationary source.

• Replacing the electromagnet hardware with a

more realistic mechanism for retrieving survivors

in a disaster scenario.

References

[1] Phan, C., and Liu, H. H. T., “A Cooperative

UAV/UGV Platform for Wildfire Detection and

Fighting,” 7th International Conference on Systems

Simulation and Scientific Computing, 2008

[2] Grocholsky, B., Keller, J., Kumar, V., Pappas, G.,

“Cooperative air and ground surveillance,”

IEE Robotics & Automation Magazine, Vol. 13,

Issue 3, pp. 16-25, 2006.

[3] Richard T. Vaughan, Gaurav S. Sukhatme, Javi

Mesa-Martinez, and James F. Montgomery, "Fly spy:

lightweight localization and target tracking for

cooperating ground and air robots," In Proceedings

of the International Symposium on Distributed

Autonomous Robotic Systems, Knoxville, Tennessee,

Oct 2000.

[4] Grocholsky, B., Bayraktar, S., Kumar, V., and

Pappas, G. “UAV and UGV collaboration for active

ground feature search and localization.” In Proc. of

the AIAA 3rd

Unmanned Unlimited Technical

Conference, 2004.

[5] D.Comaniciu and P.Meer, “Robust analysis of

feature spaces: Color image segmentation," IEEE

Conference on Computer Vision and Pattern

Recognition, pp. 750 - 755, 1997.

[6] W.Skarbek and A.Koschan, “Colour image

segmentation - a survey,” Technical report, Technical

University of Berlin, 1994.

[7] J.L.Baker, D.Campbell, and A.Bodnarova, “Colour

image segmentation using optimal fuzzy clustering,"

IASTED Artificial Intelligence and Soft Computing,

pp. 63 – 68, 2003.

[8] J.L.Baker, D.Campbell, A.Bodnarova, and

V.Chandran, “Hue-intensity segmentation for stereo

correspondence," IASTED Robotics Applications,

pp. 195 – 200, 2003.

[9] Ohta, Y.I.; Kanade, T.; and Sakai, T., "Color

Information for Region Segmentation", Computer

Graphics and Image Processing, 13, pp.222-241,

1980.

[10] Fu Chang, Chun-Jen Chan, Chi-Jen Lu. “A

linear-time component-labeling algorithm using

contour tracing technique”. Computer Vision and

Image Understanding, September 2003.

[11] Sonka, M., Hlavac, V., Boyle, R.: “Image

Processing, Analysis and Machine Vision.” PWS

Publishing, New York 1999

[12] A. Tsalatsanis & K. Valavanis & N. Tsourveloudis

“Mobile Robot Navigation Using Sonar and Range

Measurements from Uncalibrated Cameras”. Journal

of Intelligent and Robotic Systems, Vol. 48, Issue 2,

pp. 253-284, 2007.

[13] Sahin, E., Gaudiano, P.: “Mobile Robot Range

Sensing Through Visual Looming”. In: Proceedings

of IEEE ISIC, pp. 370–375, September 1998

[14] Nelson, W., “Continuous-curvature paths for

autonomous vehicles,” IEEE International

Conference on Robotics and Automation, 1989.

[15] Khatib, O., “Real-Time Obstacle Avoidance for

Manipulators and Mobile Robots,” The International

Journal of Robotics Research, Vol. 5, No. 1, pp. 90-

98, 1986.

[16] Mellon, I. C., Stentz, A., “Optimal and efficient

path planning for unknown and

dynamic environments,” International Journal of

Robotics and Automation, 1995.

[17] Coulter, R. C., “Implementation of the Pure

Pursuit Path Tracking Algorithm,” Technical Report,

Robotics Institute, Carnegie Mellon University, 1992.

[18] Pham, H.N., “A Comprehensive Architecture for

the Cooperative Guidance and Control of

Autonomous Ground and Air Vehicles,” Australian

Centre for Field Robotics, University of Sydney,

2007.

[19] Chengqing, L., Hang, M., Krishnan, H., Yong, L.

S., “Virtual Obstacle Concept for Local-minimum-

recovery in Potential-field Based Navigation,”

Proceedings of the 2000 IEEE International

Conference on Robotics & Automation, 2000.