Omni-vision based autonomous mobile robotic platform Zuoliang Cao*a, Jun Hu *a, Jin Cao**b, Ernest L. Hall***c

aTianjin University of Technology, Tianjin / China; bFatwire Corporation, New York /USA; cUniversity of Cincinnati, Cincinnati/ USA

ABSTRACT

As a laboratory demonstration platform, TUT-I mobile robot provides various experimentation modules to demonstrate

the robotics technologies that are involved in remote control, computer programming, teach-and-playback operations.

Typically, the teach-and-playback operation has been proved to be an effective solution especially in structured

environments. The path generated in the teach mode and path correction in real-time using path error detecting in the

playback mode are demonstrated. The vision-based image database is generated as the given path representation in the

teaching procedure. The algorithm of an online image positioning is performed for path following. Advanced sensory

capability is employed to provide environment perception. A unique omni directional vision (omni-vision) system is

used for localization and navigation. The omni directional vision involves an extremely wide-angle lens, which has the

feature that a dynamic omni-vision image is processed in real time to respond the widest view during the movement.

The beacon guidance is realized by observing locations of points derived from over-head features such as predefined

light arrays in a building. The navigation approach is based upon the omni-vision characteristics. A group of ultrasonic

sensors is employed for obstacle avoidance.

Keywords: Mobile robot, omni directional vision, navigation, teach-and-playback, image database.

1. INTRODUCTION

The robotics laboratory is a major research division of Tianjin University of Technology (TUT). It is equipped with an

industrial robot work cell and a mobile robotic platform with a series of lab-demo system modules. The TUT mobile

robotic platform, as a wheel-based demo system, was developed to adapt to current education and research needs. The

emphasis of design concepts is placed on the following considerations.

(1) Mobile robotic devices promise a wide range of applications for industry automation in which stationary robots

cannot produce satisfactory results because of limited working space. Such applications include material transfer and

*Zlcao@eyou.com; phone 86 22 23688585; fax 86 22 23362948; http://www.tjut.edu.cn; Tianjin University of Technology, Tianjin,

300191. China; **Jincao@farwire.com; phone 1 516 3289473; fax 1 516739-5069; http://www.Fatwire.com; Fatwire Co. 330 Old

Country Road, Suite 207 Mineola, New York, USA 11501; ***hallel@email.uc.edu; phone 1 513 5562730; fax 1 513 5563390;

http://www.uc.edu; Univ. of Cincinnati , Cincinnati, USA, OH 45242

tool handling in factories or automatically piloted carts for delivering loads such as mail around office buildings and

service in hospitals. Most of the advanced manufacturing systems use some kind of computer transportation system.

(2) The facilities of the laboratory support the education of Robotics and Mechatronics which include a series of the

primary and typical robotics courses such as Kinematics, Dynamics, Motion control, Robot programming, Machine

intelligence.

(3) The laboratory experiments are the major resources for studying and researching the future technologies. Some

particular application demonstrations are necessary to create basic concepts and approaches in new development areas

to cope with the needs for research experience and capability.

Navigation techniques is one of the most common and important factors in robotics research. The use of a fixed

guidance method is currently the most common and reliable technique. An alternative method uses a signal-carrying

wire buried in the floor with on-board inductive coil or a reflective stripe painted on the floor with an on-board

photoelectric device. In the laboratory, a magnetic strip is paved on the floor and the vehicle is equipped with a

detecting sensor to direct the vehicle to follow it. Recent preferred approach employs laser beam reflecting or digital

imaging units1 to navigate a free-range vehicle.

It is feasible and desirable to use sensor-driven, computer controlled and software programmed automatic system for the

AGV development. A series of special sensors may be employed for mobile robots. Various methods have been tested

for mobile control using ultrasonic2, 3, infrared, optical and laser emitting units4. The data fusion, fuzzy logic5, 6, neural

network7 and machine intelligence technologies8~10 have been developed.

Machine vision capability may have a significant effect on many robotics applications11, 12. An intelligent machine, such

as autonomous mobile robot, must be equipped with vision system to collect vision information which is used to adapt

to its environment. With regard to the limitation of the view scope, the current imaging system affects the performance

of the robotics system. The development of the omni directional vision system appears to have more advantages. The

omnidirecitonal vision navigation program is an example.

The mobile robotic platform, as a lab-demo system, is featured by its hybrid (in the sense of variety of actuators and

sensors), reconfigurable, easily accessible, changeable controller and some advanced features. Both laboratory hardware

modules and advanced computer software packages provide an appropriate environment supplemented by a graphics

simulation system support. The two laboratory phases form an experimentation base from fundamentals of robot

operation through integration with new control technologies. The advanced experiment is divided into two phases as

following:

Phase one: robotic control, navigation, obstacle avoidance, sensory fusion.

Phase two: computer vision, image processing, path tracking and planning, machine intelligence.

TUT-I mobile robotic platform can be operated through four modes: human remote control, programmed control,

teach-and-playback, and automatic path planning in structured environments. Typically, the main function is

teach-and-playback operation for indoor transportation. The autonomous mobile robots can be considered as automated

path planning, free-ranging vehicle. However, the cooperation of autonomous capability with a supervising operator

appears to be the engineering compromise that provides a framework for the applications of mobile robots especially in

the structured environments. The operation mode of teaching path and playback is an effective solution not only for

manipulator arms but also for vehicles. The vehicle records the beacon data progressively in an on-board computer

memory during a manually given teaching mode trip along with a desired course. On subsequent unmanned trips, the

vehicle directs itself along the chosen course by observing the beacon and comparing the data. The steering correction

will allow the vehicle automatically to follow the taught course. The path generation by teaching mode and path

correction by real-time path error detecting in the playback mode are demonstrated by the TUT-I robotic system. The

vision-based image database is generated as the given path representation in the teaching procedure. The algorithm of

the image processing is performed for path following. The path tracking involves an online positioning method.

Advanced sensory capability is employed to provide environment perception. A unique omni-vision system is used for

localization and navigation. The omni directional vision involves an extremely wide angle lens with a CCD camera,

which has the feature that a dynamic omni-vision image can be responded to in real time. The beacon guidance is

realized by observing locations of points derived from over-head features such as predefined light array in a building.

The navigation approach is based upon the omni-vision characteristics. A group of ultrasonic sensor is employed to

avoid obstacles. The other sensors are utilized for system reliability. The multi-sensory data fusion technique is

developed for collision-free trajectory pilot.

TUT-I mobile platform can be decomposed into five distinct subsystems: locomotion with control/man-machine

interface, sensors for obstacle avoidance, omni-vision module with image processor, navigator with central controller

and power source units. The paper provides a concise series of descriptions in path planning, obstacle avoidance,

control strategies, navigation, vision systems, ranging systems and various application modules.

2. CONFIGURATION OF THE OMNIDIRECTIONAL VISION NAVIGATION UNITS

The platform comprises an unmanned vehicle, which is comprised of two driven wheels and two front free swiveling

wheels. The chassis contains the necessary elements to power, propel and steer other on-board equipment. Near the

chassis, a number of ultrasonic ranging sensors are mounted. The ultrasonic devices act to sense the presence of the

objects within its path and prevent a collision. As a further precaution, safety switches will stop the vehicle if it contacts

anything.

Omni directional vision means that an entire hemispherical field of view is seen simultaneously. This is realized by

means of an extremely wide-angle optical image device, called a fisheye, with a CCD camera. The omnidirectionan

vision guidance is a new and unique navigation technique. The omni-vision appears to have a definite significance in

navigation application for various autonomous guided vehicles. Omni-vision automated guiding system referred by

overhead light consists of the following five components shown in Fig.1.

(1) Fisheye lens, CCD camera with an automated electric

shutter. An electronic shutter control system uses a single

chip microcomputer. It is suitable for TK-60 CCD camera.

When the illumination light changes, the system still

enables the camera have better output image.

(2) A camera stand with 5 coordinate degrees of freedom.

(3) Beacon tracker: an image-processing computer for

real-time data acquisition of the targets.

(4) Omni-image distortion corrector.

(5) Navigator, which includes three function modules: Path

Generator, Path Error Detector and Corrector.

System will perform path planning and tracking on the

teach-and-playback mode under the environments by

referring to overhead lights. The system guides itself by

referring to overhead visual targets that are universally

found in the buildings. A group of predefined overhead

lights is usually selected as landmark, it is not necessary

to install any special features. Since at least two points can

determine the vehicle’s position and orientation, the beacon group should consist of at least two lights as guiding targets

in each frame of image. However, the algorithms may be adjusted to handle any specified number of targets. The

number must be greater than two. This would produce a more accurate and robust system. Even if the target may be lost

from time to time, the vehicle would still have at least the minimum number of targets for guidance.

The tracker is real-time digital image processor. Although multiple targets can be multiplexed through a single target

tracker, a multi-target tracker is used to eliminate the need for the multiplexing software. Up to four image windows or

gates can be used to track the apparent movement of the overhead lights simultaneously. The beacon tracker is the major

on-board hardware. It outputs the coordinates of a pair of lights that are shown on the monitor screen and placed a

tracking gate around each of them.

3. TEACHING AND PLAYBACK OPERATION MODE BASED ON IMAGE DATABASE

For industrial manipulators the teaching and playback operation mode through a teach pendant handled by operator is a

traditional technique. For an automated guided vehicle, it appears to be an unusual method but still to be an effective

solution. The vision-based image database is employed as the desired path generator. In the teaching mode, the operator

manually causes the vehicle to move forward along the desired path at a selected speed. The selected targets with their

gates will move downward on the monitor screen as the vehicle passes underneath the overhead lights. Then the datum

is recorded as described in the section in the reference database. When a target is nearly out of the field of view, the

frame will be terminated and a pair of targets will be selected. The process will be repeated. Some combination of

vision frames and position tracking will continue until the desired path has been generated and the reference data for

that track is recorded in memory. In the playback mode thereafter, the vehicle automatically maintains itself in the

desired path by comparing between the analogous record and the vehicle path of movement and steering corrections

made to bring the path errors: Ex, the lateral error, Ey, the error along the path and Ea, the angle orientation error to zero,

thereby keeping the vehicle on the intended course. As the value of Ey diminishes and approaches to zero the next

reference frame will be called up. The cycle will be

repeated. The procedure will cause the vehicle to

follow the desired path.

A database with the tree structure shown in Fig.2 is

built up to record the data stream. The vision-based

image database is three-layer data structure. The

data structure is a group of array pointer. Each

array element points the next layer’s node. The array index means the serial number of the sampling, which represents

frames, fields and records respectively. The method creates an exclusive property that the desired path is defined by an

image database.

There are three coordinate systems: the beacon coordinate system, the vehicle coordinate system and the image

coordinate system. On the basis of the principle of the coordinate conversion, the path errors can be easily calculated

and obtained when the vehicle departs from the desired path as detected from the recorded locations of geometric points

derived from the various elements of the existing pattern. At this point it is possible to compare the observed target

coordinates (Tx, Ty) with the reference target coordinates (Rx, Ry) and determine the angle error Ea, the lateral position

error Ex and the longitudinal position error Ey.

4. LENS DISTORTION AND PIXEL CORRECTION

For vision-based measurement, in order to obtain the real world position of the vehicle the lens distortion and camera

chip pixel distortion correction have to be considered. The target

image coordinates supplied by the video tracker require correction

for two inherent errors before they can provide accurate input for

guiding the vehicle. One is lens distortion error and another is the

pixel direction error.

The wide angle lens used in this system has considerable distortion

and consequently the image coordinates of the overhead lights as

reported by the tracker are not proportional to the true real world

coordinates of those lights. The lens distortion will cause the image

range Ir vary as a function of the zenith angle B and depending on

the lens used, the function may be linear, parabolic or trigonometric.

The distortion is determined experimentally by laboratory measurements from which the function is determined.

The lens used herein the distortion correction refers to a linear formula as shown in Fig.3

Ir = KB

Where K is a constant derived from the distortion measurement. The correction factor for a specific lens distortion is

provided by the lens correction software.

The pixels in the CCD chip are different scale between X and Y direction. This causes the targets coordinates to be

different in X and Y axis. It is necessary to divide the Y coordinates by a factor R which is the ratio of pixel length

toheight. This brings the X and Y coordinates to the same scale. The following equations give this conversion:

dx = (xi – xc)

dy = (yi – yc)/R

where (xc, yc) is origin coordinates of the camera center. Then we know the image range Ir.

The image coordinates (dx, dy,) on focal plane are converted into the target coordinates (Tx, Ty,) at the known height H.

show in Fig .3.

Tr = H tan B

It is known that B = Ir / K

Tr = H tan (Ir / K)

Where Tr is the target range in real world coordinate system.

Since the azimuth angle C to a target point is invariant in the image and real world coordinates system. The target world

coordinates (Tx, Ty) can be calculated as:

Tx = dx Tr/Ir

Ty = dy Tr/Ir

In order to transfer the image coordinates from an origin at the camera centerline to the vehicle centerline, the

calibration of the three coordinates as discussed above is necessary to determinate the center point of the coordinates.

5. THE MOTION CONTROL MODULES

A programmable two-axis motor controller is used. A non-programmable model may be chosen. The necessary

programming modules can be incorporated into the vehicle computer. The controller must selectively control velocity or

position of two motors, utilize appropriate feedback, such as from optical encoders and coupled tachogenerators to

either the motor or the wheels, which involves a two freedom closed-loop servo system.

The vehicle has two self-aligning front wheels and two driving rear wheels. Velocity mode control is used when the

vision system is guiding the vehicle. In this mode, steering is accomplished by the two driving motors turning at

different speeds due to the inputs of VL and VR (left and right wheel velocities) from the computer. The wheel encoders

provide local feedback for the motor controller to maintain the velocity profile programmed during the teaching mode.

By speeding one wheel and slowing another one equally by an amount dV, the motor control strategy steers the vehicle

to return to its desired path. Since the sample frequency is large enough, we can consider the system as a continuous

feedback system. In practice, the conventional PID compensator can be designed to achieve desired performance

specifications. A simple control formula is used as followings:

dV = k1 Ex + k2 Ea

where Ex and Ea are output from the path error measurement circuit. K1 and K2 are the constraints which can be

mathematically calculated from significant parameters of vehicle dynamics and kinematics or be determined

experimentally.

VL = Vm + dV

VR = Vm – dV

where Vm is velocity at the centerline of the vehicle. The sign of dV and the magnitude of dV will determine the

turning direction by giving a turn radius. The control formula will bring the vehicle back onto course in the shortest

possible time without overshooting. The following block diagram represents the closed-loop transfer function. For the

output Ex, it involves a PD compensation shown in Fig.4.

Where D is the diameter of the rear drive wheel, w is the distance between two bear wheel, kn is a constant related to

motor output, V0 is the velocity of the mass center of the vehicle. The right diagram is a simplified unit feedback system

derived from left diagram. We could select a pair of the optimal values of K1 and K2 through calculation or experiments

in practice to result in a desired performance of the system.

6. CONCLUSION

The photographs of the TUT-I omni-vision based autonomous mobile robotic platform are shown in Fig.5. The

technique points of the application for omni-vision based autonomous mobile robotic platform include following

properties:

(1) The omni-vision provides an entire scene using a fish eye lens. It appears useful in a variety of applications for

robotics. An overall view is always required for safe and reliable operation of a vehicle. While conventional method

with camera scanning appears generically deficient, dynamic omni-vision is considered as a definite advantage

particularly for mobile navigation.

(2) The preferred approach is that an unmanned vehicle guides itself by referring to overhead visual targets such as

predefined overhead lights. Overhead lights are universally found in structured environments and not easily blocked by

floor obstacles. The guidance system does not require the installation of any special equipment in the work area. The

point matrix pattern of the beacon as an environment map is very simple to be processed and understood.

(3) The teach-and-playback operation mode seems appropriate not only for robotic manipulator but also for other

vehicles as well. Vision-based image database, as a teaching path record or a desired path generator, create a unique

technique to expand robot capability.

Fig. 5: TUT-I Omni-vision based autonomous mobile robotic platform

ACKNOWLEDGMENTS

The author gratefully acknowledges the support of K.C.Wong education foundation, Hong Kong.

REFERENCES

1. E.L. Hall, “Fundamental principles of robot vision,” Handbook of Pattern Recognition and Image Processing:

Computer Vision, Academic Press, New York, pp.543-575, 1994.

2. Qing-ha0 Meng, Yicai Sun, and Zuoliang Cao, “Adaptive extended Kalman filter (AEKF)-based mobile robot

localization using sonar,” Robotica, Vol.18, pp.459-473.2000

3. Gordon Kao, and Penny Probert, “Feature extraction from a broadband sonar sensor for mapping structured

environments efficiently,” The International Journal of Robotics Research, Vol.19, No.10, pp.895-913, 2000.

4. Xiaoqun Liao, Jin Cao, Ming Cao, Tayib Samu, and Ernest Hall, “Computer vision system for an autonomous

mobile robot,” Proc. SPIE Intelligent Robots and Computer Vision Conference, November, Boston, 1998.

5. T.I Samu, N. Kelkar, and E.L. Hall, “Fuzzy logic system for three dimension line following for a mobile robot,”

Proc. of Adaptive distribute parallel computing symposium, Dayton, Oh. pp137-148, 1996.

6. Hong Xu, and Zuoliang Cao, “A three-dimension-fuzzy wall-following controller for a mobile robot”, ROBOT,

Vol.18, pp.548-551 1996.

7. Minglu Zhang, Shangxian Peng, and Zuoliang Cao, “The artificial neural network and fuzzy logic used for the

avoiding of mobile robot,” China mechanical engineering, vol.18, pp.21-24, 1997.

8. Zuoliang Cao, Yuyu Huang, and E.L Hall “Region filling operations with random obstacle avoidance for mobile

robot,” Journal .of Robotic Systems, 5(2), pp.87-102, 1988.

9. Zvi Shiller, “Online suboptimal obstacle avoidance,” The International Journal of Robotics Research, Vol.19, No.5,

pp.480-497, 2000.

10. Alain Lamber, and Nadine Le Fort-Piat, “Safe task planning integrating uncertainties and local maps federations,”

The International Journal of Robotics Research, Vol.19, No.6, pp.597-611, 2000.

11. Liming Zhang, and Zuoliang Cao, “Teach-playback based beacon guidance for autonomic guided vehicles,”

Journal of Tianjin Institute of Technology, Vol.12 No.1, pp.28-31, 1996.

12. Liming Zhang, and Zuoliang Cao, “Mobile path generating and tracking for beacon guidance,” 2nd, Asian

Conference on Robotics, 1994.

Deleted: ¶

¶

<sp>¶

¶

¶

¶

¶

¶

¶

¶

¶

¶

<sp>¶

¶

¶

¶

¶

¶

¶

¶

¶

¶

¶

¶

¶

<sp>¶

¶

¶

¶

¶

¶

¶

¶

¶

¶

¶

¶

¶

Fig. 5: TUT-I

Omni-vision based autonomous mobile ... [1]

Page 11: [1] Deleted Ernie