machine vision methods for autonomous micro‐robotic systems

19
Machine vision methods for autonomous micro-robotic systems B.P. Amavasai, F. Caparrelli, A. Selvan, M. Boissenin, J.R. Travis and S. Meikle Microsystems and Machine Vision Laboratory, Materials and Engineering Research Institute (MERI), School of Engineering, Sheffield Hallam University, Sheffield, UK Abstract Purpose – To develop customised machine vision methods for closed-loop micro-robotic control systems. The micro-robots have applications in areas that require micro-manipulation and micro-assembly in the micron and sub-micron range. Design/methodology/approach – Several novel techniques have been developed to perform calibration, object recognition and object tracking in real-time under a customised high-magnification camera system. These new methods combine statistical, neural and morphological approaches. Findings – An in-depth view of the machine vision sub-system that was designed for the European MiCRoN project (project no. IST-2001-33567) is provided. The issue of cooperation arises when several robots with a variety of on-board tools are placed in the working environment. By combining multiple vision methods, the information obtained can be used effectively to guide the robots in achieving the pre-planned tasks. Research limitations/implications – Some of these techniques were developed for micro-vision but could be extended to macro-vision. The techniques developed here are robust to noise and occlusion so they can be applied to a variety of macro-vision areas suffering from similar limitations. Practical implications – The work here will expand the use of micro-robots as tools to manipulate and assemble objects and devices in the micron range. It is foreseen that, as the requirement for micro-manufacturing increases, techniques like those developed in this paper will play an important role for industrial automation. Originality/value – This paper extends the use of machine vision methods into the micron range. Keywords Cybernetics, Robotics, Nanotechnology, Image sensors, Artificial intelligence Paper type Research paper 1. Introduction The IST-FET MiCRoN[1] project comprises a consortium of eight academic partners located in seven European countries. MiCRoN has its roots in a prior EU project, Miniman[2] (Bu ¨ erkle et al., 2001) which was completed in January 2001. In Miniman, single dm 3 -sized micro-robots were developed to perform a generic set of The Emerald Research Register for this journal is available at The current issue and full text archive of this journal is available at www.emeraldinsight.com/researchregister www.emeraldinsight.com/0368-492X.htm The Microsystems and Machine Vision Laboratory (MMVL) is a division within the Materials and Engineering Research Institute at Sheffield Hallam University. The principal focus of the group is to investigate and develop vision-based techniques aimed at a variety of real-time applications which include microrobotic systems, biological applications, MEMS, nanotechnology, scanning electron microscopy (SEM) and scanning probe microscopy (SPM) applications. Machine vision methods 1421 Kybernetes Vol. 34 No. 9/10, 2005 pp. 1421-1439 q Emerald Group Publishing Limited 0368-492X DOI 10.1108/03684920510614740

Upload: s

Post on 18-Dec-2016

212 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Machine vision methods for autonomous micro‐robotic systems

Machine vision methods forautonomous micro-robotic

systemsB.P. Amavasai, F. Caparrelli, A. Selvan,M. Boissenin, J.R. Travis and S. MeikleMicrosystems and Machine Vision Laboratory,

Materials and Engineering Research Institute (MERI),School of Engineering, Sheffield Hallam University, Sheffield, UK

Abstract

Purpose – To develop customised machine vision methods for closed-loop micro-robotic controlsystems. The micro-robots have applications in areas that require micro-manipulation andmicro-assembly in the micron and sub-micron range.

Design/methodology/approach – Several novel techniques have been developed to performcalibration, object recognition and object tracking in real-time under a customised high-magnificationcamera system. These new methods combine statistical, neural and morphological approaches.

Findings – An in-depth view of the machine vision sub-system that was designed for the EuropeanMiCRoN project (project no. IST-2001-33567) is provided. The issue of cooperation arises when severalrobots with a variety of on-board tools are placed in the working environment. By combining multiplevision methods, the information obtained can be used effectively to guide the robots in achieving thepre-planned tasks.

Research limitations/implications – Some of these techniques were developed for micro-visionbut could be extended to macro-vision. The techniques developed here are robust to noise andocclusion so they can be applied to a variety of macro-vision areas suffering from similar limitations.

Practical implications – The work here will expand the use of micro-robots as tools to manipulateand assemble objects and devices in the micron range. It is foreseen that, as the requirement formicro-manufacturing increases, techniques like those developed in this paper will play an importantrole for industrial automation.

Originality/value – This paper extends the use of machine vision methods into the micron range.

Keywords Cybernetics, Robotics, Nanotechnology, Image sensors, Artificial intelligence

Paper type Research paper

1. IntroductionThe IST-FET MiCRoN[1] project comprises a consortium of eight academic partnerslocated in seven European countries. MiCRoN has its roots in a prior EU project,Miniman[2] (Buerkle et al., 2001) which was completed in January 2001. In Miniman,single dm3-sized micro-robots were developed to perform a generic set of

The Emerald Research Register for this journal is available at The current issue and full text archive of this journal is available at

www.emeraldinsight.com/researchregister www.emeraldinsight.com/0368-492X.htm

The Microsystems and Machine Vision Laboratory (MMVL) is a division within the Materialsand Engineering Research Institute at Sheffield Hallam University. The principal focus of thegroup is to investigate and develop vision-based techniques aimed at a variety of real-timeapplications which include microrobotic systems, biological applications, MEMS,nanotechnology, scanning electron microscopy (SEM) and scanning probe microscopy (SPM)applications.

Machine visionmethods

1421

KybernetesVol. 34 No. 9/10, 2005

pp. 1421-1439q Emerald Group Publishing Limited

0368-492XDOI 10.1108/03684920510614740

Page 2: Machine vision methods for autonomous micro‐robotic systems

micromanipulation tasks under either an optical microscope or in a scanning electronmicroscope (SEM). However, Miniman lacked in the ability to perform co-operativelyand was limited by its physical size. The results achieved in the Miniman project formthe basis of the MiCRoN project, specifically in the development of technologies for thefinal Miniman V micro-robot, shown in Plate 1.

The main goal of MiCRoN is to develop a micro-robotic cluster that is able toperform tasks autonomously. Each of the micro-robot units is equipped with on-boardelectronics and wireless communication capabilities. The micro-robots arepre-programmed to perform tasks associated with assembly and manipulation in themicrometre range, with the possibility to extend operations to the sub-micrometrerange.

One of the key motivations behind this project is given by the increasing need inindustry for flexible and re-programmable tools that can work in specific micro-scaleenvironments such as under an optical microscope. The use of micro-robots toaccomplish these tasks offers a valid alternative to bulky and expensive lab equipmentwhich is generally adapted for a single task.

The application of machine vision methods as an active feedback element to thecontrol system adds a further degree of flexibility to the system. Object models to bemanipulated by the robots can be acquired off-line, stored and then re-used duringthe execution of each task. Algorithms can be customised and optimised so thatreal-time control performance can be achieved, although this is often dependent on thecomplexity of the task at hand.

Within MiCRoN, we foresee that the requirement for future industrial applicationswill at least be as complex as the robotic platforms in the macro world. However, froma design standpoint there are several caveats. Amongst others, from a scaling point ofview, small forces will be amplified and at micro scales many objects become “sticky”.For imaging, the signal-to-noise ratio is reduced as magnification increases.Furthermore, as we reach the limits of optical imaging the quality of imagesobtained at these scales degrades considerably.

Our role within MiCRoN is to develop algorithms for the automatic recognition andtracking of objects under a purpose built image acquisition system. These objects caneither be the parts that need to be assembled/handled by robot-mounted tools or the

Plate 1.The Miniman Vmicro-robot that forms thebasis for miniature roboticresearch in the MiCRoNproject

K34,9/10

1422

Page 3: Machine vision methods for autonomous micro‐robotic systems

tools themselves. Generally, the type of tool attached to the robots is specific to thetask. The object information from the vision system is used by the control system tocontrol the robots during the execution of pre-planned tasks.

This paper is structured as follows. In Section 2 a detailed description of theMiCRoN hardware system is given. In Section 3, the vision subsystem is presented:four different algorithms developed for several task scenarios are described anddiscussed in detail in subsections 3.1-3.4. Section 4 presents the results currentlyobtained by the MiCRoN vision system. Finally, Section 5 draws some conclusionsfrom the results described in the previous section.

2. System descriptionThe micro-robots developed for MiCRoN are approximately 1 cm3 in size and consist ofa piezo-electric module that produces high velocity (several mm/s) locomotioncombined with a nanometric resolution. The robots navigate on a flat, horizontalsurface by translation and rotation. The main task of the locomotion system is to bringthe on-board tools to the work area or to transfer the micro-objects to the next station.

The main common features of the MiCRoN micro-robots are:. A wireless communication transmission system between the robot unit and the

control unit through an adhoc infrared link (Figure 1(a)).. A set of markers used for the global positioning system which is based on the

application of projected Moire fields (Figure 1(b)).. On-board electronic circuitry for the activation of the piezo-electric devices. This

will control the robot’s motion, generate and amplify the driving signals for allactuators and tools and pre-process the signals from the on-board sensors(Figure 1(c)).

Figure 1.Micro-robot configuration.

Machine visionmethods

1423

Page 4: Machine vision methods for autonomous micro‐robotic systems

. An interface for the hosting of a number of tools which are interchangeable(Figure 1(d)).

. A coil which provides power to the on-board electronics. The wireless poweringmechanism is based on the principle of induction and makes use of a speciallydesigned power floor on which all the active robots operate (Figure 1(e)).

. A piezo-electric system used for both locomotion and manipulation (Figure 1(f)).

Two early MiCRoN robot prototypes working co-operatively are shown in Plate 2. Themicro-robots are equipped with a variety of on-board tools built within the projectconsortium. These include:

. a functionalised AFM (atomic force microscopy) tip that is able to qualitativelyresolve objects down to atomic sizes;

. a syringe chip that is built specifically to inject liquid into cells;

. a gripper tool that is able to handle micron-sized objects; and

. a micro-needle that uses electrostatic forces to grasp micro-objects.

In order to assess the performance and evaluate the capabilities of the micro-roboticsystem, two different scenarios have been devised. The first demonstration involvesthe soldering of benchmark parts of size 50 £ 50 £ 10mm3. These benchmarkparts are to be manipulated using gripper tools mounted on two MiCRoN robotsand the assembly is to be monitored by a micro-camera mounted on a larger mobilerobot.

The second demonstration is derived from the field of biological and biomedicalnano-manipulation. This experiment is composed of three major steps. In the first step,one robot isolates and captures a single cell suspended in a solution using a glassmicropipette. This robot transports the cell to a pre-defined position where it is releasedand fixed to the object slide by an electric field trap. At this position, the cell is held in

Plate 2.Two early MiCRoN robotprototypes manipulating ahuman hair

K34,9/10

1424

Page 5: Machine vision methods for autonomous micro‐robotic systems

place for two procedures to be performed: the first one is an AFM measurementperformed by a second robot while the second one involves the injection of a liquid intothe cell by a third robot using a customised syringe chip. This demonstration isperformed under an inverted optical microscope.

In the first demonstration, a Miniman-IV robot will be equipped with a customisedminiature CCD camera which will be used to gather local positioning information of thelocation of tools as well as the parts to be manipulated. The required precision dependslargely on the task at hand; however, an imaging resolution of 1mm/pixel is needed tomeet the requirements of the planned demonstration.

The specification of the system and its field of application in the micro- andnano-range impose very limiting constraints to the selection of the optical system. Toachieve a pixel resolution of 1mm/pixel with currently available CCD or CMOSimaging chips, an optical system with a magnification of 6-10 £ is required. Althoughmicroscope objectives and high resolution CCD and CMOS imaging chips are largelyavailable in the market, the existing limits in size (5-6 cm in length) and weight (lessthan 100 g) severely restrict the choice. A Panasonic GP-CX261 1/4 in. CCD camerahead coupled with a custom designed lens system that provides a magnification of 5 £was selected for the task. The camera head and the lens were assembled into arobot-compatible housing (Figure 2). A three-DOF translation stage with micron-levelaccuracy was also used to aid the testing of the system.

3. The vision systemOne of the problems faced with miniature lens systems is the amount of barreldistortion present in the image. This is typified by Figure 3(a). This can be correctedusing a variety of remapping or warping techniques. Given that (x, y) are thecoordinates of a pixel in the image, in Figure 3(b) the correction is performed bymapping (x, y) to (x0, y0) using the following equation:

Figure 2.A custom built camera

system with an integratedmicroscope lens system

Machine visionmethods

1425

Page 6: Machine vision methods for autonomous micro‐robotic systems

x0

y0

12cr12c

0

0 12cr12c

0@

1A x

y

!ð1Þ

where r ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffix 2 þ y 2

pis the radius from the centre pixel of the image and c is a

curvature parameter.The process of selecting c can be semi-automated by using a grid image as shown in

Figure 3. The value of c is varied and at each stage the Hough transform is observed.Owing to the nature of the Hough transform, sharper peaks will be generated in thetransform if the lines in the image are straighter. Hence, an optimal value of c thatcorrects the distortion can be found by numerically selecting the c value thatmaximises the straightness of lines on the grid image. In the instance of Figure 3, avalue of c ¼ 0:18 was obtained.

One of the main requirements for the vision algorithms that are being designed forMiCRoN is real-time performance. The vision system has to be synchronised with thecontrol system at all times. The micro-robotic system is a hard real-time system, somaximum delays and response times for every stage of processing have to be defined.We estimate an initial processing rate of no more than 10 fps in order to achieve therequirements of the project deliverables. The control system has to react, respond andalter its parameters according to this constraint.

In this section, we shall introduce some newly designed vision algorithms andmethods. The vision system itself is built over an existing vision toolkit, developedin-house, known as Mimas[3].

The new vision algorithms that will be discussed in the following subsectionsinclude:

. a robust tracking method with in-built error recovery,

. a neural network paradigm for colour segmentation,

. a fast shape-based scale-invariant recognition algorithm, and

. a customised method for locating living cells.

3.1 Robust object trackingOne of the planned tasks in MiCRoN entails guiding a glass micropipette, with a tipdiameter of 30mm, to transport cell samples and position them on a statically chargedcell holder. The task is to be carried out semi-autonomously through closed-loopcontrol. The main problems encountered are:

Figure 3.Correcting for the effectsof barrelling.

K34,9/10

1426

Page 7: Machine vision methods for autonomous micro‐robotic systems

. the translucent glass tip image is affected by the background clutter, and

. the body of the pipette and the grid on the cell holders have very similar linefeatures making it difficult for recognition methods to differentiate between thetwo different objects.

The problem was approached by making use of the statistical method based onconditional probability density estimation, known as the condensation (Isard andBlake, 1998) algorithm or, more commonly, as particle filtering. The initial observationmeasure for the Condensation algorithm was obtained through normalised correlationbetween the model (pipette tip) image and the scene image.

Condensation is a filtering algorithm developed using a probabilistic densityfunction (pdf) to model the probable locations of objects. One of its unique features isthat no functional assumptions (e.g. Gaussianity or unimodality) are made about theunderlying pdf. In a cluttered scene environment, the probability of locating an objectin the scene gives rise to a multi-modal pdf. This is due to the fact that quite similarobjects in the background produce probability measures that are comparable to theobject itself. Therefore, these probability measures are of similar value to theprobability measure at the actual location of the object itself. Since condensation canhandle multi-modal pdfs, it is well suited for cluttered scene evaluation. TheCondensation filter belongs to the class of Bayesian filters. This implies that it is basedon sampling the posterior distribution estimated in the previous frame[4] andpropagating these samples or particles to form the prior distribution for the currentframe[5].

The key idea of particle filtering is to approximate the probability distribution byparticles which are weighted samples of the current state. A particle p can be describedas an element of P where

pl;w [ P; where ðl;wÞ [ Rn £ Rþ; n [ N* ð2Þ

We introduce the following notation:. pl refers to the features value for the particle p;. pl i the particle p’s ith feature;. pw the particle’s weight; and. p (n) is the nth particle.

The particle features are correlated with those of the tracked object in the given sceneimage. The spatial location features of the particle give the predicted location of theobject that is being tracked. Thus, the particles can be seen as the hypothetical state ofthe object being tracked and this hypothesis is quantified by the measure associatedwith it (edge and correlation measure). This measure is then integrated into the weightof the particle. The higher the weight, the more probable it is that the particle describesthe state of the tracked object.

To predict the features of the particles, earlier measures on the previous state andthe system model are taken into account. In the tracking scenario, the two features of aparticle are the x and y location of the tracked object. The weight of each particle couldbe due to the result of the correlation of the template image and the part of the imagecentered on the particle.

Machine visionmethods

1427

Page 8: Machine vision methods for autonomous micro‐robotic systems

In the tracking of the pipette, pure correlation of the grey scale template imageproduced very poor results. The results were improved by filtering the image with anedge filter and correlating it with an edge template. This procedure may be consideredequivalent to shape matching. The evolution of the particle set is described bypropagating each sample according to the kinematics of the object.

The position L of the object is estimated at each time step by the weighted mean ofthe particles:

L ¼

XNn¼1

pðnÞl £ pðnÞw

XNn¼1

pðnÞw

ð3Þ

Through this, particle filtering provides a robust tracking framework, as it modelsuncertainty. It is able to consider multiple state hypotheses simultaneously. Since lesslikely object states have the opportunity to temporarily influence the tracking process,particle filters can deal with short-lived occlusions (Nummiaro et al., 2002). Figure 4 is asimplified flow chart that sums up the steps involved.

Plate 3 and Figure 5 show the results of implementing the particle filter on an imagesequence that contains a translucent micropipette.

Owing to the problems of ill-defined features and background clutter, the aboveapproach to tracking using pure condensation was not fully successful. Further issuesthat needed to be addressed were:

. features of the translucent pipette tip were not well defined and changedaccording to the incident light angle; and

Figure 4.Particle filtering stages

K34,9/10

1428

Page 9: Machine vision methods for autonomous micro‐robotic systems

. objects from the background, e.g. the cell holder, became visible through thetranslucent pipette tip.

To address these issues, a two-layered decision process was designed, with a feedbackmechanism between the second layer and the first. The first layer decides upon theprobable locations of the object based on the information from the previous frame. Thisis very similar to classic particle filtering, as described earlier. The second layer furtherfilters those locations to arrive upon a more accurate list of possible locations of theobject. This second filtering stage makes use of the information in the current frame.

Figure 5.The weights measure for

this graph and subsequentgraphs represent edge

correlation measure.

Plate 3.Top left corner is the

template image. Smallblack dots are the

particles.

Machine visionmethods

1429

Page 10: Machine vision methods for autonomous micro‐robotic systems

For the next iteration, the probable locations from the second filtering stage is fed backinto the first filter.

To ensure robust tracking, the issues of feature corruption and background clutterwere addressed as follows:

. feature corruption: a multi-variate feature measure was developed by fusingedge features with grey scale value-based features.

. background clutter: a distance penalisation measure was incorporated whileevaluating the probable location of the objects.

Plate 4 and Figure 6 show the new results of implementing our modified version of theparticle filter algorithm on an image sequence that contains a translucent micropipettecrossing some non-translucent electrodes. As can be seen, in spite of backgroundclutter, the pipette tip is localised correctly.

To address the issue of real-time processing, it was assumed that there is only oneinstance of the object present in the scene. Distractions owing to background clutterwere further reduced by limiting the probable object locations within the vicinity of thepreviously tracked location. To implement this, the particle locations (i.e. probablelocations) are re-initialised to be in the vicinity of the tracked location of the object.Re-initialisation of the probable locations is performed only for those frames whichgive a very high probability value for the probable tracked location.

3.2 Colour segmentationAlthough segmentation by colour is not a new field of investigation, the use of colourcues has traditionally not been robust. This is mainly due to the three individual RGBcomponents that can vary almost independently depending on the brightness andcolour of the light that falls onto the surface that is being imaged. In addition, manyexisting segmenters are based on the principle of look-up-tables. This contributes to

Plate 4.In spite of indistinctivefeatures of the pipette, itslocation is found.

K34,9/10

1430

Page 11: Machine vision methods for autonomous micro‐robotic systems

the inflexibility of the segmenter to either interpolate or extrapolate information.Furthermore, these methods produce a solution outside the statistical frameworkmaking it difficult to fuse into existing systems (Cıger and Placek, 2001).

For solutions that fall within a statistical framework, methods based onunsupervised k-means clustering are preferable (Lucchese and Mitra, 1999). Many ofthese methods tend to make use of unique colour spaces, the most common of which isthe CIE L *a*b* colour space which is designed to encompass all colours that a humaneye can see. This colour space is represented by three variables, namely luminance andthe colour values on the red-green and blue-yellow axis. When applied to real-worldsituations, problems with these methods exist:

. clustering is an iterative process and so it is computationally expensive;

. unsupervised clustering is non-deterministic, so it may fail, and/or the timerequired to produce a result cannot be estimated;

. the use of unsupervised clustering leads to a non-unique index for the object ofinterest, so further human interpretation is required to identify the index thatrepresents the object of interest; and

. conversion between colour spaces adds to the computational cost.

A method for segmenting images using a committee of multilayered-perceptron (MLP)type neural networks has been developed. Each network takes the form of a 3-12-1configuration, i.e. consisting of three units on the input layer, 12 units on the singlehidden layer and one unit on the output layer. The number of inputs equals the numberof components of the RGB triplet. The transfer functions for the hidden and the output

Figure 6.

Machine visionmethods

1431

Page 12: Machine vision methods for autonomous micro‐robotic systems

layers are hyperbolic tangents. The network essentially takes the following form:

y ¼ fXnj¼1

woutj f

Xmi¼1

whidji xi þ bhid

! !þ b out

!ð4Þ

where y is the output of the network, x are the inputs to the network, f( · ) is the tanhtransfer function, w out and w hid are weights for the output and the hidden layers,bhid ¼ b out ¼ 21 are the biases, m ¼ 3 is the number of input units and, finally,n ¼ 12 is the number of hidden units.

The training dataset consists of colour examples from real images in the workenvironment. A real image is partitioned into colours that are to be recognised andthose that are to be ignored. The training set is constructed with the aid of a k-meansCIE L*a*b* algorithm (Woelker, 1996). A sample result following this clustering stepis shown in Figure 7. Here, the third index has been manually identified to be the onethat represents the object of interest. The RGB components of the object of interest andthe background are extracted for the training set, and a target training value isassigned.

By observing the sensitivity of the trained network, other networks can be trainedto bolster the overall sensitivity. The brightness and intensity of these images arevaried and these are used to train new networks. The optimisation algorithm used onequation (4) is QuickProp (Fahlman, 1988) with a training rate of h ¼ 1 £ 1023 anddmax ¼ 0:1:

The outputs from the multiple networks are combined with a winner-takes-alllayer which decides on the overall output of the network. In order to make use ofthis committee of networks, the networks are combined into a blob analysisparadigm, so that the object boundary can be determined.

The outputs from the networks are essentially a probabilistic measure so thesevalues are first prior-debiased so that they can be thresholded at the midpoint, i.e. atvalue 0.5. This means that we can allocate equal probability of whether a pixelconstitutes the object of interest or not. These thresholded values are then fed into ablob finder algorithm, which is, in essence, a recursive implementation of connectedcomponent analysis. Subsequently, small blobs are removed as these are assumed to bemainly due to either mis-classification or noise. The larger recognised blobs can betracked using a variety of tracking methods. Three instances of the results obtained areshown in Plate 5.

Figure 7.Construction of trainingset samples using k-meansCIE L *a *b * colour spaceclustering

K34,9/10

1432

Page 13: Machine vision methods for autonomous micro‐robotic systems

The images used in Plate 5 are derived from a real image sequence which consists of amoving pair of micro-grippers. Using shape information only to locate themicro-grippers is difficult and time-consuming. Shapes are derived from featuressuch as edges and corners. The highly textural gel-pack in the background contributesto a large number of unwanted features that will either skew the results of therecognition process or dramatically increase the search (recognition) time. Thetraditional way of designing a vision algorithm to recognise and track themicro-grippers is to use a distinctive set of markers that can be correlated against areference set. However, this is not ideal if the objective is to design a generic solution.

The micro-grippers were intentionally sputtered with a thin layer of gold in order toimprove their visibility in conditions of low light. The gold layer also causes the colourof the micro-grippers to be distinctly different from the gel-pack and the micro-lenses inthe scene. Hence, the use of colour as a cue for segmentation is ideal.

On a 3.0 GHz Pentium 4 PC, it took approximately 9.76 s for the k-means clusteringalgorithm to converge and to produce clusters that have to be subsequently interpretedmanually. On the other hand, our neural network segmenter is able to segment andrecognise the micro-grippers in approximately 700 ms which is an order of magnitudefaster. Owing to the fact that a trained neural network essentially consists ofmultipliers and adders, the algorithm also has the potential of being implementeddirectly onto hardware FPGAs.

3.3 Shape recognitionTo build a robust vision system, some types of generic shape recognition method arerequired. Most methods for shape recognition are either too slow to operate in real-timeor do not fulfill objectives for invariance.

The pairs-of-lines (POL) (Meikle et al., 2004) method was developed specifically toaddress these concerns. The POL method of recognition is invariant to translation,rotation and scale. Scale invariance is especially important for the vision system to beused in a 3D environment.

The recognition scheme is shown in Figure 8. First, edge strings are extractedfrom both a scene image and the model image using Canny’s algorithm (Canny,1986) with post processing. The strings are then transformed into a series ofstraight lines, using Ballard’s (1981) straight-line recursive split algorithm. Thesestraight lines approximate the original edge strings. Next, a search is made forpairs of lines in the scene which could potentially match pairs of lines in themodel, allowing for occlusion, scale and orientation changes in the model. Pairswhich match enable a computation of the possible position and orientation of the

Plate 5.Identification of object ofinterest using the neuralnetwork segmenter and

blob analysis.

Machine visionmethods

1433

Page 14: Machine vision methods for autonomous micro‐robotic systems

model in the scene to be made. Results from the comparison of all possible linepairs are collated in a 2D histogram. Finally, results are extracted from thehistogram and translated back to the input scene domain. This yields estimates ofthe position, scale and orientation of the model in the scene. Multiple instances ofthe same model in the scene can also be found.

In order to obtain scale invariance, consider the image of Figure 9 which shows acomparison between two pairs of lines which share a similar intersection angle. Forclarity, the pairs of lines have been shown in a similar orientation, however, this is notgenerally the case during recognition.

Figure 8.Using pairs of lines for fastreal-time recognition

Figure 9.

K34,9/10

1434

Page 15: Machine vision methods for autonomous micro‐robotic systems

The task is to compute the range of possible centroid positions of the model in thescene, given that these two pairs of lines could match. There are errors inherent in themeasurement of the position of the edges in the scene. Let this be labelled E. If it isconsidered that Ma1 is to map to Sa1 and Ma2 is to map to Sa2 and so on, the followingexpressions for the possible scale of the model in the scene can be produced:

min1 ¼Sb1

Mb1 þ Eð5Þ

min2 ¼Sb2

Mb2 þ Eð6Þ

max1 ¼Sa1

Ma1 2 Eð7Þ

max2 ¼Sa2

Ma2 2 Eð8Þ

Here, min1 and max1 are the minimum and maximum scale changes which areconsistent with the observed lines, if only Ma1 and Mb1 are considered as mapping toSa1 and Sb1. Similarly, if we consider that Ma2 and Mb2 map to Sa2 and Sb2 then wearrive at min2 and max2 being the bounding scale changes.

As the lines from the scene and model have been considered individually in this lastcalculation, the algorithm must compute the range of scales which is consistent withboth lines. This is simply the values which are common to both scale changesmin1

2max1 and min22max2. Usually this means that the larger of (min1, min2) is the

lowest consistent scale and the lower of (max1, max2) is the largest consistent scale.Referring back to Figure 9, if there are any model scales consistent with both lines

then the range of possible model centroid positions in the scene can be determined.The results of this scale invariant method are shown in Figure 10. Notice that the

size of the cogwheels in the scene image on the right is varied. The white lines in thescene image show the boundary of the objects that have been recognised. In the case ofthe cogwheel on the lower left, the orientation has not been correctly detected but eventhen the location of the centroid has been correctly identified.

Figure 10.Model of cogwheel (left)

and scene image withmultiple cogwheels of

varied size (right)

Machine visionmethods

1435

Page 16: Machine vision methods for autonomous micro‐robotic systems

In order to achieve real-time performance, the search method can be optimised. Thiscan either be done algorithmically, i.e. by implementing a more efficient anddeterministic search routine, or through parallelisation.

3.4 An adhoc method for cell identificationIn instances where generic vision methods are not viable, customised methods havebeen designed to perform specific tasks. An example is the demonstration that requiresthe micro-robot to work on a cluster of live cells. The entire imaging process developedis shown in Figure 11. It is largely based on methods developed in the field of imagemorphology.

Figure 11.Finding live cell clustersusing morphologicaloperators

K34,9/10

1436

Page 17: Machine vision methods for autonomous micro‐robotic systems

The boundaries of the live cells first have to be localised. This is done by first injectingdiacetyl-fluorescence into the aqueous solution that contains the cells. This causes onlylive cells to be visible under fluorescent lighting.

The image is then automatically thresholded using Otsu’s optimal thresholder(Otsu, 1979). The method is based on using the grey-level histogram of an image andthen selecting the lowest point between two peaks where each peak represents aseparate class. The criterion function involves minimising the ratio of thebetween-class variance to the total variance.

The thresholded image produces clusters of cells that can be labelled. In order to doso, connected component analysis is performed. The following morphological equationsummarises the process

I k ¼ ðI k21%BÞ> C ð9Þ

where k ¼ 1; 2; 3; . . . represents the pixel count in the image, Ik is the k-th pixel in theimage I, B is a morphological structuring element, C is the present cluster that theresult of the morphological dilation (given by %) intersects with.

The areas of the clusters detected above are then computed. An area filter is thenapplied to filter out clusters smaller than a pre-specified area as these fall into thecategory of either false positives or general noise.

The boundary of the clusters is required for the approach of the micro-robot. Inorder to identify the boundaries in real-time, a fast and well-known method foridentifying boundaries using morphological operators is used. It is essentially writtenas a two-stage process as follows (in mathematical morphology notation):

E ¼ I*B bðI Þ ¼ I 2 E ð10Þ

where E is the erosion process, I the original image and b(I) the extracted boundaryitself.

In the case of E, the equation above is equivalent to an image I being eroded by thestructuring element B and so can be written as an intersection of all translations of theimage I by the vector 2b [ B such that:

I*B ¼\b[B

I2b ð11Þ

In our case, the structuring element, B, is simply a 3 £ 3 matrix of ones.Now that the locations of the live cells and their boundaries have been identified, the

cells can be labelled, as shown in the final image in Figure 10.

4. Current resultsThe MiCRoN project is still under way and the specifications of the demonstrationsillustrated in Section 2 may still be subject to minor hardware modifications. Thealgorithms presented in this paper were initially designed in an early stage of theproject, when both demonstrations were still being defined in detail. For this reason,whilst the algorithms address the main issues that will be faced in the final set-ups,they may still be subject to alterations or fine-tuning.

Machine visionmethods

1437

Page 18: Machine vision methods for autonomous micro‐robotic systems

The results obtained with the current algorithms are presented within eachsubsection. They have been tested using image sets which are as close as possible tothe operating environments in the final set-ups.

The robust object tracking in subsection 3.1 will be used in the biologicaldemonstration for tracking the glass pipette in its approach to the cell. This will becombined with the adhoc method shown in subsection 3.4 for the recognition of livecells. A third algorithm for the recognition/tracking of the customised syringe chip isstill under development.

For the assembly demonstration (soldering task), either the colour segmentationmethod in subsection 3.2 or the shape recognition algorithm in subsection 3.3 will beemployed for the recognition/tracking of the robot-mounted micro-grippers. The choicewill largely depend on the image quality of the micro-camera-based acquisition systemand the amount of clutter present in the workspace where the soldering procedure is totake place. We are also investigating alternative methods that may be used with themicro-parts to be soldered in the case the ones presented in this paper do not seem tomeet all the needed requirements.

5. ConclusionsWhen several robots with different tools have to work in the same work environment,issues related to co-operation arise. In order to co-ordinate movement, vision systemsplay a very important role in providing feedback. This is even more important in themicro-world where it is difficult to develop and mount sensors directly on the robotsthemselves, mainly due to physical limitations.

As the need for micro-manufacturing increases, there will also be an increasing needfor technologies that will enable operation automation. Methods commonly used in themacro-world may not be as robust when applied to the micro-world. Hence, newtechnologies and methods need to be researched and developed. In this paper we havepresented a number of hardware and software issues, which arise when moving theapplication domain of robots from the macro- to the micro-world. Several new andoriginal algorithms used for calibration, recognition and tracking have been developedand presented, with results accompanying each instance.

We have adopted various techniques developed in the field of modern artificialintelligence, namely neural networks, particle filtering and morphology. Moreimportantly, we anticipate that the methods we have developed will support the widercontext of making robots autonomous. This fits in very nicely with an early quote byJohn McCarthy in 1956 on the definition of artificial intelligence: “making a machinebehave in ways that would be called intelligent if a human were so behaving”.

Notes

1. IST Project No. IST-2001-33567, www.shu.ac.uk/mmvl/micron/

2. ESPRIT 4 Project No. 33915, www.shu.ac.uk/mmvl/miniman/

3. www.shu.ac.uk/mmvl/mimas/

4. p(Xt21jZt21) is the probability of states up to time t 2 1 given all the measurements to timet 2 1. Where xt is the state of the modelled object, Xt ¼ {x1, . . . ,xt} its history and similarlyfor zt the set of image features.

5. p(xtjZt21) is the probability of states at time t given all the measurements zi to time t 2 1.

K34,9/10

1438

Page 19: Machine vision methods for autonomous micro‐robotic systems

References

Ballard, D.H. (1981), “Strip trees: a hierarchical representation for curves”, Communications ofthe ACM, Vol. 24 No. 5, pp. 310-21.

Buerkle, A., Schmoeckel, F., Kiefer, M., Amavasai, B.P., Caparrelli, F., Selvan, A.N. and Travis,J.R. (2001), “Vision based closed-loop control of mobile microrobots for micro handlingtasks”, Proceedings of SPIE Vol. 4568: Microrobotics and Microassembly III, Boston, MA,October, pp. 187-98.

Canny, J. (1986), “A computational approach to edge detection”, IEEE Transactions on PatternAnalysis and Machine Intelligence, Vol. 8 No. 6, pp. 679-98.

Cıger, C. and Placek, J. (2001), “Non-traditional image segmentation and filtering”, paperpresented at 17th Spring Conf. on Comp. Graphics, April, pp. 25-8.

Fahlman, S.E. (1988), “Faster-learning variations on back-propagation: an empirical study”, inTouretzky, D., Hinton, G. and Sejnowski, T. (Eds), Proceedings of the 1988 ConnectionistModels Summer School, Morgan Kaufmann, pp. 38-51.

Isard, M. and Blake, A. (1998), “Condensation – conditional density propagation for visualtracking”, Int. J. Computer Vision, Vol. 29 No. 1, pp. 5-28.

Lucchese, L. and Mitra, S.K. (1999), “Unsupervised segmentation of color images based onk-means clustering in the chromaticity plane”, Proceedings of IEEE Workshop onContent-based Access of Image and Video Libraries (CBAIVL’99), Fort Collins, pp. 74-8.

Meikle, S., Amavasai, B.P. and Caparrelli, F. (2004), “Towards real-time object recognition usingpairs of lines”, in preparation.

Nummiaro, K., Koller-Meier, E. and Van Gool, L.J. (2002), “Object tracking with an adaptivecolor-based particle filter”, paper presented at DAGM Symposium, pp. 353-60.

Otsu, N. (1979), “A threshold selection method from gray-level histograms”, IEEE Trans. Syst.,Man, Cybern., Vol. 9 No. 1, pp. 62-6.

Woelker, W. (1996), “Image segmentation based on an adaptive 3D analysis of the CIE-L *a *b *

color space”, Proceedings of SPIE ’96 – Visual Communications and Image Processing ’96,Vol. 2727, pp. 1197-203.

Further reading

Amavasai, B.P., Caparrelli, F., Selvan, A.N., Meikle, S. and Travis, J.R. (2003), “Control of acluster of miniature robots using cybernetic vision”, IEEE SMC Chapter Conference onCybernetics Intelligence – Challenges and Advances, Reading, September, pp. 80-6.

Gonzalez, R.C. and Woods, R.E. (1993), Digital Image Processing, Addison-Wesley, Reading, MA.

Selvan, A.N., Boissenin, M., Amavasai, B.P., Caparrelli, F. and Travis, J.R. (2003), “Trackingtranslucent objects in cluttered scenes”, paper presented at IEEE SMC Chapter Conferenceon Cybernetics Intelligence – Challenges and Advances, Reading, MA, pp. 110-8.

Machine visionmethods

1439