large 3d free surface measurement using a mobile coded light-based stereo vision system

A

bmsscbmcmpp©

K

1

cmlwlpamaofib

0d

Sensors and Actuators A 132 (2006) 460–471

Large 3D free surface measurement using a mobile codedlight-based stereo vision system

Junhua Sun ∗, Guangjun Zhang, Zhenzhong Wei, Fuqiang ZhouSchool of Instrumentation Science and Optoelectronics Engineering, Beijing University of Aeronautics and Astronautics, Beijing 100083, China

Received 14 November 2005; received in revised form 11 February 2006; accepted 28 February 2006Available online 4 April 2006

bstract

A mobile coded light-based stereo vision system for measuring large 3D free surface is proposed. The system combines the technology ofinocular stereo vision and multi-line structured light, which simplifies the process of finding correspondence. It can be freely moved aroundeasured objects, and acquires local 3D data in various viewpoints. The methods of extracting light stripe are outlined. A novel approach, pyramid

ub-grate projecting approach, is developed for encoding structured light. The approach can subdivide a high-density light grate into multipleparse sub-grates. Each of the sub-grates can be effortlessly encoded by the time-multiplexing coding method. Given a point on left image, theorrespondent point on right image is the intersection of coded light stripes with the same code and epipolar line. Then, the depth is estimatedy triangulation. Moreover, a new registering 3D data based on a planar baseline target is detailed. The mathematic model of this registrationethod is given. An objective function about all parameters of rotation matrix and transmission vector being established, and those parameters

an be found by Levenberg-Marquardt optimization Algorithm. However, the transmission vector is scaled by an unknown coefficient. Then the

resent registering method costs much less time, and is suitable for all kinds of objects. Some experimental results are presented to demonstrate theroposed technology. The results show that high-resolution 3D data of free surface has been acquired and 3D data have been perfectly registered.

2006 Elsevier B.V. All rights reserved.

; Ster

sccu(ec[oaec

eywords: 3D free surface measurement; Stereo vision; Coded structured light

. Introduction

How to efficiently acquire 3D profile of large objects is still ahallenging problem [1]. Traditionally, the 3D acquisition setupsainly include 3D coordinate measurement machine, theodo-

ite, total station positioning system and laser tracker, all ofhich are of high accuracy, but usually time-consuming and of

ow-level automatization. With the rapid development of com-uter technology, the theory of computer vision begins to bepplied to the field of precision measurement. Stereo vision, aost fundamental method in the field of computer vision, plays

n important role in 3D object model recovery from a series

∗ Corresponding author. Tel.: +86 10 82316930; fax: +86 10 82314804.E-mail address: bimsensor@126.com (J. Sun).

madc

st

924-4247/$ – see front matter © 2006 Elsevier B.V. All rights reserved.oi:10.1016/j.sna.2006.02.048

eo match; Registration; Pyramid; Planar baseline target

ions [2]. Many constraints have been established to reduce errororrespondences, and finally to find right ones. Some genericonstraints are listed as follows: (1) epipolar constraints; (2)niqueness constraints; (3) disparity continuity constraints; and4) sequence consistency constraints. Stereo algorithms that gen-rate dense depth measurements can be roughly divided into twolasses, namely, global and local algorithms. Global algorithms3] rely on iterative schemes that carry out disparity assignmentsn the basis of the minimization of a global cost function. Theselgorithms yield accurate and dense disparity measurements butxhibit a very high computational cost. Local algorithms [4–7]alculate the disparity at each pixel on the basis of the photo-etric properties of the neighbor pixels. Compared to global

lgorithms, local algorithms yield significantly less accurateisparity maps, though they can be used in many real-time appli-

While by using structured light technique, active vision canimplify the search for correspondences, it poses the problemhat multiple modules can hardly work simultaneously due to

ctuators A 132 (2006) 460–471 461

ifvt3mH

atwLcppiio

lt(mitap[tifpratepie[

aadcapocTfrs

p3w

tdrS

2

Tvutpcdotrl

a[pus

mioabsccan only measure part of it once time. So the vision sensor movesaround the object to N positions, and measures N parts of it atN viewpoints. However, all the 3D data at various viewpointsare in the sensor coordinate frame. We propose a method of

J. Sun et al. / Sensors and A

nterference of the different structured light patterns [1]. There-ore, active 3D acquisition setups capture 3D data from a singleiewpoint currently. The system and object are then rotated orransmitted with respect to each other in order to capture wholeD data of objects. At this working style, a rotating or trans-itting platform with high precision is absolutely necessary.owever, they are expensive and limit the measuring range.Here, we design a low-cost vision sensor that can efficiently

cquire dense 3D data of large objects. The sensor combines theechnologies of stereo vision and multi-line structured light, ofhich the latter simplifies the process of finding correspondence.eft camera and right camera in stereo vision simultaneouslyapture multi-line structured light patterns and their coded lightatterns on a measured object. According to these coded lightatterns, every light stripe in images captured by the two camerass encoded with a unique code. Given a light stripe point on leftmage, the correspondent point on right image is the intersectionf light stripe with the same code and epipolar line.

Currently, there are many methods for encoding structuredights. Salvi et al. [8] classify the coding methods into threeypes: (1) time-multiplexing; (2) neighborhood codification; and3) direct codification. Time-multiplexing [9–11] is one of theost commonly used coding strategies based on temporal cod-

ng. In this case, a set of patterns is successively projected ontohe measuring surface. The codeword for a given pixel is usu-lly formed by the sequence of illumination values for thatixel across the projected patterns. Neighborhood codification12–13] tends to concentrate all coding scheme in a unique pat-ern. The codeword that labels a certain point of the patterns obtained from a neighborhood of the points around it. Asor direct codification [14], there are certain ways of creating aattern so that every pixel can be labeled by the information rep-esented on it. In order to achieve this, it is necessary to use eitherlarge number of color values or introduce periodicity. In order

o acquire more dense 3D data, we propose a novel approach forncoding multi-line structured light named pyramid sub-graterojecting approach, which subdivides a high-density light gratento multiple sparse sub-grates. Each of the sub-grates can beffortlessly encoded by the time-multiplexing coding method8].

Moreover, an important issue is how to register 3D data oflarge object. The iterative closest point (ICP) [15] algorithm

nd its improved versions [16–18] are popular for registering 3Data, but they are time-consuming. Furthermore, due to its highost, we give up rotating or transmitting platforms, and developmethod of registering 3D data by a planar target instead. Thisrocess is accomplished through moving the sensor around anbject to some viewpoints, and ensuring that one of the camerasan capture the planar target between two adjacent viewpoints.his method can avoid marking on objects, some of which are

orbidden to mark like cultural relics. Compared with ICP algo-ithms, the present registration method costs less time, and isuitable for all kinds of objects.

This paper is organized as follows: Section 2 introduces therinciple of the mobile coded light-based stereo system. Sectionoutlines the method of extracting light stripe. In Section 4,e detail our methods about finding correspondences, including

Fp

Fig. 1. Structure of measurement system.

wo parts: encoding stripes and solving intersection. Section 5escribes the method of registering 3D data. The experiment andesults are given in Section 6. Finally, the conclusion is given inection 7.

. System structure and principle

The structure of measurement system is shown in Fig. 1.wo Charge Coupled Device (CCD) cameras compose a stereoision system. A Digital Light Processing (DLP) projector issed for projecting structured light patterns, simultaneously cap-ured by the two cameras. The two CCD cameras and the DLProjector constitute a vision sensor. The image acquisition cardonverts the analog signals obtained from CCD cameras intoigital signals, and send into computer, then the images arebtained. The software comprises many modules such as sys-em calibration, extracting light stripes, finding correspondence,egistration, reconstruction, display, and so on. The structuredight patterns are controlled by the computer.

Before the system works, cameras’ internal parametersnd structure parameters between cameras must be calibrated19–20]. Then, we may begin to measure an object. The DLProjector will project different structured light patterns to assists to find correspondences. Once the correspondences areolved, the 3D data is effortlessly triangulated.

Limited by its measurement range, the vision sensor can onlyeasure part of a large object once time. Therefore, our sensor

s fixed on a tripod, which can freely move around the measuredbject. 3D data acquired in adjacent viewpoints is registered byplanar target. Fig. 2 shows the principle of 3D data registrationased on planar baseline target. The 3D coordinate frame of theensor is established on left camera, and is identical to the 3Damera coordinate frame. For a large object, the vision sensor

ig. 2. The sketch map about the principle of 3D data registration based onlanar baseline target.

462 J. Sun et al. / Sensors and Actuators A 132 (2006) 460–471

registering 3D data based on a planar baseline target, on whichsome feature points are set. In the part of Experiment, we willdetail the design of the target. As shown in Fig. 2, when the visionsensor moves to position k, it measures local surface of the objectfirstly, then the target is placed on the measuring area, finally,the left camera captures the target keeping its position k. Then,the vision sensor moves to position k + 1, while the target is keptstill. After the left camera captures the target again, the target isremoved, and the vision sensor begins to measure the new partof surface. Because the left camera captures the feature pointson the target at position k and k + 1, we use these feature pointsto establish an optimization object function about all parametersof coordinate rotation matrix and transmission vector betweenthe sensor coordinate frames at position k and k + 1. Then thecoordinate frame transformation matrix Mk+1,k from position kto position k + 1 can be solved. Let the sensor coordinate frameat position 1 be the global coordinate frame, then the 3D datameasured at position k + 1 can be unified under it through thetransformation of M2,1·M3,2· . . . ·Mk+1,k. In the same way, allthe 3D data at various positions can be unified under the globalcoordinate frame. Thus, the 3D data of the whole surface undera global coordinate frame can be obtained.

3. Extracting light stripe

It is important for the system to extract the exact centers ofthe light stripe because it can affect the precision of 3D measure-ment. The usual approach detects light stripes by considering thegray values of the image. For example, the point with the high-est brightness is considered as the center point of light stripe.Furthermore, the stripe’s centroids are often used as the centerpoint. None of these methods can extract light stripe in sub-pixel.In this paper, a method of extracting light stripe in sub-pixel isgiven.

The method to extract light stripe in sub-pixel is to regardimage pixel gray value z as the function about image pixel coor-dinate (x, y), namely z(x, y), and extract light stripes from it byusing differential geometric properties. The basic idea behindthese algorithms is to locate the positions of ridges and ravinesin the image function. Based on this idea Steger [21] brings for-ward a method extracting light stripe in sub-pixel. Let s(t) be thelight stripe, s′(t) be the first derivative, namely the tangent direc-tion, and n(t) = (nx, ny)T be the norm direction. Along n(t), thosepoints whose first derivatives are zero and second derivatives’absolute value are the maximum are considered as the centersof the light stripe. Let r be the locally estimated value of z(x0,y0), and rx, ry, rxx, rxy, ryy be the locally estimated derivativesat point (x0, y0) that are obtained by convolving the image withGaussian convolution masks. Then, the Taylor polynomial ofthe image function z(x, y) at point (x0, y0) is given by

f (tnx + x0, tny + y0) = r + tnxrx + tnyry + 1

2t2n2

xrxx

+ t2nxnyrxy + 1

2t2n2

yryy. (3.1)

Tm

H

Ie

t

T

(

wtis

4

l

Fig. 3. Light stripes image.

he norm direction n(t) is calculated through a so-called Hessianatrix defined as

(x, y) =(

rxx rxy

rxy ryy

). (3.2)

ts eigenvector corresponding to the maximum eigenvalue isqual to the n(t). Let (∂f/∂t)(tnx + x0, tny + y0) = 0, we can obtain

= − nxrx + nyry

n2xrxx + 2nxnyrxy + n2

yry. (3.3)

he center points of light stripes are

px, py) = (tnx + x0, tny + y0), (3.4)

here −1/2 ≤ tnx ≤ 1/2, −1/2 ≤ tny ≤ 1/2 are required. Fig. 3 ishe image of a measured object when multi-line structured lights projected on it. Fig. 4 shows the extracted centers of lighttripes by this approach.

. Finding correspondence

In order to simplify stereo match, the technology of multi-ine structured light is applied to stereo vision system to increase

Fig. 4. The extracted centers of stripes.

J. Sun et al. / Sensors and Actuators A 132 (2006) 460–471 463

Fs

fswpTcda

4

nst

4

bs1blp

ww

iaTwgTgsg

tgttiplET‘

mltsjected pattern. The symbol 0 corresponds to black intensity while1 corresponds to full illuminated white. Therefore, a light stripecenter pixel’s codeword is obtained only when the sequence is

ig. 5. The images of a Venus statue projected by light stripes; (a) 64 lighttripes; and (b) 128 light stripes.

eature information on the measured surface. Firstly, multi-linetructured light must be encoded. Then, given a light stripe pointith certain code on left image, we search the correspondentoint along the light stripe with the same code on right image.he searching process is stopped when the point meets epipolaronstraint. Thus, the correspondences are found. We will intro-uce our coding strategy and the matching method, respectively,s follows.

.1. Coding strategy

This paper proposes a high-density grate coded light methodamed pyramid sub-grate projecting used for 3D profile mea-urement. It can remarkably increase resolution of 3D informa-ion of the measuring surface.

.1.1. Pyramid sub-grate projectingFig. 5 is the images of a Venus statue on which is projected

y light stripes. There are 64 light stripes in left image, and thosetripes are fairly dense, but can be distinguished, while there are28 light stripes in right image, and the stripes are too dense toe distinguished. If there are N (N = 2n, n is a positive integer)ight stripes, we can use the method named pyramid sub-grate

Firstly, the high-density grate (we name it as original grate)ith N light stripes are divided into two sub-grates, either ofhich has N/2 light stripes. Each of stripes in sub-grate is located

Fig. 6. The pyramid sub-grate projecting method.

n the same position as in the original grate, and the distances ofdjacent stripes increase to twice of those in the original grate.hen, if the light stripes in sub-grates are still not distinguished,e can continue to divide the two sub-grates into the second-rade sub-grates according to the above-mentioned method.hus, the original grate is divided into four second-grade sub-rates, each of which includes N/4 light stripes. Repeat thisubdivision process till the last-grade sub-grates can be distin-uished.

Let M be the number of the last-grade sub-grates, and Nlast behe number of stripes in each last-grade sub-grate. An originalrate with N light stripes is subdivided m times according tohe mentioned method, then M = 2m, and Nlast = N/M. In ordero distinguish all the last-grade sub-grates, a m bits binary digits used for denoting each of them. Fig. 6 shows the method ofyramid sub-grate projecting. The original grate in Fig. 6 has 16ight stripes, which are subdivided twice into M = 4 sub-grates.ach of the sub-grates is denoted by a m = 2 bits binary digit.hus, four second-grade sub-grates are denoted by ‘00’, ‘01’,

10’, ‘11’, and have Nlast = N/M = 16/4 = 4 stripes, respectively.Then, each of last-grade sub-grates is encoded by the time-

ultiplexing coding method, in which only two illuminationevels are used, which are coded as 0 and 1. Every stripe cen-er pixels of the pattern has its own codeword formed by theequence of 0 and 1 corresponding to its value in every pro-

Fig. 7. The time-multiplexing coding method.

464 J. Sun et al. / Sensors and Actuat

Fig. 8. Estimating the codeword according to coding patterns.

cwt

4

wtc

wybvbtfw

tmlC

mt

iae

otos

pshtPtowatr

4

ossltltt

p

F

Fig. 9. Horizontal and vertical light stripes.

ompleted. Fig. 7 shows this method. A last-grade sub-grateith Nlast stripes need log2 Nlast coding patterns to encoding

hem.

.1.2. Estimating codewordTheoretically, the symbol 0 corresponds to black intensity

hile 1 corresponds to full white. But in many factual cases,he symbol 0 does not correspond to full black, and 1 does notorrespond to full white.

One of the most commonly used methods to estimate code-ord is based on a single threshold. Let θ be the threshold, (x,

) be one pixel of light stripes on original grate, k be the num-er of the coding patterns, fi(x, y) (i = 1, 2, . . . , k) be the grayalues of correspond points on various coding patterns, and ainary digit C be the codeword of a given pixel. If fi(x, y) > θ,hen Ci = 0, where Ci is the ith bit of C. This method is suitableor objects whose reflectivities are fairly identical, and is strictith environment.Calculating gray differences of the pixels with the same posi-

ion at original grate and coding patterns, respectively, is anotherethod to estimate codeword. Let f0(x, y) be the gray value of

ight stripe point on original grate. If |f0(x, y) − fi(x, y)| < θ, theni = 1, otherwise Ci = 0. This method doesn’t require that the

wicp

ig. 10. Coded structured light matching method (pl and pr are a pair of corresponde

ors A 132 (2006) 460–471

easured object’s reflectivity would be identical, but requireshat the environment illumination is consistent.

Because the above-mentioned two methods are largely lim-ted by environment and illumination, we propose the methods follows to estimate codeword in order to reduce the ratio ofrror codeword.

Firstly, the accurate location of every stripe in original grater sub-grate image is found. Using Steger’s method, we extracthe center points of light stripe, and link them, which can throwff some interferential points. Then, we obtain the center pointet Q.

Then, select a point P ∈ Q as center, and select M continuousoints along horizontal and vertical direction, respectively, ashown in Fig. 8. Let ah be the average of M points’ gray alongorizontal direction, and av be the average along vertical direc-ion. θ is a given threshold. Let fi(P) be the gray value of the point

on the ith coding pattern. If |ah − fi(P)| > θ or |av − fi(P)| > θ,hen Ci = 1, otherwise Ci = 0. It can be seen from Fig. 9 that ifnly if ah or av is taken as the estimating condition for a code-ord, it is likely to be an error codeword when the light stripe is

long horizontal or vertical direction. According to this method,he four light stripes in Fig. 7 are coded as ‘00’, ‘01’, ‘10’, ‘11’,espectively, from left to right.

.2. Matching

Fig. 10 shows the method of finding correspondences basedn coded structured light. Having being encoded, every lighttripe in left and right image has a unique codeword. The lighttripes with the same codeword in two images are spatiallyocated at the same position. Let Li and Ri be the point set withhe codeword i (i = 1, 2, . . . , N, N is the number of stripes) ineft and right images, respectively. Select any point pl ∈ Li, andhe correspond point pr in right image meets pr ∈ Ri. Accordingo epipolar constraint [22], we have

˜ Tl · E · pr = 0, (4.1)

here pl and pr are normalized coordinates of the points onmages, and E is the essential matrix. In term of above twoonstraints, we can uniquely ascertain the correspondent pointr in right image of the point pl in left image.

nce points; lr and ll are corresponding epipolar lines of pl and pr, respectively).

J. Sun et al. / Sensors and Actuat

5

rml3fgrs

5

Ofwrza(nip

mih{

wm

EO

P

w

aEe⎧⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎩F

(

ITata

T

Ti

f

w

r(u

F

wti

5v

uoaccording to Eq. (5.3). Then, the distances between all feature

Fig. 11. The model of 3D data registration.

. Registration

For our measurement system, an important issue is how toegister multiple 3D data sets of a large object. Limited by theeasurement range, the vision sensor must measure multiple

ocal parts of a large object in various positions. However, everyD data set obtained in various positions is in sensor coordinaterame. Therefore, we must register multiple 3D data sets in alobal coordinate frame. In this paper, we develop a method ofegistering 3D data by a planar target, and detail the model andolution.

.1. Modeling and solving

Fig. 11 shows the model of 3D data registration. Letc1Xc1Yc1Zc1 and Oc2Xc2Yc2Zc2 be left camera coordinate

rames, and O1X1Y1 and O2X2Y2 be image coordinate frames,hen the vision sensor is located at position k and position k + 1,

espectively. P is a given point in 3D space, and let Pc1 = (xc1, yc1,c1)T and Pc2 = (xc2, yc2, zc2)T be its coordinates in Oc1Xc1Yc1Zc1nd Oc2Xc2Yc2Zc2, while Pc1 = (xc1, yc1, zc1, 1)T and Pc2 =xc2, yc2, zc2, 1)T be their correspond homogeneous coordi-ates. p1 = (X1, Y1)T and p2 = (X2, Y2)T are the correspondmage coordinates of point P in O1X1Y1 and O2X2Y2, while˜ n1 = (Xn1, Yn1, 1)T and pn2 = (Xn2, Yn2, 1)T are their nor-

alized homogeneous coordinates. Based on the pinhole cameramaging principle and the perspective projection principle, weave

ρ1 · pn1 = I · Pc1

ρ2 · pn2 = I · Pc2, (5.1)

here ρ1 and ρ2 are nonzero coefficients, and I is a 3 × 3 identityatrix.According to the coordinate transformation theory in

uclidean space, the transformation from Oc1Xc1Yc1Zc1 toc2Xc2Yc2Zc2 can be denoted as

˜ c2 =[

R T

0 1

]Pc1, (5.2)

psp

ors A 132 (2006) 460–471 465

here R =

⎡⎢⎣

r1 r2 r3

r4 r5 r6

r7 r8 r9

⎤⎥⎦ is 3 × 3 orthogonal rotation matrix,

nd T=(Tx, Ty, Tz)T is 3 × 1 translation vector. According to theqs. (5.1) and (5.2), the 3D coordinate of P in Oc1Xc1Yc1Zc1 isxpressed as

xc1 = zc1Xn1

yc1 = zc1Yn1

zc1 = Tx − Xn2Tz

Xn2(r7Xn1 + r8Yn1 + r9) − (r1Xn1 + r2Yn1 + r3)

= Ty − Yn2Tz

Yn2(r7Xn1 + r8Yn1 + r9) − (r4Xn1 + r5Yn1 + r6)

.

(5.3)

rom the Eq. (5.3), we can obtain the equation

Tx − Xn2Tz)[Yn2(r7Xn1 + r8Yn1 + r9) − (r4Xn1 + r5Yn1 + r6)]

−(Ty − Yn2Tz)[Xn2(r7Xn1 + r8Yn1 + r9)

−(r1Xn1 + r2Yn1 + r3)] = 0. (5.4)

t is a nonlinear equation with 12 unknown parameters: Tx, Ty,z, r1, r2, r3, r4, r5, r6, r7, r8 and r9. For the parameters Tx, Ty,nd Tz, the Eq. (5.4) is homogeneous. Let T′ = αT. According tohe established coordinate frames, Tx �= 0. So we make α = 1/Tx,nd then obtain

′ = (1 T ′y T ′

z)T. (5.5)

hus, the number of unknown parameters of Eq. (5.4) is 11. Its denoted as

(x) = 0, (5.6)

here x = (T ′y, T ′

z, r1, r2, r3, r4, r5, r6, r7, r8, r9)T.Moreover, because the rotation matrix R is orthogonal, so r1,

2, . . . , r9 must meet six orthogonal restrictions, denoted as hi(x)i = 1, 2, . . . , 6). Then, x is found by minimizing the followingnrestrictive objective function

(x) =n∑

i=1

f 2i (x) + M ·

6∑i=1

h2i (x), (5.7)

here M is a punishment coefficient, n (n ≥ 5) is the number ofhe feature points on planar target. The nonlinear minimizations conducted with the Levenberg-Marquardt Algorithm.

.2. Ascertaining the unknown coefficient of the translationector

The rotation matrix R and translation vector T with annknown coefficient having been estimated, the 3D coordinatef every feature point on planar target can be easily calculated

oints can be easily calculated. However, the distances are alsocaled by the unknown coefficient. Given two point Pi and Pj onlanar target, let D′

ab be the scaled distance between Pi and Pj,

466 J. Sun et al. / Sensors and Actuat

ac

α

Ft

Tpbt

T

ImtSsoe

Fig. 12. The designed vision sensor.

nd Dab the true distance. Thus, the unknown coefficient can be

= ±D′ab

Dab. (5.8)

ig. 13. Planar targets (a) the planar target for registration; and (b) a planararget which is prone to be influenced by illumination.

L

L

wl(

F

ws

ors A 132 (2006) 460–471

he sign of α is estimated by calculating real coordinate of aoint on target. Eq. (5.8) indicates the coefficient α is calculatedy known baseline distance between feature points on planararget. Then, the translation vector is given by

= T ′

α. (5.9)

n order to improve the capability of withstanding noise andake the algorithm more robust, we introduce the distance rela-

ive control. According to the designed planar target (detailed inection 6), the feature points are the corners of black and whitequares with the same size. Theoretically, all the side-lengthsbtained from the 3D scaled coordinate of feature points arequal. Due to noise, factually, the side-lengths are not equal. Letk denote the side-length dispersedness as follows

k = |D′2k − D′2

l |, (5.10)

here D′k and D′

l are, respectively, the kth and lth scaled side-ength. Therefore, the unrestrictive objective function in Eq.5.7) is modified as follows

(x) =n∑

f 2i (x) + M ·

6∑h2

i (x) +m∑

wi · Li(x), (5.11)

here wi is a weight coefficient, and m is the number of selectedquare sides. Then, the nonlinear minimization is also conducted

Fig. 14. Two sub-grates.

ctuat

w

α

Ai

6

6

3

Lpi

atdsa

J. Sun et al. / Sensors and A

ith the Levenberg-Marquardt Algorithm. And we have

¯ = ±(∑m

i=1D′i/D)

m. (5.12)

ccording to the Eqs. (5.9) and (5.12), the real translation vectors obtained.

. Experimental results

.1. Measuring a Venus statue

Fig. 12 shows the factual vision sensor. Two MINTRON68P CCD cameras constitute a stereo vision sensor. A NEC-

iAom

Fig. 15. All the cod

ors A 132 (2006) 460–471 467

T170 + DLP projector is used for projecting structured lightatterns. Its sensing distance is 700–900 mm, measuring ranges approximately 400 mm × 300 mm, image size 768 × 576.

The planar target for registration is shown in Fig. 13. Blacknd white squares are processed on a piece of planar glass, andhey are alternant along both horizontal direction and verticalirection. These corners of the squares are feature points and theide-length of squares is baseline length. There are 11 squareslong each side of the target, and the side-length of these squares

ing patterns.

468 J. Sun et al. / Sensors and Actuat

Fig. 16. Light stripe centers of the first sub-grate.

Fig. 17. 3D reconstruction result at the first position. (a) Using 128 light stripes;and (b) using 64 light stripes.

cia

tmmWogilmf

aalgt0l1l

Fwi

ors A 132 (2006) 460–471

orners. Fig. 13(b) shows another target, which is prone to benfluenced by illumination. (2) It is easy to automatically extractll the corners because they are coplanar.

The measured object is a Venus statue, which is about 500 mmall. The projector projects 128 light stripes, and the vision sensor

easures the statue at two positions. The vision sensor is firstlyoved to the first position and measures the head of statue.ithout loss of generality, we give the coding process of the

riginal grate captured by left camera. Using the pyramid sub-rate projecting method, the 128 light stripes are subdividednto two sub-grates shown in Fig. 14, either of which has 64ight stripes. Then, the two sub-grates are encoded by the time-

ultiplexing coding method. Fig. 15 shows the coding patternsor the first sub-grate.

The light stripe centers of the first sub-grate are extracteds shown in Fig. 16. According to the six coding patterns, theyre encoded as 111111, 111110, 111101, . . . , and 000000 fromeft to right. Likewise, the light stripe centers of the second sub-rate are also encoded as the same results. In order to distinguishhe two sub-grates, the seventh bit binary digit is introduced.

denotes the first sub-grate, while 1 the second. Thus, theight stripe centers of original grate are encoded as 0111111,111111, 0111110, 1111110, . . . , 0000000, and 1000000 fromeft to right. There are 40242 light stripe centers having been

ig. 18. The captured images of the target at the measurement area; (a) capturedhen the vision sensor is at first position; and (b) captured when the vision sensor

s at second position.

J. Sun et al. / Sensors and Actuators A 132 (2006) 460–471 469

F e secr isterin

ei0

3rs

ta

ig. 19. 3D reconstruction result. (a) 3D reconstruction result measured at thegistering; and (c) 3D reconstruction result measured at two positions after reg

ncoded in our experiment, and the number of error codewordss approximately 300, so the error codeword ratio is only about.75%.

The Venus statue is then triangulated after matching. TheD data is shown in Fig. 17, of which (a) shows theeconstruction result using 128 light stripes, while (b) 64tripes.

tmFm

ond position; (b) 3D reconstruction result measured at two positions beforeg.

When the measurement is completed at the first position,he vision sensor is moved to the second position and measuresnother part of the statue. According to Section 2, the planar

ent areas, and is captured by left camera at various positions.ig. 18 shows the captured images of the target at the measure-ent area. Using the proposed registration method, we obtain

470 J. Sun et al. / Sensors and Actuat

Table 1The result of stereo vision sensor performance

Index True valuel0 (mm)

Measuredvalue l (mm)

Error �l = l − l0(mm)

Relative error�l/l0 (%)

1 9 9.0177 0.0177 0.1962 9 8.9612 −0.0388 −0.4313 9 8.9701 −0.0299 −0.3314 9 8.9526 −0.0474 −0.5265 9 9.0085 0.0085 0.0946 9 8.9844 −0.0156 −0.1737 9 8.9413 −0.0587 −0.6528 9 8.9796 −0.0204 −0.2269 9 8.9453 −0.0547 −0.607

0.0367a 0.4074a

a RMS.

t

R

T

TFb

6

i

Fig. 20. The registration errors of x, y, z coordinates.

atif

as

iwiot

7

sblAgeraclMtisVFs

thco

ors A 132 (2006) 460–471

he registration matrixes as follows

=

⎛⎜⎝

0.999437 −0.033544 −0.000305

0.033227 0.991169 −0.128375

0.004609 0.128293 0.991726

⎞⎟⎠ ,

=

⎛⎜⎝

6.874403

0.671961

−11.668427

⎞⎟⎠ .

he reconstruction result at the second position is shown inig. 19(a). Fig. 19(b) and (c) show all 3D data at two positionsefore registering and after registering, respectively.

.2. Testing performance

To test the system performance, we conducted another exper-ment by measuring the planar target. The planar target is fixed,nd the vision sensor is moved to two positions to measure thearget, ensuring that all the feature points on target are measuredn various places. The images captured by left camera are usedor registering 3D data.

The errors of the calculated distances between feature pointsre used for evaluating the performance of the stereo vision sen-or. The results are listed in Table 1.

Having been registered, 3D data of the feature points obtainedn two positions should be theoretically equal. According to this,e evaluate the registration performance. Fig. 20 shows the reg-

stration errors of x, y, z coordinates. The registration RMS errorsf x, y, z coordinates are 0.038 mm, 0.022 mm, 0.135 mm respec-ively.

. Summary and future work

We have developed a mobile coded light-based stereo visionystem for measuring large 3D free surface. The system com-ines the technologies of stereo vision and multi-line structuredight, which simplifies the process of finding correspondence.

novel approach for encoding structured light, pyramid sub-rate projecting approach, is proposed. We can use this approachasily to encode high-density structured light. The experimentesults have indicated that the ratio of error codeword is onlybout 0.75%. Given a light stripe point on one left image, theorrespondent point on right image is the intersections of codedight stripes with same code and corresponding epipolar line.

oreover, a new registering 3D data based on a planar baselinearget have been developed. Compared with ICP algorithm andts improved ones, our registering method costs less time, and isuitable for all kinds of objects. 3D reconstruction results of aenus statue have been given in Section 6. It can be seen fromig. 19 that we have acquired high-resolution 3D data of freeurface, and 3D data have been perfectly registered.

Increasing the resolution of CCD camera and DLP projec-

ctuat

Bamgacatfo

A

e

R

[

[

[

[

[

[

[

[

[

[

[

[

[

[

B

JmrHheC

GP1Utjai

ZBhE1osv

Fo

J. Sun et al. / Sensors and A

Finally, we must indicate some issues about registering. (1)ecause there is common area of the two adjacent measurementreas, it is clear that the larger the common area is, the lower theeasurement efficiency is. But the proportion of the planar tar-

et accounting for the whole image is proportional to registeringccuracy. So, the registering accuracy and the measurement effi-iency are ambivalent, and their relation will be quantitativelynalyzed in future. (2) For a large surface, repetitious registra-ions are necessary to measure the whole surface. There will beairly large accumulative registering errors. This problem is alsone of our next research directions.

cknowledgment

This research has been supported by the National Nature Sci-nce Fund of PRC under Grant 50375012.

eferences

[1] A. Griesser, Thomas P. Koninckx, L. Van Gool, Adaptive real-time 3Dacquisition and contour tracking within a multiple structured light system,in: Proceeding of the 12th Pacific Conference on Computer Graphics andApplications, October 2004, pp. 361–370.

[2] M. Bleyer, M. Gelautz, A layered stereo matching algorithm using imagesegmentation and global visibility constraints, Photogrammetry RemoteSensing 59 (2005) 128–150.

[3] V. Kolmogorov, R. Zabih, Computing visual correspondence with occlu-sions using graph cuts, in: Proceedings of International Conference onComputer Vision, 2001.

[4] G. Van der Val, M. Hansen, M. Piacentino, The ACADIA Vision processor,in: Proceedings of 5th International Workshop on Computer Architecturefor Machine Perception, Padova, Italy, 2001, pp. 31–40.

[5] L. Di Stefano, S. Mattoccia, Fast stereo matching for the VIDET systemusing a general purpose processor with multimedia extensions, in: Proceed-ings of 5th International Workshop on Computer Architecture for MachinePerception, Padova, Italy, 2000, pp. 356–362.

[6] H. Hirschmuller, P. Innocent, J. Garibaldi, Real-time correlation basedstereo vision with reduced border errors, Int. J. Comput. Vision 47 (1–3)(2002) 229–246.

[7] K. Muhlmann, D. Maier, J. Hesser, R. Manner, Calculating dense disparitymaps from color stereo images, an efficient implementation, Int. J. Comput.Vision 47 (1–3) (2002) 79–88.

[8] J. Salvi, J. Pages, J. Batlle, Pattern codification strategies in structured lightsystems, Pattern Recognit. 37 (4) (2004) 827–849.

[9] J.L. Posdamer, M.D. Altschuler, Sureface measurement by space-encodedprojected beam systems, Comput. Graph. Image Process 18 (1) (1982)1–17.

10] D. Skocaj, A. Leonardis, Range image acquisition of objects with non-uniform albedo using structured light range sensor, in: Proceedings of the15th International Conference on Pattern Recognition, vol. 1, 2000, pp.778–781.

11] C. Rocchini, P. Cignoni, C. Montani, P. Pingi, R. Scopigno, A low cost 3Dscanner based on structured light, in: A. Chalmers, T.-M. Rhyne (Eds.),EG 2001 Proceedings, vol. 20 (3), Blackwell Publishing, Oxford, 2001,pp. 299–308.

12] N.G. Durdle, J. Thayyoor, V.J. Raso, An improved structured light tech-

13] L. Zhang, B. Curless, S.M. Seitz, Rapid shape acquisition using color struc-tured light and multi-pass dynamic programming, in: International Sympo-

2mAma

ors A 132 (2006) 460–471 471

sium on 3D data Processing Visualization and Transmission, Padova, Italy,2002, pp. 24–26.

14] T. Miyasaka, K. Kuroda, M. Hirose, K. Araki, High speed 3-D measure-ment system using incoherent light source for human performance analysis,in: Proceedings of the 19th Congress of the International Society for Pho-togrammetry and Remote Sensing, Netherlands, Amsterdam, 2000, pp.65–69.

15] Paul J. Besl, Neil D. McKay, A method for registration of 3-D shape, IEEETrans. Pattern Anal. Mach. Intell. 14 (2) (1992) 239–256.

16] K. Pulli, Multiview registration for large data sets, in: Proceedings ofSecond International Conference on 3D Digital Imaging and Modeling(3DIM’99), pp.160–168.

17] Y. Chen, Y.-P. Hung, G. Medioni, Object modeling by registrationof multiple range images, Image Vision Comput. 10 (3) (1992) 145–155.

18] Z. Zhang, Iterative point matching for registration of free-form curves andsurfaces, Int. J. Comput. Vision 13 (2) (1994) 119–152.

19] R.Y. Tsai, A versatile camera calibration technique for high accuracy 3Dmachine vision metrology using off-the-shelf TV cameras and lenses, IBMRes. Rep. RC51342 (1985).

20] Z. Zhang, A flexible new technique for camera calibration, Technical report,MSR-TR-98-71 Microsoft Research, December 1998.

21] C. Steger, Unbiased extraction of curvilinear structures from 2D and3D images, Dissertation, Fakultat fur Informatik, Technische UniversitatMunchen, 1998.

22] O. Faugeras, S. Maybank, Motion from point matches: multiplicity of solu-tions, Int. J. Comput. Vision 4 (3) (1990) 225–246.

23] C. Harris, M. Stephens. A combined corner and edge detector, in: Proceed-ings of 4th Alvey Vision Conference, Manchester, UK, 1988, pp. 147–151.

iographies

unhua Sun received BS, MS degrees from the Department of Electronic Infor-ation Engineering of Beijing Institute of Machinery, China, in 1997 and 2003,

espectively. From 1997 to 2000, he worked in Changjiang machine tool works inubei province, China, acting as an electrical engineer. From 2003 to this day,e is a doctorate student in the School of Instrumentation Science and Opto-lectronics Engineering at Beijing University of Aeronautics and Astronautics,hina. His research interests are precision measurement and machine vision.

uangjun Zhang received BS, MS and PhD degrees from the Department ofrecision Instrumentations Engineering of Tianjin University, China, in 1986,989, and 1991, respectively. He was a visiting professor at North Dakota Stateniversity, US, from 1997 to 1998. He is currently a professor and Dean of

he School of Instrumentation Science and Optoelectronics Engineering at Bei-ing University of Aeronautics and Astronautics, China. His research interestsre laser precision measurement, machine vision, optical sensing and artificialntelligence.

henzhong Wei received his BS degree form the Automation Department ofeijing Institute of Petro-Chemical Technology, China, in 1997, and receivedis MS and PhD degrees from the School of Automation Science and Electricalngineering at Beijing University of Aeronautics and Astronautics, China, in999 and 2003, respectively. He is now an associate professor in the Schoolf Instrumentation Science and Optoelectronics Engineering at Beijing Univer-ity of Aeronautics and Astronautics, China. His research interests are machineision and artificial intelligence.

uqiang Zhou received the BS, MS and PhD degrees in Measuring Technol-gy and Instrumentations from Tianjin University, China, in 1994, 1997, and

pplications.