Statistics in the Image Domain for Mobile Robot Environment Modeling

Luz A. Torres-Mendez and Gregory DudekCenter for Intelligent Machines, McGill University

Montreal, Quebec H3A 2A7, CAlatorres,dudek@cim.mcgill.ca

Abstract

This paper addresses the problem of estimating denserange maps of indoor locations using only intensity imagesand sparse partial depth information. Unlike shape-from-shading, we infer the relationship between intensity andrange data and use it to produce a complete depth map.We extend prior work by incorporating geometric informa-tion from the available range data, specifically, we add sur-face normal information to reconstruct surfaces whose vari-ations are not captured in the initial range measurements.In addition, the order on which we synthesize range valuesis based on a best-first approach that uses edge informa-tion from the intensity images, and isophotes lines from theavailable range. Our method uses Markov Random Fieldsto learn the statistical relationships from the available in-tensity and from those sparse regions where both range andintensity information is present. In contrast to classicalap-proaches to depth recovery (i.e. stereo, shape from shad-ing), we make only weak assumptions regarding specificsurface geometries or surface reflectance functions since wecompute the relationship between existing range data andthe images we start with. Preliminary results on real datademonstrate that our method works reasonably well whenincorporating geometric information.

1 Introduction

Surface depth recovery is one of the most classic visionproblems, both because of its scientific importance and itspragmatic value. Many standard “shape-from” methods,however, are based on strong assumptions regarding scenestructure or reflectance functions. While several elegant al-gorithms for depth recovery have been developed, the useof laser range data in many applications such as roboticshas become commonplace due to their simplicity and reli-ability (but not their elegance, cost or physical robustness).In robotics, the fusion of range data with visual informationfor navigation and mapping is particularly appealing, as it

can be used for several important applications. However, itis often hampered by the fact that range sensors that providecomplete (2-1/2D) depth maps with a resolution akin to thatof a camera, are prohibitively costly or otherwise impracti-cal. Stereo cameras can produce volumetric scans that areeconomical, but they often require calibration or producerange maps that are either incomplete or of limited reso-lution. A particular common simplifying assumption is torepresent 3D structure as a 2D “slice” through the world.However, in practice this is not sufficient to capture struc-tures of interest.

We seek to reconstruct suitable 3D models from sparserange data sets while simultaneously facilitating the dataacquisition process. It has been shown by Leeat al. [13]that although there are clear differences between opticaland range images, they nevertheless have similar second-order statistics and scaling properties. Of course, the in-timate connection between irradiance and surface normalsis the basis of classical shape-from-shading. Our motiva-tion is to exploit this fact and also that both video imagingand limited range sensing are ubiquitous readily-availabletechnologies while complete volume scanning remains pro-hibitive on most mobile platforms. It is important to high-light that we are not simply inferring a few missing pixels,but synthesizing a complete range map from as little as fewlaser scans across the environment.

Our methodology is to learn a statistical model of the (lo-cal) relationship between the observed range data and thevariations in the intensity image and use this to computethe unknown range data. We approximate thecompositeof range and intensity at each point as a Markov process.Unknown range data is then inferred by using the statis-tics of the observed range data to determine the behavior ofthe Markov process. The presence of intensity where rangedata is being inferred is crucial since intensity data providesknowledge of surface smoothness and variations in depth.Our approach learns that knowledge directly from the ob-served data, without having to hypothesize constraints thatmight be inapplicable to a particular environment.

In this paper, we extend our prior work [22, 23] by in-

1

corporating geometric information to our framework fromthe available range data. More specifically, we add sur-face normal information to be able to reconstruct surfaceswhose variations are not captured in the initial range mea-surements. We also have improved the order in which werecover depth values. This order plays a very important rolein the quality of the results.

In the following section we consider relevant prior work.Section 3 describes our method to infer range data and theimprovements to our algorithm. Section 4 tests the pro-posed algorithm on different configurations of experimen-tal data. Section 5 gives details about the incorporation ofsurface normal information to our framework. Finally, inSection 6 we give some conclusions and future directions.

2 Previous Work

We base our range estimation process on the assump-tion that the pixels constituting both the range and intensityimages acquired in an environment, can be regarded as theresults of pseudo-random processes, but that these randomprocesses exhibit useful structure. In particular, we exploitthe assumption that range and intensity images are corre-lated, albeit potentially complicated ways. Secondly, we as-sume that the variations of pixels in the range and intensityimages are related to the values elsewhere in the image(s)and that these variations can be efficiently captured by theneighborhood system of a Markov Random Field. Boththese assumptions have been considered before [5,6,10,24],but they have never been exploited in tandem.

Digital inpainting [2, 3] is similar to our problem, al-though our domain and approach are quite different. Bakerand Kanade [1] used a learned representation of pixel vari-ation for perform resolution enhancement of face images.The processes employed to interpolate new high-resolutionpixel data is similar in spirit to what we describe here, al-though the application and technical details differ signifi-cantly. The work by Freeman [9, 21] on learning the rela-tionships between intrinsic images is also related.

In prior work [22, 23], we performed reconstruction byfirst recovering the values of those voxels for which we canmake the most reliable inferences, based on the number offilled neighbors and, on the existence of probably depth dis-continuities (indicated by edges on both intensity and rangeimages). However, the extraction of edges is not alwayseasy, and it depends on the scene (textures, changes in il-lumination, etc.). While the connections between intensityedges and depth discontinuities is not assured, in our ap-proach we use it only to bias the reconstruction sequenceso that spurious edges (due to albedo, for example) do notcause difficulties. In this work, we have incorporated infor-mation regarding curvilinear iso-intensity structures inten-sity (known as isophotes) from the available range. These

isophotes, as edge information, indicate regions with depthdiscontinuity. Thus, as we reconstruct, we select first thosevoxels for reconstruction that have the largest degree ofboundary constraint and that do not contain any isophotesnor any edge in its surrounding voxels.

The inference of 3D models of a scene is a problem thatsubsumes a large part of computer vision research over thelast 30 years. In the context of this paper we will consideronly a few representative solutions.

In general, the process of building a full 3D model of areal environment can be divided onto two processes: ac-quisition of measurements in 3D and synthesis of usefulgeometric models from measurements. In some cases, forexample when models are generated manually, these stepsmay be combined. In other cases the processes of collectingsets of 3D points (often referred to as range scans), combin-ing them onto surfaces and then generating suitable modelsfor graphics applications entail distinct computations. Inthis paper we focus only on the processes of obtaining 3Ddata.

Over the last decade laser rangefinders have become af-fordable and available but their application to building full3D environment models, even from a single viewpoint, re-mains costly or difficult in practice. In particular, whilelaser line scanners based on either triangulation and/or time-of-flight are ubiquitous, full volume scanners tend to bemuch more complicated and error-prone. As a result, theacquisition ofdense, complete3D range maps is still a prag-matic challenge even if the availability of laser range scan-ners is presupposed.

Most of prior work on synthesis of 3D environmentmodels uses one of either photometric data or geometricdata [4,8,11,16] to reconstruct a 3D model of an scene. Forexample, Fitzgibbon and Zisserman [8] proposed a methodthat sequentially retrieves the projective calibration ofacomplete image sequence based on tracking corner and/orline features over two or more images, and reconstructseach feature independently in 3D. Their method solves thefeature correspondence problem based on the fundamentalmatrix and tri-focal tensor, which encode precisely the ge-ometric constraints available from two or more images ofthe same scene from different viewpoints. Related workincludes that of Pollefeys et. al. [16]; they obtain a 3Dmodel of an scene from image sequences acquired froma freely moving camera. The camera motion and its set-tings are unknown and there is no prior knowledge aboutthe scene. Their method is based on a combination of theprojective reconstruction, self calibration and dense depthestimation techniques. In general, these methods derive theepipolar geometry and the trifocal tensor from point corre-spondences. However, they assume that it is possible to runan interest operator such as a corner detector to extract fromone of the images a sufficiently large number of points that

can then be reliably matched in the other images.Shape-from-shading is related in spirit to what we are

doing, but is based on a rather different set of assumptionsand methodologies. Such method [12, 15] reconstruct a 3Dscene by inferring depth from a 2D image; in general, thistask is difficult, requiring strong assumptions regarding sur-face smoothness and surface reflectance properties.

Recent work has focus on combining information fromthe intensity and range data for 3d model reconstruction.Several authors [7, 14, 17, 19, 20] have obtained promisingresults. Pulli et al. [17] address the problem of surface re-construction by measuring both color and geometry of realobjects and displaying realistic images of objects from arbi-trary viewpoints. They use a stereo camera system with ac-tive lighting to obtain range and intensity images as visiblefrom one point of view. The integration of the range datainto a surface model is done by using a robust hierarchi-cal space carving method. The integration of intensity datawith range data has been proposed [19] to help define theboundaries of surfaces extracted from the 3D data, and thena set of heuristics are used to decide what surfaces shouldbe joined. For this application, it becomes necessary to de-velop algorithms that can hypothesize the existence of sur-face continuity and intersections among surfaces, and theformation of composite features from the surfaces.

However, one of the main issues in using the above con-figurations is that the acquisition process is very expensivebecause dense and complete intensity and range data areneeded in order to obtain a good 3D model. As far as weknow, there is no method that bases its reconstruction pro-cess on having a small amount of intensity and/or range dataand synthetically estimating the areas of missing informa-tion by using the current available data. In particular, sucha method is feasible in man-made environments, which, ingeneral, have inherent geometric constraints, such as planarsurfaces.

3 Methodology

Our objective is to infer a dense range map from an inten-sity image and a limited amount of initial range data. At theoutset, we assume that resolution of the intensity and rangedata is the same and that they are already registered (in prac-tice this registration could be computed as a first step, butwe omit this in the current presentation.) Note that while theprocess of inferring distances from intensity superficiallyresembles shape-from-shading, we do not depend on priorknowledge of reflectance or on surface smoothness or evenon surface integrability (which is a technical preconditionfor most shape-from-shading methods, even where not ex-plicitly stated).

We solve the range data inference problem as an extrap-olation problem by approximating thecompositeof range

and intensity at each point as a Markov process. Unknownrange data is then inferred by using the statistics of the ob-served range data to determine the behavior of the Markovprocess. Critical to the processes is the presence of intensitydata at each point where range is being inferred. Intuitively,this intensity data provides at least two kinds of informa-tion: (1) knowledge of when the surface is smooth, and (2)knowledge of when there is a high probability of a varia-tion in depth. Our approach learns that information fromthe observed data, without having to fabricate or hypoth-esize constraints that might be inapplicable to a particularenvironment.

From previous experiments we know that reconstructionacross depth discontinuities is often problematic as thereis comparatively little constraint for probabilistic inferenceat these locations. Such locations are often identified withedges in both the range and intensity maps. In our previousexperiments we used only edge information from the in-tensity images to lead the reconstruction process. We havenoticed that our results can be improved if we also add in-formation about linear structures from the available rangedata. These linear structures are called isophotes (all nor-mals forming same angle with direction to eye). Thus, aswe recover augmented voxels, we defer the reconstructionof augmented voxels close to intensity discontinuities (in-dicated by edges) and/or depth discontinuities (indicatedbythe isophotes) as much as possible.

In standard Markov Random Field methods, the assump-tion is that the field is updated in either stochastically or inparallel according to an iterative schedule. In practice, sev-eral authors have considered more limited update schedules.In our work, we restrict ourselves to a single update at eachunknown measurement only. In our algorithm we synthe-size depth valueR(x, y) sequentially (although this doesnot preclude parallel implementations). From previous ex-periments we know that the reconstruction sequence (theorder in we choose the next depth value to synthesize) hasa significant influence on the quality of the final result. Forexample, with the onion-peel ordering, the problem was thestrong dependence from the previous assigned voxel. Ourreconstruction sequence is then, to first recover the valuesof those augmented voxels for which we can make the mostreliable inferences, based on essentially two factors: 1) thenumber of neighboring voxels with already assigned rangeand intensity and 2) the existence of intensity and/or depthdiscontinuities (i.e. if an edge or a linear structure exists).Priority values are computed based on these two factors andare assigned to each voxel for reconstruction, such that aswe reconstruct, we select the voxel with the maximum pri-ority value. If more than one voxel shares the same priorityvalue, then the selection is done randomly.

3.1 The MRF model for range synthesis

Markov Random Fields (MRF) are used here as a modelto synthesize range. We focus on our development of aset of augmented voxels V that contain intensity (eitherfrom grayscale or color images), edge (from the intensityimage) and range information (where the range is initiallyunknown for some of them). Thus,V = (I,E,R), whereIis the matrix of known pixel intensities,E is a binary ma-trix (1 if an edge exists and0 otherwise) andR denotes thematrix of incomplete pixel depths. We are interested onlyin a set of such augmented voxels such that one augmentedvoxel lies on each ray that intersects each pixel of the in-put imageI, thus giving us a registered range imageR andintensity imageI. Let Zm = (x, y) : 1 ≤ x, y ≤ m denotethem integer lattice (over which the images are described);then I = Ix,y, (x, y) ∈ Zm, denotes the gray levels ofthe input image, andR = Rx,y, (x, y) ∈ Zm denotes thedepth values. We modelV as an MRF. Thus, we regardIandR as a random variables. For example,R = r standsfor Rx,y = rx,y, (x, y) ∈ Zm. Given a neighborhoodsystem N = Nx,y ∈ Zm, where Nx,y ⊂ Zm de-notes the neighbors of(x, y), such that,(1) (x, y) 6∈ Nx,y,and (2) (x, y) ∈ Nk,l ⇐⇒ (k, l) ∈ Nx,y. An MRF over(Zm,N ) is a stochastic process indexed byZm for which,for every(x, y) and everyv = (i , r) (i.e. each augmentedvoxel depends only on its immediate neighbors),

P (Vx,y = vx,y |Vk,l = vk,l, (k, l) 6= (x, y))

= P (Vx,y = vx,y |Vk,l = vk,l, (k, l) ∈ Nx,y), (1)

The choice ofN together with the conditional probabil-ity distribution ofP (I = i) andP (R = r), provides a pow-erful mechanism for modeling spatial continuity and otherscene features. On one hand, we choose to model a neigh-borhood Nx,y as a square mask of sizen × n centered atthe augmented voxel location(x, y). This neighborhood iscausal, meaning that only those augmented voxels alreadycontaining information (either intensity, range or both) areconsidered for the synthesis process. On the other hand,calculating the conditional probabilities in an explicit formis an infeasible task since we cannot efficiently representor determine all the possible combinations between aug-mented voxels with its associated neighborhoods. There-fore, we avoid the usual computational expense of samplingfrom a probability distribution (Gibbs sampling, for exam-ple), and synthesize a depth value from the augmented voxelVx,y with neighborhoodNx,y, by selecting the range valuefrom the augmented voxel whose neighborhoodNk,l mostresembles the region being filled in, i.e.,

Nbest = argmin ‖ Nx,y −Nk,l ‖,

(k, l) ∈ A

(2)

whereA = Ak,l ⊂ N is the set of local neighborhoods,in which the center voxel has already assigned a depthvalue, such that1 ≤

√

(k − x)2 + (l − y)2) ≤ d. For eachsuccessive augmented voxel this approximates the maxi-mum a posteriori estimate;R(k, l) is then used to specifyR(x, y). The similarity measure‖ . ‖ between two genericneighborhoodsNa andNb is defined as the weighted sum ofsquared differences (WSSD) over the partial data in the twoneighborhoods. The ”weighted” part refers to applying a2-D Gaussian kernel to each neighborhood, such that thosevoxels near the center are given more weight than those atthe edge of the window.

3.2 Range Synthesis Ordering

Our reconstruction sequence depends entirely on the pri-ority values that are assigned to each augmented voxel onthe boundary of the region to be synthesized. The pri-ority computation is biased toward those voxels that aresurrounded by high-confidence voxels, that are not on aisophote line, and whose neighborhood does not representan intensity discontinuity, in other words, whose neighbor-hood does not have any edges on it. Furthermore, edge in-formation is used to defer the synthesis of those voxels thatare on an edge to the very end.

Figure 1: The notation diagram.

Figure 1 shows the basic notation used to explain howour algorithm reconstructs the unknown depth values (nota-tion similar to that used in the inpainting literature [3]).Theregion to be synthesized, i.e., thetarget region is indicatedby Ω, and its contour is denotedδΩ. Only on those re-gions where depth discontinuities are not detected, the con-tour evolves inward as the algorithm progresses. The inputintensity (Φi) and the input range (Φr) both together formthe sourceregion, and is indicated byΦ. This regionΦ isused to calculate the local statistics for reconstruction.LetVx,y be an augmented voxel with unknown range locatedat the boundaryδΩ andNx,y be its neighborhood, whichis a n × n square window centered atVx,y. Then, for allaugmented voxelsVx,yεδΩ, we compute their priority value(which is going to determine the order in which they are

filled) as follows:

P (Vx,y) = C(Vx,y)D(Vx,y) + 1/(1 + E). (3)

whereE is the number of edges found inNx,y; C(Vx,y) isthe confidenceterm, D(Vx,y) the data term. These termsare defined as follows:

C(Vx,y) =

∑

p,qεNx,y∩Ω C(Vp,q)

|Nx,y|,

where |Nx,y| is the total number of augmented voxels inNx,y. At the beginning, the confidence of each augmentedvoxel is assigned1 if its intensity and range values are filledand0 if the range value is unknown. This confidence termC(Vx,y) may be thought of as a measurement of the amountof reliable information surrounding the voxelVx,y. Thus,as we reconstruct, we synthesize first those voxels whoseneighborhood has more of their voxels already filled, withadditional preference given to voxels that were synthesizedearly on. The data termD(Vx,y) is computed using the

Figure 2: The diagram shows how priority values are com-puted for each voxelVx,y on δΩ. Given the neighboorhoodof Vx,y, nx,y is the normal to the contourδΩ of the targetregionΩ and∇⊥

Ix,yis the isophote (direction and range) at

voxel locationx, y.

available range data in the neighborhoodNx,y, as follows(see Fig. 2):

D(Vx,y) =α

|∇⊥Ix,y

· nx,y|.where

α is a normalization factor (e.g.α = 255, in a typicalgray-level image),nx,y is a unit vector orthogonal to theboundaryδΩ at voxelVx,y. This term reduces the priorityof a voxel in whose neighborhood an isophote ”flows” into,thus altering the sequencing of the extrapolation process.This term plays an important role in our algorithm becauseit prevents the synthesis of voxels lying near a depth dis-continuity. Note, however, that it does not explicitly alterthe probability distribution associated the voxel (exceptbydeferring its evaluation), and thus has only limited risk forthe theoretical correctness of the algorithm.

Once all priority values of each augmented voxel onδΩhave been computed, we find the voxel with the highest pri-ority. We then use our MRF model to synthesize its depth

value. After a voxel has been augmented (i.e. it has in-tensity and range data), the confidence of theC(Vx,y) =C(Vk,l), i.e. it is assigned the confidence of the voxel whichmost resemble the neighborhood ofVx,y (see Eq. 2).

4 Experimental Results

In this section we show experimental results conductedon data acquired in a real-world environment. We useground truth data from a widely available database1 [18]which provides color images with complex geometry andpixel-accurate ground-truth disparity data. We also showpreliminary results on data collected by our mobile robot,which has a video camera and a laser range finder mountedon it. We start with the complete range data set as groundtruth and then hold back most of the data to simulate thesparse sample of a real scanner and to provide input to ouralgorithm. This allows us to compare the quality of our re-construction with what is actually in the scene.

Figure 3: The input intensity image and the associated groundtruth range. Since the unknown data are withheld from genuineground truth data, we can estimate our performance.

Our first set of experiments is interesting because it re-sembles what is obtained by sweeping a one-dimensionalLIDAR sensor. We show different subsamplings on thesame range image in order to compare the results. Figure 3displays the input color intensity image and the ground truthrange image from where we hold back the data to simulatethe samples. In Figure 4, two experiments are shown. Thefirst column displays the initial range data. The percentageof unknown range from top to bottom are 65% and 62%,respectively. The last column show the synthesized rangeimages when incorporating isophote constraints to our algo-rithm. For comparison purposes, the middle column showsthe synthesized range images without using informationabout the isophotes. The regions enclosed by the red rect-angles show where our algorithm performed poorly withoutusing isophote constraints. The mean absolute residual er-rors (MAR) in the grey-level range (i.e. 0 for no error and255 for maximum error) are, from top to bottom,10.5 and

1http://www.middlebury.edu/stereo

12.2, when no using isophote constraints compared to6.5and7.3, when using isophote constraints.

Figure 4: Results on real data. Two cases are shown. The ini-tial range images are in the first column with their percentage ofunknown range indicated below each. To compare results, themiddle column shows the synthesized range images without us-ing isophote information and the last column show the improvedsynthesized results with the incorporation of isophote constraints.

It can be seen that our algorithm was able to capturethe underlying structure of the scene by being able to re-construct object boundaries efficiently, even with the smallamount of range data given as an input.

We now show another set of experiments in Figure 5.The first row show the input color intensity image and itsassociated ground truth range (for comparison purposes).Three cases of subsampling are shown in the subsequentrows. The percentage of unknown range are 79%, 70% and62%, respectively. The MAR errors are5.94, 5.44 and7.54,respectively.

Once again, our algorithm was capable of reconstruct-ing the whole range map. However, it can be seen that thereconstruction was not good in regions containing surfacessloping away (for example walls). This is due to the factthat the very limited amount of input range does not covermuch of the changes in depth, and our algorithm fails byassigning already identical depth values instead of differ-ent depths at each point. Therefore, the initial range datagiven as an input is crucial to the quality of the synthesis,that is, if no interesting changes exist in the range and in-tensity, then the task becomes difficult. As it is now, ourmethod requires that the input range measurements in theform of clusters of measurements scattered over the image.This form of sampling is best since it allows local statisticsto be computed, but also provides boundary conditions atvarious locations in the image. However, clumps per se atnot available from most laser range scanners. To solve thisproblem, we have incorporated surface normal informationfrom voxels with known range values to generate new depth

Figure 5:Results on real data. In the first row are the input colorimage and ground truth range. The subsequent rows show threecases, the initial range images are in the left column and the syn-thesized results after running our algorithm, in the right column.

values accordingly. In the next section we give details aboutthe computation of surface normals from the available rangeand how we incorporate this information to our method.

5 Computing surface normals

Surface normal inference is accomplished using a stan-dard plane fitting approach based on an assumption of localbilinearity. The first step is to fit a plane to the pointsri

which lie on them × m neighborhood of every range pointRk,lεΦr. The normal vector to the computed plane is theeigenvector associated with the smallest eigenvalue of them × m matrix A =

∑N

i ((ri − c)T · (ri − c)), wherec isthe center of gravityor centroidof the set of range pointsri. The smallest eigenvalueevs of the matrix A is a mea-sure of the quality of the fit, expressing the deviation of therange pointsri from the fitted plane. We assign normalsonly to those range pointsRk,l whose deviation is below agiven threshold. In our experiments we use a neighborhood

of size5 × 5 and a threshold for the plane fitness of0.1.A confidence value related to the normal computation is as-signed to each augmented voxelVk,l. This confidence valueis calculated based on the smallest eigenvalue:1/evs. Theincorporation of surface normal information to our algo-rithm is simple. We just modify the similarity measure be-tween neighborhoods, so that it is based now on the normalsinstead on the range values. This similarity is computedonly when comparing neighborhoods of augmented voxelshaving already normal information assigned, otherwise it isdone as before. Anewdepth valueRx,y of the augmentedvoxelVx,y is computed using the following equation:

Rx,y =n · P − nxx − nyy

nz

,

where P are the(x, y, z) coordinates of the augmentedvoxel Vk,l whose neighborhood most resemble the neigh-borhood ofVx,y andn is the associated normal vector atvoxelVk,l.

We run some experiments incorporating the surface nor-mal information. In order to compare the results, Figure 6shows the synthesized range image for Case 3 of Figure 5.

Figure 6: Results when using surface normal information. Theinitial range data is the Case 3 of Figure 5.

Another experiment is shown in Figure 7. The left imageis the initial range data and the input intensity image is thatof Figure 3. In order to compare the results, the middle im-age shows the synthesized range image using the algorithmof the previous section and the right image displays the syn-thesized range image when incorporating surface normal in-formation to our algorithm.

Figure 7: Comparison results when using surface normal infor-mation from the initial range data.

These preliminary results show that our method can ac-complish the propagation of geometric structure when nor-

mal information, from the neighborhoods to be comparedwith, are available. However, there are some regions wherethis propagation was not achieved. A key factor in the com-putation of the normals is the size of the neighborhood con-taining the range points to fit the plane, which in turn de-pends on the amount of initial range and on the types ofsurfaces captured by this initial range.

We now show some preliminary results on data collectedin our own building. We use a mobile robot with a videocamera and a laser range finder mounted on it, to navi-gate the environment. For our application, the laser rangefinder was set to scan a 180 degrees field of view horizon-tally and 90 degrees vertically. As it was mentioned pre-viously, the input intensity and available range data needsto be already registered. Range and intensity are differenttype of data, their sampling resolution are not the same. Weachieved the registration of the intensity and range data inasemi-automatic way, by using crosscorrelation on the videoframes and then manually selecting those corresponding re-gions from the range image. Details about this registrationstep is not in the scope of this paper. We are currently seek-ing to have a fully automatic way of registering both type ofdata.

Figure 8 shows the input intensity and the ground truthrange data (for comparison purposes) on the first row. Thesecond row displays the input range image with 66% of un-known range and the synthesized range data after runningour algorithm. It can be seen that the whole depth map ofthe scene was recovered. Surface normal information wasof great use to smoothly generate the new depth values onthe walls, floor and ceiling.

6 Summary and Conclusions

In this paper we have presented an approach to depthrecovery that allows good quality scene reconstruction to beachieved in real environments using only monocular imagedata and a limited amount of range data. This methodologyis related to extrapolation and interpolation methods and isbased on the use of learned Markov models.

We have improved our previous results by adding infor-mation about linear structures (isophotes) from the availablerange in order to drive the reconstruction on the boundaryof objects. In addition, we have incorporated surface nor-mal information in the reconstruction process. The prelim-inary results demonstrate that the fidelity of the reconstruc-tion is improved. Perhaps more important, the robustnessof the reconstruction algorithm regarding incomplete inputdata should be substantially improved. On the other hand,the computation of the surface normals is not an easy taskby itself since it depends on the amount of initial range data.There are also some parameters that need to be carefully se-lected, such as the size of the neighborhood to fit a plane.

Figure 8: Results on real data collected from our mobile robot.

However, the results shown here demonstrate that is a viableoption to obtain a good model of the environment.

References

[1] S. Baker and T. Kanade. Limits on super-resolution and howto break them.IEEE Transactions on Pattern Analysis andMachine Intelligence, 24(9):1167–1183, 2002.

[2] M. Bertalmio, L. Vese, G. Sapiro, and S. Osher. Simultane-ous structure and texture image inpainting. InIEEE Com-puter Vision and Pattern Recognition (CVPR), 2003.

[3] A. Criminisi, P. Perez, and K. Toyama. Object removal byexamplar-based inpainting. InIEEE CVPR, 2003.

[4] P. Debevec, C. Taylor, and J. Malik. Modeling and render-ing architecture from photographs: A hybrid geometry andimage-based approach. InSIGGRAPH, pages 11–20, 1996.

[5] A. Efros and W. Freeman. Image quilting for texture syn-thesis and transfer. InSIGGRAPH, pages 1033–1038, 2001.

[6] A. Efros and T. Leung. Texture synthesis by non-parametricsampling. InICCV (2), pages 1033–1038, September 1999.

[7] S. El-Hakim. A multi-sensor approach to creating accuratevirtual environments.Journal of Photogrammetry and Re-mote Sensing, 53(6):379–391, December 1998.

[8] A. Fitzgibbon and A. Zisserman. Automatic 3d model acqui-sition and generation of new images from video sequences.In Proceedings of European Signal Processing Conference,pages 1261–1269, 1998.

[9] W. Freeman, E. Pasztor, and O. Carmichael. Shape recipes:scene representations that refer to the image.Vision SciencesSociety Annual Meeting, pages 25–47, 2003.

[10] S. Geman and D. Geman. Stochastic relaxation, gibbs distri-butions, and the bayesian restoration of images.IEEE Trans-actions on PAMI, 6:721–741, 1984.

[11] A. Hilton. Reliable surface reconstruction from multiplerange images. InECCV, 1996.

[12] B. Horn and M. Brooks.Shape from Shading. MIT Press,Cambridge Mass., 1989.

[13] A. Lee, K. Pedersen, and D. Mumford. The complex statis-tics of high-contrast patches in natural images, 2001. privatecorrespondence.

[14] M. Levoy, K. Pulli, B. Curless, S. Rusinkiewicz, D. Koller,L. Pereira, M. Ginzton, S. Anderson, J. Davis, J. Ginsberg,J. Shade, and D. Fulk. The digital michelangelo project: 3dscanning of large statues. InSIGGRAPH, July 2000.

[15] J. Oliensis. Uniqueness in shape from shading.Int. Journalof Computer Vision, 6(2):75–104, 1991.

[16] M. Pollefeys, R. Koch, M. Vergauwen, and L. V. Gool. Au-tomated reconstruction of 3d scenes from sequences of im-ages.ISPRS Journal Of Photogrammetry And Remote Sens-ing, 55(4):251–267, 2000.

[17] K. Pulli, M. Cohen, T. Duchamp, H. Hoppe, J. McDonald,L. Shapiro, and W. Stuetzle. Surface modeling and displayfrom range and color data.Lecture Notes in Computer Sci-ence 1310, pages 385–397, September 1997.

[18] D. Scharstein and R. Szeliski. High-accuracy stereo depthmaps using structured light. InIEEE CVPR, volume 1, pages195–202, 2003.

[19] V. Sequeira, K. Ng, E. Wolfart, J. Goncalves, and D. Hogg.Automated reconstruction of 3d models from real environ-ments.ISPRS Journal of Photogrammetry and Remote Sens-ing, 54:1–22, February 1999.

[20] I. Stamos and P. Allen. 3d model construction using rangeand image data. InCVPR, June 2000.

[21] A. Torralba and W. Freeman. Properties and applications ofshape recipes. InIEEE CVPR, 2003.

[22] L. Torres-Mendez and G. Dudek. Reconstruction of 3d mod-els from intensity image and partial depth. 2004. To appearin the AAAI Proceedings, July 24-29, San Jose, California.

[23] L. Torres-Mendez and G. Dudek. Statistical inference andsynthesis in the image domain for mobile robot environmentmodeling. page 8. IEEE Press, September 2004. To appearin the Proceedings of the IEEE/RSJ Conference on Intelli-gent Robots and Systems (IROS).

[24] L. Wei and M. Levoy. Fast texture synthesis using tree-structured vector quantization. InSIGGRAPH, pages 479–488, July 2000.