high-accuracy stereo depth maps using structured light by: d. scharstein & r. szeliski
DESCRIPTION
High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski. Presented by: Ali Agha March 02, 2009. Outline. Sterevision overview Motivation & Contribution Structured light & method overview Related work Disparity computation Results Conclusion Future work. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/1.jpg)
Presented by: Ali AghaMarch 02, 2009
![Page 2: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/2.jpg)
OutlineSterevision overviewMotivation & ContributionStructured light & method overviewRelated workDisparity computationResultsConclusionFuture work
![Page 3: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/3.jpg)
STEREO VISIONWhen 3D information of a scene is needed
![Page 4: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/4.jpg)
Depth from Disparity
(xR-xL)/f = b/z
disparity=dRL =(xR-xL)
![Page 5: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/5.jpg)
Motivation of the presented paper“A taxonomy and evaluation of dense two-frame
stereo correspondence algorithms”. Intl. J. Comp. Vis., 2002.
http://www.middlebury.edu/stereo/.
Tsukuba Venus
![Page 6: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/6.jpg)
need for more challenging scenes
accurate ground truth information??
Motivation of the presented paper
![Page 7: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/7.jpg)
Contributions of this work
A method for acquiring high-complexity stereo image pairs with pixel-accurate correspondence information.
Does not require the calibration of the light sources
High resolution in comparison with range sensors
![Page 8: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/8.jpg)
Process PipelineThis method uses structured light and
consists of the following stages:
Acquire all desired views under all illuminations.
Rectify the imagesDecode the light patterns at each pixel to
compute correspondences.Compute the view and illumination disparity
and combine them
![Page 9: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/9.jpg)
Structured lightStructured-light techniques rely on
projecting one or more special light patterns onto a scene, usually in order to directly acquire a range map of the scene
http://en.wikipedia.org/wiki/File:1-stripesx7.svg
![Page 10: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/10.jpg)
Structured lightA pair of cameras and one or more light
projectors are used
http://en.wikipedia.org/wiki/File:1-stripesx7.svg
![Page 11: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/11.jpg)
Related Work in Decoding light patterns
J. Batlle, E. Mouaddib, and J. Salvi. Recent progress in coded structured light as a technique to solve the correspondence problem: a survey. Pat. Recog., 31(7):963–982, 1998.
![Page 12: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/12.jpg)
Related work-CODED STRUCTURED LIGHT TECHNIQUES
Posdamer-Daltschuler 1981-82-87
![Page 13: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/13.jpg)
Related work-CODED STRUCTURED LIGHT TECHNIQUES
Inokuchi, Sato and Matsuda 1984
8 bits temporally Gray-coded pattern projection
8 bits temporally binary-coded pattern projection
![Page 14: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/14.jpg)
Gray CodeUsing such binary images requires log2(n)
patterns to distinguish among n locations.
![Page 15: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/15.jpg)
Decoding the light patternsUsing average of all-white and all-black
In practice, the only reliable way is to project both the code pattern and its inverse.
In surfaces with widely varying reflection properties, use two different exposure times (0.5 and 0.1 sec.).
If this largest difference is still below a threshold, the pixel is labeled “unknown”
![Page 16: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/16.jpg)
Disparity computationView disparitiesIllumination disparities
Definition:views – the images taken by the camerasIlluminations – the structured light patterns
projected onto the scene.
![Page 17: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/17.jpg)
View disparitiesAssuming rectified views leads simple 1D
search
Practical issues:OcclusionUnknown code values (due to shadows or reflections).A perfect matching code value may not exist
(interpolation errors)Several perfect matching code values may exist (limited
resolution)
![Page 18: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/18.jpg)
View disparitiesThe first problem (partial occlusion) is
unavoidableThe number of unknown code values can be
reduced by using more than one illumination source
As a final consistency check, we establish disparities dLR and dRL independently and cross-check for consistency.
![Page 19: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/19.jpg)
View disparities
scene under illumination view disparities
![Page 20: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/20.jpg)
Illumination disparitiesdisparity between the cameras and the
illumination sources.
The difference in our case is that we can register these illumination disparities with our rectified view disparities dLR without the need to explicitly calibrate the illumination sources (video projectors).
![Page 21: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/21.jpg)
Illumination disparitiesRelationship between the left view L and
illum. source 0.
Each pixel whose view disparity has been established can be considered a (homogeneous) 3D scene point S=[x,y,d,1] with projective depth d = dLR(x, y).
The pixel’s illumination disparity (u0L, v0L)
P = M0LS in which P = [u0L v0L 1]
![Page 22: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/22.jpg)
Practical IssuesA small number of pixels with large disparity
errors can strongly affect the least-squares fit.
Outlier detection by iterating the above process.
Only those pixels with low residual errors are selected as input to the next iteration.
![Page 23: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/23.jpg)
Illumination disparitiesGiven the projection matrix M0L, we can now
solve equation for dLR at all pixelsNote that these disparities are available for
all points illuminated by source 0, even those that are not visible from the right camera.
![Page 24: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/24.jpg)
Combining the disparity estimatesRemaining task is to combine the 2N + 2
disparity maps.Create combined maps for each of L and R
separatelyWhenever there is a majority of values within close
range, we use the averageotherwise, the pixel is labeled unknown.
L and R maps are checked for consistency, for unoccluded pixels,
dLR(x, y) = − dRL(x + dLR(x, y), y),
![Page 25: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/25.jpg)
Combined disparityMost stereo implementations work with much
smaller image sizes. So, we downsample the images and disparity maps to quarter size (460 × 384).
Note that for the downsampled images, we now have disparities with quarter-pixel accuracy.
![Page 26: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/26.jpg)
Unknown DisparitiesA remaining issue is that of holes, i.e.,
unknown disparity valuesSmall holes can be filled by interpolationLarge holes may remain in areas where no
illumination codes were available to begin with.
Two main sources: surfaces that show very low reflectionareas that are shadowed under all illuminations.
![Page 27: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/27.jpg)
ResultsTwo different scenes, Cones and Teddy.
![Page 28: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/28.jpg)
ExperimentsIn experimental setup, a single digital camera
(Canon G1) translating on a linear stage, and one or two light projectors illuminating the scene from different directions.
![Page 29: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/29.jpg)
Results
![Page 30: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/30.jpg)
VerificationTo verify that stereo data sets are useful for
evaluating stereo matching algorithms, several of the algorithms from the Middlebury Stereo Page has been ran on our new images.
![Page 31: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/31.jpg)
Conclusiona new methodology to acquire highly precise
and reliable ground truth disparity measurements
camera-projector disparities, which can be used as an auxiliary source of information to increase the reliability of correspondences and to fill in missing data.
![Page 32: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/32.jpg)
Considerations for Future workExploiting in navigation
Field of view is limited by the range of light projector
Investigate the number of projected patterns which directly affect the speed of the method
In daylight or dark placesInvisible lights
![Page 33: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/33.jpg)
Thank you
Questions??
![Page 34: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/34.jpg)
![Page 35: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/35.jpg)
![Page 36: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/36.jpg)
Related work-CODED STRUCTURED LIGHT TECHNIQUES
Posdamer-Daltschuler 1981-82-87
![Page 37: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/37.jpg)
Related work-CODED STRUCTURED LIGHT TECHNIQUES
Inokuchi, Sato and Matsuda 1984
8 bits temporally Gray-coded pattern projection
8 bits temporally binary-coded pattern projection
![Page 38: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/38.jpg)
Related work-CODED STRUCTURED LIGHT TECHNIQUES
Sato, Yamamoto and Inokuchi 1986-87
proposed to use a Liquid Crystal Devicewhich allows an increasednumber of columns to be projected witha high accuracy. The system also improves the
codedspeed, against a slide projector, so the LCD
can beelectronically controlled.
![Page 39: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/39.jpg)
If an object has a high textural contrast or any high
reßected surface regions, then, some pattern segmentation
errors can be produced. – Solution?
The problem of a light projector is sometimes a result
of heat irradiation onto the scene
![Page 40: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/40.jpg)
Related work-CODED STRUCTURED LIGHT TECHNIQUES
Hattori and Sato 1995replace the light projector with a
semiconductorlaser, which gives a high power illuminationwith low heat irradiation. The proposed
system,named Cubiscope
The Cubiscope system
![Page 41: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/41.jpg)
Related work – Carrihill-HummelkLook at the notes
![Page 42: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/42.jpg)
Related work – Boyer-KakColour
![Page 43: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/43.jpg)
Related work – Le Moigne-Waxman
not-coded grid patterns
![Page 44: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/44.jpg)
Related work – Morita-Yakima-Sakata
![Page 45: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/45.jpg)
Related work – Vuylsteke-Oosterlinck
![Page 46: High-Accuracy Stereo Depth Maps Using Structured Light by: D. Scharstein & R. Szeliski](https://reader035.vdocuments.mx/reader035/viewer/2022062722/56813ad6550346895da31070/html5/thumbnails/46.jpg)