stereoscopic imaging for slow-moving autonomous vehicle by: alexander norton advisor: dr. huggins...
TRANSCRIPT
![Page 1: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/1.jpg)
Stereoscopic Imaging for Slow-Moving Autonomous Vehicle
By: Alexander NortonAdvisor: Dr. Huggins
April 26, 2012
Senior Capstone ProjectFinal Presentation
Bradley University ECE Department
![Page 2: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/2.jpg)
Presentation Outline
Project Overview Stereoscopic Imaging Overview Previous Work Functional and System Description Completed Work Results Suggestions for Future Work
![Page 3: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/3.jpg)
Project Overview
The goal of this project was to design a stereoscopic imaging system using two low cost digital cameras that could calculate depth information from sets of images which could then be used to navigate an autonomous vehicle
Two modes of operation: calibration mode and run mode
![Page 4: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/4.jpg)
Stereoscopic Imaging Overview
The use of two horizontally aligned cameras separated by a fixed distance that take a pair of images at the same time
Calibrate cameras so they act like pin hole cameras Determine corresponding pixel groups Find the disparity (offset in the x coordinate) between the
corresponding pixel groups. Use triangulation to find distance to pixel groups This depth information can be used to create a 3-D
terrain map
![Page 5: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/5.jpg)
Previous Work
BirdTrak (Brian Crombie and Matt Zivney, 2003)
Bradley Rover(Steve Goggins, Rob Scherbinski, Pete Lange, 2005)
NavBot (Adam Beach, Nick Wlaznik, 2007)
SVAN (John Hessling, 2010)
![Page 6: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/6.jpg)
System Description
System block diagram Subsystem block diagrams
CamerasComputerSoftware
Modes of operationCalibration modeRun mode
![Page 7: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/7.jpg)
System Block Diagram
![Page 8: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/8.jpg)
Cameras Subsystem
![Page 9: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/9.jpg)
Computer Subsystem
CPUUser Input
Camera 1 Image capture signal
Movement instructions
Display 3D map on screen
Camera 2 Image capture signal
![Page 10: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/10.jpg)
Calibration Mode
Take images of calibration rig in several orientations
Use OpenCV to compute extrinsic and intrinsic camera parameters
Compute the intrinsic and extrinsic parameters for
the stereo cameras
Compute the rectification transformation
that makes the camera optical axes parallel
Position calibration rig in front of cameras
Estimate the relative position and orientation of the stereo camera “heads”
![Page 11: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/11.jpg)
Run Mode
![Page 12: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/12.jpg)
Necessity of Calibration
Produces the rotation and translation matrices needed to rectify sets of images
Rectification makes the stereo correspondence more accurate and more efficient
Failing to calibrate the cameras is the reason for why past groups have failed to get accurate results and useful system.
![Page 13: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/13.jpg)
Completed Work Calibration mode software
Input is a list of sets of images of a chessboard, and the number of corners along the length and width of the chessboard
Read in the left and right image pairs, find the chessboard corners, and set object and image points for the images where all the chessboards could be found
Given this list of determined points on the chessboard images, the code calls stereoCalibrate() to calibrate the cameras
![Page 14: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/14.jpg)
Calibration Mode Software
This calibration yields the camera matrix M and the distortion vector D for the two cameras; it also yields the rotation matrix R, the translation vector T, the essential matrix E, and the fundamental matrix F
The accuracy of the calibration is assessed by the software using “epipolar” geometry.
![Page 15: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/15.jpg)
Calibration Mode Software
The code then moves on to computing the rectification maps using stereoRectify()
The rectification maps are used when processing sets of images obtained in run mode
![Page 16: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/16.jpg)
Calibration Mode Software Matrices
Rotation matrix R, Translation Vector T : extrinsic matrices, put the right camera in the same plane as the left camera, which makes the two image planes coplanar
Fundamental matrix F: intrinsic matrix, relates the points on the image plane of one camera in pixels to the points on the image plane of the other camera in pixels
![Page 17: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/17.jpg)
Calibration Mode Software Matrices
Essential Matrix E: intrinsic matrix, relates the physical location of the point P as seen by the left camera to the location of the same point as seen by the right camera
Camera matrix M, distortion matrix D: intrinsic matrices, calculated and used within the function
![Page 18: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/18.jpg)
Completed WorkRun Mode Software Uses the matrices obtained from
calibration Rectifies each set of images to correct for
distortions Computes and displays the disparity map
![Page 19: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/19.jpg)
Calibration Mode Results
Output showing found chessboard corners
![Page 20: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/20.jpg)
Calibration Mode Results
Output rectified chessboard images
![Page 21: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/21.jpg)
Calibration Mode Results
Command window showing calibration results
![Page 22: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/22.jpg)
Run Mode Results
Output rectified set of images after cameras have been calibrated
![Page 23: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/23.jpg)
Run Mode Results
Output disparity map of rectified set of images
![Page 24: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/24.jpg)
Theoretical Run Mode Results
One image from a set of sample images
Disparity map obtained from the set of sample images
![Page 25: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/25.jpg)
Results
Wrote working code using OpenCV libraries and functions
Successfully grab images Some outputs of calibration are correct Unable to accurately compute the disparity
map of an image with a simple target in front of a plain background.
![Page 26: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/26.jpg)
Possible Errors
Incorrect calibration results Cameras could have internal flaws that
cannot be corrected with sufficient accuracy.
Correspondence calculation could have errors.
![Page 27: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/27.jpg)
Suggestions for Future Work
Investigate the mathematics underlying the OpenCV functions
Develop methods to find and correct for errors that occur as a result of incorrect calibrations and/or correspondence computations.
![Page 28: Stereoscopic Imaging for Slow-Moving Autonomous Vehicle By: Alexander Norton Advisor: Dr. Huggins April 26, 2012 Senior Capstone Project Final Presentation](https://reader036.vdocuments.mx/reader036/viewer/2022062422/56649e6b5503460f94b68d23/html5/thumbnails/28.jpg)
Questions??