Vision Based Landing Site Determination

Submitted By :

Manish Tripathi

Enrollment No: A47171209021

Amity University

Human Depth Perception

The Overall System

Literature Review

• Projective Geometry for 3D scene analysis Camera Projection

I. Pin Hole Camera Model (Ideal Projection)

II. Actual Camera Model (Distortions Due to Camera and sensors)

III. Coordinate Transformation

Epipolar Geometry I. Epipolar Constraint

II. Fundamental Matrix/ Homography Matrix / Essential Matrix

III. Stereo Camera Geometry

Correspondence Matching

Triangulation and 3D Reconstruction

• Planar Segmentation

• Landing Site Features

Projective Geometry for 3D scene analysisCamera Projection Pinhole Camera Model [ Ideal Projection ]

Zero Aperture

All rays follow a Straight Line and pass from one point only

Actual Camera Tangential And Radial Distortions due to non zero lens Aperture

Can be removed using good quality Camera and sensor

Or by software

Projective Geometry for 3D scene analysis Camera Projection Continued…

Coordinate Transformation

Epipolar Geometry

Projective Geometry for 3D scene analysis

• Epipolar Geometry Continued… Fundamental Matrix [ Relation In terms of World Coordinates]

Essential Matrix [ Relation In terms of Pixel Coordinates ]

Homography Matrix [ Planar Orientation ] [For Rectification]

• Triangulation for 3D construction3D Range Map of Every Pixel

Projective Geometry for 3D scene analysisCorrespondence Matching

Area Based Block Matching (Local Method)

Graph Cut ( Global Matching)

Based On Energy Minimization (Slower Method)

Planar Segmentation

• Equation of Plane in 3D

ax+by+cz+d=0; [a, b, c] are Normal coefficients

[d] is distance from origin (Camera centre)

• RANSAC Algorithm Iteratively selects 3 Points and tries to fit the plane model using its distance

from the fourth randomly selected fourth point within a certain Threshold and would join all the points within that range.

Used Extensively to fit Mathematical models( Planes, Lines etc).

Orientation -> Perpendicular to the View Axis within a certain Tolerance angle

Landing Site Detection

• Identify landing sites

Hazard free

Terrain is suitable

Large enough to fit UAV

• Assumptions:

The camera mounted perpendicular to plane of the ground, pointing straight down.

The vertical axis of the camera image plane is aligned with the principal axis of the UAV.

• General Approach:

Generate 3D terrain map from consecutive images

Determine surface roughness and slope

Choose area that fits footprint of helicopter and minimizes roughness and slope using Plane Fitting on 3D Point cloud

Analysis and Method Used

• Stereo Setup Selection of Baseline

Too small: large depth error

Too large: difficult search problem, increase in outliers

Project analysis and Block Level Design

Disparity Estimation ResultsSequence of images captured by one camera

Two Cameras with non-planar placement

Exact Co-planar positioning of Camera

• Exact Co-planar positioning of Camera

High Accuracy

Disparity Estimation• Most important part of the whole process.

• Comes Under The Process of Correspondence Matching

• Area Based Block Matching based on minimisation of Sum of Absolute Differences (SAD) i.e.

disparity(δ) = arg[min(∑w(IL(i,j)-IR(x+i,y+j)]- (x,y)L

Faster than the Graph Cut method.

• Window Size Selection:

Small : Sensitive to noise; Thus less accurate but faster calculation

Large : Less sensitive to noise but Finer details get lost .

Thus I selected window size as 7X7 to get better results

• Disparity Range should be optimum to get balance between time for execution and Accuracy as each gray level value corresponds to one range

Disparity Estimation

• Result using stereo dataset available at Vision.Middlebury website

• Result using Raw images captured by my setup

CALIBRATION• Converting Disparity to Range requires the Projection

Matrix(f{focal length, Sx,y,Ox,y,relative orientation}

• Calibration is done to find out:Focal length of lens along X axis and Y axis

Lens displacement along X axis and Y axis

3 numbers that describe radial distortion

2 numbers that describe tangential distortion

The position and orientation of the camera in the real world (x, y and z) to calculate real world distance from pixel values in m,cm,mm etc.

CALIBRATION

• For Calibration we capture many images of a standard pattern like a Chessboard Pattern at different orientations from the two lenses :

Straight line pattern and all individual corners can be easily detected.

Independent of nonlinearity in the image acquisition.

Rectification

• To remove Image Distortions from the values calculated in Calibration

• To achieve epipolar constraint that is all conjugate pairs should lie on same rows so as to reduce search domain for point correspondences to a single row .

• We remap to remove distortions and find the new Camera Matrix.

Z-Coordinate Reconstruction

• Input :Disparity at each pixels of the overlapping region

X,Y coordinate of each pixel

Projection matrix ‘Q’ calculated from calibration to convert disparity to distance

• Output:Z coordinates at each pixel.

Formula :Z=T*ƒ/d (=δ)

3D Reconstruction

• Creating a Range Map of the image under consideration

• Done using Point Cloud Library

Planar Segmentation

• Objective :Finding planes that are perpendicular to the view angle that is normal along

z-axis.

Uses RANSAC Algorithm

Calculate Distance to the plane detected.

Contour Generation• Outline representing the shape or form of the image.

Software Tools

C/C++ in Visual Studio 2010 Compiler.

OpenCV 2.4.7 Library to do computer vision like image acquisition , calibration, Rectification, Block Mathing

Point Cloud Library 1.6.0 to do 3D Reconstruction and planar segmentation

The whole process can be implemented in MATLAB 2013a in its computer vision toolbox but is slow comparatively.

MATLAB for plot analysis and contour generation.

Inferences• 3D Range Map of the view under interest is generated.

• Plane perpendicular to the z-axis and its distance is also calculated.

• The Contour of the Image is also created.

Future scope for research • Comparing the footprint of landing strip suitable for landing

with plane detected in the image.

• Working on Multi-baseline Stereo imagery using single camera to get higher accuracy in Depth Perception

• Using Monocular cues to improve the depth.