Vision-based Landing of an Unmanned Air Vehicle

O. Shakernia, R. Vidal, S. SastryDepartment of EECS, UC Berkeley

ICRA 2004 2

Applications of Vision-based Control

Fire Scout

Global HawkPredator

SR/71

UCAV X-45

ICRA 2004 3

Goal: Autonomous landing on a ship deck

ChallengesHostile environments

Ground effectPitching deckHigh winds, etc

Why vision?Passive sensor Observes relative motion

ICRA 2004 4

Simulation: Vision in the loop

ICRA 2004 5

Vision-Based Landing of a UAV

Motion estimation algorithmsLinear, nonlinear, multiple-viewError: 5cm translation, 4° rotation

Real-time vision systemCustomized software Off-the-shelf hardware

Vision in Control LoopLanding on stationary deckTracking of pitching deck

ICRA 2004 6

Vision-based Motion Estimation

Pinhole Camera

Landing target

Image plane

Feature Points

Current pose

ICRA 2004 7

Pose Estimation: Linear Optimization

Pinhole Camera:Epipolar Constraint:Planar constraint:

More than 4 feature pointsSolve linearly forProject onto to recover

ICRA 2004 8

Pose Estimation: Nonlinear Refinement

Objective: minimize error

Parameterize rotation by Euler angles

Minimize by Newton-Raphson iteration

Initialize with linear algorithm

ICRA 2004 9

Multiple-View Motion Estimation

Multiple View Matrix

Rank deficiency constraint

Pinhole Camera

ICRA 2004 10

Multiple-View Motion Estimation

n points in m views

Equivalent to finding s.t.

Initialize with two-view linear solution

Least squared solution:

Use to linearly solve forIterate until converge

ICRA 2004 11

Real-time Vision SystemAmpro embedded Little Board PC

Pentium 233MHz running LINUX440 MB flashdisk HD robust to vibrationRuns motion estimation algorithmControls Pan/Tilt/Zoom camera

Motion estimation algorithmsWritten and optimized in C++ using LAPACKEstimate relative position and orientation at 30 Hz

UAV Pan/Tilt Camera Onboard Computer

ICRA 2004 12

Hardware Configuration

On-board UAVVision System

Vision Computer

RS232

RS232

Vision Algorithm

Frame Grabber

Camera

WaveLAN to Ground

Navigation SystemNavigation Computer

RS232 RS232

Control & Navigation

INS/GPS

WaveLAN to Ground

ICRA 2004 13

Feature Extraction

Acquire ImageThreshold HistogramSegmentationTarget DetectionCorner DetectionCorrespondence

ICRA 2004 14

Pan/Tilt to keep features in image centerPrevent features from leaving field of viewIncreased Field of ViewIncreased range of motion of UAV

Camera Control

ICRA 2004 15

Ground Station

Comparing Vision with INS/GPS

ICRA 2004 16

Motion Estimation in Real Flight Tests

ICRA 2004 17

Landing on Stationary Target

ICRA 2004 18

Tracking Pitching Target

ICRA 2004 19

Conclusions

ContributionsVision-based motion estimation (5cm accuracy)Real-time vision system in control loopDemonstrated proof of concept prototype: first vision-based UAV landing

ExtensionsDynamic vision: Filtering motion estimatesSymmetry-based motion estimationFixed-wing UAVs: Vision-based landing on runwaysModeling and prediction of ship deck motionLanding gear that grabs ship deckUnstructured environments: Recognizing good landing spots (grassy field, roof top etc)