generating seamless stereo mosaics from aerial video

04/19/23 1

Generating Seamless Stereo

Mosaics from Aerial Video

Zhigang ZhuAllen R. Hanson, Harpal S. Bassali

Howard Schultz, Edward M. Riseman

Computer Vision Lab

Computer Science Department

University of Massachusetts at Amherst

zhu@cs.umass.edu

http://www.cs.umass.edu/~zhu

04/19/23 2

Introduction

ObjectivesDevelop methods to automatically generate geo-referenced stereo mosaics from video sequences

DefinitionsFree Mosaic: composite of video sequence by registering overlapping frames

• subject to drift relative to the terrain• constrains: pure rotation or planar scenes

Geo-Mosaic: use 3D instrumentation to constrain mosaic to world coordinates Stereo Mosaics : a pair of mosaics from a single camera

• seamless under motion parallax• preserve 3D information• can be viewed in 3D directly

04/19/23 3

Important Issues in 3D Video Mosaicing

Representationcompact representation for a large-scale 3D sceneorthogonal (DEM), perspective (mosaic), parallel-perspective

Computationhow expensive are the computations of the algorithms?Goal: affordable, efficient and robust mosaicing

Accuracyaccurate for 3D viewing and 3D reconstruction ?

04/19/23 4

Geometry, Representation and Properties

Sensor motion is pure translation

Sensor

Image Plane

“Right” Mosaic “Left” Mosaic

04/19/23 5

Re-Organizing the images….

04/19/23 6

Recovering Depth from Mosaics

parallel-perspective stereo mosaics

Depth accuracy independent of depth

Two views from different perspective stereo

P(X,Y,Z)

Height H from Laser Profiler

GPS

)(y

y

y

y

d

ydH

d

BFZ

disparity

baseline

Fixed !

displacement

04/19/23 7

Stereo mosaics of Amazon rain forest

166-frame telephoto video sequence -> 7056*944 mosaics

a

b

c

04/19/23 8

Stereo viewing

Red: Right view; Blue/Green: Left view

04/19/23 9

Computation: how expensive in in real world

application?

Pro-processingArbitrary motion other than a 1D translation / 3D translationCamera orientation estimation and image rectification

Mosaicing Image sequence is not dense enough for seamless mosaicshow to generate parallel-perspective projection ?

3D recoveryhow expensive is the match in stereo mosaics?Baseline and epipolar geometry of stereo mosaics

Post-processingparallel-perspective 3D mosaics to an orthogonal DEM

04/19/23 10

Step 1. Pro-ProcessingMotion estimation & rectification

Camera pose estimation using navigation instrumentation and bundle adjustment

only sparse tie points widely distributed in the two images are needed

Image rectificationtransformation on two narrow slices in each frame

Dominant motion direction

3D path of the camera: 3D rotation + 3D translation

Original image frames

Dominant motion direction Y

Rectified image frames

Path of the camera: 3D translation

04/19/23 11

Step 2. 3D Mosaicingseamless mosaicing with motion parallax

Front slit : Both slits are sub-images of m columns (m>=1)Rear slit

Perspective image

Multi-perspective mosaics

Parallel-perspective mosaics

Rays of left view

Left view mosaic

……

Rays of right view

Right view mosaic

……

Rays of right view

Right view mosaic

……

Rays of left view

Left view mosaic

……

View interpolationRay interpolation

(1)

(2)

In a multi-perspective projection mosaic, each sub-image is full perspective, but sub-images from different frames will have different viewpoints. This may cause seams in the mosaic due to motion parallax.

Geometric Seams -Clearly Visible to Human Eyes, especially along depth boundaries

Introduce Error in Height Estimation

04/19/23 12

PRISM parallel ray interpolation for stereo mosaicing

)(

)2/(

21

1

yyiy

xxxi

yy

yyi

tts

stt

syy

dytt

2

)2

( 11

yyii

y

y

xxii

dty

dy

s

sxtx

Interpolated view:

Mosaic coordinates:

- Take a slice of certain width from each frame- Perform local registration between the overlapping slices- Generate parallel interpolated views between two known views- Re-project the point back to the mosaic

(Tx, Ty)

IP of 1st fixed line

1st fixed line

y0= dy/2

(Tx+Sx, Ty+Sy)

IP of 2nd fixed line

2nd fixed line

(X,Y,Z)

(x1, y1)

(x2, y2)

IP of interpolated fixed line

(Txi,Tyi)

(xi, yi)

04/19/23 13

Comparison : 2D mosaic & 3D mosaic

2D mosaic from dense image sequence ( 4-pixel interframe motion)2D mosaic from sparse image sequence ( 40-pixel interframe motion)

3D mosaic from sparse image sequence ( 40-pixel interframe motion)

04/19/23 14

Step 3. 3D reconstructionepipolar geometry of stereo mosaics

epipolar curveylr

lrrlxlr dyy

yyyybxx

),(

Epipolar curve1D search

Near horizontal line

Coverged pairsmall search region

xl

yl

left mosaic

xr

yr

right mosaic

dy/2

(xl,yl)

(xl,yl

)

txl(yl)

(xr,yr

)txr(yr)- txl(yl)dy/2

txr(yr)

04/19/23 15

Epipolar Geometry in Real Stereo mosaics

“Left” Mosaic

04/19/23 16

Epipolar Geometry in Real Stereo mosaics

“Right” Mosaic

04/19/23 17

Epipolar Geometry in Real Stereo mosaics

Depth Map: Brighter is higher elevation

04/19/23 18

Step 4. Post-Processing from 3D mosaic to DEM

Just a transformation !!

(X,Y,Z) world coordinates(xl,yl) left mosaic coordinates(xr,yr) right mosaic coordinates

H : a reference height (depth)F: focal length of the camerady: distance between left and right slitsby: adaptive baseliney = yr - yl

2 ,)1( lr

ly

yy

F

HYx

d

y

F

HX

)1(

yy

y

d

yH

d

bHZ

04/19/23 19

3D Rendering Result

04/19/23 20

Motion Refinement for Geo-Mosaic- when geo-data is not accurate

Extended Kalman Filter (EKF) approach

Unconstrained

-Image Match

-Accumulating error

Ground TruthGeo Meaursement

- absolute error EKF EsimationFlying path

Time Update

“Predict” from

image registration

Measurement Update

“correct”

by geo-data Motion parameter

A: Warp matrix t : translation

04/19/23 21

Geo-Reference Mosaicsframe-by frame mosaicing

Geo-mosaic

Free Mosaic

04/19/23 22

Geo-Reference Mosaicsglobal warping from free mosaic

Free mosaic

Geo-mosaic from free mosaic

square

straight

Frame-by-frame geo-mosaic

diamond

curve

04/19/23 23

Methods Summarydistribute the computations in four steps

Pro-processingMotion estimation: sparse tie points distirbuted in entire frames

Rectification and Mosaicing (PRISM)

Process two narrow slices

3D recovery (Terrest)stereo match only in two mosaics

Post-processingjust a coordinate transformation

04/19/23 24

Accuracy of 3D from Stereo Mosaics

Adaptive baselines and fixed disparity - uniform depth resolution

Ray interpolation between two successive views is similar to image rectification

3D recovery accuracy is comparable to that of a perspective stereo with an optimal baseline

04/19/23 25

Next Steps

Camera calibration and bundle adjustmentsGeometric and Photometric SeamlessnessUsing Structure Information and video –photo matchingError analysis in geo-referenced mosaic and 3D reconstruction using ground truth