SENSORS IN MOBILE DEVICES FOR CAPTURING PANORAMAS AND ENVIRONMENT MAPS

Maarten Van Lier2e Master Computerwetenschappen

Use Case Found an awesome view

View too large for one picture Take pictures with smartphone app

360° left to right / full spherical view App combines pictures into panorama

Reasonably fast

Problem Description Make panoramas and environment

maps Find alignment between image pairs Combine pictures

Problem Description (2) On smartphone!

Processing power = low Efficiency = necessary!

Interactivity Help user taking pictures

View result on smartphone Within a reasonable time

Standard approach Take pictures

Partially overlapping Find neighboring pictures Find alignment between neighbors

Intensity based alignment Feature based alignment

Composite pictures

Find AlignmentPixel Based Feature Based Find transformation

With lowest misregistration

With highest intensity match

Minimize error function Search options

Full search Hierarchical Incremental

Detect features Recognizable points Caracterized with vector SIFT, SURF, …

Match features Find same points in images Using distance between vectors

Find transformation Transforms features to

corresponing features RANSAC for outliers Find Homography

Smartphone Sensors But smartphones have sensors!

Accelerometer, compass, gyroscope Determine orientation of device

Use accelerometer & compass Use orientation!

For estimated picture location (and for real time “preview”)

The Overlap Approach Take pictures

Guide user with 3D preview of estimated panorama Save sensor data on shutter

Find overlapping regions Using saved sensor data

Detect & extract features From overlap regions

Match features Between corresponding overlaps

Find alignment Composite pictures

Taking Pictures

Taking Pictures

Taking Pictures

Taking Pictures

Find Overlap Regions Using estimated picture locations

To find neighboring pictures To determine estimated overlap

Bounding circle around picture center Bounding box around overlap region

Axis aligned vs non axis aligned

Find Overlap Regions (2)

Find Overlap Regions (3)

Find Overlap Regions (4)

Find Overlap Regions (5)

Find Overlap Regions (6)

Detect & Extract Features Only from overlapping regions of image Large overlap

Many features => good alignment Expensive detection & extraction

Small overlap Fewer features => bad alignment Cheaper detection & extraction

For actual panoramas & env maps Expected: large overlap regions Actual speed gain may not be very large

Match Features Neighboring pictures

Using estimated locations (sensor data) Features in same overlap region Features estimated to be close to

eachother Use estimated 3D or polar feature locations But not yet implemented

Match Features

Likely match

Unlikely match

Likely match

Possible false match

Overlap Regions Test

Overlap Regions Test (2)

Overlap Region Test (3) Recommended for most panorama apps:

About 20% on each side=> about 70-80% when on all sides

But only an estimate=> needs to be investigated & tested!

Here: on average 62% overlap Because overlap at (nearly) all sides

Expected: a 30-40% drop in time when extracting features only from overlap regions

Results: Timings (6 pictures)Standard Approach Overlap Approach

Find features 2383 ms for 4998

features Matching features &

calculate homographies 18661 ms

Total: 21044 ms

Calculate Overlap Regions 9 ms for 12 overlaps

Find Features 1617 ms for 3153

features Matching features &

calculate homographies 8252 ms

Total: 9878 => 53% less!

Results Standard Approach

Results Overlap Approach

So… What’s Next? Improve stitched result

Increase overlap region Paper! Second semester:

Optimizations Port to smartphone

Check timings on smartphone More testing & results Try something similar for pixel based alignment

Initial location using sensors + incremental refinement And maybe HDR Writing thesis text, making poster

References Image Alingment and Stitching: A

Tutorial (Richard Szeliski, 2006)

SURF: Speeded Up Robust Features (Herbert Bay, Tine Tuytelaars, Luc Van Gool,

2006) Recognising Panoramas

(M. Brown, D. G. Lowe, 2003) BoofCV

http://boofcv.org

Questions and suggestions?