1

Real-time Image Stitching with OpenCV

Advisor: Ching-Hwa ChengFeng Chia University

Priscilla ChengNicholas Eu

2

What is Image Stitching?

Panorama, Google Maps Streetview Minimally Invasive Surgery (MIS)

Image stitching provides one cohesive, enlarged view of surgical area

3

Our Task

Difficulties Existing research minimal Real time video, not still images

Use OpenCV library with C++ Develop and improve video stitching

application Stitch up to 4 cameras Improve stitching stability/speed

4

Stitching ProcessFeature Detection and Extraction

Feature Matching

Find Homography Matrix

Perspective Transformation

Output

5

Stitching Algorithm

6

Frame-by-frame Video Stitching Real-time video made of many

frames Stitching process repeated on each

frame Loop: extract frame stitch images

output

Left Frame Right Frame

7

Stitching Process1. Feature detection and extraction

• Use a feature detector to find feature points in overlapping regions

https://courses.cs.washington.edu/courses/cse455/09wi/Lects/lect6.pdf

8

Stitching Process

SURF Detector Algorithm

1. Feature detection and extraction

Box Filters2nd derivative of the Gaussian

9

Stitching Process2. Feature Matching

• BruteForceMatcher• Green – Good match• Red – Bad match

if distance between matches < threshold,push match pair into vector(save match pair)

10

findHomography function in OpenCV

Two images related by homography if on same plane, but different perspectives.

Describes mapping between two images with matching points using 3x3 matrix

Stitching Process3. Find Homography MatrixStitching Process

http://ieeexplore.ieee.org/ieee_pilot/articles/08tro05/tro-scaramuzza-2004490

11

Warpperspective function in OpenCV

Use the Homography matrix to combine multiple perspectives into one perspective

Stitching Process4. Perspective TransformationStitching Process

http://cebe.cf.ac.uk/learning/habitat/HABITAT1/QTVR.html

12

Stitching Process5. OutputStitching Process

+

13

Demo: 2 Videocameras

14

How can we improve?

1. Stitch up to 4 videos simultaneously Adding a camera = more processing

time = program slows down!2. Speed up program

Improve feature detection speed Reduce lag due to additional cameras

15

Stitching 3 Videos Take the output of first two stitched

frames Apply the stitching process to this

image and the 3rd frame1 2

3

16

Stitching 4 VideosProcess1. Stitch 1st and 2nd frames2. Stitch 3rd and 4th frames3. Stitch results from above steps together

12

34

17

4 Videocameras: Output

1 2

3 4

18

4 Videocameras: Output

1 2

3 4

19

Pipeline Parallelism with OpenMP

Parallel as opposed to sequential execution Utilize two threads on the CPU Thread 1 and Thread 2 execute

simultaneously Analogy: assembly line Resolve issues with speed

Before pipelining: ~1.2 seconds per frame

Pipeline ParallelismThread 1 Thread 2

21

Speed-up using OpenCL Feature detection and extraction

take up 70-80% of the total time Utilizes GPU, multiple coresSURF_OCL: OpenCL implementation

of SURF detector and extractor Improves speed and stability

nvidia.com

ctimes.com.tw

22

Other Improvements For fixed camera angles Limit feature detection to just half

the frame Essentially, don’t try to find matches

outside specified region

23

Locking Homography Calculations

Lock = fixed Homography matrix values

Unlock = recalculate values every frame

24

Accomplishments

Reduce each frame’s average processing time to ~0.6 seconds

About 2x speedup after parallelizing Avoid exponential time increase

Processing Time

Number of

Cameras

Before ParallelismAfter Parallelism

25

Demo: 4 Videocameras

26

Future Work

Improve accuracy Seam removal Optimize algorithm of feature

detection functions Test on endoscopic cameras at very

close range

27

Thank you for listening!