SEOUL | Oct.7, 2016

Gwang-Soo Hong, Sookmyung women’s university

고속 스테레오 매칭 알고리즘 및 GPU 기반 병렬처리 기법 연구

2

CONTENTS

Stereo vision

Conventional stereo matching

Fast stereo matching

Evaluation

3

STEREO VISON

4

STEREO VISION Vision-Based Driver Assistance

http://ae-plus.com/technology/autoliv-develops-stereo-camera-system

Pedestrian detection

Vehicle detection

Lane detection

Stereo camera

5

STEREO VISION Computer stereo vision

Stereo vision is the extraction of 3D information from images. 3D information can be extracted by examination of the relative positions of objects in the two image planes.

Left image Right image

Disparity

6

STEREO VISION Relationship between disparity and depth

Triangulation

10/11/2016

𝑍: depth 𝑓: focal length 𝑂𝑅 , 𝑂𝑇: center positions of lens 𝐵: baseline 𝑃: object

𝑍: 𝐵 = 𝑍 − 𝑓: 𝑥

disparity

7

STEREO VISION Overview of a stereo vision system

10/11/2016

Stereo matching

Calibration Rectification

Triangulation

Stereo pair

Rectified stereo pair

Disparity map

Depth map

offline

online

8

CONVENTIONAL STEREO MATCHING

9

STEREO MATCHING Taxonomy for stereo algorithms [1]

1) Matching cost computation

2) Cost aggregation

3) Disparity computation / optimization (local and global algorithm)

4) Disparity refinement

10/11/2016

[1] D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms” Int. Jour.

Computer Vision, 47(1/2/3):7–42, 2002

10

STEREO MATCHING Matching cost computation

Basic Idea

Start at pixel 𝑝, consider its neighborhood defined by a square window

Compare with neighborhoods around pixels on the epipolar line for best match of pixel neighborhoods 10/11/2016

𝑑𝑚𝑎𝑥 𝑑𝑚𝑖𝑛

Left image Right image

𝑑𝑚𝑎𝑥

𝑑𝑚𝑖𝑛

Initial matching cost

𝑝

11

STEREO MATCHING Cost aggregation

Basic Idea

Color differences and a variation exist in the depth discontinuous

The variation in the disparity value is small between adjacent pixels

10/11/2016

Initial matching cost Aggregated matching cost

12

STEREO MATCHING Disparity computation and refinement

Disparity computation

Disparity Refinement

10/11/2016

Disparities

Matc

hin

g c

ost

Select minimum matching cost

Winner-take-all [1]

• Left-Right Consistency Check

• Median filtering

13

FAST STEREO MATCHING

14

FAST STEREO MATCHING Cost volume filtering approach for stereo matching

Stereo pair Initial matching cost Matching cost filtering Disparity computation

15

FAST STEREO MATCHING Matching cost computation

10/11/2016

𝑑𝑚𝑎𝑥

𝑑𝑚𝑖𝑛

Initial matching cost

𝑑𝑚𝑎𝑥 𝑑𝑚𝑖𝑛

Left image Right image

𝑝

𝑝

16

FAST STEREO MATCHING Cost aggregation

10/11/2016

𝑑𝑚𝑎𝑥

𝑑𝑚𝑖𝑛

𝑑𝑚𝑎𝑥

𝑑𝑚𝑖𝑛

Initial matching cost (C) Aggregated matching cost (C’)

Filtering

Guided image filtering

Filter input C

Guide G

Filtering output C’

𝐶′ = 𝐶𝑖 − 𝑛𝑖

𝐶′ = 𝑎𝐺𝑖 + 𝑏

Guide G

17

FAST STEREO MATCHING Cost aggregation

10/11/2016

Weighted guided image filtering

Filter input C

Guide G

Filtering output C’

𝐶′ = 𝐶𝑖 − 𝑛𝑖

𝐶′ = 𝑎𝐺𝑖 + 𝑏

𝐸 = ( 𝑎𝑘 ⋅ 𝐺 𝑝 + 𝑏𝑘 − 𝐶 𝑝2 + 𝜆 ⋅ 𝑎𝑘

2)

𝑝∈𝜔𝑘

𝐸 = ( 𝑎𝑘 ⋅ 𝐺 𝑝 + 𝑏𝑘 − 𝐶 𝑝2 +

𝜆

𝑊(𝐺, 𝑝)⋅ 𝑎𝑘2)

𝑝∈𝜔𝑘

Guided image filtering

18

FAST STEREO MATCHING Cost aggregation

10/11/2016

𝐸 = ( 𝑎𝑘 ⋅ 𝐺 𝑝 + 𝑏𝑘 − 𝐶 𝑝2 +

𝜆

𝑊(𝐺, 𝑝)⋅ 𝑎𝑘2)

𝑝∈𝜔𝑘

Linear regression

19

FAST STEREO MATCHING Cost aggregation

10/11/2016

The weights in tsukuba image

20

FAST STEREO MATCHING Cost aggregation

10/11/2016

Integral image technique

21

EXPERIMENTAL RESULTS

22

COMPARATIVE EVALUATIONS OF STEREO MATCHERS

Test data and Performance of stereo matching

KITTI: http://www.cvlibs.net/datasets/kitti/index.php

Middlebury Stereo Vision: http://vision.middlebury.edu/stereo/

10/11/2016

23

EXPERIMENTAL RESULTS Quantitative Middlebury evaluation

10/11/2016

TSUKUBA VENUS CONES TEDDY AVERAGE PERCENT OF

BAD PIXELS NON-OCC

ALL DISC NON-OCC

ALL DISC NON-OCC

ALL DISC NON-OCC

ALL DISC

Prop. Algorightm 1.37 1.79 6.55 0.21 0.48 2.40 5.98 11.2 15.5 2.29 8.51 7.69 5.33

Inforpermeable[2] 1.06 1.53 5.64 0.32 0.88 4.15 5.60 13.0 14.5 2.65 9.16 7.69 5.51

CostAggr[3] 1.38 1.85 6.90 0.71 1.19 6.13 7.88 13.3 18.6 3.97 9.79 8.26 5.54

GeoSup[4] 1.45 1.83 7.71 0.14 0.26 1.90 6.88 13.2 16.1 2.94 8.89 8.32 5.80

AdaptWeight[5] 1.37 1.79 6.55 0.21 0.48 2.40 5.98 11.2 15.5 2.29 8.51 7.69 6.66

[2] C. Ciglaa and A. A. Alatanb, “Information permeability for stereo matching," Signal Processing: Image Communication, vol. 28, no. 9, pp. 1072-1088,October, 2013.

[3] C. Rhemann, A. Hosni, M. Bleyer, C. Rother and M. Gelautz, “Fast cost-volume filtering for visual correspondence and beyond," IEEE Conference on Computer Vision and Pattern

Recognition, pp. 3017-3024, June, 2011.

[5] K. 300 J. Yoon and I. s. Kwoen, “Adaptive support-weight approach for correspondence search," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 4,

pp. 650-656, April, 2006.

[4] A. Hosni, M. Bleyer, M. Gelautz and C. Rhemann, “Local stereo matching using geodesic support weights," IEEE 16th International Conference on Image Processing, pp. 2093-

2096, November, 2009.

24

EXPERIMENTAL RESULTS Qualitative Middlebury evaluation

10/11/2016

25

EXPERIMENTAL RESULTS Qualitative KITTI evaluation

10/11/2016

26

EXPERIMENTAL RESULTS Computational complexity

3.2

0.16 0.11

0

0.5

1

1.5

2

2.5

3

3.5

CPU TK1 TX1

20x

1.4x

Middlebury evaluation (450 x 375) KITTI evaluation (1242 x375)

1.9

0.08 0.06

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

CPU TK1 TX1

1.3x 23x

Experiment environments

CPU: Intel Core i7 3.40GHz

TK1: Jetson TK1 Kepler GPU

TX1: Jetson TX1 Maxwell GPU

SEOUL | Oct.7, 2016

THANK YOU