A Fast Local Descriptor for Dense Matching

Engin Tola, Vincent Lepetit, Pascal Fua

Computer Vision LaboratoryEPFL

2008-06-10

MotivationNarrow baseline : Pixel Difference + Graph Cuts*

groundtruth

pixel difference

input frame

input frame

* Y. Boykov et al. Fast Approximate Energy Minimization via Graph Cuts. PAMI’01.

MotivationWide baseline : Pixel Difference + Graph Cuts

groundtruth

USE A DESCRIPTOR

input frame

input frame

pixel difference

MotivationWide baseline : SIFT Descriptor*+ Graph Cuts

groundtruth

SIFT

250 Seconds

* D. Lowe. Distinctive Image Features from Scale-Invariant Keypoints. IJCV’04

input frame

input frame

MotivationWide baseline : DAISY Descriptor+ Graph Cuts

groundtruth

DAISY

5 Seconds

input frame

input frame

MotivationHistogram Based Descriptors: SIFT, GLOH, SURF…

- Perspective robustness- Proven good performance- Robustness to many image transformations

Cons

- No efficient implementation exists for dense computation- Do not consider occlusions

Design a descriptor that is as robust as SIFT or GLOH but can be computed much more effectively and handle occlusions.

Problem Definition

epipolar lineepipolar

line

Virtual Camera

Input Frames

descriptor

Histogram based Descriptors…SIFT Computation

…

Histogram based Descriptors…SIFT Computation

SIFT -> DAISY

SIFT

+ Good Performance- Not suitable for

dense computation

SIFT -> DAISY

SIFT Sym.SIFT

+ Gaussian Kernels : Suitable for Dense Computation

GLOH*

+ Good Performance+ Better Localization- Not suitable for

dense computation

+ Good Performance- Not suitable for

dense computation

* K. Mikolajczyk and C. Schmid. A Performance Evaluation of Local Descriptors. PAMI’04.

SIFT -> DAISY

DAISY

+ Suitable for dense computation + Improved performance:*

+ Precise localization+ Rotational Robustness

Sym.SIFT

+ Suitable for Dense Computation

GLOH

+ Good Performance+ Better Localization- Not suitable for

dense computation

* S. Winder and M. Brown. Learning Local Image Descriptors in CVPR’07

DAISY Computation

…

…

…

DAISY Computation

…

…

…

DAISY Computation DAISY : 5sSIFT : 250s

- Rotating the descriptor only involves reordering the histograms. - The computation mostly involves 1D convolutions, which is fast.

Depth Map Estimation

x

NN OZpOZxDpDOZp ),(),|)(()|,( :1:1

Descriptors

Occlusion

Depthmap Evidence Smoothness Prior

Occlusions should be handled explicitly!

Depth Map Estimation

OZxMpxMOZxDpOZxDp mm

mNN ,|)()(,,|)(,|)( :1:1

Evidence

P. of a specific Occlusion Mask

Occlusion Masks

Depth Map Estimation

OZxMpxMOZxDpOZxDp mm

mNN ,|)()(,,|)(,|)( :1:1

Evidence

Occlusion Masks

P. of a specific Occlusion Mask

Experiments

DAISY SIFT

SURF NCCPixel Diff

Laser Scan

Comparing against other Descriptors

100

90

80

70

60

50

40

30

20

10

0

Correct Depth % for Image Pairs

ExperimentsComparison with other Descriptors

DAISY

SIFT

SURF

NCC

PIXEL

100

90

80

70

60

50

40

30

20

10

0

Correct Depth % for Image Pairs

ExperimentsComparison with other Descriptors

DAISY

SIFT

SURF

NCC

PIXEL

Correct Depth % vs Error Threshold

Herz-Jesu Sequence

87.4 % 83.9 % 83.8 %

84.9 % 91.8 % 91.8 %

90.8 %83.2 % 93.5 %

89.4 %80.2 % 90.7 %

Truly Occluded

Missed Depths

Missed Occlusions

Herz-Jesu Sequence

Ground TruthDAISY

Comparison with Strecha’05

Strecha’05: Wide baseline stereo from Multiple Views: A probabilistic Account

Strecha: 3072x2048

Comparison with Strecha’05

Strecha’05: Wide baseline stereo from Multiple Views: A probabilistic Account

768x512

Image TransformsContrast Change

Scale

Blurry Webcam Images

SIFTNCC

Image TransformsContrast Change

Scale

Blurry Webcam Images

DAISYNCC

Conclusion

DAISY:• Efficient descriptor for dense wide baseline matching.• Handles occlusions correctly. • Robust to perspective distortions.• Robust to lighting changes. • Can handle low quality imagery.

Future work:• Image-based rendering from widely spaced cameras. • Object detection and recognition.

DAISY Source Codehttp://cvlab.epfl.ch/software

Stereo Data and Ground Truthhttp://cvlab.epfl.ch/data

C. Strecha et al. On Benchmarking Camera Calibration and Multi-View Stereo for High Resolution Imagery, CVPR’08

Source Code & Data

Questions

DAISY Source Codehttp://cvlab.epfl.ch/software

Imageshttp://cvlab.epfl.ch/data

http://cvlab.epfl.ch/~tolaEngin Tola

DAISY Source Codehttp://cvlab.epfl.ch/software

Imageshttp://cvlab.epfl.ch/data

http://cvlab.epfl.ch/~tolaEngin Tola

QUESTIONS ?

Parameter Selection

THQ

=2TH

Q=4

R: 5->30R: 5->30

R: 5->30

THQ

=8

HQ=2 HQ=4 HQ=8

RQ:2->5 RQ:2->5 RQ:2->5

R: 5->30R: 5->30

R: 5->30

THQ

=2TH

Q=4

THQ

=8

HQ=2 HQ=4 HQ=8

RQ:2->5 RQ:2->5 RQ:2->5

Parameter Selection

R: 5->30R: 5->30

R: 5->30

THQ

=2TH

Q=4

THQ

=8

HQ=2 HQ=4 HQ=8

RQ:2->5 RQ:2->5 RQ:2->5

Wide BaselineNarrow Baseline

Max: 87 %> 86 %

V:328R=15, RQ=5,

THQ=8, HQ=8

V:52R=10, RQ=3,

THQ=4, HQ=4

V:104R=10, RQ=3,

THQ=4, HQ=8Max: 78%

V:328R=15, RQ=5,

THQ=8, HQ=8

V:200R=15, RQ=3,

THQ=8, HQ=8

V:104R=10, RQ=3,

THQ=4, HQ=8

> 77%

Parameter SelectionWide BaselineNarrow Baseline

R: 5->30R: 5->30

R: 5->30

TQ=2

TQ=4

TQ=8

Q:1->5 Q:1->5 Q:1->5

H=2 H=4 H=8

R: 5->30R: 5->30

R: 5->30

TQ=2

TQ=4

TQ=8

Q:1->5 Q:1->5 Q:1->5

H=2 H=4 H=8

0

100