Computer Vision GroupUniversity of California Berkeley

1

Learning Scale-Invariant Contour Completion

Xiaofeng Ren, Charless Fowlkes and Jitendra Malik

Computer Vision GroupUniversity of California Berkeley

2

AbstractWe present a model of curvilinear grouping using piecewise linear

representations of contours and a conditional random field to capture continuity and the frequency of different junction types. Potential completions are generated by building a constrained Delaunay triangulation (CDT) over the set of contours found by a local edge detector.

Maximum likelihood parameters for the model are learned from human labeled groundtruth. Using held out test data, we measure how the model, by incorporating continuity structure, improves boundary detection over the local edge detector. We also compare performance with a baseline local classifier that operates on pairs of edgels.

Both algorithms consistently dominate the low-level boundary detector at all thresholds. To our knowledge, this is the first time that curvilinear continuity has been shown quantitatively useful for a large variety of natural images. Better boundary detection has immediate application in the problem of object detection and recognition.

Computer Vision GroupUniversity of California Berkeley

3

Boundary Detection• Edge detection: 20 years after Canny• Edge detection: 20 years after Canny

• Pb (Probability of Boundary): learning to combine brightness, color and texture contrasts

• Pb (Probability of Boundary): learning to combine brightness, color and texture contrasts

• There is psychophysical evidence that we might have been approaching the limit of local edge detection

• There is psychophysical evidence that we might have been approaching the limit of local edge detection

Computer Vision GroupUniversity of California Berkeley

4

Curvilinear Continuity• Boundaries are smooth in nature

• A number of associated phenomena– Good continuation

– Visual completion

– Illusory contours

• Well studied in human vision– Wertheimer, Kanizsa, von der Heydt, Kellman,

Field, Geisler, …

• Extensively explored in computer vision– Shashua, Zucker, Mumford, Williams, Jacobs,

Elder, Jermyn, Wang, …

• Is the net effect of completion positive? Or negative? Lack of quantitative evaluation

• Boundaries are smooth in nature

• A number of associated phenomena– Good continuation

– Visual completion

– Illusory contours

• Well studied in human vision– Wertheimer, Kanizsa, von der Heydt, Kellman,

Field, Geisler, …

• Extensively explored in computer vision– Shashua, Zucker, Mumford, Williams, Jacobs,

Elder, Jermyn, Wang, …

• Is the net effect of completion positive? Or negative? Lack of quantitative evaluation

Computer Vision GroupUniversity of California Berkeley

5

Scale Invariance• Sources of scale invariance

arbitrary viewing distance hierarchy of parts

• Power laws in natural images– Lots of findings, e.g. in power spectra or wavelet

coefficients (Ruderman, Mumford, Simoncelli, …)

– Also in boundary contours [Ren and Malik 02]

• How to incorporate scale-invariance?

Computer Vision GroupUniversity of California Berkeley

6

A Scale-Invariant Representation• Piecewise linear approximation of low-level contours

– recursive splitting based on angle

• Constrained Delaunay Triangulation– a variant of the standard Delaunay Triangulation

– maximizes the minimum angle (avoids skinny triangles)

Computer Vision GroupUniversity of California Berkeley

7

The CDT Graph

scale-invariant

fast to compute

<1000 edges

completes gaps

little loss of

structure

Computer Vision GroupUniversity of California Berkeley

8

No Loss of Structure

Use Phuman the soft groundtruthlabel defined on CDT graphs:precision close to 100%

Pb averaged over CDT edges: no worse than the orignal Pb

Increase in asymptotic recall rate: completion of gradientless contours

Computer Vision GroupUniversity of California Berkeley

9

CDT vs. K-Neighbor Completion

An alternative scheme for completion: connect to k-nearest neighbor vertices, subject to visibility

CDT achieves higher asymptotic recall rates

Computer Vision GroupUniversity of California Berkeley

10

Inference on the CDT Graph

Xe

Xe

Xe

Xe

Xe

Xe

Xe

Xe

Xe

XeXeXe

Xe

Xe

Xe

Xe

Xe

Xe

Local inference:

XeXe

Global inference:

Computer Vision GroupUniversity of California Berkeley

11

Baseline Local Model

“Bi-gram” model:

contrast + continuity

binary classification (0,0) vs (1,1)

logistic classifier

“Tri-gram” model:

1 2

L LPbL

=

Xe

Computer Vision GroupUniversity of California Berkeley

12

Global Model w/ Conditional Random Fields

• Graphical model with expoential potential functions

edge potentials exp(i)

junction potentials exp(j)

• Inference with loopy belief propagation

• Maximum likelihood learning (convex) with gradient descent

converges < 10 iterations

Computer Vision GroupUniversity of California Berkeley

13

Junctions and Continuity• Junction types (degg,degc):• Junction types (degg,degc):

baXf cgba degdeg),(

degg=1,degc=0 degg=0,degc=2 degg=1,degc=2

• Continuity term for degree-2 junctions• Continuity term for degree-2 junctions

degg+degc=2

),(),( exp baba f

2degdeg exp cgg

degg=0,degc=0

Computer Vision GroupUniversity of California Berkeley

14

Continuity improves boundary detection in both low-recall and high-recall ranges

Global inference helps; mostly in low-recall/high-precision

Roughly speaking,

CRF>Local>CDT only>Pb

Computer Vision GroupUniversity of California Berkeley

15

Computer Vision GroupUniversity of California Berkeley

16

Computer Vision GroupUniversity of California Berkeley

17

Image Pb Local Global

Computer Vision GroupUniversity of California Berkeley

18

Image Pb Local Global

Computer Vision GroupUniversity of California Berkeley

19

Image Pb Local Global

Computer Vision GroupUniversity of California Berkeley

20

Conclusion

• Constrained Delaunay Triangulation is a scale-invariant discretization of images with little loss of structure;

• Constrained Delaunay Triangulation is a scale-invariant discretization of images with little loss of structure;

• Moving from 100,000 pixels to <1000 edges, CDT achieves great statistical and computational efficiency;

• Moving from 100,000 pixels to <1000 edges, CDT achieves great statistical and computational efficiency;

• Curvilinear Continuity improves boundary detection;

– the local model of continuity is simple yet very effective

– global inference of continuity further improves performance

– Conditional Random Fields w/ loopy belief propagation works well on CDT graphs

• Curvilinear Continuity improves boundary detection;

– the local model of continuity is simple yet very effective

– global inference of continuity further improves performance

– Conditional Random Fields w/ loopy belief propagation works well on CDT graphs

• Mid-level vision is useful.• Mid-level vision is useful.

Computer Vision GroupUniversity of California Berkeley

21

Thank You

Computer Vision GroupUniversity of California Berkeley

22