cvpr2010: context-aware saliency detection

56

Upload: zukun

Post on 21-Jun-2015

804 views

Category:

Education


4 download

TRANSCRIPT

Page 1: CVPR2010: Context-aware saliency detection
Page 2: CVPR2010: Context-aware saliency detection

Olympic weight lifter Olympic victory Olympic

achievement

Page 3: CVPR2010: Context-aware saliency detection

Olympic weight lifter Olympic victory Olympic

achievement

Description based on “salient” pixels only

Page 4: CVPR2010: Context-aware saliency detection

Itti & Koch, 2001Bruce Tsotsos, 2009Judd et al., 2009Previous talk of the session

Page 5: CVPR2010: Context-aware saliency detection

Liu et al, 2007

Page 6: CVPR2010: Context-aware saliency detection

Grab-Cut, Rother et al., 2004

Page 7: CVPR2010: Context-aware saliency detection

Our goal: Convey the image content

Page 8: CVPR2010: Context-aware saliency detection

Stas GofermanLihi Zelnik-ManorAyellet Tal

Page 9: CVPR2010: Context-aware saliency detection

1. Principles for context-aware saliency

2. A saliency detection algorithm

3. Applications:• Image retargeting

Page 10: CVPR2010: Context-aware saliency detection

1. Principles for context-aware saliency

2. A saliency detection algorithm

3. Applications:• Image retargeting• Collages

Page 11: CVPR2010: Context-aware saliency detection

Following perceptual properties

Page 12: CVPR2010: Context-aware saliency detection

Local low-level factors• Contrast• Color• …

Walther & Koch, 2006

Page 13: CVPR2010: Context-aware saliency detection

Global considerations• Maintain unique features

Hou & Zhang, 2007

Page 14: CVPR2010: Context-aware saliency detection

Local & global• Should be multi-scale

Liu et al, 2007

Page 15: CVPR2010: Context-aware saliency detection

Visual organization (Gestalt)• Few centers of gravity• Position is important!!

Our foci

Page 16: CVPR2010: Context-aware saliency detection

High-level• Faces• Objects• People• …

Judd et al, 2009

Low-level

With face detection

Page 17: CVPR2010: Context-aware saliency detection

Our result

Page 18: CVPR2010: Context-aware saliency detection

Our result

LocalWalther & Koch, 2006

GlobalHou & Zhang, 2007

Local + globalLiu et al, 2007

Page 19: CVPR2010: Context-aware saliency detection

The steps of our algorithm

Page 20: CVPR2010: Context-aware saliency detection

Principles 1-2:Unique appearance salient

salient

Not salient

Page 21: CVPR2010: Context-aware saliency detection

Principles 1-2:Unique appearance salient

Page 22: CVPR2010: Context-aware saliency detection

Principles 1-2:Unique appearance salient

( , )color i jd p p Euclidean distance between colors of patches at pi & pj

Page 23: CVPR2010: Context-aware saliency detection

Principles 1-2:Unique appearance salient

high ( , )color i jd p p j

salient

Page 24: CVPR2010: Context-aware saliency detection

Principle 3:Position is important!

Similar patches both near and far

Not salient

Page 25: CVPR2010: Context-aware saliency detection

Principle 3:Position is important!

Similar patches near

Salient

Page 26: CVPR2010: Context-aware saliency detection

Principle 3:Position is important!

( , )position i jd p p Normalized Euclidean distance between positions of pi & pj

Page 27: CVPR2010: Context-aware saliency detection

Distance between a pair of patches:

( , )( , )

1 ( , )color i j

i jposition i j

d p pd p p

c d p p

Page 28: CVPR2010: Context-aware saliency detection

Distance between a pair of patches:

( , )( , )

1 ( , )color i j

i jposition i j

d p pd p p

c d p p

salient

High j

Page 29: CVPR2010: Context-aware saliency detection

Distance between a pair of patches:

( , )( , )

1 ( , )color i j

i jposition i j

d p pd p p

c d p p

salient

High for K most similar

Page 30: CVPR2010: Context-aware saliency detection

k

rq K most similar patches at scale r

1

11 exp ( , )

i j

Kr r ri

k

S d p qK

Page 31: CVPR2010: Context-aware saliency detection

1

11 exp ( , )

i j

Kr r ri

k

S d p qK

Page 32: CVPR2010: Context-aware saliency detection

Salient at:• Multiple scales foreground• Few scales background

1

1 Mrr

i ir r

S SM

Scale 1 Scale 4

Page 33: CVPR2010: Context-aware saliency detection

Principle 3:• Few centers of gravity

Context

Page 34: CVPR2010: Context-aware saliency detection

Foci =

Include distance map

0.8iS

1 ( )focid i

X

iS

Page 35: CVPR2010: Context-aware saliency detection

Realizing Principles 1,2,3 at multiple scales

ˆ 1 ( )i i fociS S d i

Page 36: CVPR2010: Context-aware saliency detection

Principle 4:• Faces• Objects• …

Excluded from this talk

Page 37: CVPR2010: Context-aware saliency detection

Single-scale saliency

Multiple scales

Final saliency

1

11 exp ( , )

i j

Kr r ri

k

S d p qK

ˆ 1 ( )i i fociS S d i

1

1 Mrr

i ir r

S SM

X

Page 38: CVPR2010: Context-aware saliency detection

Page 39: CVPR2010: Context-aware saliency detection

Walther & Koch, 2006 Hou & Zhang, 2007

Our result

Page 40: CVPR2010: Context-aware saliency detection

Walther & Koch, 2006 Hou & Zhang, 2007

Our result

Page 41: CVPR2010: Context-aware saliency detection

Walther & Koch, 2006 Hou & Zhang, 2007

Our result

Page 42: CVPR2010: Context-aware saliency detection

Walther & Koch, 2006 Hou & Zhang, 2007

Our result

Page 43: CVPR2010: Context-aware saliency detection

Walther & Koch, 2006 Hou & Zhang, 2007

Our result

Page 44: CVPR2010: Context-aware saliency detection

Walther & Koch, 2006 Hou & Zhang, 2007

Our result

Page 45: CVPR2010: Context-aware saliency detection

Database of Hou & Zhang

Page 46: CVPR2010: Context-aware saliency detection

Liu et al, 2007

Our result

Page 47: CVPR2010: Context-aware saliency detection

Image retargetingCollage

Page 48: CVPR2010: Context-aware saliency detection

Seam Carving Our result

Page 49: CVPR2010: Context-aware saliency detection

Seam Carving Our result

Page 50: CVPR2010: Context-aware saliency detection

Seam Carving Our result

Page 51: CVPR2010: Context-aware saliency detection
Page 52: CVPR2010: Context-aware saliency detection
Page 53: CVPR2010: Context-aware saliency detection
Page 54: CVPR2010: Context-aware saliency detection
Page 55: CVPR2010: Context-aware saliency detection

New definition: Context-aware saliency

Algorithm:Based on 4 perceptual principles

Applications

salient

Not salient

Page 56: CVPR2010: Context-aware saliency detection