the university of ontario cs 433/653 algorithms for image analysis segmentation (2d)...

The University of

Ontario

CS 433/653

Algorithms for Image Analysis

Segmentation (2D)

Acknowledgements: Alexei Efros, Steven Seitz

The University of

Ontario

CS 433/653 Algorithms for Image Analysis Segmentation (2D)

Blobs• Need for blobs• Extracting blobs

– Thresholding, region growing, mean-shift– Minimum spanning trees

Object extraction• Intelligent scissors (also known as live-wire)• Snakes• Region+boundary methods

Local, greedy, and global processing

Extra Reading: Sonka at.al. Ch. 5 Gonzalez and Woods, Ch. 10

The University of

OntarioHW assignment 1

2D image segmentation• Live-wire (based on Dijkstra algorithm for shortest

paths on a graph)• Implement and experiment on two images• Bonus for comparing with local/greedy methods• Some useful code libraries and sample images will

be posted within a couple of days

The University of

OntarioGoal: Extract “Blobs”

What are “blobs”?• Regions of an image that are somehow coherent

Why?• Object extraction, object removal, compositing, etc.• …but are “blobs” objects?• No, not in general

The University of

OntarioBlob’s coherence

Simplest way to define blob coherence is as similarity in brightness or color:

The tools become blobsThe house, grass, and sky make different blobs

The University of

OntarioWhy is this useful?

AIBO RoboSoccer(VelosoLab)

The University of

OntarioIdeal Segmentation

The University of

OntarioResult of Segmentation

The University of

OntarioThresholding

Basic segmentation operation:mask(x,y) = 1 if im(x,y) > T mask(x,y) = 0 if im(x,y) < T

T is threshold• User-defined• Or automatic

Same as histogram partitioning:

The University of

OntarioSometimes works well…

What are potentialProblems?

The University of

Ontario…but more often not

Adaptive thresholding

The University of

OntarioRegion growing

Is this same as global threshold?

• Start with initial set of pixels K (initial seed(s))• Add to pixels p in K their neighbors q if |Ip-Iq| < T • Repeat until nothing changes

The University of

OntarioRegion growing

The University of

Ontario

What can go wrongwith region growing ?

Region growth may “leak” through a single week spot

in the boundary

The University of

OntarioRegion growing

See region leaks into sky due to a weak boundary

between them

The University of

Ontario

Due to Pedro Felzenszwalb and Daniel Huttenlocher

Motivating example

This image has three perceptually distinct regions

Difference along border between A and B is less then differences

within C

A CB Q: Where would image thresholding fail?

Q: Where would region growing fail?

A: Region A would be divided in two sets and region C will be split into a large number of arbitrary small subsets

A: Either A and B are merged or region C is split into many small subsets Also, B and C are merged

The University of

OntarioColor-Based Blob Segmentation

Automatic Histogram Partitioning• Given image with N colors, choose K• Each of the K colors defines a region

– not necessarily contiguous

• Performed by computing color histogram, looking for modes

• This is what happens when you downsample image color range, for instance in Photoshop

The University of

OntarioFinding Modes in a Histogram

How Many Modes Are There?• Easy to see, hard to compute

The University of

OntarioMean Shift [Comaniciu & Meer]

1. Initialize random seed, and fixed window

2. Calculate center of gravity ‘x’ of the window (the“mean”)

3. Translate the search window to the mean

4. Repeat Step 2 until convergence

Iter

ativ

e M

ode

Sea

rch

x

o

x x

mode

The University of

OntarioMean-Shift

The University of

OntarioMean-shift results

More Examples: http://www.caip.rutgers.edu/~comanici/segm_images.html

The University of

OntarioIssues:

Although often useful, all these approaches work only some of the time, and are considered rather “hacky”.

Can’t even handle our tiger:

Problem is that blobs != objects!

The University of

OntarioExtracting objects

How could this be done?

The University of

OntarioExtracting objects

Many approaches proposed• color cues• region cues• contour cues

We will consider a few of these Today:

• Intelligent Scissors (contour-based)– E. N. Mortensen and W. A. Barrett, Intelligent Scissors for Image Composition, in

ACM Computer Graphics (SIGGRAPH `95), pp. 191-198, 1995

The University of

OntarioIntelligent Scissors

The University of

OntarioIntelligent Scissors

Approach answers a basic question• Q: how to find a path from seed to mouse that follows object boundary

as closely as possible?• A: define a path that stays as close as possible to edges

The University of

OntarioIntelligent Scissors Basic Idea

• Define edge score for each pixel– edge pixels have low cost

• Find lowest cost path from seed to mouse

seed

mouse

Questions• How to define costs?• How to find the path?

The University of

OntarioPath Search (basic idea) Graph Search Algorithm

• Computes minimum cost path from seed to all other pixels

Note: diagonal “paths” are scaled by a factor .

2 … Why?

The University of

Ontario

Good segmentation method shouldhave Invariance to Image Rotation

After object rotation

L

L

Path’s cost along the top boundary Path’s cost along the top-left boundary

147*2 105*2 After diagonal links are adjusted by2

145*2*2

2 2 2 2 2 2 2

2

2

2

2

2

image gradient scores

The University of

Ontario

Instead of nodes, image gradient scores can be assigned directly to graph edges

Graph• node for every pixel p• link between every adjacent pair of pixels, p,q• cost c for each link

Note: each link has a cost• this is a little different than the figure before

where each pixel (graph node) had a cost

p

qc

Treat the image as a graph

The University of

OntarioDefining the costs

Treat the image as a graph

Want to hug image edges: how to define cost of a link?

p

qc

• the link should follow the intensity edge– want intensity to change rapidly to the link

• c - |difference of intensity to the link|

T

T

The University of

OntarioDefining the costs

p

qc

c can be computed using a cross-correlation filter• assume it is centered at p

Also typically scale c by its length• set c = (max-|filter response|)

– where max = maximum |filter response| over all pixels in the image

The University of

OntarioDefining the costs

p

qc 1

-1w-1-1

11

c can be computed using a cross-correlation filter• assume it is centered at p

Also typically scale c by its length• set c = (max-|filter response|)

– where max = maximum |filter response| over all pixels in the image

Why no scalar?2

The University of

Ontario

(see Cormen et.al. “Introduction to Algorithms”, p.595) Dijkstra’s shortest path algorithm

0531

33

4 9

2

link cost

1. init node costs to , set p = seed point, cost(p) = 02. expand p as follows:

for each of p’s neighbors q that are not expanded• set cost(q) = min( cost(p) + cpq, cost(q) )

ALG

OR

ITH

M

The University of

OntarioDijkstra’s shortest path algorithm

4

1 0

5

3

3 2 3

9

531

33

4 9

2

11

1. init node costs to , set p = seed point, cost(p) = 02. expand p as follows:

for each of p’s neighbors q that are not expanded• set cost(q) = min( cost(p) + cpq, cost(q) )

– if q’s cost changed, make q point back to p• put q on the ACTIVE list (if not already there)

ALG

OR

ITH

M

The University of

OntarioDijkstra’s shortest path algorithm

4

1 0

5

3

3 2 3

9

531

33

4 9

2

15

233

3 2

4

1. init node costs to , set p = seed point, cost(p) = 02. expand p as follows:

for each of p’s neighbors q that are not expanded• set cost(q) = min( cost(p) + cpq, cost(q) )

– if q’s cost changed, make q point back to p• put q on the ACTIVE list (if not already there)

3. set r = node with minimum cost on the ACTIVE list4. repeat Step 2 for p = r

ALG

OR

ITH

M

The University of

OntarioDijkstra’s shortest path algorithm

3

1 0

5

3

3 2 3

6

531

33

4 9

2

4

3 1

4

52

33

3 2

4

1. init node costs to , set p = seed point, cost(p) = 02. expand p as follows:

for each of p’s neighbors q that are not expanded• set cost(q) = min( cost(p) + cpq, cost(q) )

– if q’s cost changed, make q point back to p• put q on the ACTIVE list (if not already there)

3. set r = node with minimum cost on the ACTIVE list4. repeat Step 2 for p = r

ALG

OR

ITH

M

The University of

OntarioDijkstra’s shortest path algorithm

3

1 0

5

3

3 2 3

6

531

33

4 9

2

4

3 1

4

52

33

3 2

4

2

1. init node costs to , set p = seed point, cost(p) = 02. expand p as follows:

for each of p’s neighbors q that are not expanded• set cost(q) = min( cost(p) + cpq, cost(q) )

– if q’s cost changed, make q point back to p• put q on the ACTIVE list (if not already there)

3. set r = node with minimum cost on the ACTIVE list4. repeat Step 2 for p = r

ALG

OR

ITH

M

The University of

OntarioPath Search (basic idea)

A

B

Dijkstra algorithm

- processed nodes (distance to A is known)- active nodes (front)

- active node with the smallest distance value

The University of

OntarioDijkstra’s shortest path algorithm

Properties• It computes the minimum cost path from the seed to every

node in the graph. This set of minimum paths is represented as a tree

• Running time, with N pixels:– O(N2) time if you use an active list– O(N log N) if you use an active priority queue (heap)– takes < second for a typical (640x480) image

• Once this tree is computed once, we can extract the optimal path from any point to the seed in O(N) time.

– it runs in real time as the mouse moves

• What happens when the user specifies a new seed?

The University of

OntarioLivewire extensions

Directed graphs Restricted search space

• Restricted domain (e.g. near a priori model)• Restricted backward search

Different edge weight functions• Image-Edge strength• Image-Edge Curvature • Proximity to known approximate model/boundary

Multi-resolution processing

The University of

OntarioResults

http://www.cs.washington.edu/education/courses/455/03wi/projects/project1/artifacts/index.html