from pixels to “blobs” - computer...
TRANSCRIPT
![Page 1: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/1.jpg)
1
From Pixels to “Blobs”
15-463: Rendering and Image Processing
Alexei Efros
Today
Blobs
• Need for blobs
• Extracting blobs
• Image Segmentation
Working with binary images
• Mathematical Morphology
• Blob properties
Further Reading:
• Gonzalez and Woods, Ch. 9 & 10
![Page 2: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/2.jpg)
2
Goal: Extract “Blobs”
What are “blobs”?• Regions of an image that are somehow coherent
Why?• Object extraction, object removal, compositing, etc.
• …but are “blobs” objects?
• No, not in general
Blob’s coherence
Simplest way to define blob coherence is as similarity in
brightness or color:
The tools become blobsThe house, grass, and sky make
different blobs
![Page 3: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/3.jpg)
3
The meaning of a blob
Other interpretations of blobs are possible, depending
on how you define the input image:
• Image can be a response of a particular detector
– Color Detector
– Face detector
– Motion Detector
– Edge Detector
gx2+gy2
Why is this useful?
AIBO
RoboSoccer
(VelosoLab)
![Page 4: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/4.jpg)
4
Ideal Segmentation
Result of Segmentation
![Page 5: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/5.jpg)
5
Thresholding
Basic segmentation operation:
mask(x,y) = 1 if im(x,y) > T
mask(x,y) = 0 if im(x,y) < T
T is threshold
• User-defined
• Or automatic
Same as
histogram
partitioning:
As Edge Detection
gx2+gy2 gx2+gy2 > T
![Page 6: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/6.jpg)
6
Sometimes works well…
What are potential
Problems?
…but more often not
Adaptive thresholding
![Page 7: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/7.jpg)
7
Region growing
• Start with initial set of pixels K
• Add to K any neighbors, if they are within similarity threshold
• Repeat until nothing changes
Is this same as global threshold?
What can go wrong?
Color-Based Blob Segmentation
Automatic Histogram Partitioning
• Given image with N colors, choose K
• Each of the K colors defines a region
– not necessarily contiguous
• Performed by computing color histogram, looking for modes
• This is what happens when you downsample image color range, for
instance in Photoshop
![Page 8: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/8.jpg)
8
Finding Modes in a Histogram
How Many Modes Are There?
• Easy to see, hard to compute
Mean Shift [Comaniciu & Meer]
Iterative Mode Search1. Initialize random seed, and fixed window
2. Calculate center of gravity of the window (the “mean”)
3. Translate the search window to the mean
4. Repeat Step 2 until convergence
![Page 9: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/9.jpg)
9
Mean-Shift
Mean-shift results
More Examples: http://www.caip.rutgers.edu/~comanici/segm_images.html
![Page 10: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/10.jpg)
10
Issues:
Although often useful, all these approaches work only
some of the time, and are considered rather “hacky”.
Can’t even handle our tiger:
Problem is that blobs != objects!
Image Segmentation
Detour into Computer Vision…
"I stand at the window and see a house, trees, sky. Theoretically I might say there were 327 brightnessesand nuances of colour. Do I have "327"? No. I have sky,
house, and trees." --Max Wertheimer
From Pixels to Objects, not mere “blobs”…
What Defines an Object?• Subjective problem, but has been well-studied
• Gestalt Laws seek to formalize this:
– proximity, similarity, continuation, closure, common fate
![Page 11: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/11.jpg)
11
Region-Based Segmentation
We Want Regions
• why not build this in as a constraint?
q
Images as Graphs [Shi & Malik]
Graph G = (V, E, W)
• node for every pixel
• edge between every pair of pixels, p,q
• weight wpq for each edge
– wpq measures similarity
» similarity: difference in color and position (or other things)
p
wpq
w
![Page 12: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/12.jpg)
12
Segmentation by cutting a Graph
Break Graph into Segments
• Delete edges that cross between segments
• Easiest to break edges that have low weight:
– similar pixels should be in the same segments
– dissimilar pixels should be in different segments
w
A B C
Cuts in a graph
Edge Cut• set of edges whose removal makes a graph disconnected
• cost of a cut:
AB
Normalized Cut• a min cut penalizes large segments
• fix by normalizing for size of segments
Vol(A) Vol(B)
![Page 13: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/13.jpg)
13
The Normalized Cut (NCut) criterion
NP-Hard!
Given a Graph G = (V, E, W)• Find A in V that minimizes
Normalize Cut in Matrix Form
),(
),( ;
11)1(
)1)(()1(
11
)1)(()1(
)B(
)BA,(
)A(
B)A,(B)A,(
0
∑
∑ >=
−
−−−+
+−+=
+=
i
x
T
T
T
T
ii
iik
k
xx
k
xx
Vol
cut
Vol
cutNcut
i
D
D
D
WD
D
WD
.1)(,}1,1{ in vector a is
);,(),( :i node from weightsof sum theis
;),( :matrix weight theis ,
Aii
jiii
wji
N
j
ji
∈⇔=−
=
=
∑
xx
WDD
WW
![Page 14: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/14.jpg)
14
Eigenvalue Problem
After lot’s of math, we get:
.01},,1{ with ,),( =−∈−
= DyyDyy
W)y(DyT
TT
i bBANcut
This is a Rayleigh Quotient• Solution given by “generalized” eigenvalue problem:
• Solved by converting to standard eigenvalue problem:
Subtleties• optimal solution is second smallest eigenvector
• gives real result—must convert into discrete values of y
DyW)y(D λ=−
yDzzzW)D(DD 2
1
2
1
2
1
==−−−
whereλ ,
Interpretation as a Dynamical System
Weights are Springs
• eigenvectors correspond to
vibration modes
Movie by Serge Belongie
![Page 15: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/15.jpg)
15
Interpretation as a Dynamical System
Weights are Springs
• eigenvectors correspond to
vibration modes
Movie by Serge Belongie
Segmentation result
![Page 16: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/16.jpg)
16
Color Image Segmentation
balus_movie.mpg
![Page 17: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/17.jpg)
17
Binary Image Processing
The result of all these operations is a binary mask
Binary images are handy in many cases (sprite
extraction, compositing, etc).
Binary image processing is a well-studied field, based on set theory, called Mathematical Morphology
Preliminaries
![Page 18: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/18.jpg)
18
Preliminaries
Preliminaries
![Page 19: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/19.jpg)
19
Basic Concepts in Set Theory
A is a set in , a=(a1,a2) an element of A, a∈A
If not, then a∉A
∅: null (empty) set
Typical set specification: C={w|w=-d, for d ∈ D}
A subset of B: A⊆B
Union of A and B: C=A∪B
Intersection of A and B: D=A∩B
Disjoint sets: A∩B= ∅
Complement of A:
Difference of A and B: A-B={w|w ∈ A, w ∉ B}=
Z2
Ac = {w | w ∉ A}
A ∩ Bc
Preliminaries
} ,|{)(
} ,|{ˆ
AaforzaccA
BbforbwwB
z ∈+==
∈−==
![Page 20: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/20.jpg)
20
Dilation and Erosion
Two basic operations:• A is the image, B is the “structural element”, a mask akin to a kernel
in convolution
Dilation :
(all shifts of B that have a non-empty overlap with A)
Erosion :
(all shifts of B that are fully contained within A)
}])[(|{
})(|{
AABzBA
ABzBA
z
z
⊆∩=⊕
≠∩=⊕)
)φ
})(|{ ABzBA z ⊆=Θ
Dilation
![Page 21: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/21.jpg)
21
Dilation
Erosion
![Page 22: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/22.jpg)
22
Erosion
Original image Eroded image
Erosion
Eroded once Eroded twice
![Page 23: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/23.jpg)
23
Opening and Closing
Opening : smoothes the contour of an object, breaks narrow
isthmuses, and eliminates thin protrusions
Closing : smooth sections of contours but, as opposed to opning, it
generally fuses narrow breaks and long thin gulfs, eliminates
small holes, and fills gaps in the contour
Prove to yourself that they are not the same thing. Play around
with bwmorph in Matlab.
BBABA ⊕Θ= )(o
BBABA Θ⊕=• )(
OPENING: The
original image
eroded twice and
dilated twice
(opened). Most
noise is removed
Opening and Closing
CLOSING: The
original image
dilated and then
eroded. Most
holes are filled.
![Page 24: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/24.jpg)
24
Opening and Closing
Boundary Extraction
)()( BAAA Θ−=β
![Page 25: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/25.jpg)
25
Boundary Extraction
Region Filling
....3,2,1 )( 1 =⊕= − kABXXc
kk I
![Page 26: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/26.jpg)
26
Extraction of Connected Components
,....3,2,1 )( 1 =⊕= − kABXX kk I
First Step : Run Length Encoding
Segment each image row into groups of similar pixels
called runs
• Runs store a start and end point for each contiguous row of
color
Original image RLE image
![Page 27: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/27.jpg)
27
Second Step : Merging Regions
Final Results
Runs are merged into multi-row regions
Image is now described as contiguous regions instead of just pixels
![Page 28: From Pixels to “Blobs” - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2004_fall/www/Lectures/BlobProcessing.pdfFrom Pixels to “Blobs” ... Closing : smooth sections of](https://reader035.vdocuments.mx/reader035/viewer/2022070922/5fbad70c7143bc64ea436d0a/html5/thumbnails/28.jpg)
28
Blob Properties
Now that we have nice, clean blobs, what can we do
with them?
Compute Statistics:
• Area
• Perimeter
• Aspect ratio
• Center of mass
• best-fitting ellipse
• Average color
• Etc.
All this can be used to classify blobs and decide if they
hold the objects we are interested in.