![Page 1: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/1.jpg)
CS376 Computer VisionLecture 5: Texture
Qixing Huang
Feb. 6th 2019
![Page 2: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/2.jpg)
Today: Texture
What defines a texture?
Slide Credit: Kristen Grauman
![Page 3: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/3.jpg)
Includes: more regular patterns
Slide Credit: Kristen Grauman
![Page 4: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/4.jpg)
Includes: more random patterns
Slide Credit: Kristen Grauman
![Page 5: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/5.jpg)
Scale and texture
Slide Credit: Kristen Grauman
![Page 6: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/6.jpg)
Texture-related tasks
• Shape from texture
– Estimate surface orientation or shape from image texture
Slide Credit: Kristen Grauman
![Page 7: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/7.jpg)
Shape from texture
• Use deformation of texture from point to point to estimate surface shape
Pics from A. Loh: http://www.csse.uwa.edu.au/~angie/phdpics1.html Slide Credit: Kristen Grauman
![Page 8: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/8.jpg)
Analysis vs. Synthesis
Images:Bill Freeman, A. Efros
Why analyze texture?
Slide Credit: Kristen Grauman
![Page 9: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/9.jpg)
Texture-related tasks
• Shape from texture
– Estimate surface orientation or shape from image texture
• Segmentation/classification from texture cues
– Analyze, represent texture
– Group image regions with consistent texture
• Synthesis
– Generate new texture patches/images given some examples
Slide Credit: Kristen Grauman
![Page 10: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/10.jpg)
Slide credit: Kristen Grauman
![Page 11: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/11.jpg)
What kind of response will we get with an edge detector for these images?
Images from Malik and Perona, 1990Slide Credit: Kristen Grauman
![Page 12: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/12.jpg)
…and for this image?Image credit: D. Forsyth
Slide Credit: Kristen Grauman
![Page 13: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/13.jpg)
Why analyze texture?
Importance to perception:
• Often indicative of a material’s properties
• Can be important appearance cue, especially if shape is similar across objects
• Aim to distinguish between shape, boundaries, and texture
Technically:
• Representation-wise, we want a feature one step above “building blocks” of filters, edges.
Slide Credit: Kristen Grauman
![Page 14: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/14.jpg)
Psychophysics of texture
• Some textures distinguishable with preattentive perception–without scrutiny, eye movements [Julesz 1975]
Same or different?
Slide Credit: Kristen Grauman
![Page 15: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/15.jpg)
Slide Credit: Kristen Grauman
![Page 16: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/16.jpg)
Slide Credit: Kristen Grauman
![Page 17: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/17.jpg)
Slide Credit: Kristen Grauman
![Page 18: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/18.jpg)
Capturing the local patterns with image measurements
[Bergen & Adelson, Nature 1988]
Scale of patterns influences discriminability
Size-tuned linear filters
Slide Credit: Kristen Grauman
![Page 19: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/19.jpg)
Texture representation
• Textures are made up of repeated local patterns, so:
– Find the patterns
• Use filters that look like patterns (spots, bars, raw patches…)
• Consider magnitude of response
– Describe their statistics within each local window, e.g.,
• Mean, standard deviation
• Histogram
• Histogram of “prototypical” feature occurrences
Slide Credit: Kristen Grauman
![Page 20: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/20.jpg)
Texture representation: example
original image
derivative filter responses, squared
statistics to summarize patterns in small
windows
mean d/dxvalue
mean d/dyvalue
Win. #1 4 10
…
Slide credit: Kristen Grauman
![Page 21: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/21.jpg)
Texture representation: example
original image
derivative filter responses, squared
statistics to summarize patterns in small
windows
mean d/dxvalue
mean d/dyvalue
Win. #1 4 10
Win.#2 18 7
…
Slide credit: Kristen Grauman
![Page 22: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/22.jpg)
Texture representation: example
original image
derivative filter responses, squared
statistics to summarize patterns in small
windows
mean d/dxvalue
mean d/dyvalue
Win. #1 4 10
Win.#2 18 7
…
Slide credit: Kristen Grauman
![Page 23: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/23.jpg)
Texture representation: example
original image
derivative filter responses, squared
statistics to summarize patterns in small
windows
mean d/dxvalue
mean d/dyvalue
Win. #1 4 10
Win.#2 18 7
Win.#9 20 20
…
…
Slide credit: Kristen Grauman
![Page 24: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/24.jpg)
Texture representation: example
statistics to summarize patterns in small
windows
mean d/dxvalue
mean d/dyvalue
Win. #1 4 10
Win.#2 18 7
Win.#9 20 20
…
…
Dimension 1 (mean d/dx value)
Dim
en
sio
n 2
(m
ean
d/d
y va
lue
)
Slide credit: Kristen Grauman
![Page 25: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/25.jpg)
Texture representation: example
statistics to summarize patterns in small
windows
mean d/dxvalue
mean d/dyvalue
Win. #1 4 10
Win.#2 18 7
Win.#9 20 20
…
…
Dimension 1 (mean d/dx value)
Dim
en
sio
n 2
(m
ean
d/d
y va
lue
)
Windows with small gradient in both directions
Windows with primarily vertical edges
Windows with primarily horizontal edges
Both
Slide credit: Kristen Grauman
![Page 26: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/26.jpg)
Texture representation: example
original image
derivative filter responses, squared
visualization of the assignment to texture
“types”
Slide credit: Kristen Grauman
![Page 27: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/27.jpg)
Texture representation: example
statistics to summarize patterns in small
windows
mean d/dxvalue
mean d/dyvalue
Win. #1 4 10
Win.#2 18 7
Win.#9 20 20
…
…
Dimension 1 (mean d/dx value)
Dim
en
sio
n 2
(m
ean
d/d
y va
lue
)
Far: dissimilar textures
Close: similar textures
Slide credit: Kristen Grauman
![Page 28: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/28.jpg)
Texture representation: example
Dimension 1
Dim
en
sio
n 2
a
b=
−=
−+−=
2
1
2
222
211
)(),(
)()(),(
i
ii babaD
bababaD
Slide credit: Kristen Grauman
![Page 29: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/29.jpg)
Texture representation: example
Dimension 1
Dim
en
sio
n 2
a
b
a
b
Distance reveals how dissimilar
texture from window a is from
texture in window b.
b
Slide credit: Kristen Grauman
![Page 30: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/30.jpg)
Texture representation: window scale
• We’re assuming we know the relevant window size for which we collect these statistics.
Possible to perform scale selection by looking for window scale where texture description not changing.
Slide credit: Kristen Grauman
![Page 31: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/31.jpg)
Filter banks
• Our previous example used two filters, and resulted in a 2-dimensional feature vector to describe texture in a window.
– x and y derivatives revealed something about local structure.
• We can generalize to apply a collection of multiple (d) filters: a “filter bank”
• Then our feature vectors will be d-dimensional.
– still can think of nearness, farness in feature space
Slide Credit: Kristen Grauman
![Page 32: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/32.jpg)
Filter banks
• What filters to put in the bank?
– Typically we want a combination of scales and orientations, different types of patterns.
Matlab code available for these examples: http://www.robots.ox.ac.uk/~vgg/research/texclass/filters.html
scales
orientations
“Edges” “Bars”
“Spots”
Slide Credit: Kristen Grauman
![Page 33: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/33.jpg)
Multivariate Gaussian
=
90
09
=
90
016
=
55
510
Slide credit: Kristen Grauman
![Page 34: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/34.jpg)
Filter bank
Slide credit: Kristen Grauman
![Page 35: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/35.jpg)
Imag
e fr
om
htt
p:/
/ww
w.t
exas
exp
lore
r.co
m/a
ust
inca
p2
.jpg
Slide credit: Kristen Grauman
![Page 36: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/36.jpg)
Showing magnitude of responses
Slide credit: Kristen Grauman
![Page 37: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/37.jpg)
Slide credit: Kristen Grauman
![Page 38: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/38.jpg)
Slide credit: Kristen Grauman
![Page 39: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/39.jpg)
Slide credit: Kristen Grauman
![Page 40: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/40.jpg)
Slide credit: Kristen Grauman
![Page 41: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/41.jpg)
Slide credit: Kristen Grauman
![Page 42: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/42.jpg)
Slide credit: Kristen Grauman
![Page 43: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/43.jpg)
Slide credit: Kristen Grauman
![Page 44: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/44.jpg)
Slide credit: Kristen Grauman
![Page 45: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/45.jpg)
Slide credit: Kristen Grauman
![Page 46: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/46.jpg)
Slide credit: Kristen Grauman
![Page 47: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/47.jpg)
Slide credit: Kristen Grauman
![Page 48: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/48.jpg)
Slide credit: Kristen Grauman
![Page 49: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/49.jpg)
Slide credit: Kristen Grauman
![Page 50: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/50.jpg)
Slide credit: Kristen Grauman
![Page 51: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/51.jpg)
Slide credit: Kristen Grauman
![Page 52: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/52.jpg)
Slide credit: Kristen Grauman
![Page 53: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/53.jpg)
Slide credit: Kristen Grauman
![Page 54: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/54.jpg)
Slide credit: Kristen Grauman
![Page 55: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/55.jpg)
You try: Can you match the texture
to the response?
Mean abs responses
FiltersA
B
C
1
2
3
Slide Credit: Derek Hoiem
![Page 56: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/56.jpg)
Representing texture by mean abs
response
Mean abs responses
Filters
Slide Credit: Derek Hoiem
![Page 57: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/57.jpg)
[r1, r2, …, r38]
We can form a feature vector from the list of responses at each pixel.
Slide credit: Kristen Grauman
![Page 58: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/58.jpg)
d-dimensional features
. . .
2d 3d
=
−=d
i
ii babaD1
2)(),(Euclidean distance (L2)
Slide credit: Kristen Grauman
![Page 59: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/59.jpg)
Example uses of texture in vision:
analysis
![Page 60: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/60.jpg)
Classifying materials, “stuff”
Figure by Varma & Zisserman
Slide Credit: Kristen Grauman
![Page 61: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/61.jpg)
Texture-related tasks
• Shape from texture
– Estimate surface orientation or shape from image texture
• Segmentation/classification from texture cues
– Analyze, represent texture
– Group image regions with consistent texture
• Synthesis
– Generate new texture patches/images given some examples
![Page 62: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/62.jpg)
Texture synthesis
• Goal: create new samples of a given texture
• Many applications: virtual environments, hole-filling, texturing surfaces
![Page 63: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/63.jpg)
The Challenge
• Need to model the whole spectrum: from repeated to stochastic texture
repeated
stochastic
Both?
Alexei A. Efros and Thomas K. Leung, “Texture Synthesis by Non-parametric Sampling,” Proc. International Conference on Computer Vision (ICCV), 1999.
Slide Credit: Kristen Grauman
![Page 64: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/64.jpg)
Markov Random Field
First-order MRF:• probability that pixel X takes a certain value given the values of
neighbors A, B, C, and D:
D
C
X
A
B
Source: S. Seitz
![Page 65: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/65.jpg)
Texture Synthesis [Efros & Leung, ICCV 99]
• Can apply 2D version of text synthesis
Texture corpus (sample)
Output
Slide Credit: Kristen Grauman
![Page 66: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/66.jpg)
Texture synthesis: intuition
We want to insert pixel intensities based on
existing nearby pixel values.
Sample of the texture(“corpus”)
Place we want to insert next
Distribution of a value of a pixel is conditioned on its neighbors alone.
Slide credit: Kristen Grauman
![Page 67: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/67.jpg)
Synthesizing One Pixel
– What is ?
– Find all the windows in the image that match the neighborhood
– To synthesize x• pick one matching window at random
• assign x to be the center pixel of that window
– An exact neighbourhood match might not be present, so find the best matches using SSD error and randomly choose between them, preferring better matches with higher probability
p
input image
synthesized image
Slide from Alyosha Efros, ICCV 1999
![Page 68: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/68.jpg)
Neighborhood Window
input
Slide from Alyosha Efros, ICCV 1999
![Page 69: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/69.jpg)
Varying Window Size
Increasing window size
Slide from Alyosha Efros, ICCV 1999
![Page 70: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/70.jpg)
Synthesis results
french canvas rafia weave
Slide from Alyosha Efros, ICCV 1999
![Page 71: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/71.jpg)
white bread brick wall
Synthesis results
Slide from Alyosha Efros, ICCV 1999
![Page 72: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/72.jpg)
p
Image Quilting [Efros & Freeman 2001]
• Observation: neighbor pixels are highly correlated
Input image
non-parametric
sampling
B
Idea: unit of synthesis = block
• Exactly the same but now we want P(B|N(B))
• Much faster: synthesize all pixels in a block at once
Synthesizing a block
Slide from Alyosha Efros, ICCV 1999
![Page 73: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/73.jpg)
Input texture
B1 B2
Random placement
of blocks
block
B1 B2
Neighboring blocks
constrained by overlap
B1 B2
Minimal error
boundary cut
![Page 74: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/74.jpg)
min. error boundary
Minimal error boundary
overlapping blocks vertical boundary
_ =
2
overlap error
Slide from Alyosha Efros
![Page 75: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/75.jpg)
Slide Credit: Kristen Grauman
![Page 76: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/76.jpg)
Slide Credit: Kristen Grauman
![Page 77: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/77.jpg)
(Manual) texture synthesis in the media
Slide Credit: Kristen Grauman
![Page 78: CS376 Computer Vision Lecture 5: Texturehuangqx/CS376_Lecture_5.pdf · texture in window b. Slide credit: Kristen Grauman. Texture representation: window scale •We’re assuming](https://reader033.vdocuments.mx/reader033/viewer/2022060605/6059e62659844473fd495d13/html5/thumbnails/78.jpg)
(Manual) texture synthesis
in the media
Slide Credit: Kristen Grauman