texture synthesis and image analogies - computer...
TRANSCRIPT
![Page 1: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/1.jpg)
Texture synthesis and image analogies
15-463, 15-663, 15-862Computational Photography
Fall 2017, Lecture 9http://graphics.cs.cmu.edu/courses/15-463
![Page 2: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/2.jpg)
Course announcements
• Please take Doodle for second make-up lecture, link on Piazza.
• Homework 3 is out.- Due October 12th.- Shorter, but longer bonus component.
![Page 3: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/3.jpg)
Overview of today’s lecture
• Reminder: non-local means.
• Texture synthesis.
• Texture by non-parametric sampling.
• Image quilting.
• Inpainting.
• Texture transfer.
• Image analogies.
• Deep learning teaser.
![Page 4: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/4.jpg)
Slide credits
Most of these slides were adapted from:
• Kris Kitani (15-463, Fall 2016).
Some slides were inspired or taken from:
• Fredo Durand (MIT).• James Hays (Georgia Tech).
![Page 5: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/5.jpg)
Reminder: non-local means
![Page 6: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/6.jpg)
Redundancy in natural images
![Page 7: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/7.jpg)
Non-local means
𝑤(𝑝, 𝑞)
𝑤(𝑝, 𝑟)𝑤(𝑝, 𝑠)
ො𝑥 𝑖 =1
𝐶𝑖
𝑗
𝑦(𝑗) 𝑒−𝑆𝑆𝐷 y 𝑁𝑖 −y 𝑁𝑗
2𝜎2
𝑤 𝑖, 𝑗
No need to stop at neighborhood. Instead search everywhere in the image.
![Page 8: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/8.jpg)
Last couple of classes: adding things to the image
![Page 9: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/9.jpg)
This class: removing things from the image
![Page 10: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/10.jpg)
This class: removing things from the image
![Page 11: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/11.jpg)
Texture synthesis
![Page 12: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/12.jpg)
• Depicts spatially repeating patterns• Appears naturally and frequently
Texture
![Page 13: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/13.jpg)
• Large variety of textures
Texture
![Page 14: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/14.jpg)
Goal: create new samples of a given texture.Applications:• hole filling• virtual environments• view expansion• texturing surfaces• ….
Texture synthesis
![Page 15: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/15.jpg)
Input
How would you do texture synthesis for this sample?
![Page 16: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/16.jpg)
tiling random
Input
How would you do texture synthesis for this sample?
![Page 17: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/17.jpg)
Basic idea:• Compute statistics of input texture (e.g., histogram of edge filter responses).• Generate a new texture that keeps these same statistics.
Heeger and Bergen, “Pyramid-based texture analysis/synthesis,” SIGGRAPH 1995Simoncelli and Portilla, “Texture characterization via joint statistics of wavelet coefficient magnitudes,” ICIP 1998
Approach 1: probabilistic modeling
![Page 18: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/18.jpg)
Probability distributions are hard to model well.
Any other ideas?
Approach 1: probabilistic modeling
input
output
![Page 19: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/19.jpg)
Texture by non-parametric sampling
![Page 20: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/20.jpg)
Efros and Leung, “Texture synthesis by non-parametric sampling,” ICCV 1999
What are sampling from?
N(p)
Neighborhood of p
Run template matching, get N best matches, and sample one at random.
Approach 2: sample from the image
![Page 21: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/21.jpg)
Efros and Leung, “Texture synthesis by non-parametric sampling,” ICCV 1999
N(p)
Neighborhood of p
Run template matching, get N best matches, and sample one at random.
Approach 2: sample from the image
• Similar nearby images define a non-parametric PDF• By selecting a random sample, we are sampling from this PDF
![Page 22: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/22.jpg)
How do you define patch similarity?
Implementation details
![Page 23: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/23.jpg)
How do you define patch similarity?• Gaussian-weighted SSD (emphasis on nearby pixels).
In what order should you synthesize?
Implementation details
![Page 24: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/24.jpg)
How do you define patch similarity?• Gaussian-weighted SSD (emphasis on nearby pixels).
In what order should you synthesize?• Onion-peel ordering – pixels with most neighbors are synthesized first.
How do you synthesize from scratch?
Implementation details
![Page 25: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/25.jpg)
How do you define patch similarity?• Gaussian-weighted SSD (emphasis on nearby pixels).
In what order should you synthesize?• Onion-peel ordering – pixels with most neighbors are synthesized first.
How do you synthesize from scratch?• Pick a small patch at random from source.
Implementation details
![Page 26: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/26.jpg)
“I spent an interesting evening recently with a grain of salt.”Claude Elwood Shannon
(1916–2001)
• Generate English-sounding sentences by modeling the probability of each word given the previous words (n-grams)
• Large “n” will give more structured sentences
Ideas from information theory
![Page 27: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/27.jpg)
input
Size of neighborhood window matters a lot
![Page 28: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/28.jpg)
patch size
Size of neighborhood window matters a lot
![Page 29: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/29.jpg)
While image not filled
1.Get unfilled pixels with filled neighbors
2.Sort by number of filled neighbor
3.For each pixel
a)Get top N matches of visible neighbor
(Patch Distance: Gaussian-weighted SSD)
b)Randomly select one of the matches
c)Copy pixel value
Texture synthesis algorithm
![Page 30: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/30.jpg)
French canvas rafia weave
Examples
![Page 31: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/31.jpg)
white bread brick wall
Examples
![Page 32: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/32.jpg)
Homage to Shannon
![Page 33: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/33.jpg)
Hole filling
![Page 34: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/34.jpg)
Image extrapolation
![Page 35: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/35.jpg)
Texture synthesis using non-parametric sampling:
• Very simple
• Surprisingly good results
• Synthesis is easier than analysis!
• But very slow
Summary
Why is it so slow and how could we make it faster?
![Page 36: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/36.jpg)
Image quilting
![Page 37: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/37.jpg)
Observation: neighboring pixels are highly correlated.
Idea: Instead of single pixels, synthesize entire blocks
• Exactly analogous procedure as before, except we now sample P(B | N(B))
• Much faster since we synthesize all pixels in a block at once
p
non-parametricsampling
input image
B
synthesizing a block
Efros and Freeman, “Image Quilting for Texture Synthesis and Transfer,” SIGGRAPH 2001
Summary
![Page 38: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/38.jpg)
input texture
random placement of blocks
block
B1 B2B1 B2
Dealing with boundaries
neighboring blocksconstrained by overlap
![Page 39: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/39.jpg)
input texture
random placement of blocks
block
B1 B2B1 B2
Dealing with boundaries
B1 B2
neighboring blocksconstrained by overlap
minimal errorboundary cut
How can we achieve this?
![Page 40: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/40.jpg)
minimum error boundary
overlapping blocks vertical boundary
_ =
2
overlap error
Dealing with boundaries
How can we compute this boundary efficiently?
![Page 41: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/41.jpg)
Examples
![Page 42: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/42.jpg)
Examples
![Page 43: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/43.jpg)
Examples
![Page 44: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/44.jpg)
Examples
![Page 45: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/45.jpg)
Examples
![Page 46: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/46.jpg)
46
Examples
![Page 47: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/47.jpg)
Failure case (Chernobyl tomatoes)
![Page 48: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/48.jpg)
Portilla & Simoncelli
Wei & Levoy Quilting
Xu, Guo & Shum
input image
Examples
![Page 49: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/49.jpg)
Portilla & Simoncelli
Wei & Levoy Quilting
Xu, Guo & Shum
input image
Examples
![Page 50: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/50.jpg)
It even made the news
![Page 51: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/51.jpg)
Inpainting
![Page 52: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/52.jpg)
Criminisi et al., “Object removal by exemplar-based inpainting,” CVPR 2003
Inpainting natural scenes
![Page 53: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/53.jpg)
image with hole raster-scan order
Toy inpainting example:
onion-peel
Any ideas on how to do better filling?
Key idea: Filling order matters
![Page 54: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/54.jpg)
image with hole raster-scan order
Toy inpainting example:
onion-peel gradient-sensitive order
Key idea: Filling order matters
Gradient-sensitive order: Fill a pixel that
• is surrounded by other known pixels; and
• is a continuation of a strong gradient or edge.
![Page 55: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/55.jpg)
original with hole onion-peel fill gradient-sensitive
Examples
![Page 56: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/56.jpg)
onion-peel gradient-sensitive
Examples
![Page 57: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/57.jpg)
Texture transfer
![Page 58: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/58.jpg)
Try to explain one object with bits and pieces of another object
+ =+
Efros and Freeman, “Image Quilting for Texture Synthesis and Transfer,” SIGGRAPH 2001
Texture transfer
How would you do this?
![Page 59: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/59.jpg)
Texture transferSame as texture synthesis, except search for texture blocks by comparing with target image patches (“constraints”)
constraint
texture sample
constraint
texture sample
![Page 60: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/60.jpg)
Some less creepy examples
![Page 61: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/61.jpg)
=+
Some less creepy (?) examples
![Page 62: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/62.jpg)
Some less creepy examples
![Page 63: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/63.jpg)
Image analogies
![Page 64: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/64.jpg)
A A’
B B’
Hertzmann et al., “Image analogies, ” SIGGRAPH 2001
Image analogiesWhy stop at textures?
given pair of image analogies
input image synthesized image
![Page 65: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/65.jpg)
Image analogies
![Page 66: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/66.jpg)
A A’
B B’
How would you do this?
![Page 67: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/67.jpg)
Define a similarity between A and B
For each patch in B:
1.Find a matching patch in A, whose
corresponding A’ also fits in
well with existing patches in B’
2.Copy the patch in A’ to B’
Implementation:
A A’
B B’
How would you do this?
Algorithm is run iteratively (coarse-to-fine)
![Page 68: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/68.jpg)
Blurring by analogies
unfiltered source (A) filtered source (A’)
unfiltered target (B) filtered target (B’)
![Page 69: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/69.jpg)
Edges by analogies
unfiltered source (A) filtered source (A’)
unfiltered target (B) filtered target (B’)
![Page 70: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/70.jpg)
Artistic filters
unfiltered source (A) filtered source (A’)
unfiltered target (B) filtered target (B’)
![Page 71: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/71.jpg)
Colorization
unfiltered source (A) filtered source (A’)
unfiltered target (B) filtered target (B’)
![Page 72: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/72.jpg)
“Texture by numbers”
unfiltered source (A) filtered source (A’)
unfiltered target (B) filtered target (B’)
![Page 73: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/73.jpg)
“Texture by numbers”
![Page 74: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/74.jpg)
Super-resolution
unfiltered source (A) filtered source (A’)
unfiltered target (B) filtered target (B’)
![Page 75: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/75.jpg)
A A’
Super-resolution
unfiltered target (B) filtered target (B’)
![Page 76: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/76.jpg)
Deep learning teaser
![Page 77: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/77.jpg)
Step 1: forward pass input image
Step 2: define loss wrt forward pass responses
Step 3: update white noise image according to gradient descent
A return to parametric models
![Page 78: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/78.jpg)
Texture synthesis examples
![Page 79: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/79.jpg)
Texture synthesis examples
![Page 80: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/80.jpg)
Texture synthesis examples
![Page 81: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/81.jpg)
Texture synthesis examples
![Page 82: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/82.jpg)
Texture synthesis examples
![Page 83: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/83.jpg)
Texture synthesis examples
![Page 84: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/84.jpg)
Parameter number matters
![Page 85: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/85.jpg)
Style transfer examples
![Page 86: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/86.jpg)
![Page 87: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/87.jpg)
![Page 88: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/88.jpg)
![Page 89: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/89.jpg)
![Page 90: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/90.jpg)
![Page 91: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/91.jpg)
![Page 92: Texture synthesis and image analogies - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/... · 2017. 10. 3. · • Szeliski textbook, Section 10.5. • Efros](https://reader034.vdocuments.mx/reader034/viewer/2022052015/602d3a9fea6db313946b17f7/html5/thumbnails/92.jpg)
References
Basic reading:• Szeliski textbook, Section 10.5.• Efros and Leung, “Texture Synthesis by Non-parametric Sampling,” ICCV 1999.• Efros and Freeman, “Image Quilting for Texture Synthesis and Transfer,” SIGGRAPH 2001.• Hertzmann et al., “Image analogies,” SIGGRAPH 2001.• Criminisi et al., “Object removal by exemplar-based inpainting,” CVPR 2003.
the titles of the above four papers should be self-explanatory.
Additional reading:• Gatys et al., “Texture Synthesis Using Convolutional Neural Networks,” NIPS 2015.
texture synthesis using deep learning.• Gatys et al., “Image Style Transfer Using Convolutional Neural Networks,” CVPR 2016.
implementing image analogies using deep learning.• Luan et al., “Deep Photo Style Transfer,” arXiv 2017.
implementing photo-realistic style transfer using deep learning.