3d material style transfer
DESCRIPTION
3D Material Style Transfer. Chuong H. Nguyen 1 , Tobias Ritschel 2 , Karol Myszkowski 1 , Elmar Eisemann 2 , Hans-Peter Seidel 1 1 Max-Planck-Institut Informatik 2 Telecom ParisTech / CNRS . Background – Material Perception. [ Ce Liu et al. 2010]. Background - Mood. Background. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/1.jpg)
Eurographics 2012, Cagliari, Italy
3D Material Style TransferChuong H. Nguyen1, Tobias Ritschel2, Karol Myszkowski1, Elmar Eisemann2, Hans-Peter Seidel1
1Max-Planck-Institut Informatik 2Telecom ParisTech / CNRS
![Page 2: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/2.jpg)
Eurographics 2012, Cagliari, Italy
Background – Material Perception
[Ce Liu et al. 2010]
![Page 3: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/3.jpg)
Eurographics 2012, Cagliari, Italy
Background - Mood
![Page 4: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/4.jpg)
Eurographics 2012, Cagliari, Italy
Background
![Page 5: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/5.jpg)
Eurographics 2012, Cagliari, Italy
Motivation
Scene without MaterialsGuide Image Scene with
Materials
![Page 6: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/6.jpg)
Eurographics 2012, Cagliari, Italy
Previous Works - Colorization
Color TransferReinhard et al: Color Transfer between Images, IEEE CG&A 2001
ColorizationWelsh et al: Transfering Color to Greyscale Images, Siggraph 2001Irony et al: Colorization by Examples, EGSR 2005
![Page 7: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/7.jpg)
Eurographics 2012, Cagliari, Italy
Previous Works – Detail Synthesis
Improving the RealismJohnson et al: CG2Real: Improving the Realism of Computer Generated Images using a Large Collection of Photographs, IEEE TVCG 2010
AnalogyHertzmann et al: Image Analogies, Siggraph 2001Mertens et al: Texture Transfer Using Geometry Correlation, EGSR 2007
![Page 8: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/8.jpg)
Eurographics 2012, Cagliari, Italy
Previous Work – Materials
Image-Based Material EditingKhan et al: Image-Based Material Editing , Siggraph 2006
Material PerceptionVangorp et al: The Influence of Shape on the Perception of Material Reflectance , Siggraph 2007
![Page 9: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/9.jpg)
Eurographics 2012, Cagliari, Italy
Problem statement
Guide Source3D Scene
Objects Materials
![Page 10: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/10.jpg)
Eurographics 2012, Cagliari, Italy
Material Extraction
Guide Source3D Scene
Objects Materials
![Page 11: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/11.jpg)
Eurographics 2012, Cagliari, Italy
Material ExtractionOriginal
Diffuse
Specular
1st K-Mean Merging 2nd K-Mean
Extracted MaterialsTexture
Diffuse and Specular ExtractionYang et al: Real-time Specular Highlight Removal Using Bilateral Filtering, ECCV, 2010
![Page 12: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/12.jpg)
Eurographics 2012, Cagliari, Italy
12
Material Extraction Results
![Page 13: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/13.jpg)
Eurographics 2012, Cagliari, Italy
Material Assignment
Guide Source3D Scene
Objects Materials
![Page 14: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/14.jpg)
Eurographics 2012, Cagliari, Italy
Material Assignment Goal
Guide Source
Objects Materials
1.
2.
(Image Cost)
(Geometry Cost)
![Page 15: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/15.jpg)
Eurographics 2012, Cagliari, Italy
Material Assignment Goal – Image Cost
Guide Source
Objects Materials
1.
2.
(Image Cost)
(Geometry Cost)
![Page 16: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/16.jpg)
Eurographics 2012, Cagliari, Italy
Material Assignment - Image CostGuide Source3D Scene
∑ ϴ
![Page 17: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/17.jpg)
Eurographics 2012, Cagliari, Italy
Material Assignment Goal – Geometry Cost
Guide Source
Objects Materials
1.
2.
(Image Cost)
(Geometry Cost)
![Page 18: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/18.jpg)
Eurographics 2012, Cagliari, Italy
3D Scene
Material Assignment - Geometry CostGuide Source
![Page 19: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/19.jpg)
Eurographics 2012, Cagliari, Italy
Geometry Cost – Shape Descriptor2D Angular Radial Transform (ART)Bober et al: MPEG-7 visual shape descriptors, IEEE Circuits and Sys. Vid. Tech. 11, 6 (2001)
35 coefs (B)
35 coefs (A)35 coefs (C)
35 coefs (G)
35 coefs (H) 35 coefs(E) 35 coefs (D)
35 coefs (F)
35 coefs (O)
ϴ ¿min {‖𝐴−𝑂‖,‖𝐵−𝑂‖,‖𝐶−𝑂‖,.. } ,…
![Page 20: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/20.jpg)
Eurographics 2012, Cagliari, Italy
Cost Function = Image Cost + Geometry Cost
ϴ
ϴ
ϴ
+
…
ϴ
ϴ
ϴ
…
![Page 21: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/21.jpg)
Eurographics 2012, Cagliari, Italy
Material Assignment
Guide Source3D Scene
Objects Materials
![Page 22: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/22.jpg)
Eurographics 2012, Cagliari, Italy
Optimization – Simulated Annealingϴ
ϴ
ϴ
+
…
ϴ
ϴ
ϴ
…
ϴ
ϴ
ϴ
+
…
ϴ
ϴ
ϴ
…
otherwise
assignment ith
assignment i+1th
![Page 23: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/23.jpg)
Eurographics 2012, Cagliari, Italy
Results: Pebbles
![Page 24: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/24.jpg)
Eurographics 2012, Cagliari, Italy
Results: Temple
![Page 25: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/25.jpg)
Eurographics 2012, Cagliari, Italy
Results: Kitchen
![Page 26: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/26.jpg)
Eurographics 2012, Cagliari, Italy
Results: Bedroom
![Page 27: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/27.jpg)
Eurographics 2012, Cagliari, Italy
Perceptual Study – Step 1
![Page 28: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/28.jpg)
Eurographics 2012, Cagliari, Italy
Perceptual Study – Step 2 60% best
![Page 29: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/29.jpg)
Eurographics 2012, Cagliari, Italy
Limitations• Plausible material extraction• Image segmentation errors• Fix lighting
![Page 30: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/30.jpg)
Eurographics 2012, Cagliari, Italy
Conclusion and Future Work
• Conclusion– Plausible material extraction from images– An automatic system for 3D material style transfer
• Future work– Material database for different objects– Better object recognition techniques
![Page 31: 3D Material Style Transfer](https://reader035.vdocuments.mx/reader035/viewer/2022081420/56816385550346895dd46d4e/html5/thumbnails/31.jpg)
Eurographics 2012, Cagliari, Italy
Thank you!