interactive brdf estimation for mixed-reality applications
DESCRIPTION
Interactive BRDF Estimation for Mixed-Reality Applications. Martin Knecht, Georg Tanzmeister, Christoph Traxler, Michael Wimmer Institute of Computer Graphics and Algorithms Vienna University of Technology. Motivation. Goal of our mixed reality framework - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/1.jpg)
Interactive BRDF Estimation for Mixed-Reality Applications
Martin Knecht, Georg Tanzmeister, Christoph Traxler, Michael Wimmer
Institute of Computer Graphics and Algorithms
Vienna University of Technology
![Page 2: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/2.jpg)
Motivation
Goal of our mixed reality frameworkLight interaction between real and virtual objects
Materials of real objects must be known
Martin Knecht 2
![Page 3: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/3.jpg)
Problem Statement
Material estimation should not need any preprocessing
Use Kinect sensor and fish-eye lense camera for data acquisition
Should run at interactive framerates
Use GPU wherever possible
Should estimate Phong parameters Used in mixed reality framework
3Martin Knecht
![Page 4: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/4.jpg)
Similar to pipeline of Zheng et al. 2009
Estimation Pipeline
4Martin Knecht
![Page 5: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/5.jpg)
Input Data for Estimation
5Martin Knecht
![Page 6: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/6.jpg)
Highlight Removal
6Martin Knecht
![Page 7: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/7.jpg)
Highlight Removal
7Martin Knecht
![Page 8: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/8.jpg)
Diffuse Reflectance Estimation
8Martin Knecht
![Page 9: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/9.jpg)
Diffuse Reflectance Estimation
9Martin Knecht
Inverse shading:
![Page 10: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/10.jpg)
Clustering
10Martin Knecht
![Page 11: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/11.jpg)
Clustering 1/5
Assumption: similar color same material
Same material same specular parameters
Clustering executed on the diffuse estimation
Novel hybrid CPU/GPU K-Means1) Initialize cluster centers
2) Assign pixel to nearest cluster center
3) Calculate new cluster centers
4) Repeat steps 2 & 3
11Martin Knecht
![Page 12: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/12.jpg)
Clustering 2/5
1) Initialize cluster centersRandom cluster centers
Exploit temporal coherenceReuse of cluster centers of previous frame
12Martin Knecht
![Page 13: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/13.jpg)
Clustering 3/5
2) Assign pixel to nearest cluster center
13Martin Knecht
Cluster ShaderRGBC1
RGBC2
...
...
Cluster 1 Cluster 2 Cluster 6 Bitmask 1 Bitmask 2...
![Page 14: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/14.jpg)
Clustering 4/5
3) Calculate new cluster centers
1x1 Mipmap is the average over all pixel
New cluster center:
14Martin Knecht
1x1 RGBD
1x1 Bitmask
TRGBD
T*
![Page 15: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/15.jpg)
Clustering 5/5
4) Repeat steps 2 & 3
Repeat until no pixel changes cluster Standard stopping criteria too conservative
Max. 20 iterations
Check variance change of distances
15Martin Knecht
![Page 16: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/16.jpg)
Clustering 5/5
16Martin Knecht
![Page 17: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/17.jpg)
Specular Reflectance Estimation
17Martin Knecht
![Page 18: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/18.jpg)
Specular Reflectance Estimation
Done on a per cluster basis same material
CPU based nonlinear function solver
Variables: Specular parameter
Light positions
Evaluation of objective function done on GPUSimilar mipmap method used as for clustering
18Martin Knecht
![Page 19: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/19.jpg)
Specular Reflectance Estimation
19Martin Knecht
![Page 20: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/20.jpg)
Results - Estimation
20Martin Knecht
Diffuse component
Phong shaded image
Specular component+
![Page 21: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/21.jpg)
Results – Timings
BRDF estimation runs at ~2.8 fps
Two tasks with major impact K-Means clustering
Specular estimation
Martin Knecht 21
Normal estimation 0,57 msHighlight removal 0,94 msDiffuse estimation 0,23 msK-Means 39,08 msSpecular estimation 315,76 ms
< 0.5 %
~ 11 %
~ 88,5 %
![Page 22: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/22.jpg)
Differential Instant Radiosity
22Martin Knecht
![Page 23: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/23.jpg)
Results – Mixed Reality Integration
23Martin Knecht
![Page 24: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/24.jpg)
Limitations
Kinect sensor does not work everywhere
Bright objects are discarded from estimation
Shadows are not considered
No estimation of optimal amount of clusters
No integration of data over time
Simplifications lower quality of estimation
24Martin Knecht
![Page 25: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/25.jpg)
Conclusion & Future work
BRDF estimation without any preprocessing
Hybrid CPU/GPU K-Means implementation
Runs at interactive framerates
Future workImprove speed specular estimation
Improve quality BRDF estimation
Exploit temporal coherence more often
25Martin Knecht
![Page 26: Interactive BRDF Estimation for Mixed-Reality Applications](https://reader035.vdocuments.mx/reader035/viewer/2022062322/568143f1550346895db079b1/html5/thumbnails/26.jpg)
Thank you for your attention!
Supported by grand from the FFG-Austrian Research Promotion Agency under the Program “FIT-IT Visual
Computing” Project Nr.: 820916
26