neural renderingintrotodeeplearning.com/2020/slides/6s191_mit_deep... · 2021. 1. 5. ·...

Neural Rendering

Chuan Li

Lambda Labs

Collaborators: Thu Nguyen-Phuoc, Bing Xu, Yongliang Yang, Stephen Balaban, Lucas Theis, Christian Richardt, Junfei Zhang, Rui Wang, Kun Xu, Rui Tang

Model Pictures

Forward (Computer Graphics)

Model Pictures

Forward (Computer Graphics)

Inverse (Computer Vision)

Integral of the incident radians

BRDF

32K SPP Ray Tracing (90 mins 12 CPU Cores)The Tungsten Renderer

P0

P1

P0

P1

R01 | T

01

P0

P1

Inverse (Computer Vision)

R01 | T

01

P0

P1

P2

R12 | T12

Inverse (Computer Vision)

R01 | T

01

P0

P1

Building Rome in a DaySameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz and Richard Szeliski

Model Pictures

Sub-module

End-2-End

Differentiable Rendering

1 SPP

2048 SPP

Sub-modules

Mastering the game of Go with deep neural networks and tree searchDavid Silver et al.

Sub-modules

Value Network

Mastering the game of Go with deep neural networks and tree searchDavid Silver et al.

Sub-modules

Value Network

Policy Network

Mastering the game of Go with deep neural networks and tree searchDavid Silver et al.

2^15 SPP4 SPP

Value Networks

Denoising

2^15 SPP

Value Networks

Denoising

Policy Networks

Same SPP

4 SPP

2^15 SPP

Value Networks

Denoising

Policy Networks

Same SPP

4 SPP

4 SPP Denoised1 sec 2080 Ti

32K SPP Ray Tracing90 mins 12 cores CPU

Adversarial Monte Carlo denoising with conditioned auxiliary feature modulationB Xu et al. Siggraph Asia 2019

loss

DecoderEncoderInput x

Ref

Output

Adversarial Monte Carlo denoising with conditioned auxiliary feature modulationB Xu et al. Siggraph Asia 2019

L1 VGG Loss

L1 VGG Loss + GAN

Adversarial Monte Carlo denoising with conditioned auxiliary feature modulationB Xu et al. Siggraph Asia 2019

loss

DecoderEncoderDiffuseInput x Diffuse

Output

DecoderEncoderSpecular

Input x SpecularOutput Ref

Output

Adversarial Monte Carlo denoising with conditioned auxiliary feature modulationB Xu et al. Siggraph Asia 2019

Auxiliary

loss

DecoderEncoderDiffuseInput x Diffuse

Output

DecoderEncoderSpecular

Input x SpecularOutput Ref

Output

Albedo, normal, depth

Auxiliary

Conv

LeakyReLU

Conv

x

Element-wiseBiasing

Conv

Auxiliary

LeakyReLU

Conv

Conv

LeakyReLU

Conv

x

Element-wiseBiasing

Element-wiseScaling

Conv

Auxiliary

LeakyReLU

Conv

Conv

LeakyReLU

Conv

x

Element-wiseBiasing (OR)

Element-wiseScaling (AND)

Denoise comparison 4 SPP

Adversarial Monte Carlo denoising with conditioned auxiliary feature modulationB Xu et al. Siggraph Asia 2019

2^15 SPP

Value Networks

Denoising

Policy Networks

Same SPP

4 SPP

Neural Importance SamplingThomas Müller et al. ACM Transactions on Graphics 2019

incidence radiance map

Neural Importance SamplingThomas Müller et al. ACM Transactions on Graphics 2019

Neural Importance SamplingThomas Müller et al. ACM Transactions on Graphics 2019

Neural Importance SamplingThomas Müller et al. ACM Transactions on Graphics 2019

Model Pictures

Sub-module

End-2-End

Differentiable Rendering

Ray TracingImage Centric

RasterizationObject Centric

Ray TracingImage Centric

RasterizationObject Centric

Visibility

Ray TracingImage Centric

RasterizationObject Centric

Shading

Depth Map Voxel Point Cloud Mesh

Memory Good Very Poor Poor Very Good

NN friendly Great Yes No Enemy

Depth Map Voxel Point Cloud Mesh

Memory Good Very Poor Poor Very Good

NN friendly Great Yes No Enemy

Depth Map Voxel Point Cloud Mesh

Memory Good Very Poor Poor Very Good

NN friendly Great Yes No Enemy

Depth Map Voxel Point Cloud Mesh

Memory Good Very Poor Poor Very Good

NN friendly Great Yes No Enemy

RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018

Neural Voxels

32 x 32 x 32 x 16

3D Encoder

Neural Voxels

RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018

Neural Voxels

32 x 32 x 32 x 16

3D Encoder

3D-2D

32 x 32 x 512

Neural Pixels

VisibilityNeural Voxels

RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018

Neural Voxels

32 x 32 x 32 x 16

3D Encoder

3D-2D

32 x 32 x 512

Neural Pixels

VisibilityNeural Voxels

RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018

Neural Voxels

32 x 32 x 32 x 16

3D Encoder

3D-2D

32 x 32 x 512

Neural Pixels

2D Decoder

ShadingNeural Voxels Visibility

MSE pixel loss

RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018

RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018

Contour

Toon

Ambient OcclusionRenderNet: A deep convolutional network for differentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018

RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018

RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018

3D Encoder

3D-2D Neural Pixels

2D Decoder

TextureNetwork

NeuralTexture Voxels

or

Neural Voxels

Channel-wise Concatenation

64 x 64 x 64 x 4

64 x 64 x 64 x 1

RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018

RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018

Same shape, different textures

Same texture, different shapes

RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018

Depth Map Voxel Point Cloud Mesh

Memory Good Very Poor Poor Very Good

NN friendly Great Yes No Enemy

Rasterization a RGB point cloud

Neural Point-Based GraphicsKA Aliev et al, arxiv 2019

Rasterization a neural point cloud

(First three PCA dimensions of the neural descriptor)

Neural Point-Based GraphicsKA Aliev et al, arxiv 2019

Rasterization a neural point cloud

(First three PCA dimensions of the neural descriptor)

Neural Point-Based GraphicsKA Aliev et al, arxiv 2019

Neural Point-Based GraphicsKA Aliev et al, arxiv 2019

RB

G ra

ster

izat

ion

Neu

ral r

aste

rizat

ion

Neural 3D Mesh RendererH Kato et al, CVPR 2018

Deferred Neural Rendering: Image Synthesis using Neural Textures

J Thies et al, Siggraph 2019

Model Pictures

Sub-module

End-2-End

?

TargetApproximation

TargetApproximation RenderedApproximation

Loss

Back-propagate

TargetRenderedApproximation

Approximation

Loss

TargetRenderedApproximation

UpdatedApproximation

Back-propagate

Loss

TargetRenderedApproximation

UpdatedApproximation

Back-propagateFor Free

Loss

TargetRenderedApproximation

UpdatedApproximation

Back-propagate

Expensive

Loss

TargetRenderedApproximation

DecoderEncoder

Reconstruction Rendering

Human perception imposes coordinate frame on objects

Inductive Bias: Separate Appearance from Pose

Learning 3D representation from natural images without 3D supervision

HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019

Conditional GANs

HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019

Info GANs

Conditional GANs

HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019

HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019

RenderNet3D Generator

HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019

RenderNet3D Generator

3D StyleGAN

HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019

RenderNet3D Generator

3D StyleGAN

HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019

RenderNet3D Generator

3D StyleGAN

HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019

RenderNet3D Generator

HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019

RenderNet3D Generator

Real/Fake

HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019

RenderNet3D Generator

A representation that is unbreakable under 3D rigid-body transformations

HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019

HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019

HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019

HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019

z1 z2

Shape Controller Texture Controller

HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019

HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019

Model Pictures

Forward (Computer Graphics)

Inverse (Computer Vision)

Model Pictures

Sub-module for Ray Tracing (Value / Policy Networks)

End-2-End Rasterization (Depthmap, Voxel, Point Cloud, Mesh)

Differentiable Rendering (Representation Learning)

Thu Nguyen-Phuoc Bing Xu Yongliang Yang Stephen Balaban

Lucas Theis Christian Richardt Junfei Zhang Rui Wang Kun Xu Rui Tang