the virtual frontier - nvidia · computer graphics challenges in virtual reality. 3 nvidia research...

Dr. Morgan McGuire | NVIDIA Research

THE VIRTUAL FRONTIER:COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY

3

NVIDIA RESEARCH120 World-Class Ph.D. Researchers

4

VISION

Today, everyone is a high-performance computer user, with

GPUs in phones, tablets, desktops, game consoles, and cars

1977 HPC 2017 HPC1997 HPC

5

Power User Technology Pervasive

FUTURE VR

VISION

6

1. Virtual reality will be the new interface to computing for everyone

2. Virtual reality requires a new graphics systemsensors, algorithms, physics, rendering, AI, data structures, processors, optics, and displays

7

MODERN GRAPHICS SYSTEMS

8

VISUAL FIDELITY OF FILM CGI

Deadpool (Marvel)

9

VISUAL FIDELITY OF FILM CGI

Deadpool (Marvel)

10

FILM CGI: CONCEPT TO PHOTONS

Path TracerScreen24 Hz9 Mpix

Composite &Color Grade

(Hours)

ScriptCharactersCostumes

Sets

Direction

Renderer DisplayPreproduction

PerformanceModelingRigging

AnimationTexturingLighting

Simulation

Production

ParticlesTriangles

Sub-D

Primitives

12

ProductionPreproduction Renderer Display

3D GAME SYSTEM

Display30Hz2 MPix

PerformanceModelingRigging

AnimationTexturingLighting

Simulation

ScriptCharactersCostumes

Sets

RasterizationShadowMaps

AO ShadePost FX*

User Input

Direction

* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing

Primitives

ParticlesTriangles

HDMI, Sync

AINetwork

Simulation

13

3D GAME SYSTEM

Display30Hz2 MPix

Direction

PerformanceModelingRigging

AnimationTexturingLighting

Simulation

ScriptCharactersCostumes

Sets

User Input

* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing

ProductionPreproduction Renderer Display

ParticlesTriangles

Primitives

RasterizationShadowMaps

AO Shade PostFX*HDMI, Sync

AINetwork

Simulation

14

Primitives Renderer Display

3D GAME SYSTEM

Display30Hz2 MPix

ParticlesTriangles Rasterization

ShadowMaps

AO Shade Post FX*

AINetwork

Simulation

User Input

HDMI, Sync

* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing

15

Primitives Renderer Display

3D GAME SYSTEM

Display30Hz2 MPix

ParticlesTriangles Rasterization

ShadowMaps

AO Shade Post FX*

AINetwork

Simulation

User Input

HDMI, Sync

* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing

16Star Wars: Battlefront II (DICE)

1717 Forza Motorsport 6 (Turn 10 Studios)

18

7X THROUGHPUT INCREASE

3D GAME = 60 MPIX/S(1920 X 1080 @ MIN 30 FPS)

MODERN VR = 450 MPIX/S(3024 X 1680* @ MIN 90 FPS)

1920

1080

1512

1680

1512

* VR render resolution

19

Primitives Renderer Display

3D GAME SYSTEM

Display30Hz2 MPix

ParticlesTriangles Rasterization

ShadowMaps

AO Shade Post FX*

AINetwork

Simulation

User Input

HDMI, Sync

* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing

20

Display90 Hz5 MPix

ParticlesTriangles Rasterization

ShadowMaps

AO Shade Post FX*

AINetwork

Simulation

User Input

HDMI, Sync

* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing

3D GAME SYSTEM

Primitives Renderer Display

21

Primitives Renderer

Head Tracking

HMD

ParticlesTriangles Rasterization

ShadowMaps

Time Warp

AINetwork

Simulation

User Input and Tracking

HDMI, Sync

MODERN VR SYSTEM

Display90 Hz5 MPix

22

LENS DISTORTION

User’s ViewPredistorted Image Optics

23

Renderer

Head Tracking

HMD

ParticlesTriangles Rasterization

ShadowMaps

Time Warp +Lens Distortion

AINetwork

Simulation

User Input and Tracking

HDMI, Sync

MODERN VR SYSTEM

Display90Hz

Primitives

24

1. Virtual reality will be the new interface to computing for everyone

2. Virtual reality requires a new graphics systemsensors, algorithms, data structures, processors, and displays

3. Pascal architecture upgrades the gaming system to modern VRwarping, lens matched shading, multiprojection, stereo projection, variable resolution

25

FUTURE GRAPHICS SYSTEMS

The remainder of the talk describes active research,including new results not previously presented in public.

These are not products.

26

LIMITS OF HUMAN PERCEPTION

x (120 pixels/degree)2

≈ 400,000,000 pixels= 200 x 1080p TVs

220o Horizontalx 135o Vertical

100,000x to 1Mx beyond modern VR

+ High dynamic range (x2), photorealistic dynamic lighting (x10,000), …

x 240 Hz

Modern VR = 450 Mpix/s

Future VR = 100,000 Mpix/s

Head image from http://jeffsearle.blogspot.com/2015/09/drawing-head-from-different-angles.html

27

Display90Hz

ParticlesTriangles

FoveatedRasterization

ShadowMaps

Time Warp +Lens Distortion

AINetwork

Simulation

User Input and Tracking

HDMI, Sync

FOVEATED RENDERING

Eye Tracking

Primitives Renderer

Head Tracking

HMD

28

FOVEATED RENDERINGConventional Approach: Aliasing

Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016

29

FOVEATED RENDERINGOur Approach: Perceptually Optimized

Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016

31

Primitives

Display90Hz

ParticlesTriangles

FoveatedRasterization

ShadowMaps

Time Warp +Lens Distortion

AINetwork

Simulation

User Input and Tracking

HDMI, Sync

BEYOND TRIANGLES

PointsText

VoxelsLight Fields

Eye Tracking

Head Tracking

Renderer HMD

32McGuire et al., Real-time global illumination with light field probes, I3D 2017

33

Renderer HMDPrimitives

Display90Hz

ParticlesTriangles

AINetwork

Simulation

User Input and Tracking

HDMI, Sync

COMPUTATIONAL DISPLAYS

FoveatedRasterization

ShadowMaps

Time WarpPointsText

VoxelsLight Fields

VarifocalLens Distortion

Light Field

Eye Tracking

Head Tracking

34

COMPUTATIONAL DISPLAYSLight Field Display

GPU Output Display Prototype Observed Image

Lanman and Luebke, Near-Eye Light Field Displays, SIGGRAPH Asia 2013

35

COMPUTATIONAL DISPLAYSVarifocal Optics

Hologram

Akşit et al., Varifocal Virtuality: A Novel Optical Layout for Near-Eye Display, SIGGRAPH 2017 Emerging Technologies

Curved sunglassesLaser

36

COMPUTATIONAL DISPLAYSVarifocal Optics

Dunn et al, Wide field of view varifocal near-eye display using see-through deformable membrane mirrors, Proc. of IEEE VR 2017

37

Head Tracking

ParticlesTriangles

PointsText

2½D VideoVoxels

Light Fields

RendererPrimitives HMD

Display90Hz

Renderer

AINetwork

Simulation

User Input and Tracking

HDMI, Sync

PNEUMATIC HAPTICS

FoveatedRasterization

ShadowMaps

Time WarpVarifocal

Lens Distortion

Haptics

ParticlesTriangles

PointsText

2½D VideoVoxels

Light Fields

Primitives

Light Field

38

Display90Hz

AINetwork

Simulation

User Input and Tracking

HDMI, Sync

LOW LATENCYHierarchical Rendering

FoveatedRasterization

ShadowMaps

Time WarpVarifocal

Lens Distortion

Haptics

Remote GPU

Head Tracking

Light Field

Eye Tracking

ParticlesTriangles

PointsText

2½D VideoVoxels

Light Fields

RendererPrimitives HMD

39

LOW LATENCYHierarchical Rendering

Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015

Wearable TegraLocal GeForce

40

LOW LATENCYHierarchical Rendering

Wearable TegraCloud GRID Platform

Tesla GPULocal GeForce

Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015

41

LOW LATENCYHierarchical Rendering

Wearable Tegra

Compressed lighting data

High speed network

Cloud GRID PlatformTesla GPU

Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015

42

Primitives

Head Tracking

ParticlesTriangles

PointsText

2½D VideoVoxels

Light Fields

Renderer HMD

AINetwork

Simulation

User Input and Tracking

HDMI, Sync

LOW LATENCYBinary Frames

FoveatedRasterization

ShadowMaps

Eye Tracking

Time Warp +Varifocal

Lens Distortion

Haptics

Remote GPU

Deep FocusDisplay

16000 Hz

Light Field

43

ULTRA LOW LATENCY & HIGH THROUGHPUTBinary Frames

Lincoln et al., From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality, IEEE VR 2016

44

LOW LATENCYBinary Frames

Lincoln et al., From Motion to Photons in 80 M icroseconds: Towards M inimal Latency for Virtual and Augmented Reality, IEEE VR 2016

0.08 ms

Motion Initiated

Data Received

Pixel Transmitted

Light Emitted

1

2

3

4

C1 C2 Dly

27.560µs

C1 C3 Dly

74.360µs

C1 C4 Dly

97.312µs

C2 C3 Dly

46.802µs

Ch1

Ch3

2.00 V

2.00 V

BWCh1

Ch3

2.00 VCh2

Ch4 1.00 VΏ BW

M 20.0µs Ch1 √ 1.20 V

45

LOW LATENCYOn-HMD Warping

Hardware Warping Prototype Photographed in HMD Warped Static Point Set

46

ParticlesTriangles

AINetwork

Simulation

User Input and Tracking

HDMI, Sync

RAY & PATH TRACING

FoveatedRasterization

ShadowMaps

Eye Tracking

PointsText

VoxelsLight Fields

Time Warp +Varifocal

Lens Distortion

Haptics

Remote GPU

Deep FocusDisplay16000Hz

Denoising

Light Field

Renderer HMDPrimitives

Head Tracking

47

PATH TRACING10 rays/path

Deadpool (Marvel)

48Deadpool (Marvel)

PATH TRACING10 rays/path

Visualization of path tracing noise

1 path/pixel

49

10 rays/path1000 paths/pixel

50

DENOISING PATH TRACINGNaïve Real-time Result

Chaitanya et al., Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder, SIGGRAPH 2017

Schied et al., Spatiotemporal variance guided filtering: real-time reconstruction for path tracing, High Performance Graphics 2017

Mara et al., An efficient denoising algorithm for global illumination, High Performance Graphics 2017

51

DENOISING PATH TRACINGDenoised Real-Time Result

Chaitanya et al., Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder, SIGGRAPH 2017

Schied et al., Spatiotemporal variance guided filtering: real-time reconstruction for path tracing, High Performance Graphics 2017

Mara et al., An efficient denoising algorithm for global illumination, High Performance Graphics 2017

52

Chaitanya et al., Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder, SIGGRAPH 2017

Schied et al., Spatiotemporal variance guided filtering: real-time reconstruction for path tracing, High Performance Graphics 2017

Mara et al., An efficient denoising algorithm for global illumination, High Performance Graphics 2017

DENOISING PATH TRACING

53NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

AI GRAPHICS NVIDIA RESEARCH

AI Facial Animation

SIGGRAPH 2017

AI Denoising

AI Anti-Aliasing AI Light Transport

54

Display90Hz

ParticlesTriangles Rasterization

ShadowMaps

Time Warp +Lens Distortion

AINetwork

Simulation

User Input and Tracking

HDMI, Sync

MODERN VR SYSTEM

Primitives Renderer HMD

Head Tracking

55

Primitives

ParticlesTriangles

AINetwork

Simulation

User Input and Tracking

HDMI, Sync

FUTURE VR SYSTEM

FoveatedRasterization

ShadowMaps

Eye Tracking

PointsText

VoxelsLight Fields

Time Warp +Varifocal

Lens Distortion

Haptics

Light Field

Remote GPU

Deep FocusDisplay16000Hz

Denoising

Renderer HMD

Head Tracking

56

1. Virtual reality will be the new interface to computing for everyone

2. Pascal architecture upgrades the gaming system to modern VR

GPU warping, lens matched shading, multiprojection, stereo projection, variable resolution

3. NVIDIA is innovating for a revolutionary new future VR system

computational displays, varifocal optics, foveated & cloud rendering, light fields, binary frames, on-display warping, beam racing, haptics, path tracing, denoising

57

http://research.nvidia.com

58

59

MODERN VR EXPERIENCES

Mechanics & DesignThe Climb (Crytek)SUPERHOT (Superhot Team)

Narrative & CharactersThe Labs (Valve)

Simulation & PerformanceNVIDIA VR Funhouse

ContentGoogle Earth VR

+Batman replacing Aperture

60

PROJECTOR-BASED DISPLAY

61The Vanishing of Ethan Carter (The Astronauts)

62

CHALLENGE: FOCUS CUES

Video from Narain et al., Optimal Presentation of Imagery with Focus Cueson Multi-Plane Displays, SIGGRAPH 2015

63

BEYOND TRIANGLESLight Fields

McGuire et al., Real-time global illumination with light field probes, I3D 2017

64

NVIDIA AR/VR RESEARCH

Computational Displays Light field displays and varifocal optics

Foveated Rendering Perceptually-guided rendering for massive throughput

Ultra-Low Latency Hierarchical & binary rendering, beam racing, near-display warp

Beyond Triangles Points, voxels, light fields, and text

Path Tracing Extending ray tracing leadership to cinematic quality rendering

65

NVIDIA VRWORKS SDKSIG

HT

SO

UN

D

PH

YSIC

S &

TO

UC

H

CA

PTU

RE

BRINGING REALITY TO VR

66

NVIDIA HOLODECKHANDS-ON DEMO AT SIGGRAPH

67

NVIDIA PROJECT HOLODECKPH

OTO

REA

LIS

TIC

MO

DELS

CO

LLA

BO

RATIO

N

INTERA

CTIV

E P

HYSIC

S

68

NVIDIA VRWORKS & PASCALAccelerating Modern VR

GRAPHICS HEADSET AUDIOTOUCH & PHYSICS

PROFESSIONAL

VIDEO

69

VRWORKS & PASCALHardware Acceleration for Modern VR

Single-Pass StereoLens Matched Shading & Multiprojection

Preemption for Timewarp

70

VRWORKS & PASCAL

PhysX Flex

Hairworks Flow

VRWorks Audio

Cloth

Hardware Acceleration for Modern VR

71

FOVEATED RENDERING

High-fidelity foveal

pixels

Low-fidelity peripheral

pixels