the virtual frontier - nvidia · computer graphics challenges in virtual reality. 3 nvidia research...
TRANSCRIPT
![Page 1: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/1.jpg)
Dr. Morgan McGuire | NVIDIA Research
THE VIRTUAL FRONTIER:COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY
![Page 2: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/2.jpg)
3
NVIDIA RESEARCH120 World-Class Ph.D. Researchers
![Page 3: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/3.jpg)
4
VISION
Today, everyone is a high-performance computer user, with
GPUs in phones, tablets, desktops, game consoles, and cars
1977 HPC 2017 HPC1997 HPC
![Page 4: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/4.jpg)
5
Power User Technology Pervasive
FUTURE VR
VISION
![Page 5: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/5.jpg)
6
1. Virtual reality will be the new interface to computing for everyone
2. Virtual reality requires a new graphics systemsensors, algorithms, physics, rendering, AI, data structures, processors, optics, and displays
![Page 6: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/6.jpg)
7
MODERN GRAPHICS SYSTEMS
![Page 7: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/7.jpg)
8
VISUAL FIDELITY OF FILM CGI
Deadpool (Marvel)
![Page 8: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/8.jpg)
9
VISUAL FIDELITY OF FILM CGI
Deadpool (Marvel)
![Page 9: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/9.jpg)
10
FILM CGI: CONCEPT TO PHOTONS
Path TracerScreen24 Hz9 Mpix
Composite &Color Grade
(Hours)
ScriptCharactersCostumes
Sets
Direction
Renderer DisplayPreproduction
PerformanceModelingRigging
AnimationTexturingLighting
Simulation
Production
ParticlesTriangles
Sub-D
Primitives
![Page 10: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/10.jpg)
12
ProductionPreproduction Renderer Display
3D GAME SYSTEM
Display30Hz2 MPix
PerformanceModelingRigging
AnimationTexturingLighting
Simulation
ScriptCharactersCostumes
Sets
RasterizationShadowMaps
AO ShadePost FX*
User Input
Direction
* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing
Primitives
ParticlesTriangles
HDMI, Sync
AINetwork
Simulation
![Page 11: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/11.jpg)
13
3D GAME SYSTEM
Display30Hz2 MPix
Direction
PerformanceModelingRigging
AnimationTexturingLighting
Simulation
ScriptCharactersCostumes
Sets
User Input
* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing
ProductionPreproduction Renderer Display
ParticlesTriangles
Primitives
RasterizationShadowMaps
AO Shade PostFX*HDMI, Sync
AINetwork
Simulation
![Page 12: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/12.jpg)
14
Primitives Renderer Display
3D GAME SYSTEM
Display30Hz2 MPix
ParticlesTriangles Rasterization
ShadowMaps
AO Shade Post FX*
AINetwork
Simulation
User Input
HDMI, Sync
* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing
![Page 13: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/13.jpg)
15
Primitives Renderer Display
3D GAME SYSTEM
Display30Hz2 MPix
ParticlesTriangles Rasterization
ShadowMaps
AO Shade Post FX*
AINetwork
Simulation
User Input
HDMI, Sync
* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing
![Page 14: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/14.jpg)
16Star Wars: Battlefront II (DICE)
![Page 15: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/15.jpg)
1717 Forza Motorsport 6 (Turn 10 Studios)
![Page 16: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/16.jpg)
18
7X THROUGHPUT INCREASE
3D GAME = 60 MPIX/S(1920 X 1080 @ MIN 30 FPS)
MODERN VR = 450 MPIX/S(3024 X 1680* @ MIN 90 FPS)
1920
1080
1512
1680
1512
* VR render resolution
![Page 17: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/17.jpg)
19
Primitives Renderer Display
3D GAME SYSTEM
Display30Hz2 MPix
ParticlesTriangles Rasterization
ShadowMaps
AO Shade Post FX*
AINetwork
Simulation
User Input
HDMI, Sync
* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing
![Page 18: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/18.jpg)
20
Display90 Hz5 MPix
ParticlesTriangles Rasterization
ShadowMaps
AO Shade Post FX*
AINetwork
Simulation
User Input
HDMI, Sync
* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing
3D GAME SYSTEM
Primitives Renderer Display
![Page 19: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/19.jpg)
21
Primitives Renderer
Head Tracking
HMD
ParticlesTriangles Rasterization
ShadowMaps
Time Warp
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
MODERN VR SYSTEM
Display90 Hz5 MPix
![Page 20: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/20.jpg)
22
LENS DISTORTION
User’s ViewPredistorted Image Optics
![Page 21: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/21.jpg)
23
Renderer
Head Tracking
HMD
ParticlesTriangles Rasterization
ShadowMaps
Time Warp +Lens Distortion
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
MODERN VR SYSTEM
Display90Hz
Primitives
![Page 22: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/22.jpg)
24
1. Virtual reality will be the new interface to computing for everyone
2. Virtual reality requires a new graphics systemsensors, algorithms, data structures, processors, and displays
3. Pascal architecture upgrades the gaming system to modern VRwarping, lens matched shading, multiprojection, stereo projection, variable resolution
![Page 23: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/23.jpg)
25
FUTURE GRAPHICS SYSTEMS
The remainder of the talk describes active research,including new results not previously presented in public.
These are not products.
![Page 24: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/24.jpg)
26
LIMITS OF HUMAN PERCEPTION
x (120 pixels/degree)2
≈ 400,000,000 pixels= 200 x 1080p TVs
220o Horizontalx 135o Vertical
100,000x to 1Mx beyond modern VR
+ High dynamic range (x2), photorealistic dynamic lighting (x10,000), …
x 240 Hz
Modern VR = 450 Mpix/s
Future VR = 100,000 Mpix/s
Head image from http://jeffsearle.blogspot.com/2015/09/drawing-head-from-different-angles.html
![Page 25: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/25.jpg)
27
Display90Hz
ParticlesTriangles
FoveatedRasterization
ShadowMaps
Time Warp +Lens Distortion
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
FOVEATED RENDERING
Eye Tracking
Primitives Renderer
Head Tracking
HMD
![Page 26: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/26.jpg)
28
FOVEATED RENDERINGConventional Approach: Aliasing
Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016
![Page 27: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/27.jpg)
29
FOVEATED RENDERINGOur Approach: Perceptually Optimized
Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016
![Page 28: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/28.jpg)
31
Primitives
Display90Hz
ParticlesTriangles
FoveatedRasterization
ShadowMaps
Time Warp +Lens Distortion
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
BEYOND TRIANGLES
PointsText
VoxelsLight Fields
Eye Tracking
Head Tracking
Renderer HMD
![Page 29: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/29.jpg)
32McGuire et al., Real-time global illumination with light field probes, I3D 2017
![Page 30: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/30.jpg)
33
Renderer HMDPrimitives
Display90Hz
ParticlesTriangles
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
COMPUTATIONAL DISPLAYS
FoveatedRasterization
ShadowMaps
Time WarpPointsText
VoxelsLight Fields
VarifocalLens Distortion
Light Field
Eye Tracking
Head Tracking
![Page 31: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/31.jpg)
34
COMPUTATIONAL DISPLAYSLight Field Display
GPU Output Display Prototype Observed Image
Lanman and Luebke, Near-Eye Light Field Displays, SIGGRAPH Asia 2013
![Page 32: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/32.jpg)
35
COMPUTATIONAL DISPLAYSVarifocal Optics
Hologram
Akşit et al., Varifocal Virtuality: A Novel Optical Layout for Near-Eye Display, SIGGRAPH 2017 Emerging Technologies
Curved sunglassesLaser
![Page 33: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/33.jpg)
36
COMPUTATIONAL DISPLAYSVarifocal Optics
Dunn et al, Wide field of view varifocal near-eye display using see-through deformable membrane mirrors, Proc. of IEEE VR 2017
![Page 34: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/34.jpg)
37
Head Tracking
ParticlesTriangles
PointsText
2½D VideoVoxels
Light Fields
RendererPrimitives HMD
Display90Hz
Renderer
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
PNEUMATIC HAPTICS
FoveatedRasterization
ShadowMaps
Time WarpVarifocal
Lens Distortion
Haptics
ParticlesTriangles
PointsText
2½D VideoVoxels
Light Fields
Primitives
Light Field
![Page 35: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/35.jpg)
38
Display90Hz
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
LOW LATENCYHierarchical Rendering
FoveatedRasterization
ShadowMaps
Time WarpVarifocal
Lens Distortion
Haptics
Remote GPU
Head Tracking
Light Field
Eye Tracking
ParticlesTriangles
PointsText
2½D VideoVoxels
Light Fields
RendererPrimitives HMD
![Page 36: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/36.jpg)
39
LOW LATENCYHierarchical Rendering
Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015
Wearable TegraLocal GeForce
![Page 37: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/37.jpg)
40
LOW LATENCYHierarchical Rendering
Wearable TegraCloud GRID Platform
Tesla GPULocal GeForce
Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015
![Page 38: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/38.jpg)
41
LOW LATENCYHierarchical Rendering
Wearable Tegra
Compressed lighting data
High speed network
Cloud GRID PlatformTesla GPU
Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015
![Page 39: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/39.jpg)
42
Primitives
Head Tracking
ParticlesTriangles
PointsText
2½D VideoVoxels
Light Fields
Renderer HMD
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
LOW LATENCYBinary Frames
FoveatedRasterization
ShadowMaps
Eye Tracking
Time Warp +Varifocal
Lens Distortion
Haptics
Remote GPU
Deep FocusDisplay
16000 Hz
Light Field
![Page 40: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/40.jpg)
43
ULTRA LOW LATENCY & HIGH THROUGHPUTBinary Frames
Lincoln et al., From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality, IEEE VR 2016
![Page 41: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/41.jpg)
44
LOW LATENCYBinary Frames
Lincoln et al., From Motion to Photons in 80 M icroseconds: Towards M inimal Latency for Virtual and Augmented Reality, IEEE VR 2016
0.08 ms
Motion Initiated
Data Received
Pixel Transmitted
Light Emitted
1
2
3
4
C1 C2 Dly
27.560µs
C1 C3 Dly
74.360µs
C1 C4 Dly
97.312µs
C2 C3 Dly
46.802µs
Ch1
Ch3
2.00 V
2.00 V
BWCh1
Ch3
2.00 VCh2
Ch4 1.00 VΏ BW
M 20.0µs Ch1 √ 1.20 V
![Page 42: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/42.jpg)
45
LOW LATENCYOn-HMD Warping
Hardware Warping Prototype Photographed in HMD Warped Static Point Set
![Page 43: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/43.jpg)
46
ParticlesTriangles
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
RAY & PATH TRACING
FoveatedRasterization
ShadowMaps
Eye Tracking
PointsText
VoxelsLight Fields
Time Warp +Varifocal
Lens Distortion
Haptics
Remote GPU
Deep FocusDisplay16000Hz
Denoising
Light Field
Renderer HMDPrimitives
Head Tracking
![Page 44: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/44.jpg)
47
PATH TRACING10 rays/path
Deadpool (Marvel)
![Page 45: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/45.jpg)
48Deadpool (Marvel)
PATH TRACING10 rays/path
Visualization of path tracing noise
1 path/pixel
![Page 46: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/46.jpg)
49
10 rays/path1000 paths/pixel
![Page 47: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/47.jpg)
50
DENOISING PATH TRACINGNaïve Real-time Result
Chaitanya et al., Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder, SIGGRAPH 2017
Schied et al., Spatiotemporal variance guided filtering: real-time reconstruction for path tracing, High Performance Graphics 2017
Mara et al., An efficient denoising algorithm for global illumination, High Performance Graphics 2017
![Page 48: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/48.jpg)
51
DENOISING PATH TRACINGDenoised Real-Time Result
Chaitanya et al., Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder, SIGGRAPH 2017
Schied et al., Spatiotemporal variance guided filtering: real-time reconstruction for path tracing, High Performance Graphics 2017
Mara et al., An efficient denoising algorithm for global illumination, High Performance Graphics 2017
![Page 49: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/49.jpg)
52
Chaitanya et al., Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder, SIGGRAPH 2017
Schied et al., Spatiotemporal variance guided filtering: real-time reconstruction for path tracing, High Performance Graphics 2017
Mara et al., An efficient denoising algorithm for global illumination, High Performance Graphics 2017
DENOISING PATH TRACING
![Page 50: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/50.jpg)
53NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
AI GRAPHICS NVIDIA RESEARCH
AI Facial Animation
SIGGRAPH 2017
AI Denoising
AI Anti-Aliasing AI Light Transport
![Page 51: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/51.jpg)
54
Display90Hz
ParticlesTriangles Rasterization
ShadowMaps
Time Warp +Lens Distortion
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
MODERN VR SYSTEM
Primitives Renderer HMD
Head Tracking
![Page 52: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/52.jpg)
55
Primitives
ParticlesTriangles
AINetwork
Simulation
User Input and Tracking
HDMI, Sync
FUTURE VR SYSTEM
FoveatedRasterization
ShadowMaps
Eye Tracking
PointsText
VoxelsLight Fields
Time Warp +Varifocal
Lens Distortion
Haptics
Light Field
Remote GPU
Deep FocusDisplay16000Hz
Denoising
Renderer HMD
Head Tracking
![Page 53: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/53.jpg)
56
1. Virtual reality will be the new interface to computing for everyone
2. Pascal architecture upgrades the gaming system to modern VR
GPU warping, lens matched shading, multiprojection, stereo projection, variable resolution
3. NVIDIA is innovating for a revolutionary new future VR system
computational displays, varifocal optics, foveated & cloud rendering, light fields, binary frames, on-display warping, beam racing, haptics, path tracing, denoising
![Page 54: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/54.jpg)
57
http://research.nvidia.com
![Page 55: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/55.jpg)
58
![Page 56: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/56.jpg)
59
MODERN VR EXPERIENCES
Mechanics & DesignThe Climb (Crytek)SUPERHOT (Superhot Team)
Narrative & CharactersThe Labs (Valve)
Simulation & PerformanceNVIDIA VR Funhouse
ContentGoogle Earth VR
+Batman replacing Aperture
![Page 57: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/57.jpg)
60
PROJECTOR-BASED DISPLAY
![Page 58: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/58.jpg)
61The Vanishing of Ethan Carter (The Astronauts)
![Page 59: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/59.jpg)
62
CHALLENGE: FOCUS CUES
Video from Narain et al., Optimal Presentation of Imagery with Focus Cueson Multi-Plane Displays, SIGGRAPH 2015
![Page 60: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/60.jpg)
63
BEYOND TRIANGLESLight Fields
McGuire et al., Real-time global illumination with light field probes, I3D 2017
![Page 61: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/61.jpg)
64
NVIDIA AR/VR RESEARCH
Computational Displays Light field displays and varifocal optics
Foveated Rendering Perceptually-guided rendering for massive throughput
Ultra-Low Latency Hierarchical & binary rendering, beam racing, near-display warp
Beyond Triangles Points, voxels, light fields, and text
Path Tracing Extending ray tracing leadership to cinematic quality rendering
![Page 62: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/62.jpg)
65
NVIDIA VRWORKS SDKSIG
HT
SO
UN
D
PH
YSIC
S &
TO
UC
H
CA
PTU
RE
BRINGING REALITY TO VR
![Page 63: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/63.jpg)
66
NVIDIA HOLODECKHANDS-ON DEMO AT SIGGRAPH
![Page 64: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/64.jpg)
67
NVIDIA PROJECT HOLODECKPH
OTO
REA
LIS
TIC
MO
DELS
CO
LLA
BO
RATIO
N
INTERA
CTIV
E P
HYSIC
S
![Page 65: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/65.jpg)
68
NVIDIA VRWORKS & PASCALAccelerating Modern VR
GRAPHICS HEADSET AUDIOTOUCH & PHYSICS
PROFESSIONAL
VIDEO
![Page 66: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/66.jpg)
69
VRWORKS & PASCALHardware Acceleration for Modern VR
Single-Pass StereoLens Matched Shading & Multiprojection
Preemption for Timewarp
![Page 67: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/67.jpg)
70
VRWORKS & PASCAL
PhysX Flex
Hairworks Flow
VRWorks Audio
Cloth
Hardware Acceleration for Modern VR
![Page 68: THE VIRTUAL FRONTIER - NVIDIA · COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY. 3 NVIDIA RESEARCH 120 World-Class Ph.D. Researchers ... Real-time global illumination with light](https://reader034.vdocuments.mx/reader034/viewer/2022042710/5f5d679f2b1b83389a3e3f89/html5/thumbnails/68.jpg)
71
FOVEATED RENDERING
High-fidelity foveal
pixels
Low-fidelity peripheral
pixels