shader programming: what’s next? -...
TRANSCRIPT
![Page 1: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/1.jpg)
Tim Foley
Shader Programming: What’s Next?
![Page 2: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/2.jpg)
2
An interesting time in graphics programming
Many platforms, and even more APIs
Vulkan 1.0 released just weeks ago
Still programming shaders like it’s 2002
Author shaders in D3D9-era HLSL
Translate to GLSL, Metal, etc.
2
![Page 3: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/3.jpg)
3
Isn’t SPIR-V going to solve all of this?
Vulkan’s intermediate language (IL) for shaders
Documented binary format, consumed by drivers
Intended to support alternative front-ends
May encourage other APIs to follow suit
Not a panacea today
Somebody still has to write all the new front- and back-ends
3
![Page 4: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/4.jpg)
4
Benefits of building your own tools
A shader compiler tailored to your engine
Not a language designed for somebody else’s API
Target a wide range of platforms
Including future rendering pipelines for, e.g., VR
Transform and manipulate shaders
Example: Mechanically generate shader LODs
4
![Page 5: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/5.jpg)
5
This Talk
Two in-progress research projects
Exploring possible approaches to building shader tools
Rapid exploration of scheduling choices
In context of future/extended pipelines
Framework for rapidly building custom shader language/tools
Shaders as a performance-oriented domain-specific language (DSL)
5
![Page 6: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/6.jpg)
6
Rapid Exploration of Shader Scheduling Choices
Collaborators: Yong He, Kayvon Fatahalian (CMU)
![Page 7: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/7.jpg)
7
Pipelines with new shading rates
7
Pixels Fragment
Shader Vertex
Shader
Coarse-Rate
Shader
texture space?
quarter-resolution screen space?
“foveated” rendering?
![Page 8: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/8.jpg)
8
Shading reuse for stereo (VR)
8
Pixels Per-Eye
Shader Vertex
Shader
Eye-Shared
Shader
share diffuse, compute specular per-eye?
![Page 9: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/9.jpg)
9
Pixels Fragment
Shader
Vertex
Shader
Vertex
Baking
Texture
Baking
Define engine-specific pipeline
9
Source
Assets
offline asset processing runtime (GPU) processing
![Page 10: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/10.jpg)
10
Write shaders against that pipeline
10
shader Terrain using MyEnginePipeline { @MeshVertex vec2 uv; @MaterialUniform sampler2D albedoMap1, albedoMap2; @MaterialUniform sampler2D mixMap; float mixFactor = texture(mixMap, uv); vec4 albedo { vec4 c1 = texture(albedoMap1, uv); vec4 c2 = texture(albedoMap2, uv); return mix(c1, c2, mixFactor); } // ... }
![Page 11: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/11.jpg)
11
Shader = Dataflow Graph
11
output
albedo
nDotL
lightDir
normal
mixFactor
mixMap
albedoMap1
albedoMap2
uv
![Page 12: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/12.jpg)
12
Fragment Shader
Vertex
Shader Vertex
Baking
Texture
Baking
Frame Uniform
Mesh Vertex
Material Uniform
Scheduling = Placing Computation in Stages
12
output
albedo
nDotL
lightDir
normal
mixFactor
mixMap
albedoMap1
albedoMap2
uv
![Page 13: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/13.jpg)
13
Vertex
Shader Vertex
Baking
Fragment Shader
Texture Baking
Frame Uniform
Mesh Vertex
Material Uniform
Low LOD? Older GPU? Bake texture offline.
13
output
albedo
nDotL
lightDir
normal
mixFactor mixMap
albedoMap1
albedoMap2
uv
![Page 14: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/14.jpg)
14
Fragment Shader Vertex Shader
Vertex Baking
Texture
Baking
Frame Uniform
Mesh Vertex
Material Uniform
Even lower LOD? Do lighting per-vertex.
14
output
albedo
nDotL
lightDir
normal
mixFactor
mixMap
albedoMap1
albedoMap2
uv
![Page 15: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/15.jpg)
15
What can you do with a representation like this?
Rapidly explore performance/quality trade-offs
Use a UI tool to explore different shader variants
Auto-tune scheduling choices
Meet performance constraints on a particular target
Mechanically apply LOD choices to many shaders
Guarantee that low LODs share same vertex/fragment code
15
![Page 16: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/16.jpg)
16
Video
16
![Page 17: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/17.jpg)
17
Rapid Development of Engine-Specific Shading Tools
Collaborators: Kerry Seitz, John Owens (UC Davis)
![Page 18: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/18.jpg)
18
Building engine-specific shader infrastructure
Front-end language
Integration with other systems
Back-end code generation
![Page 19: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/19.jpg)
19
Shaders as domain-specific languages (DSLs)
Performance-oriented DSLs
Halide (for image processing) is poster child
Terra: framework for easily constructing DSLs
Low-level C-like language
Use Lua scripts to extend language/compiler, generate code
Supports LLVM code generation out of the box
19
![Page 20: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/20.jpg)
20
Building Shader Tools in Terra
Write shader code directly in the Terra language
Can re-use types, functions between app (CPU) and shaders (GPU)
Shader-specific features implemented as syntax extensions
Lua scripts that run inside the compiler
4/5/201
6
20
![Page 21: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/21.jpg)
21
Building Shader Tools in Terra
Built-in support for HLSL/GLSL code generation
Additional tools can parse/inspect/manipulate shaders
Using a Lua library API
Artist UI
Statically-typed engine interface
Asset processing tools
4/5/201
6
21
![Page 22: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/22.jpg)
22
Wrapping Up
![Page 23: Shader Programming: What’s Next? - Nvidiadeveloper.download.nvidia.com/.../tfoley_shader_programming.pdf · 2 An interesting time in graphics programming Many platforms, and even](https://reader031.vdocuments.mx/reader031/viewer/2022021911/5c11b0f409d3f23b288c7cd8/html5/thumbnails/23.jpg)
23
What is the right shader infrastructure for you?
Looking ahead: opportunities and challenges
Don’t expect Khronos, Microsoft, etc. to build all the pieces
Opportunity to build what you really want
Researchers can help accelerate and guide your efforts
Programming models that are ready for new HW and SW pipelines
Frameworks to rapidly build engine-specific compilers and tools
4/5/201
6
23