Advanced Lighting

With Spherical Harmonics

What Is the Algorithm

Similar to Light Propagation

Volumes

Light Propagation Volumes

• From Crytek and Used in Crysis 2 on Pc, Xbox 360, PS3• Uses Reflective Shadow Maps• And Propagation Volumes to

Speed up Rendering

Goal• Global Illumination• The Goal is to have light “Bounce” off of

Surfaces as it does in the Real World• It allows the environment to look much

more realistic• It has the greatest effect in environments

with low ambient light

Instant Radiosity• The father of Reflective Shadow

Mapping• The idea is that lights will

“project” more lights onto other surfaces• Has led to many other

techniques

Instant Radiosity• The “projected” lights will

light other surfaces• The “Bounce” has been

achieved

But• This is incredibly slow• Time to calculate becomes, number of

lights multiplied by the number of projected lights to the power of the number of bounces multiplied by the number of surfaces affected• Plus the position of the projected

lights is determined by ray tracing

Reflective Shadow Mapping

Start by Shadow Mapping

Shadow Mapping• Create a Z-buffer or depth

buffer from the perspective of the light• Using this information you

can determine if a point of a surface is in shadow or not

Reflective Shadow Mapping

• Transform Z-buffer to world space and create lights at the Z-buffer locations• But we need more information• So when we render the initial Z-

buffer we also render colour and surface normal

Reflective Shadow Mapping• Using the colour we can

determine what colour the newly created light will be• Using the surface normal we

can determine what direction the new light should face

Reflective Shadow Mapping• Removes the Overhead of Raytracing

to find new light positions• But• Does not remove the Overhead of

Rendering hundreds or even thousands of lights

• Adds the Overhead of Rendering many Z-buffers

Imperfect Shadow Mapping

Imperfect Shadow Mapping• Same steps as Reflective Shadow

Mapping• But• When we are creating a Reflective

shadow map we create many Z-buffers in many different render targets many of which are of very low resolution

• Low resolution means less info of the geometry is needed

Imperfect Shadow Mapping• Instead render all the small

depth buffers in one large buffer• The graphics card has to change

state less• When rendering the Z-buffers

use Stochastic depth Sampling

Stochastic Sampling• We only need a little information we

can figure the rest out• So we take less depth samples• What we end up with is a depth

texture with “holes” in it• These holes are where we did not

take a sample

Imperfect Shadow Mapping• Essentially tries to remove the

cost of rendering lots of shadow maps• Works well for diffuse surfaces

(Regular)• Does Not work well for glossy

surfaces

Many LODs

Many LODs• A newer technique then

Imperfect shadow mapping• Again similar to reflective

shadow mapping• But uses a different way of

reducing the cost of rendering many depth textures

Many LODs• Both Imperfect shadow mapping

and Reflective shadow mapping use level of detail to reduce the cost of rendering depth textures

• But traditional LOD is good for displaying high quality

• And Bad at displaying low quality

Many LODs• To solve this many LODs uses a different

method• It renders points instead of triangles• The points are rendered from a Bounding

Sphere Hierarchy of the geometry• Think about it the depth texture only cares

about the shape of the object and the depth textures could be as small as 16 * 16 pixels or even smaller

Many LODs• With such a low resolution triangle

rendering becomes inefficient it requires a pixel to be shaded dozens of times more than they need to be

• If a triangle is smaller than the size of a pixel then that pixel is being written to more than four times

• Point rendering will only render that pixel once

Many LODs• Reduces the cost of rendering depth

textures about as much as Imperfect shadow mapping

• But maintains greater quality• Works well displaying diffuse

surfaces• Works well displaying glossy

surfaces

Many LODs• Also is used for rendering lots of

small reflections• And refractions• Both of which also require very

small buffers for which point rendering becomes much more efficient than triangle rendering

Temporal Coherence

Temporal Coherence• Lights don’t move around all that

often• So why do we need to calculate the

positions, colour and direction of the extra lights every frame

• Why not keep information from previous frames

Temporal Coherence• Due to floating point precision

error a slow moving reflective shadow map will “jitter”• Temporal Coherence will stop

that• And lower the average cost of

rendering

Temporal Coherence• Will reduce average cost of

rendering depth textures• Will either increase the memory

requirement or will lower the quality of the result• Will not reduce the cost of

lighting

Temporal Coherence• You can either store results from

previous frames (increasing memory requirements)• Or you can decide to only

calculate the depth textures every X frames (reducing the quality)

But• All of these extra techniques focus

on reducing the cost of calculating the depth textures

• This lowers the Overhead of the creation of the “bounce” lights

• None of these techniques reduce the Overhead of rendering these lights

Light Propagation Volumes

Light Propagation Volumes• Aims to reduce the Overhead of

rendering the lights• Uses reflective shadow mapping to

find locations for the first set of “bounce” lights

• But instead of creating lights add a virtual point light to a volume

A Volume you say

Volume• Imagine a 3D grid and at each

cell on that grid you determine what light enters and exits it• You could use that grid to

determine what light hits a surface within that grid

Volume• The only way to truly know what light

enters each cell is to render the entire environment into a cube map

• Meaning for every cell on the grid you have to render the entire environment again

• And then when you want to use that data for each pixel you are shading you have to read each and every pixel and average out the results

Volume• So instead of averaging out half an

entire cube map per pixel we use Spherical Harmonics

• We then have four numbers per colour channel so the equivalent of three pixels

• And there is a simple directX call that will convert our cube map into the Spherical Harmonics

Volume• However we still have to render the

entire environment into each grid cell• So instead we forget about getting

perfectly accurate light flow we just worry about the lights that are generated from the reflective shadow map

• We still want to end up with spherical harmonics at the end but we don’t want to have to render cube maps

Volume• So instead of cube maps we create

three Volume textures (think of them as 3d arrays of floats)

• And perform a pixel shader on them• Then perform the math to create a

light in Spherical harmonics and then add it to the grid

Volume• We have then essentially

rendered to spherical harmonics• But lights will now only affect

surfaces in the same cell of the volume as they are in• So we must Propagate

Propagate• If we use the values in the

neighbouring cells we can calculate what light from that cell should flow into this cell• If we do this multiple times then

the light will flow as far as it normally would

Propagate• As we propagate the light we can also check if

there is any geometry that would block the flow between two cells

• Giving you a shadowing effect• And if you knew the average surface normal for

that geometry then you can “bounce” the light into a corresponding cell

• This bounce would also have an altered colour based off of the average colour of the blocking geometry

Propagate• The get the blocking geometry we

use the geometry shader and compare each triangles world space position against the positions of the cells of the volume

• This is done on the geometry shader and not the vertex shader because the vertex shader does not give a surface normal

Ok so we have a Light Volume

Sampling• When we have to shade a surface

we now have the lighting info (and we didn’t have to render the entire world multiple times)• Find the closest cell to the point

you are shading (pixel shader) grad the spherical harmonic coefficients and .....

Sampling• What does the Spherical harmonics

actually give us?• The Spherical harmonics is a low

frequency representation of the light coming from all directions

• Think of being in a blurry glass ball that would be a good representation of what the Spherical harmonics gives us

Sampling• So all we need to know is

what light is coming in the direction of the surface normal we are shading• Or for specular the reflection

angle based on viewing angle and surface normal

So why is this faster than lighting normally

Light Propagation Volumes• When rendering many lights we normally

use deferred rendering• In light propagation volumes if we wanted

to add another light all we have to do is add it to the list and then on the GPU calculate Spherical harmonics in the pixel shader and that is the only extra overhead of adding an extra light (we do have the base cost of propagation though)

Light Propagation Volumes• In deferred lighting if we want to add a

new light then for every surface it touches per pixel we need to calculate its effect and add it to the light currently effecting that pixel

• With light propagation volumes adding a new light is basically only writing to three pixels where as deferred could potentially cover hundreds of pixels

Light Propagation Volumes• Since we now have a 3d volume telling us

how light is travelling in our scene we can also do other effects very easily such as

• True specular lighting: normaly the specular materials are faked like in old games like doom 3. true specular is used in battlefield 3 (they also use a grid of spherical harmonic coefficients to do their lighting)

Light Propagation Volumes• Also we can do light volumes (also in

Battlefield 3) which is the beam of light you see in a really dusty room• We can also properly light particles

(also used in Battlefield 3) if the smoke is thicker then less light would pass through it and more will bounce off it

Light Propagation Volumes• A issue with light propagation volumes is

that is it an approximation of the lighting and so it will never result in the perfect raytraced lighting

• But most people would not be able to tell the difference

• The error of the approximation will increase if the resolution of the volume texture is low

Light Propagation Volumes• Also if the resolution of the reflective shadow

maps is low then more error is introduced• The other issue is that the Spherical

Harmonic coefficients are a low frequency approximation of light flow

• Only low frequency details are captured• Greater resolution of the grid means higher

frequency details are captured but a lot of overhead is introduced

Light Propagation Volumes• So what crytek did was use an

average resolution volume and for high frequency details they use screen space ambient occlusion• This gives the scene the micro

shadows that it would other wise miss

Screen Space Ambient Occlusion

• A post process pixel shader technique• First used in crysis• It has as many different variants as there

are different kinds of Anti Aliasing• But the general idea is to use the depth

buffer to determine how much geometry is occluding that pixel from its surroundings

Screen Space Ambient Occlusion

• In recent games the shader will also take into account the surface normal so the scene looks more detailed

• The performance limiting factor is the number of samples each pixel takes from the depth buffer

• Some games make the SSAO buffer a quarter of the size to increase performance

Screen Space Ambient Occlusion

• Also one of the gears of war games uses temporal coherence to build up the SSAO over multiple frames

• A general rule is that roughly 100 samples will be needed for good results (which is completely unpractical)

• But instead the games that use SSAO take between 8 and 16 samples and blur the results

Screen Space Directional Occlusion

Screen Space Directional Occlusion

• Instead of doing 8-16 samples we do 4-10 and add a Raytrace in the direction of the surface normal

• Overhead ends up being just a little more than Screen Space Ambient Occlusion

• But normal mapping the result is automatic and you can grab the colour from the raytrace and the light that has hit that surface

Screen Space Directional Occlusion

• This can be used to replace the high frequency light bounces

• It is not 100% accurate but it is effective at faking the extra precision

• For real time rendering it is a fair compromise

• Plus just about every triple A game uses it of late (the list of game is to big for this presentation)

My PrototypeI am not doing the exact

same thing as Crytek or Dice

My Prototype• I recently worked on Spelunking Club

Session in unity• I was asked if I could make the glowing

dance floors cast light onto other objects• Propagation volumes is perfect for that it

could hand the large number of lights needed to approximate a surface casting light

My Prototype• So I decided to prototype the

propagation volumes part without the reflective shadow map part

• Then just place lots of lights on the dance floor prefab

• And pass those into the volume

Problems though• Volume textures were introduced in

directX 10, Unity only supports directX 9

• To add occlusion during propagation you need the geometry shader, the geometry shader was introduced in directX 10...Dam

Problems though• The Volume texture can be replaced by

offsetting your logic on a large 2D texture• But the benefit of the built in Volume

texture is that when you sample the volume texture it returns a filtered result (the average of a bunch of pixels)

• If I wrote my own it would add more complexity and would take more time but it is not insurmountable

Problems though• Geometry shader is needed

for the occlusion of light• You could instead do this

on the vertex shader right?

WRONG!

Problems though• The vertex shader does not give us a surface

normal• On directX 9 and opengl equivalent have low

performance for texture fetch on the vertex shader (which was fixed in directX 10)

• So we could not even check the colour of the surface

• So I decided that occlusion was not feasible

How Mine works• Create three RGBA render targets, RGBA

gives us four floats which is equivalent to four coefficients per colour channel

• The three targets each represent a different colour (RGB)

• The coefficients are zonal harmonics• Which means that they are aligned

around an axis (usually Z)

How Mine Works• Add the information for each light to an

array that is sent to a pixel shader that runs on these render targets

• Problem• The number of lights in the scene varies

but you cannot pass a non fixed length array to a shader in unity

How Mine Works• Solution• Unity uses the cg runtime (Nvidia’s

shader language)• The cg runtime allows dynamic

recompilation of shaders• This means we can recreate the shader at

runtime

How Mine Works• This is done by physically changing the

text of the shader and then recompiling• This sounds hefty but we only need to

change a few numbers (length of the fixed length array)

• And we only need to do this when a new light is created or a light is destroyed

How Mine Works• To inject we create a

Spherical Harmonic representation of the light colour facing in all directions• Then create a spherical

harmonic representation visibility which is a cone

How Mine Works• Then rotate that visibility function

to the direction of the light we are simulating

• To combine these we perform the spherical harmonic dot product on them

• This gives us the Spherical Harmonic representation of this light

How Mine Works• Combining this with the other lights from

that cell will give the full light from that cell

• To propagate we have to avoid light flowing backward into its original cell which would cause brightening artefacts

• So we need to propagate only the light going in the direction of the cell we are propagating to

How Mine Works• To do this we dot product the visibility to

that cell by the light in that cell and then propagate

• This ensures light never flows back• Then finally to sample we write a unity

surface shader which is put on all surfaces in the world

How Mine Works• With the light coefficients we do a simple

dot product between the coefficients and the surface normal to get the light flowing in that direction