Ratchet and Clank Future: Tools of DestructionTechnical Debriefing

February 2008

Tech Goals

• 60Hz

• Visuals that appeal to casual and experienced players

• Look and feel of a CG movie

• Lots of action on screen

Some Poly Counts• Kerchu City (level) 3,895,000 Polys• Metropolis (level) 8,781,000 Polys• Pirate (moby) 5,382 +/- ~600 Polys• Kerchu Boss (moby) 26,472 Polys

But…

• We need to avoid risky tech changes.

– Production time is less than 1 year

• 60Hz means we need to look for cheap and simple solutions.

• We have a limited window for making tech changes – prioritizing is essential.

What were our priorities?• Improve HDR model and image processing• Improve baked lighting model and artist workflow• Improve texture resolution• Better, faster method for dynamic shadows• Improve effects – esp. performance• More one-off effects (fur, cloaked effect, etc)• Better method for alpha sorting• Major overhaul to our tools and production pipeline

And To Get To 60Hz…

• Support new kinds of LOD

– Shader LOD

– Fade-Out LOD

– Discrete Geometry LOD

• Shader Usage Controls

• PPU / SPU – dig in and optimize

And To Get To 60Hz…

• PPU / SPU – dig in and optimize (examples)– More than 5x speedup in animation

• Re-write in assembly

– More than 2x speedup in physics• Reduced sync points, simplified pipeline (SPU shaders)

– 3x-14x speedup in particle update• Reduced sync points, straight optimizations

– Lots of other optimizations!

Some Basics

• Workstations

• Programmer Iteration

• Memory Map

• Loading and Resource Management

Workstations

• Of course, iteration time is a huge issue

• We were seeing big differences between workstations, so we set out to find the best possible hardware for our needs.

• We were surprised to find that we could improve performance by 3x to 5x over our “typical” production PCs on real-world apps (our typical PC is a 3GHz 2x Dual-Core Xeon with 4GB of ram made by Dell in 2005)

Workstations - Results• We compared all variations of high-end CPUs from AMD and Intel.

Results were not surprising – 2x Quad-Core Xeon was best (3GHz w/ 1600MHz bus).

• We compared every drive configuration we could think of from ultra high-speed SSD to 15K Raid 0 to a standard 7200rpm drive. We saw very little real-world improvement from the more exotic drive configurations.

• Performance-oriented motherboards make a huge difference (another one for the “duh” files).

Perforce• We replaced an in-house, custom-brewed system

with Perforce.• Greatly reduced the amount of time we spent

supporting a system with a better off-the-shelf alternative.

• Did require some significant changes in workflow, but in the end, these were for the better.

• We used it more extensively than ever on this project. We managed all assets (almost) and all code through perforce.

Perforce• We use code branches and integration extensively. This is a

constant source of pain, but I don’t think much of the blame lies with Perforce.

• We used asset branches much less often because of the size, but nonetheless they were still a critical part of our process for creating stable discs while production was ongoing.

• Late in the project we encountered performance issues when large quantities of localized audio data were checked in as a huge lump. – This performance problem really only lasted for a day as everyone

grabbed the data.

Perforce

• It did cause some problems with disks filling. – This is now on our radar for a better solution for

future projects.

• Overall, it has been very reliable for something we use and abuse so much

Maya• We are trying to use Maya for what it was designed for

(modeling, animation) and nothing more.• We moved all of the scene composition and gameplay editing

into our own editor (Luna).• We’re hoping to make it so that programmers and designers

never have to open Maya – though this is still months away.• We’re also trying to make our tools flexible to support other

authoring tools.

Internal Tools Overhaul• It involved an almost complete rewrite of the codebase. The biggest

changes:

• Luna now serves as a unified interface to the production tools, so artists can avoid command-line tools entirely (almost).

• Luna editor is used for scene composition, gameplay editing, assigning attributes. Maya is for modeling and animation.

• Intermediate file format – previously we had used Maya files directly. This helps us isolate our dependence on Maya while also improving performance

Internal Tools Overhaul• All assets are managed by Perforce. Nothing is live, and all changes must be

checked-in explicitly.

– This was probably the biggest change to content producers’ workflow.

– Required a significant amount of time before the workflow was accepted and benefits were apparent.

• Took some steps to create an abstraction between source assets and their final in-game engine implementation.

– Allows for greater flexibility in how the engine handles assets without affecting content producers.

• Attempted to make source asset setup more intuitive and granular. Want something to behave a certain way? Add the right attribute to it.

Internal Tools Overhaul• Switched from a system in which each class of asset was identified by a

number (moby 123) to a name-based system where content producers are responsible for organization and engine type is abstracted away.

• Automated dependency tracking. Artists don’t have to think about what needs to be rebuilt.

• Created a framework for building/packing using dlls dynamically loaded based on asset type.

– Meant a single command/button (buildtool.exe) for building everything rather than specific command line tools (zb.exe, mb.exe, tb.exe, etc.)

– Allowed for better control over build process, making it easier to integrate ubiquitous dependency and download systems.

Internal Tools Overhaul• Signature-based system for finding already-built files on the network to

improve build times.

Memory Map - VRAMSize Description

60 MB Low mip-maps of shader textures

60 MB Streaming buffer for high mip-maps of shader textures (room for about ¼ of all textures in the level)

50 MB Baked lighting data (mix of light maps and per-vertex data)

20 MB Frame buffers

15 MB Actor (moby) geometry

8 MB Scratch buffers (shadow maps, image processing, etc).

Memory Map – Main ramSize Description

20 MB Code, PRXs, sys heap

20 MB Static instanced geometry

5 MB Static unique geometry

20 MB Animations

5 MB Collision database

5 MB Static occlusion database

10 MB Scratch buffer

7 MB Effects buffers

10 MB Push Buffers

15 MB Gameplay instances

10 MB Scripting (Lua) Memory

10 MB Gameplay systems

40 MB Sound

Memory Management• Straightforward memory buffers

– One buffer for main RAM, one for VRAM• Allocate all available RAM

– 1MB page sizes– Leave ~2MB for SDK heap use

• Managed from a single pointer to current ram position– Alloc = increment pointer– More complex allocators can be used inside of a chunk

• Pool allocators• Arbitrary size “heap” allocators

• Minimum number of allocation tiers– System init, level load, gameplay start, current frame

Memory Management• Minimize number of allocation “tiers”

– System init– Level load– Gameplay start– Current frame

• Systems aren’t “destructed” they are just rolled-back• Manually manage systems that steal memory from each other

– e.g. Movie buffers, UI texture streaming buffers

Memory Management• Handling out-of-memory

– Keep it simple• Throw away overflow (i.e. don’t do it)• Generate a warning on-screen• Give enough information to fix problem

• Tracking “stale” pointers– Some gameplay systems do this for debug version– Most systems don’t. Not necessary: Systems that use memory are

rolled back with systems that own the memory.– Otherwise, we just track it manually (i.e. we know who needs/uses

what).

Loading and Resource Management

• We stuck with a level-based loading scheme. There is a nearly complete flush of memory between levels, and we load roughly 200 MB (with compression) during the transition.

• Within a level we only stream a few things: high mips of textures, dialogue, some sounds, cinematics and pause menu assets.

• We also use FIOS to cache the next level to the HDD while the current level is being played. This can reduce the level transition load time to 5-6 seconds.

• The texture high-mip file is also cached to the HDD to speed up texture streaming and to reduce blu-ray contention.

Some Interesting Things (?)

Improved Detail Maps• We put some effort into making a better detail map than we had in

Resistance. Our artists are even starting to use them sometimes.• It uses 4 channels (DXT5 or ARGB8888)

– Color map offset (-1..+1)– Gloss map offset (-1..+1)– Normal map delta (partial derivative dx)– Normal map delta (partial derivative dy)

• Also, there is an optional channel in the standard shader template that functions as a detail map mask – it modulates the intensity with which the detail map is applied.

Partial Derivative Normal Maps• Our previous way of encoding normal maps was the standard

approach:– We used DXT5 (by default) and store the X component in G and the Y

component in A. – The Z component is reconstructed in the pixel shader in 3 instructions.

Partial Derivative Normal Maps• Our new approach is to store the partial derivative with respect to

X and Y instead of their vector components. This allows the normal vector to be reconstructed in 2 instructions.– Store

• dx = (-nx/nz); • dy = (-ny/nz);

– Reconstruct

• nx = -dx;

• ny = -dy;

• nz = 1;

• normalize(n);

Partial Derivative Normal Maps• Better yet, it allows the detail normal map to be combined with the

main normal map using only 3 instructions (versus ~10 with the standard encoding).

• But, it yields a different error distribution…

Texture Streaming• We used as straightforward a system as we could create. Only the

top mip level is streamed – this gets the majority of the benefit with little pop-in artifacting.

• We had roughly 300 Megs of texture data on a typical level, of which 120 Megs could be in memory at once.

• We used an automated scheme for determining which mips are needed based on the distance to each asset in the scene. This was an approximation, but worked well enough.

• The biggest problem we ran into was with blu-ray contention and inadvertently starving a higher priority system like sound streaming.

Custom Shaders and Effects• These are the ones we added for RCF – inquire if there are

any that seem interesting to you:– Fur shader (actually, two different ones)– Cloaked effect (Predator-style)– Anisotropic material– Audio “equalizer” effect– Water pool shader– Underwater caustics– Dematerialization effect (dissolve into particles)– Arbitrary UV animation

Shader Usage Controls• The problem came up – how do we tune the Gpu performance late in the project

without going all the way back to the source art.• Our approach was to add knobs to tweak performance to each place where a

shader gets used on a asset. Some of them include:– Transition distances for discrete LOD– Transition distance for fade-out LOD– Fade distance for detail maps– Transition distance for shader LOD– Which attributes (normal, gloss, parallax, detail map) are disabled for this use of the shader.– Whether this geometry casts shadows– Whether this geometry receives shadows– Whether this geometry skips the depth pass when fading out

SPU “Shaders”• An idea we’re exploring is uploading small bits of custom SPU code to

execute within the context of a larger system. We’ve been calling these “shaders” even though they aren’t really related to shading.

• There are a lot of limitations the code snippet has to follow that are set by the larger system – like code size, memory workspace, hooks into the larger system, etc. But within these limitations, the code is free to do anything it wants.

• One goal of this is to provide a way for non-engine programmers to have a way to start writing code for SPUs.

• Our physics system uses this idea quite extensively.• There were some challenges initially – like debugging.

– Now we just build the a complete elf (Not for all systems, yet though)– We DMA the shader PLUS the 16 byte module header for the debugger.

Fully Asynchronous SPU Systems• We pushed several SPU systems to be as asynchronous of the main

update loop as possible.• Long-term goal of getting rid of the main update loop as we know

it. Instead, you’d have lots of independent systems running with relatively few sync points.

• Our physics system got the furthest along in this direction. It runs almost fully independently of the main loop, updating its simulations continuously and the main loop can inquire about any of the simulations whenever it wants.

• We also moved the effects update code toward this model.

Explicit SPU Scheduling• Another way we use SPUs is nearly the opposite

of this. We schedule tasks at very explicit times and we have very explicit sync points for each.

• The best example of this is during the “render” phase of the main loop – the part that builds the push buffer. We budget about 4ms for this, and the numbers on the following diagram are the rough budgets for each phase in ms.

Other Engine Improvements

• Push-buffer generation code on SPU

• Increased support for programmer-custom effects.

• Math library re-write– Rant on! Vector/Matrix math libraries suck.

– Better: stick to (large) buffer transforms of native types.

Education and Training• Tech team working on different projects• Weekly programmer presentations

– Tech team takes turns giving presentations to programmers on topics of their choice

– 30-40 internal presentations in 2007

• SPU Ninja Training– Weekly classes on low-level SPU coding

Education and Training• Guest Presentations

– E.g. Jim Buck from Twitchy Thumbs gave a talk on Rally Cross Physics (PSX)

• This year we had Embedded Systems Training

Key Features for Resistance 2• Some things we’ll be working on in the next year (if anyone

has any advice to share…)– Full-fledged streaming system– Much broader use of dynamic lights– Another crack at the dynamic shadow system – different model

for outdoor scenes versus indoor.– Shader driven effects – esp. smoke, fire, etc.– Water, lots of water– Very large scale enemies and interactions– Integrated cinematics

Contact Info

• Mike Acton – macton@insomniacgames.com

• See also Insomniac R&D:http://www.insomniacgames.com/tech