copyright replay solutions, llc 2004 game|tech 2004 building a reusable replay system jonathan lindo...
TRANSCRIPT
Copyright Replay Solutions, LLC 2004
game|tech 2004
Building A Reusable Replay System
Jonathan LindoJeff Daudel
Copyright Replay Solutions, LLC 2004
Introduction
Jonathan Lindo & Jeff Daudel Company founders Research, next-gen technology at Nortel
Networks, Lucas Arts, Secret Level, and SGI.
Replay Solutions, LLC Founded in 2003 In use with Ion Storm, Crystal Dynamics,
NVIDIA Corp.
Copyright Replay Solutions, LLC 2004
Why Have A Recording System?
Why do I need one? Great uses for Recording Systems.
Nailing down bugs Communicating bugs from QA to Dev Memory & Performance Profiling Demos
Copyright Replay Solutions, LLC 2004
What is a Recording System? VCR Functionality
Record Mode Captures data that will be necessary to execute
the program accurately at a later point in time. Replay Mode
Intercepts non-deterministic calls. Injects previously recorded data into the game.
What type of data should replay record?
Copyright Replay Solutions, LLC 2004
Sources of Non-Determinism
User input Timers Asynchronous callbacks Thread context switching DMA access timing Shared memory access Interrupts
Copyright Replay Solutions, LLC 2004
Types of Recording Systems
A brief overview of recording systems:
External I/O Recorders. Source Code Instrumenters Binary Patching Systems
Copyright Replay Solutions, LLC 2004
External I/O Recorders
Network IO, User Input, D3D Recorders
Pros: Don't require any mods to game code. Are robust to code and asset changes. Can be used for cross platform testing, demos.
Cons: May not produce accurate replays due to Timing issues,
race conditions. May be rejected by system for security reasons (anti-
replay attacks on the xbox)
Copyright Replay Solutions, LLC 2004
Source Code Instrumenters
Proxy API Libraries, Modules
Pros: Somewhat Reusable Can be easily expanded, tuned. Recordings may be portable across platforms.
Cons: Applicable only to modules for which source is available. Requires the use of specialized API for certain calls by the
developer, or a code parsing module. No support for 3rd party modules. May not produce 100% accurate replays.
Copyright Replay Solutions, LLC 2004
Binary Patching Systems
Post-compilation Binary Instrumentation
Pros: Very Reusable Deep level recording will reproduce accurate recordings. No source code mods required, can be applied to any EXE, LIB or DLL. Zero impact when disabled. Can capture low-level ASM calls.
Cons: Can be fragile when code or assets change. May require special support for certain APIs. Support required for processors and binary formats (PE, XBE, ELF) Difficult to make recordings portable between platforms. Cross module inlining (i.e. LTCG) distorts function boundaries
Copyright Replay Solutions, LLC 2004
Recording System Architecture
Game Binary
Game CodeStatic Libs, DLLs
Kernel / Operating System
Processor(s), System Hardware
Recording System
Network I/O
File I/O
Multi-Threading
User Input
Interrupts
Timers
Copyright Replay Solutions, LLC 2004
Replaying Async Callbacks
Define & Install Proxy callback handlers Record & Queue events in Proxy handlers Deliver callback events at pre-determined
Sync points, such as Proxy methods. During Replay, deliver simulated callback
events at recorded Sync points. This methodology can be applied to Windows
message callbacks, sound system callbacks, or async I/O of almost any kind.
Copyright Replay Solutions, LLC 2004
Replaying Async Callbacks
Game Binary
Kernel / Operating System
Processor(s), System Hardware
Recording System
Proxy Callback Handlers
Game Callback Handlers
Copyright Replay Solutions, LLC 2004
Replaying Thread Context Switches
Thread context switching order is mostly important when data is being exchanged between threads.
Identify the methods which are called to share data between threads. These are Thread Sync Points. Lock, TryLock, Release, EventWait, EventSignal WaitForMultipleEvents, WaitForSingleObject EnterCriticalSection, LeaveCriticalSection
By ensuring that these methods are called in the same order, we can make sure that data is processed in the same way on replay as it was on record.
Copyright Replay Solutions, LLC 2004
Replaying Thread Context Switches
Recording System
Thread A
Thread B
Shared Resource
Copyright Replay Solutions, LLC 2004
Robustness and Portability
Handling hardware configuration differences when replaying on different machines.
Syncing assets and source code in a dynamic development environment.
Copyright Replay Solutions, LLC 2004
Handling Hardware Differences
To support replaying recordings on machines with different configs, steps must be taken during recording to ensure replay compatibility.
Determine a baseline hardware config that will be supported.
Create Proxy functions for all hardware detection methods. (GPU, Sound Hardware)
Force the game to the baseline hardware config to ensure compatibility when replaying on different systems.*
*(Make this an optional setting!)
Copyright Replay Solutions, LLC 2004
Syncing Assets & Code
In a live dev environment, code and assets change daily. This makes it difficult to store valid Recordings.
When recording, save accessed assets and code relating to the build for future replays. This can be done through File I/O Proxy functions.
When replaying, load assets and code relating to the recording for reference.
This can be implemented using a simple storage and retrieval module, local or network based.
Copyright Replay Solutions, LLC 2004
Feature Additions
Fast forwarding… Any function that does not affect game state can
be skipped during Replay.
Skipping ahead to a saved game state. Grab game & system state Provide an API interface to dump and restore
game state.
Copyright Replay Solutions, LLC 2004
Memory
Accurate memory replication is required to: Reproduce Memory leaks Reproduce any code that has data sorted
by pointers (i.e. stl collection of pointers)
Typically, engines main loop involves iterating over game objects that are sorted by their memory address.
Copyright Replay Solutions, LLC 2004
Memory Isolation Provide a private heap for all game memory.
Intercept all memory requests to the OS and redirect to a controlled heap. HeapAlloc, VirtualAlloc, etc.
Isolates nondeterministic system calls from making under-the-hood allocations.
Track system memory allocations requested from the game. Console environments may ask the game to provide and
manage memory (XMemAlloc on xbox) Simulate these requests on replay
Dummy memory, but will insure game heap is preserved.
Copyright Replay Solutions, LLC 2004
Replay Performance
Importance of low impact Can be run at all times – not just when looking
for a bug Toughest bugs can occur at unexpected times Does not have to be removed from shipping
version. Captures more realistic game scenario Accessible throughout the entire QA process
Especially near the end where it is needed most!
Copyright Replay Solutions, LLC 2004
Being Efficient Record only what is nondeterministic.
Most function calls are deterministic in nature – no need to record them rand()
Optimize high-performance clock sampling using sample throttling. Besides high performance profiling, usually there
is no need to access time more than a few times a frame.
Copyright Replay Solutions, LLC 2004
Buffering Buffer replay data and flush to persistent
storage only when needed. Disk and network storage I/O is expensive. Flush periodically or when a critical event
occurs Exceptions, exit
Install exception handlers and filters to intercept these events.
Copyright Replay Solutions, LLC 2004
Practical Results Applied a replay binary replay patcher on
several titles Typical replay data can be as low as 300
KB / minute Will vary for each game Depends on game’s functionality
No noticeable drop in frame rate.