gamedev-grade debugging
DESCRIPTION
Disclaimer: the slide deck does not contain all the content, this was a mostly spoken/hands-on talk. Code used in the talk: http://github.com/inequation/ggd Video games are highly complex and indeterministic systems working closely to hardware. This means that sometimes the regular troubleshooting measure of setting up breakpoints just doesn't cut it. How to filter out noise, when our breakpoint is hit dozens of thousands of times per second? Or if our bug, despite perfect reproduction rate, manifests itself in a different way each time? How to find the culprit overwriting our precious data? This talk will present some interesting cases from the life of a game programmer along with tips and tricks to have up your sleeve when fighting them.TRANSCRIPT
inequation.orginequation.org
SpreadIT 2013 · October 19th, 2013SpreadIT 2013 · October 19th, 2013
Gamedev-grade debugging
Leszek GodlewskiFreelance [email protected]
Code snippetsCode snippets
2 inequation.org2 inequation.org
All code used in the talk available online:github.com/inequation/ggd
AgendaAgenda
3 inequation.org3 inequation.org
● Who is this guy?
● Why the talk?
● A taste of gamedev debugging
● Right tool for the job
● Noise filtering
● Memory corruption
● Questions
Who is this guy?Who is this guy?
4 inequation.org4 inequation.org
Freelance Programmer (Sep 2013 – onwards)● inequation.org● Painkiller Hell & Damnation
Linux port finalization (2013)● Unannounced project
Generalist Programmer,The Farm 51 (Mar 2010 – Aug 2013)● thefarm51.com● Painkiller Hell & Damnation
(2012-2013; Win/Linux/X360/PS3)● Deadfall Adventures
(2011-2013; Win/X360)● A few unannounced projects
AgendaAgenda
5 inequation.org5 inequation.org
● Who is this guy?
● Why the talk?
● A taste of gamedev debugging
● Right tool for the job
● Noise filtering
● Memory corruption
● Questions
Why the talk?Why the talk?
6 inequation.org6 inequation.org
Because THIS has happened to me:
http://imgur.com/yBa1OGm
Why the talk?Why the talk?
7 inequation.org7 inequation.org
Intern: I've read* at all the code andstill don't see the bug.
Me: So just debug it!
*read, as in „stare without actually running it”
Why the talk?Why the talk?
8 inequation.org8 inequation.org
Intern:
http://imgur.com/yBa1OGm© DreamWorks
Why the talk?Why the talk?
9 inequation.org9 inequation.org
The three uses of debugging
● Bug hunting (doh)●
●
Why the talk?Why the talk?
10 inequation.org10 inequation.org
The three uses of debugging
● Bug hunting (doh)● Reverse engineering●
Why the talk?Why the talk?
11 inequation.org11 inequation.org
The three uses of debugging
● Bug hunting (doh)● Reverse engineering● Testing a new feature
AgendaAgenda
12 inequation.org12 inequation.org
● Who is this guy?
● Why the talk?
● A taste of gamedev debugging
● Right tool for the job
● Noise filtering
● Memory corruption
● Questions
A taste of gamedev debuggingA taste of gamedev debugging
13 inequation.org13 inequation.org
Code: Code: 01-taste01-taste
A taste of gamedev debuggingA taste of gamedev debugging
14 inequation.org14 inequation.org
Can you spot the culprit?
// class declarationclass Crasher extends ActorComponent;var int DummyArray[1024];
// in ammo consumption codeCrash = new class'Crasher';Comp = new class'ActorComponent' (Crash);
AgendaAgenda
15 inequation.org15 inequation.org
● Who is this guy?
● Why the talk?
● A taste of gamedev debugging
● Right tool for the job
● Noise filtering
● Memory corruption
● Questions
Right tool for the jobRight tool for the job
16 inequation.org16 inequation.org
„Fix a bug for an intern, they will get stuck on the next one.
Teach them debugging, they will fix most bugs they encounter on their own.”
– Paulo Coelho
Right tool for the jobRight tool for the job
17 inequation.org17 inequation.org
Debugging tools are essential to this profession!
● When joining a new team or starting development for a new platform, demand debugging tools● Ask senior teammates● If they don't know, there must be documentation● Be proactive!● Don't give up until the debugger is fully working
● No tools? Roll your own!● You are a coder after all, right?
Right tool for the jobRight tool for the job
18 inequation.org18 inequation.org
Not all bugs are in the code
● Animation graphs● Flow graphs/visual scripts● Post-process effects
Still need a way to debug them
Right tool for the jobRight tool for the job
19 inequation.org19 inequation.org
Code: Code: 02-tools02-tools
AgendaAgenda
20 inequation.org20 inequation.org
● Who is this guy?
● Why the talk?
● A taste of gamedev debugging
● Right tool for the job
● Noise filtering
● Memory corruption
● Questions
Noise filteringNoise filtering
21 inequation.org21 inequation.org
There are parts of code executed thousands of times each frame
● Object transformation● Collision detection
Also rarer, but still impractical to track
● Setting materials on objects● Attaching and detaching of components
Noise filteringNoise filtering
22 inequation.org22 inequation.org
Code: Code: 03-material03-material04-ragdoll04-ragdoll05-assert05-assert
AgendaAgenda
23 inequation.org23 inequation.org
● Who is this guy?
● Why the talk?
● A taste of gamedev debugging
● Right tool for the job
● Noise filtering
● Memory corruption
● Questions
Memory corruptionMemory corruption
24 inequation.org24 inequation.org
Look closely:
// class declarationclass Crasher extends ActorComponent;var int DummyArray[1024];
// in ammo consumption codeCrash = new class'Crasher';Comp = new class'ActorComponent' (Crash);
Memory corruptionMemory corruption
25 inequation.org25 inequation.org
Look closely:
// class declarationclass Crasher extends ActorComponent;var int DummyArray[1024];
// in ammo consumption codeCrash = new class'Crasher';Comp = new class'ActorComponent' (Crash);
Memory corruptionMemory corruption
26 inequation.org26 inequation.org
● UnrealScript object construction syntaxnew <class> [(<template object>)];
● But:sizeof(Crasher) > sizeof(ActorComponent)
● Verdict:
Memory corruptionMemory corruption
27 inequation.org27 inequation.org
● UnrealScript object construction syntaxnew <class> [(<template object>)];
● But:sizeof(Crasher) > sizeof(ActorComponent)
● Verdict:BUFFER OVERFLOW!
Memory corruptionMemory corruption
28 inequation.org28 inequation.org
But this can happen anywhere! How to find it?
● Use a memory fence● Many related techniques● Allocate additional space in front and behind actual allocations
● Then protect them from writing...● Or write a byte pattern and periodically assert its consistency
● Also it's useful to log stack traces● Memory and CPU overhead!
● Use a debug memory allocator (dmalloc)● Use a memory debugger (Valgrind)● Use a memory analysis tool (HeapInspector)
Memory corruptionMemory corruption
29 inequation.org29 inequation.org
Memory fences
malloc
Regular allocation Fenced allocation
malloc
TakeawayTakeaway
30 inequation.org30 inequation.org
● You can't be an effective programmer without debugging tools
● If there are no tools, make some● Noise filtering techniques save your time
● Time is not only money – nerves are just as important!● Know your machine (physical or virtual) down to the metal● Instruction opcodes, registers etc. come in handy● Tons of resources available
● Random crashes and/or content glitches may indicate memory corruption
● Memory corruption can be defeated!
inequation.orginequation.org
Questions?
SpreadIT 2013 · October 19th, 2013SpreadIT 2013 · October 19th, 2013
inequation.orginequation.org
Thank you!
[email protected]@TheIneQuation
SpreadIT 2013 · October 19th, 2013SpreadIT 2013 · October 19th, 2013