aaa automated testing… for aaa games

Download AAA Automated Testing… For AAA Games

Post on 21-Mar-2016




0 download

Embed Size (px)


AAA Automated Testing… For AAA Games. Francesco Carucci Technical Director. How much does a game cost?. Development costs spiraling up EA Exec: “Cost of games is crazy!” Some examples: Killzone 2: > 40 M$ Halo 3: > 30 M$ GTA 4: > 100 M$ (rumored) - PowerPoint PPT Presentation


  • AAA Automated TestingFor AAA GamesFrancesco Carucci Technical Director

  • How much does a game cost?Development costs spiraling upEA Exec: Cost of games is crazy!Some examples:Killzone 2: > 40 M$Halo 3: > 30 M$GTA 4: > 100 M$ (rumored) Clearly, this is not sustainable in the long term

  • Development cost partitioningTo keep it simple:Producing assetsWriting code (game code, technology, tools)Debugging and maintaining codeCode dictates cost of producing assetsBetter tools means less time to create assetsMore efficient code means less time spent optimizing assets

  • Solutions?Better production frameworksSCRUM or some variation of itMore experienced teamsHigher senior/junior ratio actually costs lessBetter tools (3rd party engines, off the shelf packages)Out sourcing (yay for assets, nay for code)Higher quality code

  • Code is not the KingBut its a first class citizen and we want to keep its cost downHigher code quality reduces Cost Of OwnershipLess bugsLess over engineeringHigher code quality means code easier to change and iterate uponHigher code quality means more time to be spent on polishing the gameHigher code quality means less money spent to make a game!And a bigger chance to ship on time and hit that market sweet spot

  • Change Prevention ProcessMany teams follow a change management process whose main goal is to prevent feature creepIts effectively a Change Prevention ProcessBut you cant design fun up frontGames must be developed iteratively until fun is foundSo?Embrace Change

  • Cost of ChangeGame Design requirements changeThe aim system is good, but I would like it more peachy, you know a bit more of this, but slightly less of that, more rocking, less oompfh. Can you change it?Traditionally Cost Of Change rises sharply over timeNo, I cant change it, It would break X and Y and probably Z too, I dont really knowWe want to keep Cost Of Change as flat as possibleYes we can!

  • Enter Automated TestingAutomated Testing has been a broadly accepted tool for several decades in the Software Industry aimed at reducing cost of developing working codeIn fact (warning: Boring Slides Ahead)

  • Assessing test-driven development at IBMIn a software development group of IBM Retail Store Solutions, we built a non-trivial software system based on a stable standard specification using a disciplined, rigorous unit testing and build approach based on the test- driven development (TDD) practice. Using this practice, we reduced our defect rate by about 50 percent compared to a similar system that was built using an ad-hoc unit testing approach. The project completed on time with minimal development productivity impact. Additionally, the suite of automated unit test cases created via TDD is a reusable and extendable asset that will continue to improve quality over the lifetime of the software system. The test suite will be the basis for quality checks and will serve as a quality contract between all members of the team.

  • An initial investigation of test driven development in industry

    Simposium on Applied Computing Test Driven Development (TDD) is a software development practice in which unit test cases are incrementally written prior to code implementation. In our research, we ran a set of structured experiments with 24 professional pair programmers. One group developed code using TDD while the other a waterfall-like approach. Both groups developed a small Java program. We found that the TDD developers produced higher quality code, which passed 18% more functional black box test cases. However, TDD developer pairs took 16% more time for development. A moderate correlation between time spent and the resulting quality was established upon analysis. It is conjectured that the resulting high quality of code written using the TDD practice may be due to the granularity of TDD, which may encourage more frequent and tighter verification and validation. Lastly, the programmers which followed a waterfall-like process often did not write the required automated test cases after completing their code, which might be indicative of the tendency among practitioners toward inadequate testing. This observation supports that TDD has the potential of increasing the level of testing in the industry as testing as an integral part of code development

  • GoogleStandard Google practices include unit tests and functional tests, continuous builds, last known good build, release branches with authorized bug fix check-ins, focused teams for release engineering, production engineering and QA, bug tracking and logging of production runs.

  • Professionalism and Test-Driven Development

    Test-driven development is a discipline that helps professional software developers ship clean, flexible code that works, on time. In this article, the author discusses how test-driven development can help software developers achieve a higher degree of professionalism

  • Unit Testing in GamesIn our experience, the introduction of automated tests and continuous integration makes development teams more efficient and results in more reliable, and often quite simply, better software. Additionally, it reduces the pressure and workload on development teams by reducing the effort for manual testing, and allows bugs to be found earlier in the development process. Certainly, automated tests alone won't make your game a hit. But almost as certainly, they will make life easier for developers, artists, project leaders, producers and even players.

  • Unit Testing in Games (contd)Test-driven development can be a very effective development technique. We have successfully applied it to game development in a variety of situations, and were convinced of the many benefits it has provided us. Right now, the idea of writing code without writing tests first feels quite alien to most of us, and we treat TDD like the scaffolding in building construction: a necessary tool that will not be shipped to the customer but that helps tremendously during development

  • Unit Testing in Games (contd)Wir haben in den letzten fnf Jahren ausschlielich positive Erfahrungen mit automatisierten Tests und Continuous Integration gesammelt. Die automatisierten Tests sichern Stabilitt auf hohem Niveau, whrend Continuous Integration zustzlich eine deutliche Arbeitserleichterung darstellt. Der Aufwand fr die Einrichtung und Wartung der Verfahren ist berschaubar und hat sich schnell amortisiert. Wenn man die anfngliche Skepsis berwindet, mchte man unserer Erfahrung nach diese Art der automatisierten Qualittssicherung sehr schnell nie mehr missen.

  • Whats an Automated Unit TestA snippet of code that exercise a Unit Under Test (method of a class), assert its correctness and is executed automatically every time the system is builtRunning unit test is effectively a compilation stepCompiler checks if the code is syntactically correctUnit tests check if the code is semantically correct

  • Test First vs Test LastWriting automated tests is accepted to produce higher quality code at lower costMore tests == less time spent debuggingShould tests be written after production code? (Test-Last)Should tests be written before production code? (Test-First)This observation supports that TDD has the potential of increasing the level of testing in the industry as testing as an integral part of code developmentTest Driven Development == Test-First

  • Test-Driven DevelopmentWrite a failing test before the code is writtenWrite the simplest code that could possibly make the test passRefactor the code to eliminate all possible duplications and code smells Red Green RefactorThats the coding rhythm

  • A Failing TestFirst write a failing test:CRY_TEST (FireButtonPressAndRelease) { StartFire(); ASSERT_IS_TRUE(IsFiring()); } The test is dictating the interface and the contract of the unit being written:When StartFire method is invoked, the weapon is firing then implement the minimal code to make it passbool IsFiring() const { return true; }

  • Another Failing TestWrite another failing test to dictate more behavior:CRY_TEST (FireButtonPressAndRelease) { StartFire(); StopFire(); ASSERT_IS_FALSE(IsFiring()); } When StartFire method is invoked, and then StopFire method is invoked, the weapon is not firing.The actual word explanation is redundant, the whole information needed is in the code itself in readable form!

  • Another Failing Test (contd.)Minimal implementation:bool StartFiring() {m_isFiring = true; } bool StopFiring() { m_isFiring = false; } bool IsFiring() const { return m_isFiring; }

  • How does it workTests are small, simple, straightforward and test only one conditionLong and complicated tests must be avoidedTests are self documentingTests describe the design of the code to be writtenContractInvariantsPost-conditionsCalling conventions A TDD cycle should take few minutes (often less than a minute)30 minutes to write a test means a bad testBreak bigger tests in smaller steps (help test coverage)Begin with baby steps, increase step size when becoming more comfortable

  • Why TDD?Small and clearly defined steps lead to the final


View more >