Download - Game Programming 06 - Automated Testing
![Page 1: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/1.jpg)
Game ProgrammingAutomated Testing in Games
Nick Prühs
![Page 2: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/2.jpg)
Objectives
• To learn how to properly set up automated testing for your games
• To get an overview of common unit testing frameworks and tools
2 / 49
![Page 3: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/3.jpg)
Unit Testing
• Method by which individual units of source code are tested to
determine if they are fit for use
• Unit of source code is the smallest testable part of an application
(e.g. method)
• Created by programmers during the development process
• Usually split up into three parts:
▪ Arrange
▪ Act
▪ Assert
3 / 49
![Page 4: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/4.jpg)
Unit Testing
• Ideally, each test case is independent from the others
• Substitutes such as mocks can be used to assist testing a module in
isolation (e.g. database, mails)
• Can be implemented as part of automated builds
4 / 49
![Page 5: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/5.jpg)
Unit Testing with NUnit
• Unit Testing framework for all .NET languages
• Initially ported from JUnit
• Written entirely in C#
• Stand-alone tools and R# integration
5 / 49
![Page 6: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/6.jpg)
Setting up NUnit
1. Add new Class Library project to the solution.
2. Add reference to bin/framework/nunit.framework.dll.
6 / 49
![Page 7: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/7.jpg)
NUnit Test Class Design
• Public
• Default constructor
7 / 49
![Page 8: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/8.jpg)
NUnit Test Method Design
• [Test] attribute
• Return void
• No parameters
8 / 49
![Page 9: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/9.jpg)
NUnit Test Example
C#
9 / 49
namespace LevelEditor.Tests{
using LevelEditor.Model;
using NUnit.Framework;
public class MapTest{
[Test]public void TestMapConstructor(){
// Arrange.const int Width = 32;const int Height = 16;
// Act.Map map = new Map(Width, Height);
// Assert.Assert.AreEqual(Width, map.Width);Assert.AreEqual(Height, map.Height);
}}
}
![Page 10: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/10.jpg)
NUnit Assertions
• AreEqual, AreNotEqual
• AreSame, AreNotSame
• IsTrue, IsFalse
• Greater, GreaterOrEqual
• Less, LessOrEqual
• IsEmpty, IsNotEmpty
• IsNull, IsNotNull
• Contains
• Fail, Inconclusive
10 / 49
![Page 11: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/11.jpg)
Expected Exceptions
C#
11 / 49
[Test]
[ExpectedException(typeof(ArgumentOutOfRangeException))]
public void TestNegativeWidth()
{
Map map = new Map(-10, 20);
}
![Page 12: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/12.jpg)
SetUp and TearDown
C#
12 / 49
public class MapTest{
private const int Height = 16;private const int Width = 32;
private Map map;
[SetUp]public void SetUp(){
this.map = new Map(Width, Height);}
[Test]public void TestTileIndexer(){
// Arrange.const int X = 1;const int Y = 2;
// Act.var mapTile = new MapTile(X, Y, "Desert");this.map[X, Y] = mapTile;
// Assert.Assert.AreEqual(mapTile, this.map[X, Y]);
}}
![Page 13: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/13.jpg)
Hint
Override Equals in types whose objects you need to compare!
13 / 49
![Page 14: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/14.jpg)
NUnit Test Tool
14 / 49
![Page 15: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/15.jpg)
NUnit Console
Console Output
15 / 49
D:\Dev\Repositories\SAE-ToolDevelopment\Vendor\NUnit-2.6.3\bin>nunit-console-x86.exe ..\..\..\Source\LevelEditor.Tests\bin\Debug\LevelEditor.Tests.dll
NUnit-Console version 2.6.3.13283
Copyright (C) 2002-2012 Charlie Poole.
Copyright (C) 2002-2004 James W. Newkirk, Michael C. Two, Alexei A. Vorontsov.
Copyright (C) 2000-2002 Philip Craig.
All Rights Reserved.
Runtime Environment -
OS Version: Microsoft Windows NT 6.2.9200.0
CLR Version: 2.0.50727.7905 ( Net 3.5 )
ProcessModel: Default DomainUsage: Single
Execution Runtime: net-3.5
...
Tests run: 3, Errors: 0, Failures: 0, Inconclusive: 0, Time: 0.046059608766156 seconds
Not run: 0, Invalid: 0, Ignored: 0, Skipped: 0
![Page 16: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/16.jpg)
NUnit & Visual Studio
16 / 49
![Page 17: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/17.jpg)
NUnit & Visual Studio
17 / 49
![Page 18: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/18.jpg)
Advantages of Unit Testing
✓ Finds problems early
▪ Test Driven Development
✓ Facilitates changes
▪ Can be run before each commit or build
▪ In combination with source control, can identify the revision (and
originator) that broke the code
18 / 49
![Page 19: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/19.jpg)
Limits of Unit Testing
• Won’t catch every error in the program
• Won’t catch integration errors
• Combinatorial problem
▪ Every boolean decision statement requires at least two tests
• Can’t test non-deterministic or concurrency problems
19 / 49
![Page 20: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/20.jpg)
Limits of Unit Testing
• Setting up realistic and useful tests is a challenge
• Test case failures need to be reviewed daily and addressed
immediately
• Embedded system software presents a unique challenge
▪ Software is being developed on a different platform than the one
it will eventually run on
20 / 49
![Page 21: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/21.jpg)
Test Driven Development
1. Write an (initially failing) automated test case that defines a desired
improvement or new function.
2. Produce the minimum amount of code required to pass that test.
3. Refactor the new code to acceptable standards.
21 / 49
![Page 22: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/22.jpg)
Advantages of TDD
✓ Client-first development
✓ Taking small steps
✓ All written code is covered by at least one test
✓ Can lead to more modularized code
22 / 49
![Page 23: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/23.jpg)
Limits of TDD
• Support of the entire team is essential
• Test are typically created by the developer who is writing the code
being tested, and may therefore share the same blind spots with the
code
• Maintenance overhead
23 / 49
![Page 24: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/24.jpg)
Unity Test Tools
• Released by Unity in December 2013
• Partly integrated since Unity 5.3
• Completely open-source
• Available on the Asset Store and Bitbucket
• Based on NUnit
24 / 49
![Page 25: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/25.jpg)
Unity Test Tools
• Unit tests are discovered using reflections
• Can be run automatically on recompile
26 / 49
![Page 26: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/26.jpg)
Unity Test Tools
27 / 49Unity NUnit Test Runner Window
![Page 27: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/27.jpg)
Unity Test Tools
• Integration tests allow testing integration of components, game
objects and assets
• Each test suite is a separate scene containing a game object with a
TestRunner attached
• Each test is a separate game object with a TestComponent attached
▪ Everything beneath that object in the hierarchy is considered to
belong to that test
• The CallTesting behaviour can modify the test result without having
to actually write additional code
28 / 49
![Page 28: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/28.jpg)
Unity Test Tools
When you run the tests, the following steps are performed, in order:
1. Play mode is enabled.
2. The first or next test object is activated.
3. Wait until the test has finished (or a timeout has occurred).
4. The current active test gets deactivated.
5. If there are more tests in the queue, go to step 2.
6. Report results and finish test run.
29 / 49
![Page 29: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/29.jpg)
Unity Test Tools
30 / 49Unity Integration Test Scene
![Page 30: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/30.jpg)
Unity Test Tools
• Assertions check invariants – conditions you expect to be always
true
• You can specify when to check these conditions
• If any assertion fails, an exception is thrown
• You can automatically cause the game in that case by enabling
Error pause in the Unity console window
31 / 49
![Page 31: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/31.jpg)
Unity Test Tools
32 / 49
Unity Assertion Component
![Page 32: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/32.jpg)
Unity Test Tools
Just like with NUnit, Unity integration tests can be run from command
line:
"C:\Program Files (x86)\Unity\Editor\Unity.exe“
–batchmode
-projectPath D:\Temp\UnityTest
-executeMethod UnityTest.Batch.RunIntegrationTests
-testscenes=NpruehsScene
-targetPlatform=StandaloneWindows
-resultsFileDirectory=D:\Temp\Results
33 / 49
![Page 33: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/33.jpg)
NSubstitute
• Creates substitutes for interfaces
• Saves you from having to use stubs, mocks, spies, test doubles
34 / 49
public interface ICalculator{
int Add(int a, int b);}
![Page 34: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/34.jpg)
NSubstitute
• Creates substitutes for interfaces
• Saves you from having to use stubs, mocks, spies, test doubles
35 / 49
ICalculator calculator = Substitute.For<ICalculator>();
![Page 35: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/35.jpg)
NSubstitute
• Allows you set up return values to method calls.
• Great for mocking database connections or email plugins.
36 / 49
calculator.Add(1, 2).Returns(3);
![Page 36: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/36.jpg)
37 / 49
![Page 37: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/37.jpg)
Unit Testing in C++
• googletest is a platform-independent framework for writing C++ tests
• Used for a variety of Google projects, including Chromium and
Google Protocol Buffers
• googlemock allows you to write and mock C++ mock classes
38 / 49
![Page 38: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/38.jpg)
Unit Testing in C++
1. Create a new Win32 console application.
2. Add the projects you want to test to the additional include
directories.
3. Add the gtest root folder and gtest-a.b.c/include folder to the
additional include directories.
4. In your test source file, #include "src/gtest-all.cc“.
5. Provide a main entry point like this:
39 / 49
GTEST_API_ int main(int argc, char**argv){
testing::InitGoogleTest(&argc, argv);return RUN_ALL_TESTS();
}
![Page 39: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/39.jpg)
Unit Testing in C++
Test method signatures are automatically generated by the TESTmacro:
40 / 49
TEST(TestSuiteName, TestName){
// Arrange.
// Act.
// Assert.}
![Page 40: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/40.jpg)
Unit Testing in C++
Assertions are made using macros as well.
41 / 49
TEST(TestArithmetic, TestAdd){// Arrange.auto i = 3;auto j = 4;
// Act.auto k = i + j;
// Assert.EXPECT_EQ(7, k);
}
![Page 41: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/41.jpg)
Unit Testing in C++
Run the tests by just starting the console application…
42 / 49
![Page 42: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/42.jpg)
Unit Testing in C++
… or by using the UI of gtest-gbar.
43 / 49
![Page 43: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/43.jpg)
Unit Testing in C++
googlemock provides macros for mocking methods.
44 / 49
#pragma once
#include "gmock/gmock.h"
class ICalculator{public:virtual int Add(int x, int y) = 0;
};
class MockCalculator : public ICalculator{public:MOCK_METHOD2(Add, int(int x, int y));
};
![Page 44: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/44.jpg)
Unit Testing in C++
Just like NSubstitute, it allows you to specify results:
45 / 49
#include "src/gtest-all.cc"#include "src/gmock-all.cc"
#include "MockCalculator.h"
using ::testing::Return;
TEST(TestArithmetic, TestCalculator){
// Arrange.MockCalculator calculator;EXPECT_CALL(calculator, Add(3, 4))
.WillRepeatedly(Return(7));
// Act.auto sum = calculator.Add(3, 4);
// Assert.EXPECT_EQ(7, sum);
}
![Page 45: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/45.jpg)
Unit Testing in UE 4
Unreal Engine 4 defines unit tests with macros, too:
46 / 49
IMPLEMENT_SIMPLE_AUTOMATION_TEST(FSetResTest, "Windows.SetResolution", ATF_Game)bool FSetResTest::RunTest(const FString& Parameters){
FString MapName = TEXT("AutomationTest");FEngineAutomationTestUtilities::LoadMap(MapName);
int32 ResX = GSystemSettings.ResX;int32 ResY = GSystemSettings.ResY;FString RestoreResolutionString =
FString::Printf(TEXT("setres %dx%d"), ResX, ResY);
ADD_LATENT_AUTOMATION_COMMAND(FEngineWaitLatentCommand(2.0f));ADD_LATENT_AUTOMATION_COMMAND(FExecStringLatentCommand(TEXT("setres 640x480")));ADD_LATENT_AUTOMATION_COMMAND(FEngineWaitLatentCommand(2.0f));ADD_LATENT_AUTOMATION_COMMAND(FExecStringLatentCommand(RestoreResolutionString));
return true;}
![Page 46: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/46.jpg)
Unit Testing in UE 4
Tests can be run from the Unreal Frontend:
47 / 49
![Page 47: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/47.jpg)
References
• Wikipedia. Unit testing. http://en.wikipedia.org/wiki/Unit_testing, April 25, 2017.
• Poole. NUnit Quick Start. http://www.nunit.org/index.php?p=quickStart&r=2.6.4, 2015.
• Wikipedia. Test-driven development. http://en.wikipedia.org/wiki/Test-driven_development, April 21, 2017.
• Petersen. Unity Test Tools Released. http://blogs.unity3d.com/2013/12/18/unity-test-tools-released/, December 18, 2013.
• Unity Technologies. Unity Test Tools. https://bitbucket.org/Unity-Technologies/unitytesttools, March 9, 2017.
• NSubstitute. NSubstitute. http://nsubstitute.github.io/help/getting-started/, May 2017.
• Google. Google C++ Testing Framework. https://code.google.com/p/googletest/, May 19, 2017.
• Google. Google Test UI. https://code.google.com/p/gtest-gbar/, May 2, 2017.
• Epic Games. Unreal Engine Automation Technical Guide. https://docs.unrealengine.com/latest/INT/Programming/Automation/TechnicalGuide/, May 2017.
• Epic Games. Unreal Engine Automation User Guide. https://docs.unrealengine.com/latest/INT/Programming/Automation/UserGuide/, May 2017.
48 / 49
![Page 49: Game Programming 06 - Automated Testing](https://reader031.vdocuments.mx/reader031/viewer/2022031801/55a5ccb81a28ab374e8b48a6/html5/thumbnails/49.jpg)
5 Minute Review Session
• What are unit tests?
• What are the main advantages of unit testing?
• What are the most important limits of unit tests?
• Explain the process of Test Driven Development!
• What are the upsides and downsides of TDD?
• What are integration tests?
• How do you test functionality that depends on external
communication (e.g. database, mails)?