test automation 3

7 Years of Test Automation

!

@mikeb2701

Assumptions

• Testing is important

• Automating testing is important

Types of tests• Static Analysis

• Unit Testing

• Integration Testing

• Acceptance Testing!

• Performance Testing

• Testing in Live

• Exploratory Testing

The acceptance test suite as a whole both verifies that the application delivers the business value and guards against regressions or defects that break preexisting functions of the application.

— Humble, Farley, Continuous Delivery, 2010, Chapter 8.

An acceptance test may only drive and assert behaviour in the system through interfaces available to the system’s users.

Writing Acceptance Tests

@RunWith(AcceptanceTestRunner.class)@Multithreaded@AcceptanceTest(id = 29, flavour = {Flavour.PLACE_ORDER, Flavour.API})public class PlaceOrderAcceptanceTest extends DslTestCase {! @Before public void beforeEveryTest() {! adminAPI.createInstrument("name: instrument”, "orderQuantityIncrement: 0.1”,…); registrationAPI.createUser("user", "balance: 100000"); mtfMemberAPI.createMtfMemberAndFixSession("marketMaker"); publicAPI.login("user"); mtfFixAPI.login(“marketMaker"); } // end! @Test public void shouldRejectOrderIfNotLoggedIn() { publicAPI.logout(); publicAPI.placeOrder(“instrument", "side: buy", "quantity: 1.2”, "timeInForce: ImmediateOrCancel”, "unauthorised: true");!

@RunWith(AcceptanceTestRunner.class)@Multithreaded@AcceptanceTest(id = 29, flavour = {Flavour.PLACE_ORDER, Flavour.API})public class PlaceOrderAcceptanceTest extends DslTestCase {! @Before public void beforeEveryTest() {! adminAPI.createInstrument("name: instrument”, "orderQuantityIncrement: 0.1”,…); registrationAPI.createUser("user", "balance: 100000"); mtfMemberAPI.createMtfMemberAndFixSession("marketMaker"); publicAPI.login("user"); mtfFixAPI.login(“marketMaker"); } // end! @Test public void shouldRejectOrderIfNotLoggedIn() { publicAPI.logout(); publicAPI.placeOrder(“instrument", "side: buy", "quantity: 1.2”, "timeInForce: ImmediateOrCancel”, "unauthorised: true");!

@RunWith(AcceptanceTestRunner.class)@Multithreaded@AcceptanceTest(id = 29, flavour = {Flavour.PLACE_ORDER, Flavour.API})public class PlaceOrderAcceptanceTest extends DslTestCase {! @Before public void beforeEveryTest() {! adminAPI.createInstrument("name: instrument”, "orderQuantityIncrement: 0.1”,…); registrationAPI.createUser("user", "balance: 100000"); mtfMemberAPI.createMtfMemberAndFixSession("marketMaker"); publicAPI.login("user"); mtfFixAPI.login(“marketMaker"); } // end! @Test public void shouldRejectOrderIfNotLoggedIn() { publicAPI.logout(); publicAPI.placeOrder(“instrument", "side: buy", "quantity: 1.2”, "timeInForce: ImmediateOrCancel”, "unauthorised: true");!

@RunWith(AcceptanceTestRunner.class)@Multithreaded@AcceptanceTest(id = 29, flavour = {Flavour.PLACE_ORDER, Flavour.API})public class PlaceOrderAcceptanceTest extends DslTestCase {! @Before public void beforeEveryTest() {! adminAPI.createInstrument("name: instrument”, "orderQuantityIncrement: 0.1”,…); registrationAPI.createUser("user", "balance: 100000"); mtfMemberAPI.createMtfMemberAndFixSession("marketMaker"); publicAPI.login("user"); mtfFixAPI.login(“marketMaker"); } // end! @Test public void shouldRejectOrderIfNotLoggedIn() { publicAPI.logout(); publicAPI.placeOrder(“instrument", "side: buy", "quantity: 1.2”, "timeInForce: ImmediateOrCancel”, "unauthorised: true");!

@RunWith(AcceptanceTestRunner.class)@Multithreaded@AcceptanceTest(id = 29, flavour = {Flavour.PLACE_ORDER, Flavour.API})public class PlaceOrderAcceptanceTest extends DslTestCase {! @Before public void beforeEveryTest() {! adminAPI.createInstrument("name: instrument”, "orderQuantityIncrement: 0.1”,…); registrationAPI.createUser("user", "balance: 100000"); mtfMemberAPI.createMtfMemberAndFixSession("marketMaker"); publicAPI.login("user"); mtfFixAPI.login(“marketMaker"); } // end! @Test public void shouldRejectOrderIfNotLoggedIn() { publicAPI.logout(); publicAPI.placeOrder(“instrument", "side: buy", "quantity: 1.2”, "timeInForce: ImmediateOrCancel”, "unauthorised: true");!

DSL

PublicAPI

TradingUI

AdminAPI

Test

MtfMemberAPI

LoginPanelDriver

Selenium

BalancesPanelDriver

InstrumentPanelDriver

JavaAPIDriver

AdminAPIDriver

MtfFixDriver

XML/HTTP

QuickFIX

Drivers

Time Travel

ServiceService

ServiceService

Service

TimeMachine Test

Time Travel To

Time Delta Get CurrentTime

@RunWith(AcceptanceTestRunner.class)@AcceptanceTest(id = 5194, flavour = {Flavour.PLACE_ORDER, Flavour.API})@Multithreaded(tardis = true)public class CancelAndReplaceAcceptanceTest extends DslTestCase{ @Before public void setUp() { dsl.forceToUtcTimeZone(); dsl.enableTimeMachine();! dsl.createTimePoint("name: origin”, "value: <next weekday>"); dsl.createTimePoint("name: marketOpen", "value: origin plus 1 weekdayCalendarOpenOffset”);! dsl.waitUntil("marketOpen");

Pro-Tip:

Isolate as much as possible

Simple DSL

https://github.com/LMAX-Exchange/Simple-DSL

Acceptance Testing (Running)

RTAParallel

Sequential

Sequential

Sequential

RTA

RTA

RTA

Scotty RomeroServer

Jenkins

Intermittency

avp.wikia.com

Sources of Intermittency

• Bad code (under test)

• Bad test

• Environmental

Just Fix It!

Acceptance Test DB

Wrap-Up

•Write them well •Run them often •Keep them green

Thank you

Q&A