a test pyramid heresy - a fresh look at test automation strategies

@wakaleo 1

A fresh look at test automation strategiesThe Testing Pyramid Heresy

John Ferguson Smart http://johnfergusonsmart.com

http://johnfergusonsmart.com

@wakaleo

John Ferguson

Smart

John Ferguson Smart

“I help teams of smart people learn to work together more efficiently,

to deliver better software sooner”

2

https://johnfergusonsmart.com



“Test Automation is like any other tool It’s either a benefit or a hazard”

3

@wakaleo


How much are your tests worth?

4

Weekly cost vs

weekly savings$0

$20,000

$40,000

$60,000

$80,000

$100,000

$120,000

$140,000

$160,000

$180,000

1/09/2015 1/10/2015 1/11/2015 1/12/2015

Test0suite0cost0vs0savings0(per0week)

Cost0saved0par0week Test0suite0cost0per0week

Team cost is constant

Time saved increases with the number of executed tests

@wakaleo


How much are your tests worth?

5

$0

$200,000

$400,000

$600,000

$800,000

$1,000,000

$1,200,000

1/09/2015 1/10/2015 1/11/2015 1/12/2015

What0is0your0test0suite0worth?

Test0suite0ROI Test0suite0cost

Cumulated cost vs

cumulated savings

Cumulated value should increase faster than cumulated cost

@wakaleo


Now factor in high maintenance

6

20-40%Typical maintenance overhead for poorly-written test suites

@wakaleo



7

Number of tests

Main

tena

nce

over

head

@wakaleo


$0$50,000

$100,000$150,000$200,000$250,000$300,000$350,000$400,000$450,000$500,000

1/09/2015 1/10/2015 1/11/2015 1/12/2015 1/01/2016

What0is0your0test0suite0worth?

Test0suite0cost Test0suite0ROI


8

High-maintenance costs reduce the number of tests a team can write

High maintenance costs can cause your test suite to lose value

@wakaleo


Enter the Testing Pyramid

9

@wakaleo


…or the Testing Ice-cream cone

10

@wakaleo


In other words…

11

Don’t use a web test where a service or unit test could do the same job.

13

“All models are wrong, but some are useful.”

- George Box

15

The Testing Pyramid is about how a test is implemented.

But is this the best way to classify our tests?

@wakaleo

Developer TestsQA TestsBusiness Tests

Tests come from different sources

16

@wakaleo



17

@wakaleo



18

@wakaleo



19

@wakaleo

DemonstrateDescribeDiscover

Tests have different roles

20

@wakaleo



21

@wakaleo


Test (activities) that discover

22

Example Mapping

Impact Mapping

Feature Mapping

@wakaleo



23

@wakaleo


Test (activities) that describe

24

@wakaleo


Test (activities) that describe

25

@wakaleo



26

@wakaleo


Example and Feature Mapping


27

Dis

cove

r

Describe

Demonstrate

Impact Mapping

Executable specifications

Performance testsFunctional tests

Unit tests

Regression tests

28

A new testing quadrant

29


30


31


32


33

Goldilocks Test Suites

@wakaleo 34

Questions?

John Ferguson Smart http://johnfergusonsmart.com

http://johnfergusonsmart.com

a test pyramid heresy - a fresh look at test automation strategies

Technology