Beyond Agile Testing to Lean Development

James O. Coplien Gertrud & Cope

Processes and Tools over

Individuals and Interactions

Usability Testing

System Testing

Exploratory Testing

Test Testing

Why doesn’t unit testing work?

We can’t test things of business value

Not knowing what to test, we test many things that don’t need testing

Automated crap is still crap

Unit tests can’t test things of business value

Can you demonstrate that Method X of Object Y will ever be invoked in practice?

If a unit test fails, how does it affect ROI?

What’s amazing is that we continue to test… Paths that will never be exercised Methods that are never used Naive method invocation sequences

Most unit tests are unnecessary

We test scenarios that will never be exercised in the real world

We test methods that will never be invoked

We supply data that a real application would never have to process

Automated crap is still crap

Design requires intelligence: intelligence cannot be automated

Autonomation instead of automation

Testing is about exploratory inference

Get manual procedures working first and then automate the mundane parts

Automated testing is not testing

Nokia Test Score 2: Testing

No dedicated QA - 0

Unit tested - 1

Feature tested - 5

Feature tested as soon as completed - 7

Software passes acceptance testing - 8

Software is deployed - 10

Weinberg’s Law of Decomposition

Weinberg’s Law of Decomposition

Weinberg’s Law of Decomposition

Weinberg’s Law of Decomposition

Used

Int

erfa

ce

Used Interface

Weinberg’s Law of Decomposition

Used

Int

erfa

ce

Used Interface

Must unit-test!

Must unit-test! Must

unit-

test!

Weinberg’s Law of Decomposition

Weinberg’s Law of Decomposition

Wasted testing effort

Weinberg’s Law of Decomposition

Test Size

Wasted testing effort

Weinberg’s Law of Decomposition

Use Size

Test Size

Wasted testing effort

Part III: You Will Have Bugs!

Scenarios we test in the lab

Scenarios exhibiting

faults

Scenarios invoked in the field

Testing waste

Scenarios we test in the lab

Scenarios exhibiting

faults

Scenarios invoked in the field

Testing waste

Scenarios we test in the lab

Scenarios exhibiting

faults

Scenarios invoked in the field

Type 1 Muda

Type 1: Defects

Testing waste

Scenarios we test in the lab

Scenarios exhibiting

faults

Scenarios invoked in the field

Type 2 Muda

Type 1 Muda

Type 1: Defects

Type 2: Stuff customers

don’t ask for

Testing waste

Scenarios we test in the lab

Scenarios exhibiting

faults

Scenarios invoked in the field

Type 2 Muda

Type 1 Muda

Type 4 Muda

Type 1: Defects

Type 2: Stuff customers

don’t ask for

Type 4: Overprocessing

Testing waste

Scenarios exhibiting

faults

Scenarios invoked in the field

Type 1 Muda

Type 4 Muda

Type 1: Defects

Type 2: Stuff customers

don’t ask for

Type 4: OverprocessingType

1 Muda (field)

Testing waste

Scenarios exhibiting

faults

Scenarios invoked in the field

Type 1 Muda

Type 4 Muda

Type 1: Defects

Type 2: Stuff customers

don’t ask for

Type 4: OverprocessingType

1 Muda (field)

Removing tests leaves field faults

undetected!

Why do we unit test?

Because we can

It doesn’t require teamwork

We don’t have to talk with the business

We get to interact with tools instead of people

Our decisions are emotional, not rational

Reduction of WasteType 1: Better testing (e.g., going beyond contextual testing to exploratory & experience-based testing) (and better yet, better analysis and design, where the real payoffs are)

Type 2: Don’t test dead code

Type 4: Remove redundant and tautalogical tests, and tests that give no information

Type 1 in the field: Ship your tests!

Don’t take the tests out when you ship!Remember, you will have bugs in the field!

Deployment is your largest test driver

Ship your tests as part of the code

IN case of a failure, it saves you from Type 9 muda (unsafe execution)

Be Customer-Focused

Assertions and Design by Contract!

How Adding ASSERTS reduces work

0

500000

1000000

1500000

2000000

2500000

0 1 2 5 10 20 50 100

15 bugs p=0.01 15 bugs p=0.0210 bugs p=0.01 10 bugs p=0.025 bugs p=0.01 5 bugs p=0.02

number of ASSERTS

Wor

k, T

sing

lech

ecks

Overload Magazine,

October 2014

Defensive Programming

… is largely the incorporation of tests in the code

Long known for high-reliability systems

The value of asserting the obvious

Design by Contract

Assertions make code readable!

- (void) prepareForSegue: (UIStoryboardSegue*) segue sender: (id) sender { assert(segue != nil); if ([[segue identifier] isEqualToString: @"SegueAvatarPickerToPickOneLoginPlayer"]) { assert((UI_USER_INTERFACE_IDIOM() == UIUserInterfaceIdiomPhone)); // The field parentViewController_ (instance variable) is // maybe null if we are in iPhone mode and doing a real // segue, instead of a popover as we do on the iPad. So // get the right object from the segue record. KnowsyPlayerPickerCommonViewController *parentViewController = [segue destinationViewController]; [parentViewController setKnowsyData: knowsyData_]; } else { // iPad - no special processing - should be no popover assert(parentViewController_ == nil); } }

Part IV: Beyond agile testing, Lean testingWhen a test finds a bug, it’s about fixing the process — not the bug

The long-term payoff is higher!

No bug-tracking systems: we fix problems NOW.

The tester should be the team’s conscience

Other Lean-isms

Deming: Quality is inversely proportional to the number of testers

GM kept around its cars for 1 - 6 weeks for test and repair: Toyota drove them off the end of the assembly line onto the car carriers.

Conclusion

Be humble about what tests can accomplish

It’s not about testing, but value

Design tests to assess properties of business value

Go beyond unit testing to defensive programming and design-by-contract

As agile testers, focus on process first and product second