who will guard the guards themselves? how to trust your automation and avoid deceit

Who will guard the

guards themselves?

How to trust your automation and avoid deceit

Bas Dijkstra

www.ontestautomation.com

bas@ontestautomation.com

@_basdijkstra

_I build test automation solutions

_I teach test automation, service virtualization

_I write www.ontestautomation.com

TechBeacon, StickyMinds, …

Continuous Testing

DEV ??? PROD

testing testing testing

testing

“Sed quis

custodiet ipsos

custodes?”

But who will guard the guards themselves?

--- Juvenal, Satire VI, lines 347–348 ---

Confidence“Our product is

able to perform

function X”

Control Trust

“I control the

execution of X”

“I trust our

product to be able

to perform X”

I want to be confident about the ability of our

product to perform function X

Automated tests are used to build trust in our

product performing function X correctly

I need to be able to trust my automated tests

around function X

“I’m not upset that

you lied to me,

I’m upset that from

now on I can’t

believe you”

--- Friedrich Nietzsche ---

False positives

Persistent > fix it or fire it

Intermittent > ‘flaky tests’

https://youtu.be/XnkWkrbzMh0

Richard Bradshaw – Selenium Conf Berlin

False negatives

False negative: exhibit A

100%

code coverage!

False negative: exhibit B

still

If you want to trust your tests,

start building tests you can trust on

Keep your tests fresh

Do code review on your test code

Test your tests periodically

Review your suite from time to time

Don’t be afraid to throw away tests

Tools to help you

keep your tests fresh

Mutation testing

http://pitest.org/

(Java)

Property-based testing

https://github.com/pholser/junit-quickcheck

(Java)

http://blog.jessitron.com/2013/04/property-based-testing-what-is-it.html

_Email: bas@ontestautomation.com

_Blog: www.ontestautomation.com

_LinkedIn: www.linkedin.com/in/basdijkstra

_Twitter: @_basdijkstra