gerlof hoekstra - omg what have we done - eurostar 2013
DESCRIPTION
EuroSTAR Software Testing Conference 2013 presentation on "OMG What Have We Done" by Gerlof Hoekstra. See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/TRANSCRIPT
![Page 1: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/1.jpg)
Gerlof Hoekstra, Atos
OMG, What Have We Done !?
www.eurostarconferences.com
@esconfs#esconfs
![Page 2: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/2.jpg)
Gerlof Hoekstra (NL) Started as software engineer Early 90’s: Testing!
Sr Test Consultant/-Manager at Atos
From software testing to large scale integration testing◦ Complex business processes◦ Many stakeholders◦ Many (software) suppliers
![Page 3: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/3.jpg)
I had a dream…
Implemented standard test procedures
10 years later: everything still the same
Dream came true: but does this make me happy?
![Page 4: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/4.jpg)
Assembling my E2E test team
Certified test professionals or ….?
What has happened?
Somewhere, something must have gone wrong….
![Page 5: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/5.jpg)
The Test Plan Independent Testing Test Reporting & The Go / No Go Advice
Entry/exit criteria
Defect databases
Defect handling workflows
Test case specifications
Defect severity definitions
…
…
![Page 6: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/6.jpg)
Raise a
defectnew Open Assigned
Accept
defectanalyse Y fixed Deployedfix defect
in progress
Deliver fix
to test
Retest
defectRetest OK Y Validated Closed
PostponedOn hold, postpone defect N
RejectedN Accept
rejectionY
N
![Page 7: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/7.jpg)
The creation of the test plan involves the following activities:1. Establishing the assignment2. Understanding the assignment3. Determining the test basis4. Analysing the product risks5. Determining the test strategy6. Estimating the effort7. Determining the planning8. Allocating test units and test techniques9. Defining the test products10.Defining the organisation11.Defining the infrastructure12.Organising the management13.Determining the test project risks and countermeasures14.Feedback and consolidation of the plan
![Page 8: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/8.jpg)
The creation of the test plan involves the following activities:1. Establishing the assignment2. Understanding the assignment3. Determining the test basis4. Analysing the product risks5. Determining the test strategy6. Estimating the effort7. Determining the planning8. Allocating test units and test techniques9. Defining the test products10.Defining the organisation11.Defining the infrastructure12.Organising the management13.Determining the test project risks and countermeasures14.Feedbackand consolidation of the plan
Now look at this !?!
![Page 9: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/9.jpg)
Shall we strip the Test Plan to the bone and never write those big, boring and useless documents anymore?
Shall we strip the test planning process to the bone and neverdo it alone at the office but always in heavy interaction with our stakeholders?
![Page 10: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/10.jpg)
“Testing requires skills that developers generally don’t have”
“Testing should be done by independent professionals, specialists”
“You can’t let a butcher test his own meat”
Really??
Testing is part of our life …
![Page 11: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/11.jpg)
“We strictly separate development and testing…”
“We pay the developing party a fixed price for the product they deliver…”
“We pay the testing provider a fixed price for each defect they discover…”
“But we subtract that from the fee for the developing party!”
![Page 12: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/12.jpg)
Nail down entry/exit & acceptance criteria
Hire independent test supplier for acceptance testing, preferably fixed price based on:◦ function points◦ number of test cases◦ number of requirements◦ ---
Test supplier executes acceptance tests
The customer accepts
![Page 13: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/13.jpg)
Is this the way to get a product accepted by its stakeholders?
Don’t we reduce testing to just a series of contractual checks?
Does this customer realize what he is (not) getting?
If we need to ‘dig somewhat deeper’: how about the contract?
Shouldn’t we (the test providers) at least warn our customer?
![Page 14: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/14.jpg)
Numbers/Graphs/KPI’s
# test cases executed
# test cases passed
$ test cases failed
# defects found/fixed/open
Coverage %
Defect density
Defect Detection %
Lists of open defects
Traffic lights
GO/NO GO ADVICE
![Page 15: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/15.jpg)
Does this mean:
we have a quality product now?
the product complies to the design?
we have a usable product?
We have reached 95% multiple condition
coverage
What on earth does that mean??
Must be good, 95% is almost everything
Multiple condition coverage
![Page 16: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/16.jpg)
If my customer asks me (the test manager) for a go/no go advice, I consider that as a personal failure. I did not properly do my job!
If I did, what would my advice be worth anyway?
![Page 17: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/17.jpg)
We covered 95% of the area
We found 60 mines
We disarmed 58 mines
The 2 mines we did not disarm are here
These are MSM MKII mines containing 4Kg of Composition B explosives
IS THIS USEFUL INFORMATION ?
WOULD YOU APPRECIATE MY GO/NO GO ADVISE?
![Page 18: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/18.jpg)
![Page 19: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/19.jpg)
Do you consider your ‘mature’ test process as a bless or a curse?
Shouldn’t I complain and just be glad with what we have accomplished?
![Page 20: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/20.jpg)
Consider you own test plan template. If you would strip it to the bone:◦ What parts would you immediately throw out?◦ What should definitely be in there?
Consider your own test planning process. How could you make it◦ Lighter?◦ More effective?◦ More interactive?◦ Less boring?
![Page 21: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/21.jpg)
To test suppliers◦ (Ethical question): Do you join the rat race of test commoditization/
should we warn customers for the consequences of excessive separation of testing? How can we?
To customers◦ Does your purchase department really know what they are doing when
they define all those fancy KPI’s.
◦ How can you build positive incentives into test SLA’s to stimulate testers to use their brains in stead of just following procedures and recipes?
![Page 22: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/22.jpg)
Suggestions for alternative ways of reporting?
Can you imagine why I never use the traffic lights symbols in test reporting?
What would be a good alternative?
![Page 23: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/23.jpg)
![Page 24: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/24.jpg)
![Page 25: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/25.jpg)
![Page 26: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/26.jpg)
![Page 27: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/27.jpg)
![Page 28: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/28.jpg)
![Page 29: Gerlof Hoekstra - OMG What Have We Done - EuroSTAR 2013](https://reader033.vdocuments.mx/reader033/viewer/2022051516/559ec3241a28ab59038b47b5/html5/thumbnails/29.jpg)