finally… reliable software!
TRANSCRIPT
Finally… Reliable
Software!
Bryan Bakker
Eindhoven 2015
@Bryan_Bakker
Contents
Intro
Reliability
Four step model
Different steps in detail
Conclusion
© Sioux 2015 | Confidential | 2
Thanks
Thanks to:
Rob de Bie
René van den Eertwegh
Peter Wijnhoven
© Sioux 2015 | Confidential | 3
About Bryan Bakker
Test Expert
Certifications: ISTQB, TMap, Prince2
Member of ISTQB Expert Level on Test Automation
Tutor of several test related courses
Domains: medical systems, professional security
systems, semi-industry, electron microscopy
Specialties: test automation, integration testing, design
for testability, reliability testing
© Sioux 2015 | Confidential | 4
About Sioux
HERENTALS NUENEN
EINDHOVEN
UTRECHT MOSCOW
© Sioux 2015 | Confidential | 5
DANANG
Did you ever experience unreliable software?
Reliability
© Sioux 2015 | Confidential | 6
Reliability
What is reliability?
“Software Reliability is the probability of failure-
free software operation in a specified
environment for a specified period of time.” IEEE 729
In short:
Something can be functional correct
But is it reliable? © Sioux 2015 | Confidential | 7
Reliability
What is reliability?
“Software Reliability is the probability of failure-
free software operation in a specified
environment for a specified period of time.” IEEE 729
In short:
Something can be functional correct
But is it reliable? How reliable is it?
© Sioux 2015 | Confidential | 8
Four steps
© Sioux 2015 | Confidential | 9
Example: Security and Surveillance System
Cameras
Recording
Event Handling
Define user domain reliability targets:
Define customer profiles
Identify operation modes and critical
functions
Determine reliability requirements per
operation mode / critical function
First step
© Sioux 2015 | Confidential | 10
Define customer profiles
ATM security
First step
© Sioux 2015 | Confidential | 11
Define customer profiles
ATM security
Parking lot surveillance
First step
© Sioux 2015 | Confidential | 12
Define customer profiles
ATM security
Parking lot surveillance
Airport surveillance
First step
© Sioux 2015 | Confidential | 13
Identify operation modes and critical
functions
Recording mode
Playback mode
Auto-start (critical)
Out-of-the-box
User triggered
Software reset
First step
© Sioux 2015 | Confidential | 14
Determine reliability requirements per
operation mode / critical function
Segment between 0.5s and 2s missed
failure rate ≤1x per day
Playback command does not function as expected
failure rate ≤ 1x per hour of viewing
Not auto-started failure rate ≤ 3 * 10-7
failures/restart
First step
© Sioux 2015 | Confidential | 15
Define operational profile for
user functions
Decompose software reliability targets
Decompose operational profile with
probabilities per component
Second step
© Sioux 2015 | Confidential | 16
How will the product be used by
customers?
Focus development resources based
on expected usage
Operational profile: a quantitative
characterization of how a system will be used
Developed by John Musa
To be used for development and test
activities
Also very useful for reliability
Operational profile
© Sioux 2015 | Confidential | 17
Critical functions can be missed in
operational profiles.
Treat them separately!
What about?
© Sioux 2015 | Confidential | 18
Define operational profile for
user functions
Second step
© Sioux 2015 | Confidential | 19
Decompose software reliability targets
Not auto-started failure rate <= 3 * 10-7
failures/restart
Stream Handler contribution: 20%
“Target” = 0.6 * 10-7 failures / restart
Second step
© Sioux 2015 | Confidential | 20
Focus on Stream
Handler
component
Decompose operational profile with
probabilities per component
Second step
© Sioux 2015 | Confidential | 21
Playback function Occurrence Probability % Stream Handler commands
Play / Pause (toggle) 48 out of 100 48% Start / Stop Playback
Fast forward 20 out of 100 20% Start Playback
Fast reverse 20 out of 100 20% Start Playback
Setup playback windows
10 out of 100 10% Stop / Start Playback
Search and select event for playback
2 out of 100 2% Start Playback with time stamp = time of selected event
Define the engineering processes
Process steps to prevent reliability
faults
Process steps to detect reliability faults
Design choices to minimize effects of faults
Next steps
© Sioux 2015 | Confidential | 22
Measure software reliability growth
Design and execute reliability tests
Based on the operational profiles
Randomly execute test set according to the
operation profile
Extra focus on critical functions
Visualize reliability growths
Next steps
© Sioux 2015 | Confidential | 23
Reliability growth curve
Next steps
© Sioux 2015 | Confidential | 24
Conclusion
Reliability is not binary but a characteristic
that can be measured
Based on theory of John Musa
Practical 4 step approach
Reliability is not reached by accident
For a full description and worked out case
study, see next page…
© Sioux 2015 | Confidential | 25
Questions
© Sioux 2015 | Confidential | 26
Published February 2015:
www.amazon.com
ISBN: 978-1499226669
Rob de Bie
Bryan Bakker
René van den Eertwegh
Peter Wijnhoven
Backup slides
© Sioux 2015 | Confidential | 27
Reliability Curves
Bathtub curve Hardware Reliability
Sawtooth curve Software Reliability
© Sioux 2015 | Confidential | 28
