product risk analysis workshop
TRANSCRIPT
Contents of the Workshop
The theory of Product Risk Analysis
Workshop
Report Back
2
© Maarten Toonder Studio’s
Risk & Requirement Based Testing
Risk and requirements analysis are two, independent activities
• Risk analysis:
• Identified by “all” stakeholders
• Analysis byond defined features and functions
• Requirements analysis
• Functional
• Non-functional
• Risk & Requirements Matching leads to a set of requirements that
• Have a functional priority
• Are linked to product risks and
• Have a risk priority
4
Risk Analysis
PREPARATION EXECUTION
Logica ©
Evaluation
and transfer
Issue
management
Progress
management
Reporting
and advice
Test
organisation
Estimation
Risk
analysis and
Test strategy
Planning
Risk and
requirement
based testing
Risk
analysis and
Test Strategy
5
Project Risk versus Product Risk
11
Project risks
Project overrun in time & budget
Lack of system standards
Insufficient resources
Unclear requirements
Incomplete documentation
mitigation measures
in the test plan
Product risks
Unsatisfactory quality
Incorrect functionality
Not user-friendly
Difficult to maintain
Low efficiency
Difficult to install
Test strategy based
on product risks
Relative Priority
Compare:
Mobile phone backlight malfunction
versus
Space shuttle fuel system malfunction
13
MoSCoW: an example
If this
goes
wrong
it will
have
...
…financial consequences
for our customers.
16
MoSCoW: an example
If this
goes
wrong
it will
have
...
…financial consequences
for our customers.
All customers
17
MoSCoW: an example
If this
goes
wrong
it will
have
...
…financial consequences
for our customers.
Must test All customers
18
MoSCoW: an example
If this
goes
wrong
it will
have
...
…financial consequences
for our customers.
Must test All customers
One customer
19
MoSCoW: an example
If this
goes
wrong
it will
have
...
…financial consequences
for our customers.
Must test All customers
One customer Should test
20
MoSCoW: an example
If this
goes
wrong
it will
have
...
… no financial
consequences
for our customers
…financial consequences
for our customers.
Must test All customers
One customer Should test
21
MoSCoW: an example
If this
goes
wrong
it will
have
...
… no financial
consequences
for our customers
…financial consequences
for our customers.
Must test All customers
One customer Should test
Could test
All customers
One customer
Should test
22
MoSCoW: an example
If this
goes
wrong
it will
have
...
… no financial
consequences
for our customers
…financial consequences
for our customers.
Must test All customers
One customer Should test
… financial consequences
for our department.
No workaround
Workaround Won’t test
Could test
Could test
All customers
One customer
Should test
23
MoSCoW: an example
If this
goes
wrong
it will
have
...
… no financial
consequences
for our customers
…financial consequences
for our customers.
Must test All customers
One customer Should test
… no financial
consequences
for our department. Won’t test
No workaround
Workaround
Could test
… financial consequences
for our department.
No workaround
Workaround Won’t test
Could test
Could test
All customers
One customer
Should test
24
Stakeholders
A stakeholder is a person or organization that has a legitimate
interest in a product or service
26
Stakeholders
A stakeholder is a person or organization that has a legitimate
interest in a product or service
Who is responsible?
Who has a problem when things go wrong?
Who needs the system for normal operations?
Who will use the system?
Who sets regulatory rules?
27
Stakeholders - Examples
• Legal authorities like AFM, DNB, ECB,
Represented by Compliance & Legal department
• Helpdesk
• Operations
• Application Management
• (End-) users
• Marketing
• System development
• Testers
…
28
Stakeholders & Quality Attributes
Information
system
Development
Application Management
End user
Audit
Helpdesk
29
Stakeholders & Quality Attributes
Information
system
Development
Application Management
End user
Audit
Helpdesk
Requirements
on usability
Requirements on
changeability
Requirements on
security
Requirements on
maintainability
Requirements on
analysability
30
Quality Attributes ISO9126
Quality Attributes
ISO 9126
Functionality
Suitability
Accuracy
Interoperability
Security
Functionality
Compliance
Reliability
Maturity
Fault tolerance
Recoverability
Reliability
Compliance
Usability
Understandability
Learnability
Operability
Attractiveness
Usability
Compliance
Efficiency
Time behaviour
Resource utilisation
Efficiency
Compliance
Maintainability
Analysability
Changeability
Stability
Testability
Maintainability
Compliance
Portability
Adaptability
Installability
Co-existence
Replaceability
Portability
Compliance
31
Quality Attributes
ISO 25010
Functionality
Suitability
Functional
appropriateness
Accuracy
Reliability
Maturity
Availability
Fault tolerance
Recoverability
Performance
Efficiency
Time behaviour
Resource
utilization
Operability
Appropriateness
recognisability
Ease of use
User error
Protection
User interface
Aesthetics
Technical
Learnability
Technical
Accessibility
Security
Confidentiality
Integrity
Non–repudiation
Accountability
Authenticity
Portability
Adaptability
Installability
Replaceability
Compatibility
Co-existence
Interoperability
Maintainability
Modularity
Reusability
Analyzability
Modifiability
Testability
Quality Attributes ISO 25010
32
The domain and the question
• Financial organization selling mortgages
• Direct writer and via independent intermediates
• New mortgage proposal system
• Company Policy
• Raise of market share
• Low cost, high quality
• Turnover up 10%
• Operational excellence
The question is:
what are the risks related to an IT system
used within the mortgage domain
Stakeholders
Stakeholder Responsibility
Marketing & Sales Defines and sells mortgages
Application management Functional/technical management of the
system
Legal Represent the legal authorities
System development Develop and maintain the system
Tester Test the system
Operations Keeps the system up and running
Client Relations Represents the end-user community
PRAW Planning
Determine the type of PRA
In case of a PRAW:
• Plan the workshop • Requirements experts
• Stakeholders
• Scribe
• Collect documentation • High level definition study
• Business Requirements Specification*)
• Checklists and Rule sets • Customize to fit application type
*)Is used during matching phase, not before!
Test MAnager
Team
Scribe
Team
Team
Team
Test Manager
Elicitation
Matching
Planning
Logging
Balancing
Prioritization
Exit
PRAW Phases
37
Brown Paper Session
A Brown Paper Session is a session where experiences,
needs and ideas from different points of view are
brought together (on large sheets of brown paper).
39
“Copafijth” aspects
• Communicatie (Communication)
• Organisatie (Organisation)
• Personeel (Staff)
• Administratieve Organisatie (Administrative Organisation)
• Financieel (Financial)
• Informatie (Information)
• Juridisch (Legal)
• Techniek (Technical)
• Huisvesting (Housing/logistics)
This list will support the risk elicitation process
40
COPAFIJTH is a Dutch acronym that indicates all aspect of an organisation
MoSCoW: an example
Must test
Should test
Could test
Won’t test
Won’t test
Could test
Could test
Should test
Non-financial impact
own department
Financial impact
own department
Non-financial
customer impact
Financial
customer impact
If this
failure
occurs,
it has
...
Workaround
No workaround
Workaround
No workaround
One customer
All customers
One customer
All customers
42
The result
The result of a product risk analysis is an agreed and complete list
of product risks and their related priorities
44
The result
The result of a product risk analysis is an agreed and complete list
of product risks and their related priorities
45
But what’s the use?
First advandtage: Risk & Requirement Matching
Requirements
Product Risks Risk, no
requirement:
• Add requirement
• Leave out risk
Requirement, no
risk:
• Add risk and test
• Leave out requirement
Second advantage: Strategic Test Slicing Method
(STSM)
Won’t test
conditions:
• …
• …
Could test
conditions:
• …
• …
• …
• …
Should test
conditions:
• …
• …
• …
• …
• …
• ...
Must test
conditions:
• …
• …
• …
• …
OUCH!
Priorities in the test management process
50
Number of test cases
within available budget
Planning and
sequence
Priorities in the test management process
51
Number of test cases
within available budget
Planning and
sequence
Status in budget
and quality
Priorities in the test management process
52
Number of test cases
within available budget
Planning and
sequence
Status in budget
and quality
Which defect will
be solved and
which is postponed?
Priorities in the test management process
53
Number of test cases
within available budget
Planning and
sequence
Status in budget
and quality
Which defect will
be solved and
which is postponed?
Are all risks mitigated:
can we deploy?
Max 30 days
24 hours
Product Backlog
As prioritized by
Product Owner
Sprint Backlog
Backlog tasks
expanded
by team
Potentially Shippable
Product Increment
Daily Scrum
Meeting
Source: Adapted from Agile Software
Development with Scrum by Ken
Schwaber and Mike Beedle. Product Owner Scrum master Team member Stakeholder
Risk priorities in the development process
54
Product
Product risk
Requirement
PR
01
PR
02
PR
03
PR
04
PR
05
PR
06
PR
07
PR
08
PR
09
REQ 01 ● ● ● ●
REQ 02 ● ● ●
REQ 03 ●
REQ 04 ● ●
REQ 05 ● ●
REQ 06 ● ●
REQ 07 ● ●
REQ 08 ● ●
REQ 09 ● ●
REQ 10 ● ● ●
REQ 11 ●
REQ 12 ●
REQ 13 ● ● ●
REQ 14 ● ● ●
REQ 15 ● ●
Risk Matrix M
ust
Have
Sh
ou
ld H
ave
Wo
uld
Have
Must Test ShouldTest WouldTest
Risk Matrix
Product risk
Requirement
PR
01
PR
02
PR
03
PR
04
PR
05
PR
06
PR
07
PR
08
PR
09
REQ 01 ? ? ? ?
REQ 02 ? ? ?
REQ 03 ?
REQ 04 ? ?
REQ 05 ? ?
REQ 06 ? ?
REQ 07 ? ?
REQ 08 ? ?
REQ 09 ? ?
REQ 10 ? ? ?
REQ 11 ?
REQ 12 ?
REQ 13 ? ? ?
REQ 14 ? ? ?
REQ 15 ? ?
Mu
st
Have
Sh
ou
ld H
ave
Wo
uld
Have
Must Test ShouldTest WouldTest
Risk Matrix
Product risk
Requirement
PR
01
PR
02
PR
03
PR
04
PR
05
PR
06
PR
07
PR
08
PR
09
REQ 01 ? ? ? ?
REQ 02 ? ? ?
REQ 03 ?
REQ 04 ? ?
REQ 05 ? ?
REQ 06 ? ?
REQ 07 ? ?
REQ 08 ? ?
REQ 09 ? ?
REQ 10 ? ? ?
REQ 11 ?
REQ 12 ?
REQ 13 ? ? ?
REQ 14 ? ? ?
REQ 15 ? ?
Product risk
Requirement
PR
X
PR
X
PR
X
REQ X H M M
REQ X H M L
REQ X M M L
Mu
st
Have
Sh
ou
ld H
ave
Wo
uld
Have
Must Test ShouldTest WouldTest
Risk Matrix
Product risk
Requirement
PR
01
PR
02
PR
03
PR
04
PR
05
PR
06
PR
07
PR
08
PR
09
REQ 01 ? ? ? ?
REQ 02 ? ? ?
REQ 03 ?
REQ 04 ? ?
REQ 05 ? ?
REQ 06 ? ?
REQ 07 ? ?
REQ 08 ? ?
REQ 09 ? ?
REQ 10 ? ? ?
REQ 11 ?
REQ 12 ?
REQ 13 ? ? ?
REQ 14 ? ? ?
REQ 15 ? ?
Mu
st
Have
Sh
ou
ld H
ave
Wo
uld
Have
Must Test ShouldTest WouldTest
Risk Matrix
Product risk
Requirement
PR
01
PR
02
PR
03
PR
04
PR
05
PR
06
PR
07
PR
08
PR
09
REQ 01 ? ? ? ?
REQ 02 ? ? ?
REQ 03 ?
REQ 04 ? ?
REQ 05 ? ?
REQ 06 ? ?
REQ 07 ? ?
REQ 08 ? ?
REQ 09 ? ?
REQ 10 ? ? ?
REQ 11 ?
REQ 12 ?
REQ 13 ? ? ?
REQ 14 ? ? ?
REQ 15 ? ?
M
Mu
st
Have
Sh
ou
ld H
ave
Wo
uld
Have
Must Test ShouldTest WouldTest
Risk Matrix
Product risk
Requirement
PR
01
PR
02
PR
03
PR
04
PR
05
PR
06
PR
07
PR
08
PR
09
REQ 01 ? ? ? ?
REQ 02 ? ? ?
REQ 03 ?
REQ 04 ? ?
REQ 05 ? ?
REQ 06 ? ?
REQ 07 ? ?
REQ 08 ? ?
REQ 09 ? ?
REQ 10 ? ? ?
REQ 11 ?
REQ 12 ?
REQ 13 ? ? ?
REQ 14 ? ? ?
REQ 15 ? ?
M M
Mu
st
Have
Sh
ou
ld H
ave
Wo
uld
Have
Must Test ShouldTest WouldTest
Risk Matrix
Product risk
Requirement
PR
01
PR
02
PR
03
PR
04
PR
05
PR
06
PR
07
PR
08
PR
09
REQ 01 ? ? ? ?
REQ 02 ? ? ?
REQ 03 ?
REQ 04 ? ?
REQ 05 ? ?
REQ 06 ? ?
REQ 07 ? ?
REQ 08 ? ?
REQ 09 ? ?
REQ 10 ? ? ?
REQ 11 ?
REQ 12 ?
REQ 13 ? ? ?
REQ 14 ? ? ?
REQ 15 ? ?
H M M
Mu
st
Have
Sh
ou
ld H
ave
Wo
uld
Have
Must Test ShouldTest WouldTest
Risk Matrix
Product risk
Requirement
PR
01
PR
02
PR
03
PR
04
PR
05
PR
06
PR
07
PR
08
PR
09
REQ 01 ? ? ? ?
REQ 02 ? ? ?
REQ 03 ?
REQ 04 ? ?
REQ 05 ? ?
REQ 06 ? ?
REQ 07 ? ?
REQ 08 ? ?
REQ 09 ? ?
REQ 10 ? ? ?
REQ 11 ?
REQ 12 ?
REQ 13 ? ? ?
REQ 14 ? ? ?
REQ 15 ? ?
H M M
H
Mu
st
Have
Sh
ou
ld H
ave
Wo
uld
Have
Must Test ShouldTest WouldTest
Risk Matrix
Product risk
Requirement
PR
01
PR
02
PR
03
PR
04
PR
05
PR
06
PR
07
PR
08
PR
09
REQ 01 ? ? ? ?
REQ 02 ? ? ?
REQ 03 ?
REQ 04 ? ?
REQ 05 ? ?
REQ 06 ? ?
REQ 07 ? ?
REQ 08 ? ?
REQ 09 ? ?
REQ 10 ? ? ?
REQ 11 ?
REQ 12 ?
REQ 13 ? ? ?
REQ 14 ? ? ?
REQ 15 ? ?
H M M
H
H
Mu
st
Have
Sh
ou
ld H
ave
Wo
uld
Have
Must Test ShouldTest WouldTest
Risk Matrix
Product risk
Requirement
PR
01
PR
02
PR
03
PR
04
PR
05
PR
06
PR
07
PR
08
PR
09
REQ 01 ? ? ? ?
REQ 02 ? ? ?
REQ 03 ?
REQ 04 ? ?
REQ 05 ? ?
REQ 06 ? ?
REQ 07 ? ?
REQ 08 ? ?
REQ 09 ? ?
REQ 10 ? ? ?
REQ 11 ?
REQ 12 ?
REQ 13 ? ? ?
REQ 14 ? ? ?
REQ 15 ? ?
H M M
H
H
H
Mu
st
Have
Sh
ou
ld H
ave
Wo
uld
Have
Must Test ShouldTest WouldTest
Risk Matrix
Product risk
Requirement
PR
01
PR
02
PR
03
PR
04
PR
05
PR
06
PR
07
PR
08
PR
09
REQ 01 ? ? ? ?
REQ 02 ? ? ?
REQ 03 ?
REQ 04 ? ?
REQ 05 ? ?
REQ 06 ? ?
REQ 07 ? ?
REQ 08 ? ?
REQ 09 ? ?
REQ 10 ? ? ?
REQ 11 ?
REQ 12 ?
REQ 13 ? ? ?
REQ 14 ? ? ?
REQ 15 ? ?
H M M
H
H
H
M H M M
M M H
M
M M
H M
M H
M M
H L
M L
M M M
H
L
M L L
L L
M L
Mu
st
Have
Sh
ou
ld H
ave
Wo
uld
Have
Must Test ShouldTest WouldTest
Selecting from the product backlog
During the planning poker session the risk priorities are taken into
account
• Risks relate to story points
• Combining user stories efficiently
Sprint 1 Sprint 2 Sprint 3
Selecting from the product backlog
During the planning poker session the risk priorities are taken into
account
• Risks relate to story points
• Combining user stories efficiently
Sprint 1 Sprint 2 Sprint 3
Selecting from the product backlog
During the planning poker session the risk priorities are taken into
account
• Risks relate to story points
• Combining user stories efficiently
Sprint 1 Sprint 2 Sprint 3
Selecting from the product backlog
During the planning poker session the risk priorities are taken into
account
• Risks relate to story points
• Combining user stories efficiently
Sprint 1 Sprint 2 Sprint 3
Risk & Requirement Based Planning
Product risk
Requirement
PR
01
PR
02
PR
03
PR
04
PR
05
PR
06
PR
07
PR
08
PR
09
REQ 05
REQ 12
REQ 13
REQ 03 M
REQ 04 M M
REQ 08 H L
REQ 02 M M H
REQ 06 M H
REQ 07 M M
REQ 09 M L
REQ 01 M H M M
REQ 10 M M M
REQ 11 H
REQ 14 L L
REQ 15 M L
Must Test ShouldTest WouldTest
Mu
st
Have
Sh
ou
ld H
ave
Wo
uld
Have
Risk & Requirement Based Planning
Product risk
Requirement
PR
01
PR
02
PR
03
PR
04
PR
05
PR
06
PR
07
PR
08
PR
09
REQ 05
REQ 12
REQ 13
REQ 03
REQ 04
REQ 08
REQ 02 M M H
REQ 06 M H
REQ 07 M M
REQ 09 M L
REQ 01 M H M M
REQ 10 M M M
REQ 11 H
REQ 14 L L
REQ 15 M L
Must Test ShouldTest WouldTest
Mu
st
Have
Sh
ou
ld H
ave
Wo
uld
Have
Risk & Requirement Based Planning
Product risk
Requirement
PR
01
PR
02
PR
03
PR
04
PR
05
PR
06
PR
07
PR
08
PR
09
REQ 05
REQ 12
REQ 13
REQ 03
REQ 04
REQ 08
REQ 02
REQ 06
REQ 07
REQ 09
REQ 01 M H M M
REQ 10 M M M
REQ 11 H
REQ 14 L L
REQ 15 M L
Must Test ShouldTest WouldTest
Mu
st
Have
Sh
ou
ld H
ave
Wo
uld
Have
Risk & Requirement Based Planning
Product risk
Requirement
PR
01
PR
02
PR
03
PR
04
PR
05
PR
06
PR
07
PR
08
PR
09
REQ 05
REQ 12
REQ 13
REQ 03
REQ 04
REQ 08
REQ 02
REQ 06
REQ 07
REQ 09
REQ 01
REQ 10
REQ 11
REQ 14 L L
REQ 15 M L
Must Test ShouldTest WouldTest
Mu
st
Have
Sh
ou
ld H
ave
Wo
uld
Have
Risk & Requirement Based Planning
Product risk
Requirement
PR
01
PR
02
PR
03
PR
04
PR
05
PR
06
PR
07
PR
08
PR
09
REQ 05
REQ 12
REQ 13
REQ 03
REQ 04
REQ 08
REQ 02
REQ 06
REQ 07
REQ 09
REQ 01
REQ 10
REQ 11
REQ 14
REQ 15
Must Test ShouldTest WouldTest
Mu
st
Have
Sh
ou
ld H
ave
Wo
uld
Have
Risk & Requirement Based Reporting
Product risk
Requirement
PR
01
PR
02
PR
03
PR
04
PR
05
PR
06
PR
07
PR
08
PR
09
Product risk
Requirement
REQ 05
REQ 12
REQ 13
REQ 03
REQ 04
REQ 08
REQ 02
REQ 06
REQ 07
REQ 09
REQ 01
REQ 10
REQ 11
REQ 14
REQ 15
So, when do we use the PRA?
During the test process
• Planning/estimation
• Traceability
• Reporting
• Go/No-go decision
Within the (agile) development proces
• Planning/estimation
• Rankschikking of the backlog
• Progress
• Rapportage
76
Product risk
Requirement
Test condition
Test case
Defect