using jira for risk based testing - qasymphony webinar
TRANSCRIPT
September 25, 2015
Introduction to QASymphony for [INSERT COMPANY NAME]
#AgileTransformation
Agile Transformation: People, Process and Tools to Make
Your Transformation SuccessfulUsing JIRA for Risk Based Testing
#RiskBasedTesting
#JIRATesting @RyanYackAttackTest Case Management | Exploratory Testing | Test Analytics | BDD and
more...
ABOUT QASYMPHONYThe #1 Enterprise Testing Platform for
#AgileTransformation
OUR PRESENTER
Ryan YackelSenior Product Engineer
QASymphony@ryanyackattack
#RiskBasedTesting
• Serves as the “customer voice” for all QASymphony products.
• Comes to QASymphony from Macy’s, where he managed testing on large enterprise initiatives.
• Certified scrum master from Scrum Alliance and holds a Bachelor of Arts degree from Covenant College.
• Has a newborn baby, and doesn’t sleep a ton :-)
The Test Everything Mentality
Traditional vs Risk Based Testing
How can JIRA be used to track risk
Use Case of JIRA + qTest Products
Recommended Add-ons to try out
WHY ARE WE HERE?
4 #RiskBasedTesting
#AgileTransformation
SOFTWARE TESTING IN A RISKY BUSINESS
5 #RiskBasedTesting
#AgileTransformation
PRESSURE IS ON TEST TEAMS
6 #RiskBasedTesting
What’s the plan to test X new requirements?
What’s the plan to test this one requirement,
that affects 50 applications?
I guess I could create X tests, multiplied by
the 100 configurations
I guess I could create 1,000
tests, multiplied by
the 50 configurations
#AgileTransformation
TIME IS OF THE ESSENCE
7 #RiskBasedTesting
#AgileTransformation
TEST EVERYTHING
88 #RiskBasedTesting
#AgileTransformation
WHAT DO I TEST?
99 #RiskBasedTesting
Do we need to test everything?....If not, what can I cut from my test plans?
When should my testing be complete?
How do I get team convinced that we are limited in test?
Where am I going to get all my resources for this?
Deadlines
Limited Testers
Environment Availability
Data Creation for Test
#AgileTransformation
TRADITIONAL vs RISK BASED TESTING
1010 #RiskBasedTesting
TRAD
ITIO
NAL
RISK
BAS
ED
TEST
ING
September 25, 2015
Introduction to QASymphony for [INSERT COMPANY NAME]
#AgileTransformation
Agile Transformation: People, Process and Tools to Make
Your Transformation Successful
11
RISK BASED TESTING (RBT)
#RiskBasedTesting
#AgileTransformation
WHAT IS RISK?
12 #RiskBasedTesting
Risk = Damage * Frequency
Risk = Impact * Probability of Failure
A Risk is a problem or situation that has not happened yet and it may never happen in future as well. It is a possible issue or problem that
may occur.DEFINITIONS
Michael Scott poorly shoots a three pointer (Impact) * Everytime he gets the ball (Frequency) = EXTREMELY HIGH Risk
#AgileTransformation
TWO TYPES OF RISK ANALYSIS
1313
Statistical Risk Analysisa) Weighted Metrics b) Color Coding c) P1 - P4 Areas
Heuristic Risk Analysisd) Consider more
possibilities and interesting aspects of the problem
e) Prioritize a List of Risks
Distributed
Popular
NewComplex
Down Stream
Strategic
#RiskBasedTesting
(Daniele Gagliardi – eclipsecon Europe 2015)
#AgileTransformation
STATISTICAL RISK ANALYSIS
1414 #RiskBasedTesting
(Sticky Minds Risk-Based Testing: Test Only What Matters)
Marginal impact—1 Moderate impact—2 Medium impact—3 High impact—4 Critical impact—5
A cosmetic error. The progress was disrupted with manageable extensions to the short-term schedule and cost; customers were not affected
The progress was disrupted with large extensions to project schedule and cost; customers were not directly affected
An application or transaction caused a significant disruption to production
The problem caused all major and critical functioning of the application to fail. This resulted in a large loss of revenue.
Marginal—1 Moderate—2 Medium—3 High—4 Critical—5
Low complex code written on low complicated system architectural design
Moderate complex code written on moderate complicated system architectural design
Medium complex codes written on medium complicated system architectural design
There is a large amount of complex code written on a highly complicated system architectural design
This suggests extreme complex code that has been written on top of on extremely complicated architectural design
IMPACT
PROBABILITY OF FAILURE
#AgileTransformation
STATISTICAL RISK ANALYSIS
1515 #RiskBasedTesting
- P1 – Critical Impact: Must be tested; ideal candidate for automation (candidate for automated smoke testing)
- P2 – High Impact: Should be tested
- P3 – Medium Impact: Can be tested if budget and schedule permits
- P4 – Low Impact: May not be tested; no impact on application and no need of automation
(Sticky Minds Risk-Based Testing: Test Only What Matters)
#AgileTransformation
HEURISTIC RISK ANALYSIS: INSIDE OUT
1616 #RiskBasedTesting
- Vulnerabilities: What weaknesses or possible failures are there in this component?
- Threats: What inputs or situations could there be that might exploit a vulnerability and trigger a failure in this component?
- Victims: Who or what would be impacted by potential failures and how bad would that be?
Grab a developer and ask “What can go wrong here? and here? and …” :-)
(James Bach Heuristic Risk-Based Testing)
#AgileTransformation
HEURISTIC RISK ANALYSIS: OUTSIDE IN
1717 #RiskBasedTesting
(James Bach Heuristic Risk-Based Testing)
E
D
F
A
B
C
Complex: anything disproportionately large, intricate, or convoluted.
New: anything that has no history in the product.
Changed: anything that has been tampered with or "improved".
Upstream Dependency: anything whose failure will cause cascading failure in the rest of the system.
Downstream Dependency: anything that is especially sensitive to failures in the rest of the system.
Critical: anything whose failure could cause substantial damage.
Risk Lists
#AgileTransformation
HEURISTIC RISK ANALYSIS: IMPACT
1818 #RiskBasedTesting
Component Risk MatrixComponent Risk Risk Heuristics
Advanced Shipping Normal Complex, popular
GUI Higher New, complex, third-party, critical
Manifest Lower Legacy, changed,
Point of Sale Lower Legacy, no change
Purchasing Higher Upstream, Critical
Receiving Normal Recent Failure
Shipping Normal Downstream
#AgileTransformation
PLAN OF ATTACK
1919 #RiskBasedTesting
RISK IDENTIFICATION
RISK ANALYSIS
TEST PLANNING
TEST EXECUTION
TEST RESULT
P1 P2 P3 P4 P5
Exploratory Testing
#AgileTransformation
THE GOAL OF RBT
2020 #RiskBasedTesting
GOAL
Assurance of Quality
Cos
t of F
ailu
re C
ost of Testing
(Sticky Minds Risk-Based Testing: Test Only What Matters)
• Not Everything Needs to be Tested
• Finding Balance the cost of testing and Failures with Quality Assurance
• Taking Calculative Risk and Test
September 25, 2015
Introduction to QASymphony for [INSERT COMPANY NAME]
#AgileTransformation
Agile Transformation: People, Process and Tools to Make
Your Transformation Successful
21
USING JIRA TO MANAGE RISK
#RiskBasedTesting
#AgileTransformation
FLEXIBLE AND CONFIGURABLE
2222 #RiskBasedTesting
SOFTWARE DEVELOPMENT ISSUE TRACKING
RISK MITIGATION
SOFTWARE DEVELOPMENT
MARKETING GO TO MARKET
SOFTWARE DEVELOPMENT
MARKETING GO TO MARKET
PERSONAL TASK MANAGEMENT
HR ON BOARDING
#AgileTransformation
CONFIGURING JIRA ISSUES
23 #RiskBasedTesting
Option 1: Create New “Risk” Issue
#AgileTransformation
CONFIGURING JIRA ISSUES
24 #RiskBasedTesting
Create Custom Statistical Fieldsa) Probability of Failureb) Impact c) Risk Analysis
Custom Heuristic RISK Fieldsa) Risk Heuristicsb) Heuristic Analysis
Option 2: Custom Risk Fields
#AgileTransformation
RISK WITH JIRA ISSUES
25 #RiskBasedTesting
Issue Search Analysis
Statistical HeuristicCommon
#AgileTransformation
RISK WITH JIRA ISSUES
26 #RiskBasedTesting
Board Analysis
#AgileTransformation
CATAGORIZE RISK FOR TEST
27 #RiskBasedTesting
Organize Risks for Test
#AgileTransformation
PLAN AGAINST RISKS
28 #RiskBasedTesting
Map High Priority Tests w/ High Risk
#AgileTransformation
EXECUTE YOUR AGAINST RISKS
29 #RiskBasedTesting
Coordinate Line of Priority for Execution
September 25, 2015
Introduction to QASymphony for [INSERT COMPANY NAME]
#AgileTransformation
Agile Transformation: People, Process and Tools to Make
Your Transformation Successful
30
JIRA ADD-ONS
#RiskBasedTesting
#AgileTransformation
RISK REGISTER
31 #RiskBasedTesting
#AgileTransformation
VISUALIZER FOR JIRA
32 #RiskBasedTesting
#AgileTransformation
SCRIPT RUNNER FOR JIRA
33 #RiskBasedTesting
JIRA Misc Custom Fields ScriptRunner for JIRA
September 25, 2015
Introduction to QASymphony for [INSERT COMPANY NAME]
#AgileTransformation
Agile Transformation: People, Process and Tools to Make
Your Transformation Successful
34
WRAPPING IT UP
#RiskBasedTesting
#AgileTransformation
PROS AND CONS
35 #RiskBasedTesting
Management might think you are lazy
Miscalculate or mis-appropriate weight to a certain risk
Coordinating risk identifiers is a challenge
The impossible could happen
High severity defects are flushed out
Low values tests are kicked to curb
Critical areas are tested first
Confidence is boosted
Deadline anxiety is removed
#AgileTransformation
RESOURCES
36 #RiskBasedTesting
JIRA Add-Ons
• Risk Register - https://marketplace.atlassian.com/plugins/com.projectbalm.riskregister.riskregister-jira/server/overview
• Visualizer for JIRA - https://marketplace.atlassian.com/plugins/vfjc/cloud/overview
• ScriptRunner - https://marketplace.atlassian.com/plugins/com.onresolve.jira.groovy.groovyrunner/cloud/overview
• JIRA MISC Custom Fields - https://marketplace.atlassian.com/plugins/com.innovalog.jmcf.jira-misc-custom-fields/server/overview
References
• Sticky Minds Risk-Based Testing: Test Only What Matters • Xoriant – An Approach for Risk Based Testing
• James Bach - Heuristic Risk-Based Testing
• Daniele Gagliardi – eclipsecon Europe 2015
Thank You
#AgileTransformation
Thank you
#RiskBasedTesting