the quality quotient insightful strategies for quality it...
TRANSCRIPT
The Quality Quotient – Insightful Strategies for Quality IT Project
Releases
October 24, 2015
History
On September 9, 1947, the Mark II technical team of
Harvard University found a moth in the wiring of the
computer system. Computer scientist and US Navy Rear
Admiral Grace Hopper removed the moth, and taped it
in the logbook where she made an entry: "Relay #70
Panel F (moth) in relay... First actual case of bug being
found."
2
https://en.wikipedia.org/wiki/Grace_Hopper
Quality Defined
• Quality is a perceptual, conditional, and somewhat subjective
attribute and may be understood differently by different people.
• The degree to which the product/service was produced correctly, is
reliable, maintainable or sustainable.
• How good or bad something is.
Quality is when expectations are either met or exceeded (depending on how high your expectations were!)
3
Perceptions of Quality
4
Journey from perception to reality is unique - it includes scientific methods and emotional connects
Quotients, Quotients, Quotients
5
Examples of Quotients:
IQ Intelligence Quotient – How bright you are
TQ Technical/Operational Quotient – How able are you to get things done
MQ Motivational Quotient – How driven are you to achieve and grow
XQ The experience Quotient – How many of the requisite kinds of experience you’ve had
PQ People Quotient – How well you handle yourself at work with others (sometimes referred to as EQ)
LQ Leaning Quotient - How deftly you adopt new skills, behaviors and beliefs
QQ Quality Quotient – How well you incorporate strategy, capabilities, process,
talent and testing rigor
Quality Quotient FormulaQuality includes two key parameters:
“Quality of Design” and “Quality of Conformance”
Because of the two perspectives on quality, the de-facto definition
of quality needs to be enhanced to reflect practitioner and
empirical views
q + Q + S = Quality Quotient
where
q = operationally limited to product’s defect rate and reliability
Q = includes product quality, process quality and customer
satisfaction
S = establishing the right strategies, capabilities, heuristics and organizational talent
6
Top Challenges
7
• Diverse Technologies and evolving Application landscape
• Agility to meet Business Strategy and Cost Governance
• Organizational Maturity and Change Management
— Process & Standards
— Vendor Governance (outsourcing
partners, software products)
— Culture of Innovation
• Talent / Skill Mix
Use Case
Very Large Investment Fund Accounting Project
Quality was the #1 priority for the project team
8
• Total Defects = 5600; almost 2700 found by “Model Office”
• Vendor defects accounted for 17% with 44% of those classified as critical / high severity
• Four years in the making
• High complexity
• Large Integration testing
effort
• Several new teams created
This project was perceived as extremely successful due to high quality
Key Attributes of Quality Quotient
9
1. Strategy and Planning
2. Engage Early, Test Early, Test Often
3. Talent & Skill Mix
4. Process and Tools
5. Influential Communication
1. Strategy and Planning
10
• How are we influencing Scope - business acceptance, white box view, defined roles – getting agreements!
• Are our Test Objectives aligned to SMART principles
Is my Quality Strategy aligned
How is my Test Strategy governing Product and Release level - See the Forest through the Trees
How am I instilling Heuristics and Innovation– “How can we ..” culture
Are the Tools supplementing the life cycle
Are we incorporating Lessons Learned?
With 15% increase in planning efforts, test execution efforts can be reduced by 25 – 30%
Test Objectives
11
Sample Test Objectives
Test Level Objective Metric
Requirements 100% of functional, non functional requirements are testable, assessed for impact, have full test coverage and are approved by stakeholders
Execution 100% completion of high and medium test cases and 80% of low priority (time to market driven project). Include Defect Detection % - 2/3rd
defects found in 1/3rd of test execution, % of acceptable defects by phase
Traceability 100% of requirements are risk rated (assigned a risk score); all high riskrequirements are traced to test cases
Root Cause By test level (requirements, code, design, data, etc. defects found in each test level) use a guidepost to determine expectation. Industry examples: 20% requirements, 30% design, 35% coding, 15% other
Automation Percent of automated testing (API vs Non API, Regression), failure rates, # sprints behind
Non Functional Performance results meet X% variance to baseline, % improvement, % of x-priority SLA/SLO met per non functional test type
2. Engage Early, Test Early, Test Often
12
Early Engagement – Influence ecosystem and multifaceted testing needs
• Vendor assessment and proof of concept phases
• Contract phase
Test Early
• Influence requirements (functional / non-functional) - strengthens collaboration between Quality, Business and Dev teams
• Test design techniques integrated with requirements
• Testability, traceability and impact assessment for business process integration testing
Test Often
• In Sprint API automation
• Integrated “Day In The Life” as well as business regression testing
• Integrated Continuous Integration/Continuous Delivery (Jenkins/Hudson)
Sample Techniques
13
MindMap for pair-wise testing
Use cases / Test
Scenarios / Data reqs
Technique/Tool (Pairwise Algorithm,
etc..)
Unified view of disparate entities
3. Talent & Skill Mix
14
You don’t just need testers…..
• Accountable
• Analytical
• Interpersonal and politically savvy
• Problem solver
• Sixth Sense – extra sensory perception
• Influential Communicator (written, verbal, visual)
• Strategic – sees the forest
• Business Domain knowledge
• Developer, ETL and Data testing skills
• Project Management
Your natural aptitude
Your ability that comes from your knowledge, practice or
aptitude
“We know that our best people are scattered everywhere but we don’t know exactly who knows what.”
BA Tester SDET Strategic
Ta
len
tS
kil
ls
Skill Development Plans
Faster BetterSmarter
HO
W?
Setup Knowledge Sharing & adoption
• Dedicated Knowledge Management (KM) focus group - KSM and SDP(Knowledge Silo Metrics, Skill Development Plans)
• Test life cycle integration with KM framework, standards adaptation
Injection of New Roles & Tools
• Introduction of Test Architect, Technical Program Manager, Test Data Architect,
SDET’s (Software Developer Engineer in Test)
• Impact assessment, business regression, RBT, Pairwise
Injection of new skills
• Enabling non-UI Automation with SDET skills, up-skill to develop hybrid testers
• Performance Engineers
• Test Data Architects
15
4. Process and Tools
16
• (Prescriptive) Standards – lightweight with guiding principles
• Move from “Silo” department functions to a culture of embedded/integrated mindshare
• Process – focus on objectives, allow innovation with some balance• Define table stakes (i.e. IT Controls)
• Tools – Process and Engineering(continuous build and release, code coverage, etc.)
• Continuous Improvement
“We cannot have islands of excellence in a sea of slovenly indifference to standards” - John W. Gardner
Scorecard Maturity Levels
17
• Level -1 – Basic— Does not meet criteria for level 2
• Level – 2 – Operational— Example – Quality is governed by Test Approach
which is base-lined, approved and continuously refined…
• Level – 3 – Insightful— Example– Enablement of upstream defect
prevention, refined using test optimization techniques…
• Level – 4 – Influential— Example– Quality approach is a key influencer,
focus on expected business outcomes and strongly advocated…
• Level – 5 – Aspirational— Example– Quality influences IT and business
process with a “Best in QM” view…
A transformation maturity framework, focused on behavior and practices,
incorporating CMMi, TMMi framework standards
5. Influential Communication
• Quality does not stop at defect prevention
• Reminders on cost to value
• Large transformation programs may have overlapping Integration phases
• Go/No Go recommendations
18
Pre-ProductionQuality
— Architecture and Design — Build stability (First Pass Yield), root cause patterns,
corrective actions, defect rejections— Quality gates— Metrics driven reporting with dashboards— Operational readiness
Post ProductionQuality
— Production checkouts— 30 / 60 / 90 day user acceptance
Applied Example
1. Strategy and Planning— Well Defined Scope, roles and responsibilities agreement— Test objectives matured over time (defined well with vendor)— Heuristics matured over time, leveraged lessons learned
2. Engage Early, Test Early, Test Often— Leveraged mindmaps, well defined acceptance criteria, static testing and impact assessment— Traditional and API automation, maturing operational readiness, production checkouts— Business flow and Day in the Life testing
3. Talent & Skill Mix— Good use of talent, quality efficiencies helped drive integration strategy
4. Process and Tools— Leveraged maturing test and development standards and process
5. Influential Communication— Excellent communication throughout (vendor trips, business, dev, integration, architecture,
EPMO), good use of stage gates
Large Investment Fund Accounting Project
19
Conclusion
20
Quality IS in the eye of the beholder. You have a unique perspective that can influence the perceptions of quality.
• Start with a shared understanding of the objectives to align teams to the common vision.
• Put the time and effort into the Quality Strategy
• Find your own Quality Quotients that work for your organization
• Disrupt!
Quality Quotient is not about tools or people or changing technology landscape, it is about …U and I.
QUALITY