are you in control of testing, or does testing control you?
TRANSCRIPT
Are you in control of Testing, or does Testing control you?
Mike SmithIndependent Consultant
40th Year in IT!
I’ve lived through a few technology eras!
Getting the balance right can be difficult!
Trying to stay relevant
• Tool Author – Test Process Management (T-Plan)• Businessman/Entrepreneur/Tester/Innovator• UK Representative to ISTQB• Chairman ISTQB Advanced Level Syllabus Working Group• Founder member of CAT (Certified Agile Tester) SIG• Practical Assessments (R&D Project)• Web TV & Digital Marketing in sport• Professional sports governance (in my spare time!)• Writing a book!
Presentation Background
• Lack of control over projects/initiatives in IT, Business & Sport
• Common faults over this broad perspective• This presentation focuses on software testing as a form
of measurement• But it uses analogies and common characteristics across
other perspectives • I will use some examples and my experiences to help
highlight key points
Presentation Background
Concerns over effectiveness of software testing as a measurement process
The subject of my book – Measurement & Testing!
My theme for a book
“Effective Measurement and Testing Delivers Control of Anything”
Measurement & TestingMy Axioms
• Testing is more than just Testing!• Software Testing can learn from other forms of
Measurement & Testing• Other forms of Measurement & Testing can learn
from Software Testing• Agile Development is helping the case for
Measurement & Testing• Software Testing took a wrong turn when it
became obsessed with Test Cases!
Another Testing Axiom
•The answer to the ultimate question of Life, The Universe and Everything?
Answer: “42”(Hitchhikers Guide to the Galaxy – Douglas Adams)
•The answer to any question about Software Testing?
Answer: “It Depends!”(Dale Perry – SQE Inc. USA)
3 Key Words
Control
Measurement
Testing (& Software Testing)
Control – Some definitions
•The power to influence or direct people's behaviour or the course of events. (lead)
•Determine the behaviour or supervise the running of. (lag & lead)
Control vs ManagementWhat’s the difference?
Corporate Governance Definition1.Strategic Direction of the Business
1.Effective Monitoring and Control of the Management of the Business
2.Accountable to the Shareholders of the Business
Control vs Management?
Being in Control – Can you answer these questions?
•What are my measures of success?•How will I know when I’ve met them?•How sure will I be that I’m going to meet them at any point in time?•How do I know where I am against them at any point in time?•Can I manage change effectively?•For an IT project, how useful are Test Cases to help me keep in control?
Successful Outcomes?
• Reach destination port• Get there on schedule• All passengers & crew are safe• Passengers have not been (too) seasick!• Within budget (economical with fuel)• Obstacles avoided!
How will I know when I’ve met the successful outcomes?
• What are the key measures & targets?• Speed?• Direction?• Position?• Quality Attributes?
Keeping the ship under control!Key Drivers
Initiatives taken based these key goal & performance drivers
Other measures to support the key drivers for Control
There may be many management systems to support these measures
A complex measurement & management framework is needed
Simple Hierarchical Model does not support the complex relationships
What is Business Performance Measurement & Management?
• ‘Translating Strategy into Action’– Kaplan & Norton
• Use of Business Balanced Scorecard– Different perspectives
• Drives behaviour• Measures outcomes• Links actions to strategy• Leads to predictable outcomes
Dangers of Performance Measurement & Management?
Measures that are badly constructed can destroy an organisation’s performance… effort may be misdirected into unproductive actions and a great deal of time and effort can be wasted.
‘Balanced Scorecard in a week – Mike Bourne, Pippa Bourne’
5 Key principles
1. Generic application2. Objectives, Measures & Targets, Initiatives3. Separate “What” & “How”4. Cascading Scorecards
– One person’s ‘How’ is another person’s ‘What’– Measures & Targets become objectives for next
person 5. Lead & Lag Indicators
– Goal Indicators (reactive)– Performance Indicators (predicative)
Scorecard Objectives, Measures, Targets & Initiatives
InitiativesObjectives TargetsMeasures
WHY WHAT HOW
Scorecards linked bycascading ‘whats’ & ‘hows’
HOWWHAT
HOWWHAT
HOWWHAT
• Could be 1:many or
many:many
• Simplest way is 1:1
HOW
HOWWHAT
HOWWHAT
HOWWHAT
WHAT
Summary of Outcome Driven Measurement Framework/Model
• Separate “What” & “How”• Complex Relationships in Model• Information Traceability matrix Top to
Bottom• Control of Complex Organisations &
Systems• Good fit with Agile Development
Practices!
Now consider Software Testing Model
• What are the Measures & Targets that support the Software Testing Process?
• Test Cases?• Test Objectives?• Test Requirements?• Test Conditions?• Testers focus on Test Cases, but are they the
best Measures/Targets to drive behaviour?• Are they the best to manage risk and change?
The problem with Test Cases
Test(ing) & Software Testing
•The means by which the presence, quality, or genuineness of anything is determined; a means of trial (lag)
•Software testing is a process of executing a program or application with the intent of finding the software bugs. (lag) It can also be stated as the process of validating and verifying that a software program or application or product meets the business and technical requirements that guided it's design and development. (lag & lead)
•Testing is a risk reduction process (lead)
The problem with Test Cases
Test caseA set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. [After IEEE 610]
The problem with Test Cases
• How big?• When can they be defined?• They may help me Test, but do they help me
Measure?• Whose Measures are they?• Do they separate the What from the How?• History of Test Cases (‘Bottom-up’)• What else fails if my Test Case fails - traceability?• Tool problems – wrong model• What if none defined?
The problem with Test Cases
1479 test cases: but are they “useful”? Now, let’sstart with a
classificationtree
1479 test cases, so
it must be good,right? Test
specificationprocess
Documentationto agreecoverage
A Simple Hierarchical World
A Complex Relational World!
A simple world became a complex world!
Test Model described by ‘methods’ people at large pan-European financial institution.
Test RequirementsRequirements
WHY
Test Design Specifications
Test Case Specifications
WHAT WHAT HOW
• “From Requirements to Test Case Specification”
in test management tool
A complex model to support software testing
May or may notNeed complex
model:
IT DEPENDS!
Guidelines for a better model
• Need to separate What & How• Model needs Complex Relationships – many:many• Test Conditions can be Considered targets for success,
organised by Measurement Attributes– Functional, Behavioural
• Can and should be defined early• Ability to cope with change• Know where you are at any point in time• Delivers Control of Complex Systems• Good fit with Agile Development Practices!• Business Outcome Driven Development
What factors affect the level of measures and targets in a measurement framework?
• Risk• Size & Complexity• Dev Method• Test Level• Manageability & Maintainability• Tool usage• Ability to cope with change• Quality and level of artefacts• Regulatory requirements…....
What factors affect the level of measures and targets in a measurement framework?
IT DEPENDS!!Setting good measures and targets is
difficult - it requires strong analytical skills
People often neglect the –ve measures and focus on the +ve
What can the world of Measurement learn from Software Testing?
• Software Testing has been developed as a discipline for many years
• Very detailed processes and techniques compared with other forms of Measurement & Testing
• Software Test Techniques– Support Analysis and Design Activities
• Airport Example
Other Experiences
• Making a new Financial Exchange Live
• An Office Move• ISTQB Syllabus Development• Practical Assessments• Governance in Sport• Mindset change and silos!
WHAT’S MISSING?
General Model
Control
Management &Information
Systems
Measurement Framework
Measures &Targets
Objectives
Initiatives
WHAT’S MISSING?
Testing Model
Control
Tests and Test Cases
Test Requirements/Test Conditions
Measures & TargetsSeparated from TestsMany:Many Reqs:Tests
Objectives/Requirements
Initiatives
WHAT’S MISSING?
New Financial Exchange
New Market
Tests and Test Cases
Regulator Approval
SPECIFICATION
MeasurementFramework
RetrospectiveTest Conditions inComplex Model
WHAT’S MISSING?
ISTQB Syllabi (Presented in Prague 2011)
Certificates(FL,AL,EL)
LearningObjectives &
Syllabus
VISION / STRATEGY
SPECIFICATION
Business Outcomes
REQUIREMENTS
WHAT’S MISSING?
Sports Club Constitution Review
Need to Change
Old Subject Matter Experience
VISION / STRATEGY
CONSTITUTION/RULES
Terms of Reference
REQUIREMENTS
My ‘Axioms’
“Test Driven Everything”
“Business Outcome Driven Development”
“Effective Measurement and Testing Delivers Control of Anything”
“Top Down, meets Bottom Up”
References
• Testing Entities Paper – John Kent• IEEE Paper – ‘Keystone Testing Entity’ Mike
Smith & Neil Thompson• Value Flow Scorecards – Mike Smith & Neil
Thompson• Balanced Scorecard in a week – Mike & Pippa
Bourne• My book on Measurement & Testing – when?!
What can you take away from this?
• If you are building a pure hierarchical model to manage testing, think again
• Can I optimise my model to give better control without it becoming unmanageable?
• Am I relying too much on my testing tools – do I need something else?
• Can I use my analytical skills to engage with the business better?
• I encourage you to think about these concepts, even if you think you are in control…...
If you think you’ve found a good move,
look for a better one!
My Measures of Success for this talk?!
1.Standing Ovation?2.Loud Applause?3.Polite Applause?4.Silence?5.I won’t be let into Russia again?!!