successful investing in software testing! - rbcs, inc · pdf filesuccessful investing in...

Rex BlackRBCS, Inc.31520 Beck RoadBulverde, TX 78163 USA

Phone: +1 (830) 438-4830HTTP: //www.rexblackconsulting.com

E-mail: [email protected]

Successful Investing in Software Testing!

Black bear photo © by and used with permission of of Wild Things Photography, [email protected]

Champion bull, Deets, photo © by and used with permission of of The Double D Ranch, [email protected]

Rex Black Consulting Services, Inc.Technical and Project Management Expertise for Quality

July 24, 2002 Copyright (c) 2000-2002 Rex Black

Advice for the Prudent Test ProfessionalThink of testing as an investment

The project management team foregoes expenditures elsewhere……to fund your testing effort

When you consider testing as an investment, the following questions arise

What kinds of returns can I get on my testing investment?How do I pick the right tests for my portfolio?What kinds of (quality) risks should I worry about?What kinds of techniques can I apply to build ahigh-return-on-investment test system?When should I invest in testing, who does the testing, and how else does testing affect my development process?

Let’s look at some answers…



Cost of Quality: Your Initial CapitalWhat does quality cost?Ë Cquality = Cconformance + Cnonconformance

Conformance costs include testing (finding bugs, a.k.a. defects) and quality assurance (preventing bugs)Nonconformance costs include fixing bugs, retesting, dealing with angry customers, damage to company image, lost business, etc.

Can investing in testing reduce the cost of quality?Let’s look at a hypothetical case study

Conformance costs: manual, automated testing only (no QA)Nonconformance costs: fixing bugs only (no intangibles)Assume 1,000 must-fix bugs in each (quarterly) releaseAssume that a developer-found bug costs $10 to fix, a tester-found bug costs $100 to fix, and a customer-found bug costs $1,000 to fix



No Formal Testing1,000 bugs in productDevelopers find 250Customers find 750 Quality costs over 3/4 million dollars--and customers are angry!What to do?Let’s introduce formal manual testing with anindependenttest team…



Manual TestingInvest in testing teamDevelopers find 250 bugsTesters find 350 bugsCustomers find about 50% fewer bugsCosts are down...Customers are happier…Can we find more bugs and reduce costs further?Suppose we can find 40% more bugs with test automation…



Manual and Automated TestingInvest $150,000 in tools

Amortized over twelve quarterly releases

Complement manual testing with automationCustomers find 66% fewer bugsQuality costs halvedTesting ROI445%But such returns require smartinvestment...



Smart Test InvestmentsPicking the wrong stocks can result in a total loss of your investment--or worseYou won’t get big returns if you pick the wrong tests, tooTest investment returns can be zero--or even negative

Don’t waste money testing actions that customers will never perform, verifying configurations customers don’t use, and fixing bugs customers will never seeTo add insult to injury, such misguided test efforts give management a false sense of security

The smart test investments are in the tests that find bugs your customer would want fixed

Understand customer usage profiles to……test the system the way the customers use it



Testing and Usage ProfilesHigh fidelity test systems mimic customer usage



Testing and Usage ProfilesLow fidelity test systems miss key customer scenarios



Quality Risks and Your Testing Investment

Functional: missing or broken features or functionsUse cases: features okay alone, but important workflows brokenRobustness: common errors not handled properly Performance: too slow at key points or end-to-end

Localization: errors with dates, language, money, and cultureUsability: the software works, but what a pain!Volume/capacity: can’t stand large datasets, transaction flowsReliability: system crashing, hanging, or behaving strangely

Quality risks go beyond broken functionality

Quality: fitness for use; i.e., presence of customer-satisfying behaviors and absence of customer-dissatisfying behaviorsQuality risks: the potential for dissatisfying behaviorsQuality risks fall into many categories, such as…



Analyzing Quality Risks

Stakeholders use RPN to guide appropriate depth and breadth for testing

Set priority for testing each subcharacteristic with key stakeholders

Set priority for testing each quality risk with key stakeholders

Key stakeholders list possible failure modes, predict their effects on system, user, society, etc., assign severity, priority, and likelihood, then calculate risk priority number (RPN)

Functionality, Reliability, Usability, Efficiency, Maintainability, Performance (FRUEMP), then decompose into key subcharacteristics for your system

Functionality, states and transactions, capacity and volume, data quality, error handling and recovery, performance, standards and localization, usability, etc.

Start with categories, char-acteristics, or subsystems

Start with six main quality characteristics

Start with the classic quality risk categories

Failure Mode and Effect AnalysisISO 9126Informal

Target your testing investment depending on the priority of each risk



Essential Techniques

Typically testersTypically developersAll stakeholdersWho

Profilers, coverage analyzers, harnesses, debuggers, data generators, test case generators

Usually unit, component, integration

Identify bugs in functions or interfaces

Test how system works

Structural

Tools

Phase

Why

What Test what system doesTest without running

Intrusive and non-intrusive GUI tools, load generators, performance tools, data generators

Simulators, code analyzers, spell- and grammar checkers, graphic/diagramming, spreadsheets

Usually integration, system, and acceptance

During specification and development

Identify bugs in system results and behavior

Identify bugs before they’re built

BehavioralStatic

Tools, data, and cases can be shared: encourage cross-pollination of techniques



Automated or Manual?Tests well-suited for manual testing

Operations and maintenanceConfiguration and compatibilityError handling and recoveryLocalizationUsability Installation and setupDocumentation and help

Manual, automated, or combined

FunctionalUse cases (user scenarios)User interfaceDate and time handling

Tests well-suited for automated testing

Regression and confirmationMonkey (or random)Load, volume, and capacityPerformance and reliabilityStandards complianceStructural, especially API-based unit, component, integration Static complexity and code analysis

Inappropriate manual testing misleads people about test coverageInappropriate automation usually failsSome tests mix techniques



Pervasive Testing

Concurrent: testing starts with requirements and happens throughout the lifecycle

Smart test investments take months to mature

Successful testing is not a few people in a dark lab at the end of the project...

…winning test strategies pervade software efforts!

Cross-functional: many people play an active role in testing

Testers aren’t the only ones who design, write, and run tests

Collaborative: testing has dependencies throughout the project team

Teams must deliver what’s committed on time



Pervasive Testing: TimeBegin testing activities on the first day of the projectRequirements, specifications, and sometimes even source code receive independent evaluation, are inputs to test developmentTest case, tool, and data development being as soon as possiblePhased test execution, moving from structural to behavioral



Pervasive Testing: TasksFor optimal return on the testing investment, the whole development team participates in test tasks

Salespeople and marketers define product requirementsCustomer support and operations provide usage profilesDevelopers define specs and perform structural (white-box) testsTesters implement test tools and perform behavioral (black-box) tests

Promote re-usable tool development Across teams: Between development and testAcross phases: From unit to acceptance testing

Integrate testing of hardware and software componentsUse simulators if necessary for electronic and mechanical elementsHardware affects the software’s behavior, and vice versa

Include test results in the project management processUse test progress and defect metrics in the management dashboard



Pervasive Testing: TeamworkTesting, being an evaluation and assessment process, receives its test objects and systems from outside groups

Development: writes the software to be testedManagement: provides resources and budgetsRelease/configuration management: creates the software installation mediaOperations/system administration: installs and supports the testnetwork

These teams must deliver on schedule and as promised to the test team

Timely, correct delivery of items will happen when testing is seen as an important, value-added activity International Teamwork: UN Security Council



Payoff SummaryInvesting in testing can significantly reduce quality costs

Base your testing budget on a solid business caseTesting must be focused on usage profiles to catch customer-critical bugs

Build high-fidelity test systems that mimic customer usageUsage profiles involve a number of quality risks

Understand and test those quality risks critical to your customersCertain test techniques work best for certain quality risks

Create behavioral and structural, manual and automated tests

Testing tasks and support must penetrate the entire lifecycle for optimal effectiveness

Test early, involve everyone, and facilitate testingMake testing pervasive

Be bullish…about investing in software testing!

Champion bull, Friendly, photo © by and used with permission of Double D Ranch



Further ResourcesB. Beizer, Software Testing Techniques, 2nd Ed.

Van Nostrand Reinhold, 1990.B. Beizer, Black Box Testing. Wiley, 1995. R. Black, Critical Testing Processes. Addison-

Wesley, 2002.R. Black, “Investing in Testing,”

www.stickyminds.com R. Black, Managing the Testing Process, Second

Edition. Wiley, 2002.R. Black and G. Kubaczkowski, “Mission Made

Possible,” STQE, Volume 4, Issue 4.J. Bullock, “Calculating the Value of Testing,”

www.stickyminds.com.J. Campanella, ed., Principles of Quality Costs.

ASQ Quality Press, 1999.P. Crosby, Quality is Free. McGraw-Hill, 1979. M. Fewster and D. Graham, Software Test

Automation. Addison-Wesley, 1999.D. Freedman and G. Weinberg, Handbook of

Walkthroughs, Inspections, and Technical Reviews. Dorset House, 1990.

E. Hendrickson, “More Testing, Worse Quality,”www.qualitytree.com.

International Standards Organization, ISO/IEC 9126. 1991.

B. Hetzel, The Complete Guide to Software Testing. Wiley, 1988.

C. Jones, Estimating Software Costs. McGraw-Hill, 1995.

J. Juran, Juran on Planning for Quality. The Free Press, 1988.

S. Kan, Metrics and Models in Software Quality. Addison-Wesley, 1995.

C. Kaner, et al., Lessons Learned in Software Testing. Wiley, 2001.

C. Kaner, et al., Testing Computer Software, 2nd

Ed. Wiley, 1999.P. Neumann, Computer-Related Risks. Addison-

Wesley, 1995.D. Stamatis, Failure Mode and Effect Analysis.

ASQC Quality Press, 1995.K. Wiegers, Software Requirements. Microsoft

Press, 1999.

successful investing in software testing! - rbcs, inc · pdf filesuccessful investing in...

Documents