…from test team assessments...ambrose bierce, the devil’s dictionary a surprising number of our...

Six Surprising Findings……From Test Team Assessments

Si S p i F T t A tSix Surprises From Test AssessmentLife is a box of chocolates. You never know what f fyou’ll get. – Winston Groom, Forrest GumpOne of RBCS’ main services is test assessment

Review documentsAnalyze metricsInterview stakeholders and participants

We’ve performed assessments around the world d l d l t f th li tand learned a lot from these clients

Some of these things might surprise youThe following slides show six surprising findings f t t t

g p g gfrom test assessments

General observationSome examples

Six Surprising Findingswww.rbcs-us.com

Copyright (c) RBCS

M t i With t I i htMetrics Without InsightTruth, n. An ingenious compound of desirability and , g p f yappearance.—Ambrose Bierce, The Devil’s DictionaryA surprising number of our clients have lots of metrics, but can’t derive meaningful insight from metrics, but can t derive meaningful insight from those metricsSo, they make critical decisions with incomplete datadataFor example, based on financial considerations, when should testing conclude?There is an answer to that question more on that There is an answer to that question—more on that later—but most organizations can’t answer it with data


Copyright (c) RBCS

A W R d t Ship?Are We Ready to Ship?

Can you name at least four reasons why the answer is, “We need more i f i b f d id ?” Thi k b i bi l


Copyright (c) RBCS

information before we can decide?” Think about it a bit…more later

E S t P pl J p t C l iEven Smart People Jump to ConclusionsFor every complex problem there is an answer For every complex problem there is an answer that is clear, simple—and wrong. – H.L. MenckenP l f i l b i People often want one simple, obvious explanation of an outcome or problemThis problem is compounded by the This problem is compounded by the previous one, the lack of dataFrequently we hear a hypothesis during q y yp ginterviews, and it is often accepted as obvious wisdom by people at the company—without any objective proofcompany without any objective proof


Copyright (c) RBCS

“B d B R p t A i f I p i ”“Bad Bug Reports Arise from Inexperience”


Copyright (c) RBCS

Y C O t tYou Can Over-automateThe answer to the ultimate question of life the The answer to the ultimate question of life, the universe, and everything is 42 – Douglas Adams, The Hitchhiker’s Guide to the UniverseAsking a computer what the meaning of life is doesn’t make much senseAsking a computer to run non automatable Asking a computer to run non-automatable tests doesn’t make much sense eitherRun of thumb: If a test requires frequent tester q qintervention to run or to interpret its results, it’s not automatable


Copyright (c) RBCS

E pl f O t tiExamples of Over-automationFocusing on 100% regression test suite Focusing on 100% regression test suite automation for a database query toolS i lit i k i d Serious quality risks remained unaddressed in areas of usability, i t ll ti d fi tiinstallation, and configurationTrying to automate compability testingThe tests required attention every 30-60 minutes to start a new software install


Copyright (c) RBCS

Bl d d I B ttBlended Is BetterPut the lime in the coconut then you feel Put the lime in the coconut, then you feel better – Harry Nilsson, CoconutI ff ti t t t ft i fl ibl Ineffective test teams often inflexibly stuck on one test strategy or tactic

100% test automationPurely reactive testing (exploratory)

The most effective test teams use blended strategies and tactics


Copyright (c) RBCS

Bl d d V St i htBlended Versus StraightOne client that adopted an exploratory testing One client that adopted an exploratory testing strategy to blend (~20%) with an analytical strategy

Defect detection effectiveness before blending 92%Defect detection effectiveness after blending Defect detection effectiveness after blending 95%

Another client that used a similar blend had a defect detection effectiveness of 94%, which has gone up as they continue to fine-tune their strategiestheir strategies


Copyright (c) RBCS

S T t T D G t R p tSome Test Teams Do Get RespectI get no respect Rodney Dangerfield late I get no respect – Rodney Dangerfield, late comedianSome testers and test teams feel like Some testers and test teams feel like RodneyOften that’s a result of not having a Often, that s a result of not having a clearly defined mission with success criteriacriteriaTest teams with clearly defined missions and success criteria are often valuedand success criteria are often valued


Copyright (c) RBCS

P i f T t TPraise for a Test TeamQuotes from project stakeholders (not testers) about one p j ( )particularly effective test team

QA gives me an increased comfort level and confidence.The QA Team provides broad knowledge that crosses organizational boundaries.gThe QA and UAT testing period and process provide a central reference point, coordinating the various projects.QA…prevents problems rather than just detecting them.QA helps me pass the [“CIO test”] because if something bad were QA helps me pass the [ CIO test ] because, if something bad were to happen in Production, I could honestly say I had done everything I could.QA provides useful, timely, credible information that helps guide the project.the project.Bad news from QA is handled effectively, and they have a good knowledge of the big picture.

This kind of success doesn’t happen by accident, but I do believe it is possible for most test organizationbelieve it is possible for most test organization


Copyright (c) RBCS

T ti I N t C t C tTesting Is Not a Cost CenterIf you think education is expensive, try If you think education is expensive, try ignorance. —Derek Bok, former President of HarvardT ’ h If hi k i i Tester’s paraphrase: If you think testing is expensive, try shipping untested softwareQuick poll: How many of you know how Quick poll: How many of you know how much testing saved your company last year?Roughly one third of RBCS assessment clients don’t know and can’t get the data to determine the answerto determine the answer


Copyright (c) RBCS

E pl f T ti ROIExamples of Testing ROIHere are some return on investment Here are some return on investment numbers for a few clients who could calculate their team’s return on investment

I d t i l t l t 1475%Industrial control systems: 1475%Banking software: 3500%Consumer electronics: 333%Consumer electronics: 333%Internet appliance: 866%

If you’re wondering how we did this, see i l “T i ROI Wh IT my article, “Testing ROI: What IT

Managers Should Know,” on the Basic Library page of www.rbcs-us.comy p g


Copyright (c) RBCS

A t E li Q tiAnswer to Earlier QuestionBased on financial considerations, when should ,testing conclude?Marginal cost of a bug in testing: the cost, taking into account all factors, of testing until another bug into account all factors, of testing until another bug is foundAverage cost of a bug in production: the total cost of production failures, divided by the number of of production failures, divided by the number of bugs found in productionWhen the marginal cost of a bug in testing reaches the average cost of a bug in production, testing the average cost of a bug in production, testing should concludeThis rule takes into account only financial considerationsconsiderations


Copyright (c) RBCS

A t E li Q tiAnswer to Earlier QuestionSome reasons why we need more information Some reasons why we need more information on the ship decision

Don’t know if test effort was constant over timeDon’t know if all the planned tests were runDon t know if all the planned tests were runDon’t know what risks were addressed and what risks weren’t addressedD ’t k h t i t d d i Don’t know what requirements and design specification elements were coveredDon’t know how many defects were “resolved” by deferraldeferral

A balanced scorecard should be used to assess test project status, not a single chart


Copyright (c) RBCS

Q ti ?Questions?So these are some of my favorite surprisesSo, these are some of my favorite surprisesAny questions?Any favorite surprises that you’ve had when Any favorite surprises that you ve had when working on a test team?If you’re interested in more information on our assessment services, please contact me at [email protected] for your attention!Thanks for your attention!


Copyright (c) RBCS

T C t t RBCSFor over a dozen years, RBCS has delivered services in consulting, outsourcing and training for software and hardware testing Employing the industry’s most

To Contact RBCStraining for software and hardware testing. Employing the industry s most experienced and recognized consultants, RBCS conducts product testing, builds and improves testing groups and hires testing staff for hundreds of clients worldwide. Ranging from Fortune 20 companies to start-ups, RBCS clients save g g p ptime and money through improved product development, decreased tech support calls, improved corporate reputation and more. To learn more about RBCS, visit www.rbcs-us.com.Add RBCS IAddress: RBCS, Inc.

31520 Beck RoadBulverde, TX 78163-3911USA

Phone: +1 (830) 438-4830E-mail: [email protected] b b

Six Surprising Findings

Web: www.rbcs-us.com

www.rbcs-us.comCopyright (c) RBCS

…from test team assessments...ambrose bierce, the devil’s dictionary a surprising number of our...

Documents