software testers: the next generation bj rollison, test architect engineering excellence group...

Software Testers: The Next Generation

Bj Rollison, Test Architect

Engineering Excellence Group

Microsoft, Inc.

([email protected])

Microsoft® Copyright © Bj Rollison, 2005

Overview

Today’s testers State of tester’s knowledge Tester performance

The proof (Lies, damn lies and statistics) Empirical evidence

Why the current way isn’t good enough anymore Influences causing change

Tomorrow’s testers Here's what’s happening at MS Here's what you should/could do personally to help

yourself


Tester training (or lack thereof) “Less than 10% of testers have formal training in

test techniques” – Dorothy Graham “…few testers or developers have received any

training in formal methods, especially test techniques.” – Marne Hutcheson (Software Testing Fundamentals)

The average software developer reads less than one professional book per year and subscribes to no professional magazines. - Steve McConnell, DeMarco and Lister, Peopleware, 2d Ed, 1999.


Tester’s Body of Knowledge The average tester has read one book on

software testing Anecdotal evidence suggests the majority of testers

have NOT read more than one book on testing 97% read Testing Computer Software (Kaner et. al) 4% read How to Break Software (Whittaker)

Less than 25% have a degree in computer science or other engineering field

More than 75% unable to effectively code using a modern programming language


The problem is...

Universe of functionality

&bugs

Actualtestingeffort

Under-testing

Over-testing

Average30%


The venerable triangle case study“A program reads three (3) integer values. The three values are interpreted as representing the lengths of the sides of a triangle. The program displays a message that states whether the triangle is scalene, isosceles, or equilateral.” – G. Myers

Suggested answersG. Meyers – 65P. Jorgensen – 185R. Binder - 65K. Beck – 6R. Collard – 4

All answers are rightbecause there is noreal context!


Establishing the baseline

if (Side A) or (Side B) or (Side C) != int16 (signed 16 bit int) if input > maxint then error message for overflow condition else if input ! = integer value then error message for format exception

if Side A or Side B or Side C <= 0 then error message sides must be > 0

if (Side A + Side B <= Side C) or (Side B + Side C <= Side A) or (Side A + Side C <= Side C) then input does not equate to valid triangle

if (Side A = = Side B) and (Side B = = Side C) then triangle is equilateral

else if (Side A = = Side B) or (Side A = = Side C) or (Side B = = Side C) then triangle is isosceles

else triangle is scalene

*Aside from the obvious conditions above additional tests include 1 for max val for all 3 sides, 1 for min val for all 3 sides, and 3 to test for BO of sum of 2 sides


Testing the testers

Generate a set of tests using error guessing and exploratory testing methods that will: adequately* evaluate the triangle algorithm functionality implemented in C# against the specification

(Time limit 15 minutes)


Case study demographics

Total number participants = 400 0% ever had formal training in software testing

Avg. years testing experience = 2¼ years 53% – < 1 year experience (20% STE) 12% – 1 - 2 years experience (60% STE) 10% – 2 - 4 years experience (70% STE) 25% – > 4 years experience (100% STE)

Approx. 50% C/C++/C# coding skill


Case study synopsis 31% of tests covered baseline functionality

28% of the tests did not add significant value

Redundant coverage, results would not prove or disprove anything not covered by previous tests

14% of the tests missed the primary objective

These tests did not focus on the specific triangle functionality of the program

12% of the tests were incorrect assumptions

Wrong or incorrect results expected


Overall Results

0

4

8

12

16

20

24

28

Triangle Effective Redundant Off Target Avg. # Tests

31% of tests metthe primary objective

> 50% of test effort was redundantor not focused on primary objective


Results by Experience

0

4

8

12

16

20

24

28

Triangle Effective Redundant Off Target Avg Number

1-12 mon 13 - 24 mon 25 - 48 mon > 49 mon


STE Results by Experience

0

4

8

12

16

20

24

28

Triangle Effective Redundant Off Target Avg # Tests

1-12 mon. 13-24 mon. 25-48 mon. > 49 mon.


SDET Results by Experience

0

4

8

12

16

20

24

28

Triangle Effective Redundant Off Target Avg # Tests

1-12 mon. 13-24 mon. 25-48 mon.


Technical skill comparison

0

2

4

6

8

10

12

14

Triangle Effective Redundant Off Target

STE (Gp. 1) SDET (Gp. 1) STE (Gp. 2)

SDET (Gp. 2) STE (Gp. 3) SDET (Gp. 3)


Detailed Analysis

40% of the tests covered only 4 tests for the triangle (equilateral, scalene, isosceles, invalid) STE 56% more likely to execute 4 or less tests 20% STEs (1-12 month) did not test for triangle 24% of STEs did not test for invalid triangle input

Only 54% tested for specific boundary conditions Only 5% tested for overflow conditions

Less than 1% of STEs tested for overflow conditions


Skill Comparison

STE 4 times more likely to write an incorrect / invalid test

STE 1.5 times more likely to execute a redundant test

SDET 2 times more likely to exercise specific boundary conditions and overflow errors

SDET tests 2 times more likely to be an effective test


Case study summary

Black box testing is less than 35% effective Testers who lack formal training in testing

techniques are most likely to under test boundaries, exception handling routines, and critical functional areas

More than 50% of the testing effort by untrained testers results in redundant or ineffective* tests

Testers without computer/programming knowledge are less capable of improving test effectiveness efficiently


So, what’s changing?

Market pressures Increasing demand for higher quality Customer’s more aware of capabilities

Complex solutions Increasing complexity (esp. integration)

Drive quality upstream Unit testing, test driven development, etc “Good Bugs harder to find

Cost cutting Hot-fixes, service packs, shelf-life (10 years) Less testers with greater skills & knowledge


Changes at Microsoft

Hiring standards Computer science and other engineering background

Training 40 hours of hands on training for new test engineers Study groups, focus groups, etc. External conferences and seminars

Retention in testing discipline Stop the “brain-drain” Remove the “glass ceiling” for non-management roles Provide greater challenges and scope of influence


What should I do? Increase professional knowledge

Formal training, conferences, etc. Books, magazines, industry white papers

Software Testing Techniques 2nd Ed. – Boris Beizer The Art of Software Testing .– Glenford Myers Testing Object Oriented Systems – Robert Binder A Practitioners Guide to Software Test Design – Lee

Copeland

Increase technical knowledge Modern programming language

Automation Domain expertise System expertise

Questions?

Testing is our profession; Quality is our passion!™

software testers: the next generation bj rollison, test architect engineering excellence group...

Documents