~TANDEM

Developing Quality Assuranceand Testing Personnel

Mary AlexanderKeith Stobie

Technical Report 88.9Summer 1988Part Number 15368

DEVELOPING

QUALITY ASSURANCE AND

TESTING PERSONNEL

Mary Alexander

Keith Stobie

Technical Report 88.9Tandem Part No. 15368Summer 1988

Tandem Technical Report 88.'

Developing Quality Assurance and Testing Personnel

Mary AlexanderKeith Stobie

Spring 1988

ABSTRACT

Specific techniques for identifying candidates who will be

successful in QA are described, as well as approaches for selling

them on a career in QA. Counteracting the industry's perception

that QA and test are inferior fields to product development is

the major challenge. Since most people are not trained in QA,

adequate grounding must be provided. Comparison are made between

using courses provided from outside companies and using material

focused to the individual organization. Advancement of QA and

test personnel should be similar to that of product development

personnel. The types of management commitment needed and how to put

this into practice are also described.

TABLE OF CONTENTS

1 • 0 I NTRODUCT I ON • . • • • .. .•. . .. .. 1

2.0 HIRING TESTING PERSONNEL . . . • • . • .. .. 2

2.1 OPPORTUNITIES IN SOFTWARE TESTING ..•.•...•.• 2

2.2 ATTRIBUTES OF A SOFTWARE TEST DEVELOPER .•...••• 3

2.3 FINDING AND HIRING QUALIFIED CANDIDATES ••....•• 4

3.0 QUALITY ASSURANCE AND TESTING EDUCATION 6

3.1 EDUCATION BACKGROUND . • . . .. .• 7

3.2 EXTERNAL COURSES. . . . . 7

3.2.1 Customized External Course. . •. . 9

3.2.2 Internal Course from Scratch. . . . . 10

3.2.3 Management Course . . . . . . •. •. 11

3.3 CONTINUING EDUCATION . . . . . . . . . . . 11

3.4 TOOLS TRAINING. . . . . . . . . . . . . . . . . .. . . 13

3.5 TANDEM'S EXPERIENCE WITH TESTING EDUCATION. . . 14

4.0 CAREER DEVELOPMENT. . . . . . . . . . . . . . . . . .. 16

4.1 JOB CATEGORIES. . . . . . . . . . . . .. 16

4.2 CAREER PATHS. 18

4.3 COMMON HIRING MISTAKES. . . . . . .. 19

4.4 COMMON ADVANCEMENT MISTAKES . . . . . . . 20

5 • a SUMMARY • . . . • . . • . . .. 21

REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . 22

1

1.0 INTRODUCTION

As Quality Assurance (QA) and testing organizations have become an

accepted part of product development organizations throughout the

industry, development of QA and testing personnel has become an

issue. When broken down, there are three separate areas which need

to be addressed: that of successfully hiring candidates into the QA

or testing department, their training once hired, and their career

development as quality professionals. This paper addresses each of

these areas in turn.

While QA and testing can be completely different functions, the term

testing will be use in the rest of this paper to refer to both, for

brevity.

1

2.0 HIRING TESTING PERSONNEL

In order to run an effective software testin~ organiza~ion, it

is imperative that the right staff be chosen. In order to do

this, the testing manager must understand the nature of software

development in general, as well as the attributes specific to the

testing function. It must be clearly understood that software

testing is a specialty of the technical field known as software

development; development and testing are not completely separate

fields. Once the testing manager understands the requirements

which a successful candidate must possess, as well as the unique

opportunities available in software testing, the next step is to

understand the types of candidates who will be successful in the

field. Lastly, the testing manager should know where to find these

qualified candidates, and techniques for successfully recruit.ing

them.

2.1 OPPORTUNITIES IN SOFTWARE TESTING

To those unfamiliar with the field, it often seems that quality

assurance is little more than rote testing of a completed product:

a generally unchallenging and minimally fulfilling endeavor.

Actually, quite the opposite is true. A test developer has the

opportunity to influence product directions at a high level, rather

than becoming a specialist in a narrow aspect of the product. The

test developer is required to have broad product knowledge, both of

the external interfaces, and of the internal design. Acquiring this

knowledge alone would be an adequate technical challenge for most,

2

but in addition, the test developer must be able to find product

problems before the product is completed. These may be problems in

the external definition of the product, areas in which the design

is weak, or areas in which a design choice will lead to performance

degradation in the final product.

The field of formal software testing is still relatively new

compared to software development. Many of the larger problems and

issues have been defined, though some have yet to be recognized.

Some solutions have been discovered; however, many problems are

defined, but unsolved. There are partial solutions in the area of

quality metrics and test automation. While there are guidelines,

specific testing techniques are quite often more of an art than a

science. Summed up, this means there is an excellent opportunity in

testing to be at the leading edge of a new technical field: testing

technology.

2.2 ATTRIBUTES OF A SOFTWARE TEST DEVELOPER

Once the testing manager understands the nature of the position,

the next step IS to understand the characteristics of a successful

candidate. Due to the technical nature of the position, a test

developer must have the same technical expertise as would be

required of an equivalent product developer. In an applications

shop, that means a management information systems (MIS) background;

in a systems shop, that means a background in computer science (CS).

3

In general, a significant portion of the testing department should

consist of senior, experienced people, many of whom have substantial

product development experience. Since the testing department is

often called upon to act as the user representative or to assess

the risk to the customer of particular defects in the product, field

experience is also quite useful.

Beyond education and experience levels, there are personal

attributes which tend to make a person successful in testing. Good

communication skills are a must. High level problem solvers, who

can see the big picture, will describe many successful candidates.

People who enjoy solving problems, and who have a knack for finding

problems in software will do well.

2.3 FINDING AND HIRING QUALIFIED CANDIDATES

Since a testing department needs people with a variety of experience

levels, as well as diverse backgrounds, there are many _areas from

which to draw candidates.

First, look within the company at existing employees. There may be

product developers, field analysts, or technical writers who meet

the criteria. An additional benefit is that these candidates are

already familiar with the company and with the products.

New college graduates can do well in testing, provided they will be

working under the guidance of a senior test developer. Many of the

4

top schools are focusing on software engineering as well as problem

solving techniques. The idea of working at the leading edge of a

new technical field is often particularly appealing to the better

students. Lesser known schools should not be overlooked in the

search; quite often, their top students are excellent candidates.

Lastly, experienced developers can be hired from outside the

company. This will include those with product development

experience, but it is becoming easier to find qualified candidates

with testing experience as well. Ensure that your requirements are

met.

Once candidates have been identified, the recruiting process can

begin. It is important to train any in-house or professional

recruiter used so that they understand exactly the type of candidate

needed. This may take several iterations, but is well worth the

effort. Make sure the position to be filled is clearly defined:

both the tasks to be performed, and a descrip~ion of the ideal

candidate. An all-day, rigorous interview will shake out weaker

candidates. Senior people should be included on the interview

schedule. Some product developers can be used, to judge the future

compatibility of the candidate and the product developers.

Finally, DON'T SETTLE It is impossible to run an effective

sof~ware testing organization without a ~ualified staff. Software

testing can be a career opportunity; only offer this opportunity to

the right people.

5

3.0 QUALITY ASSURANCE AND TESTING EDUCATION

Testing educ~tion should be an adjunct to the basic technical

training of testing personnel. Testing people should have the

same level of technical expertise as the developers of the product

they are evaluating. Just like product developers, testing people

need continuing technical education as well as continuing testing

education.

New people to a testing organization will need the same introductory

courses that product developers take. In addition they may need

general testing courses if they are not grounded in these subjects

and they will need education in testing as it is applied in their

organization.

Education for testing personnel can come from external or internal

courses. Almost all groups can take advantage of external courses.

Usually only large experienced groups have the knowledge and the

need to develop internal courses. An internal course is most

appropriate if it will be taught regularly due to turnover or

expansion of the group. A big consideration of the internal course

is how to develop it. There are basically two paths: bring in an

external course and perhaps customize it, or develop the internal

course from scratch.

6

3.1 EDUCATION BACKGROUND

There should be some idea of the general background expected of the

technical people. For example, a systems programming organization

expects a computer science education. Some psychology background

for a user interface tester might be expected. Application specific

knowledge such as accounting, scientific mathematics, or numerical

analysis could also be appropriate. Frequently this background is

gained from college courses.

Unfortunately there are today only a few college courses oriented

specifically towards testing. Many colleges offer a software

engineering course which may include testing as one part. There are

schools offering relatively specific courses. For example, Brigham

Young University currently offers a course in which students test

commercial software submitted by major companies nationwide.

3.2 EXTERNAL COURSES

There are numerous courses and seminars offered by many vendors

through which testing people can gain a foundation in testing ideas.

Many, if not all, testing people should attend these courses. The

introductory testing courses are generally three to five days in

length. A one day course is usually insufficient except as a very

basic outline of testing concepts.

A course that matches the organization's orientation must be chosen.

If the testing group designs, writes, and runs tests, then choose a

7

testing course. If the group sets standards and audits conformance,

then choose a quality assurance or audit course. Another facet to

consider about the course's orientation is how separate the testing

group is from the product developers. A course may be oriented

towards a separate verification and validation (V&V) company, a

separate testing group in the same company, or developers testing

their own or colleagues' work.

Courses are further differentiated by the levels of testing

discussed. There should be a match between the levels of testing

dealt with in the course and those for which the testing group is

responsible. Two major levels are before and after coding. Before

coding might include design reviews or inspections, and code rE~views

or inspections. After coding there are several levels of testing

including unit testing, integration testing, systems testinq, and

acceptance testing. There is also testing conformance to standards.

All of these issues will help determine the appropriateness of a

particular course for the group. A systems testing course fo:~ V&V

companies would be inappropriate for training product developers to

do unit testing.

The level of people towards which the course is oriented is a~other

consideration. It can be for managers, technical people, or both.

Technical people and first line managers should attend a technically

oriented course.

8

Look at courses from at least three different vendors to gain an

idea of their relative differences. At least one and preferably two

people should attend the course externally to really evaluate the

course's suitability to the organization.

Should all the people be sent as one group to the same course, or

individually over time to the same or even different courses? These

are hard questions since they each have their own merit. If all

people attend the same course they will share a common framework

from which to work. On the other hand, a possible diversity of

ideas from having attended different courses may be lost.

3.2.1 Customized External Course

Bringing in a consultant to teach their standard course basically

just saves money. However for an extra cost, the course can

usually be customized and made more relevant. The amount of course

customization desired will vary. The cost of customizing the course

IS one factor in determining how much to ~hange. The matcr. between

t~e course's terminology and the organization's terminology should

be another consideration. Additional tailoring can be achieved

by mixing in 'guest speakers' from the organization to highlight

points particularly relevant to the organiza:ion. For example, if

the consultant discusses a particular class of tool and there is

a company specific version of it, have a guest speaker briefly

describe the relevant points of the tool.

9

3.2.2 Internal Course from Scratch

Developing a custom internal course from scratch is only applicable

to a large organization that will get major benefits to offset the

upfront cost of creating the course. Some of the people who might

be used to develop the course are new hires with extensive previous

testing experience, 'old timers' in the testing organization who

know the organization's way of doing business, industrious learners

who can adapt reading and studying of the testing literature to the

organization, and people with teaching experience.

There are also several tradeoffs in benefits to consider betwel~n a

custom internal course and an external course. A custom intl~rnal

course will be the most relevant for the students. There may be

more flexibility in scheduling the teaching of internal courses than

having to match an external schedule. Students will be able to

easily find the experts who wrote the course later to ask further

questions. In developing the course, a clearer focus about the

organization's testing methodology might be gained. Maintaining

an internal course to keep up with the industry (and even the

organization) is difficult. It is reasonable to expect that an

external course is kept up to date by the vendor. An internal

course reduces the influx of ideas that students might bring from an

external course and the people they meet there.

10

3.2.3 Management Course

Managers should attend an external testing management course. Only

very large organizations will have enough testing managers to

justify developing an internal testing management course. Managers

may gain new perspectives from interacting with managers from other

companies. Courses focused strictly towards testing management are

more rare than the technical or mixed courses. A management course

may vary from two to five days. A longer course allows more time

for case studies and interaction with other course participants.

An evaluation of a testing management course's appropriateness to

the organization must be conducted, just as should be done for an

external technical course on testing.

3.3 CONTINUING EDUCATION

Once people are initially educated, continuing education must be

considered. Education might include taking testing courses, but

be careful that the same introductory testing course 1S not taken

over and over again from different sources. There are far more

people looking for an introductory testing course than for advanced

testing courses; consequently there are many more introductory

courses offered. Be careful in sorting through the course listings

of vendors to find the advanced testing courses.

Beyond testing courses, there are conferences to attend. At a

conference there are opportunities to learn about advances in the

11

industry, compare experiences with others in the industry, and

evaluate vendors. Conference participants can learn about specific

areas by choosing a particular track within a conference or choosing

a conference with a narrow theme.

One should also try attending local testing society meetings in

their area, if they exist. If local societies don't exist and there

seems to be sufficient interest, a local chapter could be started.

There are several different national testing societies.

Testing education can be continued by reading the literature. The

company library should stock a large variety of reference works.

One's personal library should contain essential testing works that

have been found to be relevant. There are numerous periodicals

published, some of which are relatively specific to the tE!sting

area: others discuss testing as one aspect of their audienCE!. It

is easy to get on the mailing list for testing society and vendor

newsletters.

Finally, continuing testing education within the organization should

be considered. This can be in the form of periodic meetings to

share information among the testing community in the organization.

New tools or the results of experimental projects can be presented.

12

3.4 TOOLS TRAINING

In addition to general testing education, there is specific tools

training. Frequently training for an external tool comes from the

vendor who provided the tool. The training may be part of the

purchase of the tool. Some tools corne with their own tutorials

to allow self-paced learning. The tutorials may be sufficient, if

there are other knowledgeable users already trained who can help the

new user. Many vendors also offer consultative assistance in the

implementation of a specific concept or tool.

In addition to internal testing education, internal training on

testing tools should also be considered. The training might include

customization of the training by the original vendor of the tool.

Standalone training might be as little as a one hour lecture or

as extensive as a one week course. Internal tools can be provided

with their own tutorials to allow self-paced learning. As mentioned

earlier, internal tools ~raining can be incorporated into testing

education by using guest speakers or by adding ~ool-specific labs to

a course.

13

3.5 TANDEM'S EXPERIENCE WITH TESTING EDUCATION

Tandem is a computer systems manufacturer. Tandem's testing people

generally have a background in computer science. They have attended

various external courses, including testing management courses.

We developed our own custom internal testing course in 1984 with

a committee of 5 people working one day a week over a period of

six months. The committee drew on the knowledge of many people in

the organization to produce the course. The course did not provide

a good grounding in general testing. We believed people could

be hired with the experience or attend external courses to learn

the basics. The course was oriented to our specific life cycle,

methodologies, and tools. We attempted to show how general testing

principles are specifically applied within our organization.

The course was not maintained and a year after its first teaching

was no longer taught. We substituted a partially customized

external course supplemented with guest speakers. However, the

number of guest speakers and the student feedback asking for even

more customization grew. A project to recreate the custom internal

testing course was initiated in the summer of 1987. The new course

borrows from the old course while significantly updating it. It

is again being developed by a group of 6 people dedicating at most

one day a week and draws on the knowledge of many people in the

organization. The course has the same orientation as the original

course. It focuses on Tandem's way of applying testing as opposed

to teaching the general testing basics.

14

Tandem also has a testing talk every other week. Generally the

talks alternate between tools, such as an experimental automatic

test case generator and our terminal simulator, and testing

concepts, such as testing for fault tolerance.

15

descriptions of the

testing development.

4.0 CAREER DEVELOPMENT

Once a software testing organization has been staffed and trained,

the manager must be concerned with career development. This

involves understanding of testing job families, their differences,

and the variety of career paths open. As advancement problems are

often caused by poor hiring decisions, a discussion of common hiring

mistakes is included. Wrapping up this section is a list of the

common advancement mistakes and how to avoid them.

4.1 JOB CATEGORIES

Since testing is a specialty of the field of software development,

it is imperative that the same job codes and pay grades be used for

test developers as are used for the equivalent software developers.

This is a key factor in a manager's ability to hire and retain

qualified people.

Further, it is important to have written

development job categories as they apply to

Typically, these include:

• one or more levels of technicians. Technicians are used for

repetitive tasks such as running of test libraries, mounting

tapes, etc.

•

16

junior developer. This

requiring a bachelor's

this level works under

is generally an entry level pos:tion,

degree or equivalent. A person at

the supervision of a more senior. test

developer, participates in planning and test design, and codes

test libraries.

• one or more levels of journeyman test developer. This is a

person who is fully competent in the job, able to write a test

plan without guidance, as well as develop testing tools and new

testing techniques. A person at this level can develop a variety

of types of test, such as positive, negative, stress, volume,

limits, and failure scenarios. This level of test developer can

assess risks and determine which are the most important tests to

develop first.

• senior test developer. This person is heavily involved in

product design reviews, and is known for their ability to detect

problems well in advance of the coding stage. This person

operates at the forefront of testing technology.

It is important that developers are appropriately assigned to these

job categories, and that there be a good mix af both junior

and senior developers ln the group. ~his allows wor~ to be

assigned appropriately, and avoids the problem of a senior person

given an assignment which does not take full advantage of their

capabi:ities, or a junior person given an assignment which is far

too demanding. The testing manager needs to ensure an environment

in which employees can succeed.

17

4.2 CAREER PATHS

The most obvious career path is the in-line promotion. Since

breadth is an important component of success in testing, managers

should provide opportunities for their employees to move from one

project area to another within the testing organization. This

allows greater product knowledge, and allows an individual to work

with, and learn from, a greater number of other test developers.

The option of moving from product development to testing can

enhance a developer's career. The reverse, a move from testing to

development (and back to testing at a future time), can provide a

less experienced person with the in-depth technical experience which

is so necessary in the more senior positions.

Testing can also provide leadership opportunities, such as a project

leader, or eventually testing management. It is important to ensure

that all employees understand their career options; career planning

discussions should occur with each employee on at least an a~nual

basis.

18

4.3 COMMON HIRING MISTAKES

Quite often, an obstacle to effective career development is that the

wrong person was hired in the first place. How does this happen?

The most common mistake is that the hiring manager becomes too

eager to hire; to avoid this, recognize that hiring is a difficult

endeavor, and use restraint against becoming too eager. Wait for

the right candidate.

Some newcomers to a company view the testing department as a

way to gain entry to the company, learn the products, and move

into product development. Unfortunately, in this situation, it is

extremely difficult to recover the training investment the testing

department has made. Again, the recommendation is to screen out

these candidates during the interview process.

It is important to hire a variety of experience levels into the

group; a group with all junior developers will suffer from a lack

of leadership and direction, while a group of all senior developers

will tire of performing all the junior tasks themselves and will

move on;

Finally, do not hire people who would not be qualified to do

product development. Testing is indeed development; the same level

of technical expertise is required to have an effective testing

department.

19

4.4 COMMON ADVANCEMENT MISTAKES

A developer should be assigned the same job code whether workinq as

a product developer or as a test developer. If the standards in the

testing department are tougher than those in product development,

employees will find the idea of moving to product development and

getting a promotion to the next job code too attractive to pass up.

If the standards in the testing department are easier than those

in product development, the credibility of the test developers will

be in question, and their suggestions will not be well received by

the product developers. This will lead to dissatisfaction and low

morale, and ultimately a transfers out of the testing department.

Failure to provide appropriate training, and the assignment of

inappropriate tasks will also cause problems. Care should be taken

to provide proper testing training; it is not intuitively obvious

how to best test a product.

20

5.0 SUMMARY

Once i~ is recognized that software tes~ing is a specialty of the

technical field known as software development, it becomes clear that

testers have many characteristics in common with product developers,

as well as some unique to testing professionals. Due to the newness

of this specialty, the test manager will need to sell them on the

opportunities in software testing. It is likely that candidates

will not have a prior background in testing, and will therefore need

to be educated.

Once the testing organization has been s~affed and trained, the

manager must be concerned with career development. Since testing

1S a rapidly developing specialty, continuing education is vital.

There are a variety of career paths available~ the manager must be

sure to define and communicate these to the organization.

21

REFERENCES

[Beizer] B. Beizer, Software System Testing and Quality Assurance,

Van Norstrand Reinhold, 1984

[Stobie] K. Stobie and M. Alexander, "Plan into Practice: Software

Testing at Tandem Computers", Tandem Technical Report

87.3, Cupertino, CA, June 1987.

22

Distributed by..,TANDEM

Corporate Information Center19333 Valko Parkway MS3-07Cupertino, CA 95014-2599