project management in the testing phase1

7/30/2019 Project Management in the Testing Phase1

1/28

Project Management in the Testing Phase

"The most important considerations in software testing are in the issues of econo-mies and human psychology." + Glenford J. MyersIMat is testing? -Actiaities that make up testing-Test scheduling and types of tests-People issues in testing-Management structures for testing in global teams*Metricsfor testing.

INTRODUCTIONIn Chapter 7, we discussed various methods of ensuring the quality of aproduct. We distinguished Quality Assurance (as a means of proactivelyavoiding defects before the product is built) from Quality Control (as ameans of detecting defects after the product is built). In this chapter, wewill discuss Testing, one of the common quality control methods.

WHAT IS TESTINC?-fi The eventual goal of a software project is to have zero defects in the endproduct that goes out to customers. The defects in the final product nor-mally creep in by means of defects from various phases. For example,

defects can come from improper identification of requirements or incor-rect translation of requirements into design or through erroneous coding' Mecira-nisms like design reviews can catch the defects in the initial phases. Testing refers toactivities that are carried out to ensure that the final software product meets thcrrequirements that the product is intended to satisfy. Some of the attributes of test-ing are:


2/28

Project Management in the'festing Phast Mg,-"

-

,.-, \1'Whythe No,Test Failure Yes!

"Expected" Results iwat'a'ndt ;"isn-i;: :---- - - - - - - - - - -.I fig. tz.t What is testing?

Testing is done on a product that has already been built. It is notiike an inspection or a code review.Testing is done to detect the presence of defects in the product,that the product is defect free.o Tests are divided into physical units called the test scripts (or test cases). Eachtest script exercises the built product in a pre-defined manner to produce anactual result . Each test script also has a pre-defined expected result thatspecifies the result of executing the test is if the product behaved according toits original intent/requirements.. The actual result is compared against the pre-defined expected result. If theexpected and the actual results matctr, the test is said to have been a successand if they don't match, the test is considered a failure.. When a test failure is encountered, the product and the expected results areexamined to analyse the reasons for this failure and accordingly, appropriatechanges are made in the product, and the tests are re-run to ensure that theysucceed. It could also be that the expected results arrived at initially weiewrong, in which case they are changed and the test is then deemed successful.Also, the expected results are updated to reflect the new changes.

a desk checknot to prove


3/28

rt,Frta Managing Global Software Prolects: tt.l-av

-WHAT ARE THE ACTIVITIES THAT MAKE UP TESTINC?Testing includes the following activities:r Test specificationo Test designo Test developmento Test registrationo Test execution. Test maintenance

In the following sections, we will go into the details of each of these activities.

It this step, what need.s to be tested is finalised and documented. This can also beviewed. as "requirements specifications" for the testing activity. Some of the ques-tions that get answered during this phase are:Which hard.ware configurations should be used for testing the product? The soft-ware product is eventually designed. to run on_certain hardware platforms' In thisstep, ihe specific hardwaie platlorms on which the product should be testcd arcidentified. and documenied. Frorn a project planning perspective, this leacls to esti-mating the cost of hardware need.d fot testing and also the time needed for thetesting activities. If there are more platforms on which the product needs to betested., the test d.evelopment and test execution costs would also increase'Are there any cut-off configurations under which the product must be tested?There may be some "minimum" configurations the product is expected (or docu-mented) to run on (for example should run under 256 MB of RAM)' During therequirements stage, such configurations would have been specified. During tesispecification, it is important to re-visit these minimal configurations to ensure thattests are designed for them.Which software environments should the product be tested under? Typicaliy,product development takes place in specific software environments-for exampler,ihe specific veriions of the operating system, compilers, databases, and other utili-ties. But when the product is deployed in a customer location, the software environ-ment may not be exactly identicai to the environment under which the product wasdeveloped. The operating system or any of the other environmental software mighthave had patches apptied to them. These changes may cause the software product

\ Test Specification


4/28

Prctject Mnnagement in the Testing Phase 2Wunder test to behave differently in the deployment environments than it did on theclevelopment platform. Hence while testing, it is very important to capture anddocument the software environment under which the tests are to be developed andexecuted. This will help in narrowing down the time of resolution should a prob-lem-that did not show up during development-appear at the customer site.What are the most common scenarios that need to be tested? A software productcan be exercised by the user in several different scenarios that the software devel-oper might not have visualised. It is just impossible to test the product under all thedifferent scenarios. Hence a careful and judicious choice must be made of the sce-narios to test under that is representative of the most common ways of exercisingthe product. During test specifications, the choice of which scenarios are to be testedh.rve to be docurnented after discussions with a cross section of the typical users.What are the criteria for test completion? During the test specification stage, thecriteria for test completion should also be documented. Some examples of comple-tior-r criteria are "achieving code coverage of (say) 95o/"" or "exercising entry to (say)80%, of the routines". There may also be hard deadlines for the product to reach themarket. Such deadlines can also be specified here and they-along with criteria likecode coverage-would act as drivers for estimating the resources needed for test-i^9.\ Test DesignHaving decided what to test at a high level during the previous step, the details ofthe layout, tooling and standards required for test development are designed in thisstep. Some of the questions taken up at this stage are:For each of the scenarios identified as important at the test specification stage, canwe drill into details of what needs to be tested (the test cases) and what should bethe expected output for each such test case? For example, suppose it is decidedduring the test specification stage that limit tests have to be done for the permissiblenumber of users who could concurrently be logged iry then during the test designphase, questions like "how many users should be tested?" are answered.Are we planning to use a tool for test automation or are the tests going.to be runmanually? Once the tests are developed, they have to be run multiple times, typt-cally, once each time a new version of the product is built. This makes test executionrepetitive and labour-intensive. Using a test automation tool alleviates the problem.Most of the commonly used tools provide mechanisms to capture or "record" tests


5/28

Managing Global Software Proiectsanaging Clob at s oJt* ?!!!!g--r. tr :rlA..I:Land,the ouputs, and then play the tests to verify that the produced output andexpected outputs match. But the decision to use a test automation tool is not a post-facto decision that is taken after the tests are developed. This decision has

to betaken during the test design stage itself as it has ramifications on the cost, effort, andthe technology to be used for test development'What kind of structure or standards are to be used for development of tests? Mosttest automation tools come with their set of recommended standards and strttc-tures. Standards cover coding standards (on how the tests are coded to faciljtatefunctionality and re-usability), documentation standards (how the tests, their fr-tnc-tion, and t6eir changes are documented) , structure standards (directory strttctureof storage) and other means of translating the test design into test cases.\. Test DevelopmentThis step is akin to the coding part of development. During this activity, tests areactuall;rwritten. The specific activities differ depending on whether an automatedtool is used or whether the testing is manual'Test AutomationOne of the biggest complaints about testing is that it is considered a manual activity'This is "rp".iully true of testing of the GUI based applications wherein a faft amountof manual operations are involved. One way to alleviate this problem is the usc oftest automation tools. Such tools provide meclranisms to

o Capture or record the user inputs. Invoke the right parts of the product (or the envit'onment) in an automatcdwalr based on the user inPuts. Specify responses the product shouid produce ("Expected Results" fromFig. L7.t).o ComPare actual results and expected results'The actions of the user and the system are coded in a way that allows the testautomation tool to run the tests automatically with minimal human interventiotr'Test automation poses several intetesting challenges:The complexity of test automatior, \rl.."uses with an increase in user interaction

and. also *h"^ there is an increase in -GUI-ness" of the application. This is because


6/28

Project Management in the Testing Phasecapturing user inputs and expected screen outputs becomes tricky. Such input/output undergoes minor changes from one release to another and hence the verifi-cation of validity of observed (actual) outputs becomes difficult. One of the ways inwhich test automation tools can overcome this problem is to capture logical screeninputs and outputs instead of physical screen inputs and outputs.

The complexity of test automation increases with an increase in the number ofplatforms on which the product is supposed to run. This is because even if an appli-cation behaviour is logically identical across platforms, the look-and-feel of the GUIcontrols is different on different platforms and hence the re-use of expected resultsalcross platforms becomes almost impossible. This problem can be overcome by us-ing a generic testing framework that works across various platforms.Most tests have some environment dependency in capturing expected results.For example, the expected output may contain the current system date, terminal,idor similar environment dependent variables. The comparison of expected resultswith observed results should not flag such environmental differences as errors. Thecomparison software should have the intelligence to represent such differences andbe able to avoid flagging these as errors.

When an automated tool is used for testing, test writing comprises the followingsteps:

1. Training the test engineers on the automated testing tools to be used. Mosttesting tools have special languages in which-the tests have to be represented.Each tool also has specific ways of capturing inputs and-outputs of a test andvalidating the results.Coding the tests in the language that the tool would understand" that wouldresult in test scripts.Running the test und"er reference conditions. This step would ensure that thetest runs and that the documented expected output is indeed consistent withwhat was originally expected.4. Capturing the inputs and outputs in this run and ensuring that this matcheswith the standard expected behaviour for fufure runs.5. Registering (baselining) the test script/the input output capture into theconfiguration management system.

So, at this stage, the test scripts are available and the standard expected inputs

2.J.

and outputs are calibrated to be the actual expected inputs/outputs.


7/28

fug Global Software Proie'cts

Manual ,Testing\Alhen the testing is manual, test writing refers to writing

detailed instructions onwhat to test u'iho* to test it. Also, tJst development in this case requires veryclear and. unambiguous d.ocumentation' An examit" or how to document a manualtest case is given in the next p.;;. il .".r.":r.ho* io tu"y out a deadlock detectiontest properry while two users ui-."r, a tabre in an Inventory application' some sali-ent points that should be kept in mind. while documenting tests for manual execu-tion are:Do not assume the tester win have an in-depth knowledge of the application: Thetester may not know the details of the appiication he is testing. For example' thestatement, ..h".k if the item has reached its re-ord'er quantity level" may be veryobvious to the developer of an Inventory application but may not make sense tosomeone who is new to the application d.omain and is testing the application'Do not assume that the tester is proficient with the deta's and syntax of the un-derlying software: For example, ir u" application is built on top of a relationalDBMS supporting SQL, it may not be sufficient to te'l the tester, ,,Subtract quantityordered from quantity on hand for the prod'uct" ' You would have to give an ex-plicit SQL syntax which the tester would have to key in'Do be specific and detail oriented: When the instructions are d'ocumented, theyshourd forlow the specific syntax and the punctuation that the tester should type in'when a GUI application is b";; tested., ihe rnstructions should go into details ofwhere the mouse shourd u" ptu.za, what kind of cursor the tester would see, whathe should do (mouse click, double click, tabbing across choices' etc') and whatwould be the result of such an action'Do not assume that a, the environment is set up and is_ready for testing: It is verylikely that a test would. require some set-up t" u" done. In the example given in theinset, the required table genq has to be created and populated with necessaryrows(Item100and.200).Itisthetestd-eveloper,s,",po,.'ibilitytoensurethatspe-cific instructions for d,oing the set-up are provided.. Itls also a good idea to drop theexisting environment and' recreate it to remove any undesirable side effects of dataleft over by the previous test runs'Do run the test on an environment distinct from the one in which the test devel-opmentwascarriedoutu"ro,","gistering/baseliningthetest:Whenatestdevel-oper develops the tests, he is .trr.r-ully ""t^i"g if'9t'' i""hit test environment whichmav not match the virgin.rrrrirorr*ent in

which the tests are suPPosed to run' For


8/28

Project Management in the Testin Phase 27,5:.example, there may be specific environment variables set up in the test developerrsenvironment which may not exist in the live test envirorunent. Thus befsrebaselining the tests, it is important to validate them in a separate environment.Do have someone else run the tests by iust following your test documentation:The person running the tests routinely may not have access to the test developerwhile running the tests. Hence it is useful to have someone else test out the testdocumentation.Do clean up after your test is over: Eventually, all the tests have to be run one afterthe other to test out the complete product functionality. Hence every test shouldclean up after it-i.e., drop the unnecessary data, unset any environment.variables,etc.

System/product:Test case purpose:Test case author:Test change history:Test description:Test requirements:

'Iest set-up:

Inventory systemDeadlock [email protected] Nov 2000 ldoe Creation15 Nov 2000 ]doe Added SQL examplesTo check out proper detection of a deadlock scenarioNeed two simultaneous sessions: administrative privilegesfor setting up the test but normal,privileges for running thetests.Have your database administrator do the fotrlowing: "createa table ITEM" by the SQL statementsdrop table ITEMS ;create table ITEMS (item-code number, qty number);insert into ITEMS aalues (100,1-0);insert into ITEMS aalues Q0A,20);Also have the database administrator create two userid'sand have him grant update privileges on ITEMSthese newiy created userid's


9/28

276 Mnn.auing Gloh nl Softanre p rtti e tt sDetailed instructions for test case

From Session # 1, Invoke the querytool for the database, login totestuserl with passw ord Xy 27

Display the message "Connected,,

Jtl,rRauting Item tubl" froSession # 1 with Item code :100 Iby executing the SQL statementupdate items set qty : L000 whereitem-code:100;Display the messnge "Connected,,

Display the messnge "Record updatecl,,

From Session #2, Invoke the querjtool for the database, login totestuser2 with passw ord XyZ2Try updating Item table fromSession # 2 with Item code :200by executing the SQL statementupdate items set qty :2000where item_code :200:try updating the item_code200 row by:update items set qty :4000where item_code :200;

From the query tool alreadyunder execution in Session # 1,

From the query tool alreadyunder execution in Session # 2,try updating the item_codeL00 row by:update items set qty : 4000where item*code :100:

The tool would utnit utith no response as therow 200 is in use ("locked") bv the.transaction from Session # 2.

Tht: system is noro Ltbottt ltt enter n deadlot:kbecause Session # I transaction is zoniting forSession # 2 to release tlrc lock on Itetn Corlt. 200rah.ile Session # 2 is wniting for Session #, 7to rclease the lock on ltem Cod.e 100. youshould sce a n "Denrllock detected"\ Test RegistrationThe tests have to be.run multiple times for each build of the product. Whenerrer aproduct is tested, it.is neceltury that the correct versions of the tests be rury just asit is the correct versions of the products that have to be used. Flence tests neecl to gcrthlgugh Proper.configuration-management. When we say "tests", we refer to theentire set that characterises a test-ihe test specification, the test scripts, the testdocumentation the test inputs that are captured and the expected test outputs thatare Produced.\ Test ExecutionThis activi{z refers to the actual execution of tests against a given build of the procl-uct' This step is repeated several times, usually ",ruiy time tie product is built. The


10/28

t in the Testing Phase 277objectives of test execution are to detect any 9ld defects that have re-surfaced and toreport any new defects. (Such a test run is called regression run-see Section 17.3.6-)

If automated tools are used" for executing the tests, such test execution can bepiggybacked on the activity of building the product. If the test execution is manual,th"r., it car-r be triggered by a formal communication to the testing team uPon suc-cessful completion of a given version of the product.

\. Test MaintenanceOver time, a product evolves with its new versions. New features are introduced'some features keep getting enhanced while some other become obsolete. In addi-tion, testing techrr,rlogy rtseH can evolve with more automation and tools' Becauseof all t6ese changes, tlhe tests and/or the expected results need to be maintained.

The existing tests may need to be changed to reflect changes in product behav-iclur. For example, the error message that a product gives under certain conditionsmay change with new versions oiproducti. This would require new versions ofexpected results to be created.Existing tests may also need to be scrapped as some of the features may be de-

supported. in newer releases. For example, if a given version of an operating systemsupports a particular type of a devi.e, the tests for that version of the os-shouldi'clucle communication with that type of device. In the future versions, support forthat type of clevice may be discontinued (as it might be replac"l by another-device)'ln such an event, it does not make sense to run the tests for the de-supported device'Any new product version would have new features. It is important that as newfeatures get added. to the product, new tests for those features also get added to thetest suites.

are quite a few similarities, betweencoderdpvetlioTo Product requirements get finaliiedi,fiist#irn

gets finalised first. ,, ,,O


11/28

itru Globnl Softzunrc Projectsl$:,;.'*"1i'alq"g5ldevelopment is a means of translating the test clesign into executabie:-1::: ' ' l:t,t,'.ilst'as. Program development is a means of representing the design/I algorithms in a programming language, eventunily complied into an

l, executable code.,.,' o'Test code., just lile the program code, needs to go through properi r Tests get debugged just as code gets debugged. Debugging a test involves

TEST SCHEDUTING AND TVPES OF TESTSffiWhendothetestingactivitiesstartandwhendotheyend?Istestingaseparate and distinct phase of product development? To answer thesequestions, let us first recapitulate the intent of any product developmentactivity, which is'to avoid product defects and in case product defects docreep in, to minimise the time delay between the point of injection and the point.fcorrection of the defect. Testing being a means of uncovering the defects in iher finalproduct (as in quality control), it is important that the clefects in the prgducts bedetected reasonably close to the point of injection and not at the epcl of the clevelop-rnent phase. Thus testing is an activity that should run parallel with deveiopmentso'that defects can be detected and corrected early in lhe cycle. But the kind oftesting that needs to be done would depend upon the level or position that a prod-uct is in. In this sectiory we will discuss the different types of tests that are done afdifferent points of product development life cycle. Each of these test types mayrequire different skill sets, different profiles, and the responsibility of the ieit activi-ties would rest with different people.

The different types of testing done during a product life cycle are:o White box testingr Black box testingo Integration testing'o ,system testingc Installation testingn Regression testing. Acceptance testing


12/28

Project Management in the TestinWe will cover each one of the above in the following contexts: ":",i:'t!t. Why and how is this type of testing is done? ,'r;. Who is responsible foi this type of tests and what are the required skill se#s?-o What are some of the common challenges in this type of tests? :l\ 17.3.1 White Box Testing

",i l

Why and How is White Box Testing Done?Using program design specifications, the programrRer produces the source cod,ethat in turn gets transformed into an executable code (by the use of compilers orother similar software). The product accomplishes different functionality or optionsby executing different paths through the code. In white box testing, the program isrun with various test data that exercise as many of the various paths in the sourceprogram as possible, thereby testing as much of the functionality as possible.'Wtlitebox testing requires knowledge of (and access to) the source code of the prOdgct.Either the developer (or someone else who can understand the source code) looks atit and identifies the various paths through the program cod.e and designs test cdsesto cover as many of these paths as possible. The test data is generated-spri{rh-g${amajority of the paths are covered. Some means of widening the coverage o$'di.f. lr"ent paths through the program are: ttl',?.,r Tests that exercise the multiple conditions in a boolean expression in variouscombinations.r Tests that exercise the THEN and ELSE parts of conditional statements.o Tests that are within the various CASE conditions of a multi-way decisionstatement like the SWITCH statement in C.o Tests that take a loop construct from 0 to the maximum value expected inf{aeloop variable.

o Tests that exercise various error conditions by giving erroneous data.r Tests that check the combinations of correct and incomect values for iriprrnti . t++] tparameters.irThe rationale behind white box texting is that if we cover as mantrpfrpaths as possible through test cases and in each case, the product behalexpected" then the product is working in accordnnce with what the code

i. l.poseq to acnleve.


13/28

280 WPro.iet:tswhite Box Testing: Responsibilities and skilr setswhite box testing clearly requires an in-depth understanding of the source codeand the paths thereof. Thus the people who clesign white box tests should be totallyfamiliar with the programming language used during development. The tcst cle-signer should also be familiar with th! logic of the product, like what are theallowed inputs to a program, what are the e"xpected outputs and behaviour at t6eProgram level' In addition, because the source code itself clranges very often, thetest developer must be aware of what the changes in the various paths url" ,n that hecan modi{z the tests as needed to get adequatJ path coverage.Given all the above factors, it becomes clear that the design of white box tests arebest done by the author of the program himself as he is faniiliar with the prograr.-ming language, program logic and the various paths within a progranr. Also, as tlr.intent is to detect any defects as soon as possible, it is also adv^isable that the devcl-oPer himself executes the white box tests before baseiining the code.

Common Challenges tn White Box TestingSimple as it may sound, white box testing does present sorne interesting challe'ges.Develope/s unwillingness to develop/run tests: Developers in general prefcr nu rit-ing new code to test development or repeated test execution. They are also trndcrPressure to produce new features/versions. Hence they ur," ,rruily unwiililg todesign and run tests.Common human limitations in finding defects in one's own work: A prograrnmerwho writes his code has a biased view or ni, own way of writing. Human nature issuch that it is very tough for one to find defects in one's own work. Furthermore, astesting is meant to uncover defects in one's own product, not many people want t.admit that their work has defects! Hence by nature, there is an ilherent reluctanccon the part of the developer to test his own code.Combinatorial explosion of possible paths and time constraints: The number ofpaths that exist in a program are potentially infinite. Testing all these paths wouldbe a huge drain on time. rhe chillenge lies in testing a reasonabie subset of thislarge,number of paths.Relating common usage paths to code paths and prioritising: If a derreloper looksat the code as a set of paths he has to &ercise / traverse, he woulcl not be able tovisualise the common usage patterns of a typical user, nor would he be able tcrunderstand which paths are critical show stoppers if they don't work. Understanrl-


14/28

Proiect Management in the T'esting Phaseing the most commonly used paths or relative criticality of paths requires looking:atthe product from an external perspective and making sure that test cases are de-signecl for such paths. To bridge this important gap in white box testing we have theblack box testing.\ 17.3.2 Black Box TestingWhy and How is Black Box Testing Done?As mentioned at the end of the previous section" the primary disadvantage of-thewhite box testing is that it is based on the knowledge of the program code from thedeveloper's perspective but not on product functionality from the user perspective.Black box testing looks at the product as a black box which is supposed to behave asper the specifications originally laid down and attempts to identify, from an exter-nal perspective, any deviations from the expected behaviour of the product. Asblack box testing looks at the product from an external perspective, there are multi-ple approaches possible to design black box tests. Presented below is one,approaehthat is usually quite effective:

1. From the requirements specifications, the product is broken down into: itssalient features or logical modules. In fact, a well laid out requirementsspecification would already have identified the distinct features of theproduct. A feature is realised by a collection of source files but since black boxtesting consciously avoids looking at the program source, the test designer isallowed to look only at the external behaviour of the product and not at theimplementation in the source files.2. For each feature, tests are designed for functionality, usability, limits, errorhandling and interfaces.

Functionality testing: This type of tests exercises the basic functionality of the prod-uct from an external perspective. Some typical functionality tests are:

o Tests to ensure that proper input data is accepted.o Tests to ensure thaf the internal calculations and transformations result inappropriate intermediate results.o Tests to ensure that the external outputs match what is expected in terrns, ofcontent and layout.


15/28

282Usability testing: Some examples of usability testing are:r Does the user interface match the GuI standards of the platform?o Is there consistency in the user interface across procrtrct(s)?' Does the navigation match what is documented?' Is there consistency in the reports - both in content a'cl forrnat?Limit testing: A product feature may have in-built hard limits that are independentof the environment it is running on. some examples of such limits are the maximu'rize of a particular data item, maximum logical size of a file,maximum number .foncurrent users on the system' etc. Limit tlsting refers to creating tests to verify:r that the system is able to handle the values within and including these iim*sproperly.o that the system is able to handle the exceeding of such rimits i. a g;racefurmanner/ i'-"' uy displaying an error message .uthe, than a system crash .rome similar catastrophic outcome.Error handling tests: It is important to validate product behaviour not only for cor-ect data but also for incorrect data. The system shourd handle error conditionsgracefully and in a user friendly manner. some of the error checking tests that needo be done include:. Testing data inputs of wrong type or format.o Testing data inputs of wrong size (covered in limits tests above).o Testing inconsistent data combinations.r Testing that the right message is displayed for the appropriate errorcondition.r Testing that error messages are not given for non-error sifuations.Interfaces testing: Each module (feature) should talk to the other modules (fea-fures) through well defined interfaces. Interface testing is done to ensure that properinterfaces are passed across modules. Interfaces can be external, (like files) or inter-nal' (like internal data structures). since all the modules may not be available at thesame time for testing, it is important to document clearly what the interfaces are sohat such interface test data can be generated manualry and the tests carried out.Blaek Box Testing: Responsibirities and skiil setsBlack box testing requiresexternal functionality of a an in-depth understanding of the requirements and thefeature. Thus the people who design black box tests need

9!4ol Softutare pro


16/28

Prrtlt:t:t Managarnent in tlrc'Testittg Phase 283not be totally familiar with the programming language used during development'T6e test designer shoulcl be able to visualise how the feature would be used by theusers, what th" .o**only used functionality are, and what would be the expectedbeharviour und.er various scenarios. Hence the black box tests development team isusually different from the people who actually develop the code'Common Challenges in Black Box TestingAmbiguous or unclear expected behaviour: The success of black box testing is dic-tated ily ho* well the external behaviour of the product is understood. When theclevelopment cycles are short, it is often the case that the requirement or designspecifications leave some loose ends in terms of the expected product behaviour'

Itc''lcl also be that all the possible usage scenarios are not envisaged or documented'Hencer the tester may have his own v-iew of "what is the correct behaviour", whichmay be distinct from what the product was originally intended to do' One way toremove the resultant subjectiviiy in testing is to tighten the Requirement and De-sign to describe unambiguotrsiy the expected behaviour in various scenarios'Black box testers viewed as adversaries to development: Because of skill sets re-ciuirecl and the nature of black box testing, the black box testing grouP is kept sePa-rate, from the developmer:rt glouP. As the testing grouP is distinct and their job is tofi.d defects in the products proiuced by the developers, there.is a possibility of thetwo groups viewing uu.h oih", as ard,versaries. Some ways of tackling this feelingar:e discussed later irr the chapter'llelating common usage patterns and prioritising: when the product usage isviewecl from an external perspective, myriad potential usage patterns are possible'If one were to test all the porribl" scenarios, it would be prohibitively expensive andtime consuming. 'rhe chailenge (as in white box testing) lies in visualisaing the com-nlon usage patlerns, prioritising them and ensuring that they are tested'\ 17.3.3 Integration Testingluhy and How is Integration Testing Done?tslack box testing tests the functionality and external behaviour of an individualmodule or a featrlre. A product works as a *hote by interactionbetween the variousmodules/features. Integration testing is d.one to test if the individual parts worktogether as one single unit.


17/28

284 Manoging Gktbal SoftToare ProjectsOne of the steps we had listed in black box testing was interface testing-making

sure that each module or feature produced the right interfaces to be used by othermodules or features. We also mentioned that since all the interfaces may not beavailable at the same time, interface data may have to be generated manually fortesting purposes. When we come to integration testing, the modules that are to beintegrated are available for testing. Thus the manual test data that was used to testthe interfaces is replaced by that which is generated automatically from the variousmodules and can be used for testing how the modules would actually interact.

There are various sequences in v,zhich the modules can be integrated and tested.A couple of important drivers for deciding the order: of integration and testing are:. Availability of modules to be integrated: Each module loes through itsdevelopment schedules. If a module becomes available for testing, it wouldbe expedient to test it with other modules that are ready, which will optimisethe time needed to do the integration testing. This will be dictated by theavailabilif of WBS units and milestones during the project planning phase.. Modules with maximum impact: Among the modules that are avaiiable,integrating those with the maximum impact (i.e., those that interface withmaximum number of modules) earlier will detect the integration errors earlier.

In contrast to the above staggered approaches to integration testing, one can alsoconsider doing a Big Bang integration testing wherein all the modules are put to-gether and tested in the end. This may simulate the real life use of the product bythe customer better but would be successful only if the planning and organisation ofinterfaces is done with a high degree of perfection. Since the number of interfacesand integration points are very high in a typical real-life software product, an un-planned integration testing is very likely to lead to wasted resources by not testingthe most fypical integration points.lntegration Testing: Responsibilities and Skill SetsThe type of skills that an integration test development team has are very similar tcrthose of the black box test development team. Both have good product knowledgefrom an external perspective and both know the basic modules / features within theproduct. The one difference in the responsibilities arises from the dependencies thatexist-across modules. Flence the need to communicate across different groups andthe soft skills required is generally higher for integration test team than for the blackbox team. Also, because of the multi-group dependencies, the integration test tearnmay have a different reporting structure then that of the black box team. We wjllcover this in Section 17.5.


18/28

Project ManagenrcntCommon Challenges ln lntegration TestingVisualising integration tests: Integration testing takes the testing one step closer tothe actual product usage. Truly successful integration testing calls for a soundknowledge of how the various parts should fit into the whole. This requires some-one who understands the external product functionality as well as the internal unitsof integration. To find this combination of skill sets is not easy.Schedule conflicts because of availability: Once a module reaches a certain stageof completion, it would require some other modules in order to proceed. But be-cause of scheduling conflicts, these other modules might not be available. Thus inte-gration testing rrray come to a standstill. In order to avoid this sifuatiory one mayhave to, at times, proceed further with development and defer integration testing.Assigning responsibilities to problem areas: During white box or black box test-ing, when a problem or an anomaly is observed, there is no controversy about as-signing responsibility to fix the problem since the unit being tested is self contained.But in integration testing, we are taking modules developed by different groupsand putting them together for testing. When a problem comes up, it is sometimesvery difficult to pin-point where the problem lies and who should take the correc-tive action. If the interfaces are perfectly defined, there would be no such controver-sies. But in real life specifications, there are always unanticipated scenarios and un-defined interface values. In this case, there has to be some mechanism of assigningresponsibility of fixing the problems that get exposed.Communication: In white box and black box testing, the testing is carried out prettyciclse to the point of development as there are very few external dependencies. lnte-gration testing depends on the availability of several modules. Hence, triggeringthe start of integration testing and ensuring smooth progress and defect resolutionis dictatedby very good and well defined communication channels across the vari-ous grouPs.\ 17.3.4 System TestingWhy and How is System Testing Done?Like integration testing, system testing is aimed at exercising the product as .awhole. And unlike integration testing, the focus during system testing is on stress-ing the system under extreme conditions and ensuring that if there is any failure, itis weil managed. Some of the aspects of system testing are:


19/28

Ma n a ght g Glob al S oftut ar1l,1;9Load testing: This subjects the product to a very abnormal work load andensures thai there are no adverse effects. Son're examples of load tests at'e:testing with a high number of concurrent users and ensuring that the systemdoes not break; hooding the system with excessive input requests; trlzing 1opump in more data than what the network bandwidth or other similar:physical limits can handle, etc.configuration testing: Every product should have the specifications of thchardware configurations that it runs under. Configuration testing is a mt'attsof running the software product und.er various combinations of theseconfigurations. For exampte ir the product is certified to run with a mitrimum*"*ory of 16M8, then the tests must ensure that the product does rttn itr amachine with 16MB memory.Integrity testing: Regardless of the things that go il,rlt)rrg in the environntetlt,the froduct should not compromise the integrity of the data it handies' Fot"xu*plu, there should be adequate protection and recovery from events likepo**i failure, media crashes, "t.. k t"grity testing is aimed at testing featureslike recovery, authenticatiory etc.fl Inter-op"rulility tisting: In today's comPonentised worid, a product wotrldbe expected to inter-operate with other products, whether from the sarnevendor or from di{ferent vend,ors. While doing Inter-operability testing, tlreapproach should. be to specify which versions of which products should inter-oplrate with each otheiand test such combinations" Inter-operabilitl' testingusuallybecomes increasingly complex as the product matures (as there wouldbe more versions to test inter-operability) hnd increasingly expensive toobecause of the number of combinations possible'

System Testing: Responsibilities and Skill SetsThe skill sets of system testing encompass and transcend the skill sets of integrationtesting. The added requirement is thai in tests like load testing, the person develop-ing and running the tests must be throughly familiar with the nuts and bolts of theunderlying system software. For example, load testing has a significant depend-ence o1 th. hardware configuration and system tuning. Such skill sets are fairlyniche and difficult to come bY.Common Challenges in System TestingDefining realistic scenarios: Defining what constitutes realistic scenarios is a verycomplex task. For a general purpose software-fs6ause each customer's systenrcould be different -what constitutes a good system test for:

one customer may be

n

tr

. _ _.,.- - -., ..,.,..,,.;.'. ..,;{:t:::,:-:tlAigr*!.*",:ri'*;j*ftfiff#


20/28

Matngement in the'lestitrg Plmse 287for another customer. Even for a bespoke product, the usage patternsfrom time to time.

mix of skill sets needed: As discussed in the previous sectiory system test-requires a deiicate combination of visualising the 20,000 foot level view of howend product would be used, along with the nuts and bolts of how to exploit thefeatures of the underiying system. This is a fairly specialised skill set.needed to perform tests: Most of the system tests described above testunder extreme conditions. Tests like performance tests require high endthat are expensive to acquire.

Most problems that show up during system testing do so becausea complex combination of factors from the product, the software environmentthe hardware configuration. Thus, most "problems" found during the integra-testing tend to be difficult to reproduce and resolve.

Identification: A natural consequence of the difficulty to reproducein the system tests is that problem analysis and responsibility allocation be-a more serious issue than in the case of integration testing. Does the problemin the product or in the environment? Which should be corrected? These arequestions with no easy answers.

17.3.5 Installation Testingfar we have been concentrating on various features and the behaviour of a prod-But before the product can be used on a customer site, it has to be installedInstallation tests are are done to ensure that the product is packagedand can be installed successfully using the instructions given in the instal-documentation. The steps that are followed for carrying out the installation

are:1. Packaging: Once the product features are tested, the product needs to bepackaged for use by the customers. Packaging can be the creation of a mastercopy in a media like a CD or the creation and publishing of a location (ftp site/ URL) from where the user can download/install the product. Thepackaging has to be easy to use, customisable and conJorming to any platformor product standards.2. Documenting: Accompanying the product would be some documentation;rbout how to install the package. Such installation documentation should


21/28

2&8 Managing Global Softwarc Proiactscontain screen captures of the various screens that would come up duringinstallation, speciiy various options possible and the actions in each of theseroptions and lead the user to successfully install the product in hisenvironment from the media'3. Installing: The person performing the instaliation tests should get the mediaand the documentation listed uiorru, follow the instructions given in thedocumentation and ensures that. The screen shots and the defaults shown ' in the documentation are:

accurate. There is a match between the documentation and the acttial behaviour ofoptions chosen, installation flow, and the results ob'"':u',1i', t1o If erroneous inputs are given, the installation system hanples the errorsgracefully and gives apPropriate error messages' 'i. No extraneous or repetitive information is to.tgfot from the LlserunnecessarilY.4. Verifying: Every installation packaging should have some means ot vertfyrngthat the installation has been .orr,pt"t"J ,.r..urrfui1y. After installation' a "selftest" suite should be designed ind run. Such a self test should have clear

success criteria.

\. 17.3.6 Regression Testingso far, we have seen a number of different types of tests that need to be run atdifferent points of time during product development. As discussed earlier in ihischapter, product development joes in iterations of built-test-fix. It is obvious thatfrom one cycle to the rr"*t, the older defects shouid not re-surface'Let us illustrate this with an examPle: During a given build and test cycle' defectsDL, D2 and D3 are discovered.. The root cause is found, the necessary correctiotrsmade, and the tests are re-run during that cycle. It is confirmed that the defects arefixed. During the next build. cycle, new defects D4 and D5 are found on the testing'These defects are corrected and the tests are re-run. Now there is a thance that thefixes for D4 or D5 may break the fix made forDl/D2/D3' Such a re-appearance ofan earlier defect is termed. as a regress. In the example above, if the tests for: fixes ofD1,/D2/D3 are also run at the end of the second cycle, then a regress of D1 /D2/D3would have been caPtured.

Regression tests are then defined as those tests that are run to verify that prob-lems do not resurface (or regress)'


22/28

. ,tests for defects fixed in the last few cyclesRegression tests are usually designed to run automatically after each time a prod-

uct is built. This reduces the tedium of running the tests repeatedly. ]ust like in anyother forms of testing discussed above, the choice of what to test (and include inregression tests) should be done judiciously, balancing the knowledge of the inter'nal code and design with the common external usage scenarios. As regression testsare run every time the product is built, it is important to keep the size of these testswithin manageable limits so that they can be completed in reasonable time:,,F'iil-p$Gesomeone in the organisation should be assigned the responsibility of perie{ical,lylooking at the list of regression tests and keeping them updated by deleting,;*urt-wanted tests and by adding tests for the new features. '\ 17.3.7 Acceptance TestingWhen a product is designed for use by a specific customer, it is usually required topass certain tests that the customer considers representative of his typical environ-ment. During the time that the initial requirements are arrived at (when the contractis signed), a set of acceptance tests would be specified by the customer. The per-forrriance of a product would be considered satisfactory (by the customer) only if ithas passed the acceptance tests. Such acceptance tests typically require the productto be tried using splcific data provid.ed by the customer. Moreover, it should alsoproduce the spJciic outputs that the customer exPects to s-ee._ In addition to suc-hiunctionality based tests, the acceptance tests may also include perforrnance teststhat test the system under load from a number of users, delivering a minimumresponse time or throughPut.PEOPLE ISSUES IN TESTING

Out of all the life cycle activities in software development, theonelSc{v-ity that faces the highest number of people related issues is testitilg'okmost conunon manifestation of such people issues is thatployees d.o not want to be in the role of a tester. There is


23/28

Managing Glolt sl Softwnre Proj ectsthat'testing is a mundane job and does not require any special skills. Given a choice,most people prefer to be in development rather than in testing. There are sever:alreasons for this misconception:

1". Testing is not formally taught in many of the institutions.Testing as a branch has not received the same amount of publicity andattention as design in terms of working groups and forums.Management attention tends to be biased towards developersManagement sometimes does not "walk the talk" when it comes to acting ondefects uncovered by the testing teamPeople not seeing a career in testing, view testing as a stepping stone to

2.ao.4.

graduating to development.

Whl I e,,.mana g,i r,rjg,:19"1!! n g.,te arn s,, Ih e r e a r earnong,lSel,tesii{g rn6:'!"! 'will do: testing,lq::,staqt,with but canproject in the next six rnanths?"191'esl.people do-'not perceive'there is a career in testing, whereas they do feel thalthereiisiacareer in "development", which is equated to coding. lt is important forthe management to provide',role models who make a career irr testingoTesting is just data entry"

This misconception is carried over from the earlier days of testing rn,here one hadto do manual testing. With the advent of test automation tools, this is no longertrue. Testing can focus more on test design than test executionSuch comments can get aggravated by comments like the following from man-agement:"Let us put our best engineers in development and assign testing to rookies"It is.i,mportant that management exhibit their commitment towards the testingfunction by assigning some of the best engineers for testing and by rotating peo-ple between development and testing functions."Let us do atl the testing after development"

5.

in Managing Testingsome comffron themes that are seen

you move me to a lava programming

,, '. ': .;:. :


24/28

291Project Nlanagement in the'festing PhaseIt is very irnportant to start testing functions as early as possible in the product lifecycle. Test'specifications can be evolved right from the requirement or: design,iog"r; test developrnent can run in parallel with product developrnent'to enableearly beginning to testing',,Let us do whatever testing we can do in the nexttwo weeks,-befare,th.7lteleas3,"Testing should not always be eonstrair.red,rbV,tirne. The end criteria:{or:19.+iigslrould not be just "oh it is timerfor the releisel" but that a given amount:sf'codecoverage or bug fixes is achieved.

How does one contencl with such perceptions? A few simple principles can beacloptecl by the organisation's senior management as well as the project manager toattract and retain people who want to do testing:1. Treat the clevelopers and testers on par in terms of recognition, comPensation,etc.2. Demonstrate tlrat the testing team is equally important for example, if thetesting tearn uncovers ,"rio.r, defects in the product, ensure that the,Cevelopers clo fix the defects rather than just going ahead and releasing theprociuct.3. Do not assigr-r ali the best engineers to development; assign some to testing aswell.4. Design ancl show a career path for the discipline of testing5. Rotate peoplc. between dlvelopment and testing so that the developersunderstand the fact that testing is challenging6. lnvest in skill clevelopment of the testers. A key part in the success of a tester'sjob is soft skills. This includes:. cleveloping better attitude and pride in testing among testers andclevelopersr being able to communicate the d.efects with the sole intent of getting the

product defect free.r not projecting oneself as being an adversary to the developer.MANAGEMENT STRUCTURES FOR TESTING IN GLOBAL TEAMS

How clo you organise the testing and development teams to minimiseconflicts and deliver a time bound, high quality product? We presentbelow some possible organisational structures-initially starting with


25/28

2j92 Managing Globol Software Projectsgeneric strucfures and then bringing into the picture the new possibilities as dem-onstrated in the case of geographically distributed teams.lntegrated Testing Team Model

.lf fig, tZ.Z Integrated testing teamIn the above model (Fig. 17.2), there is one project manager wlro manages boththe development and the testing functions. By making the testing and development

teams a part of one physical team reporting to a single manager, this model presentssome advantages:-+ The project manager can multiplex the development resources with testingresources. A typical product development cycle is front loaded with the needfor development resources and rear loaded with the need for testingresources. Witt all the resources in one basket, there are opportunities forbetter load balancing and resource utilisation.=+ The chances of conflicts between the testing and development teams arereduced because the teams have many opportunities for interaction. The teamwork can further be enhanced by job rotation and rnultiplexing of testers anddevelopers.However, this model also has some problems that one should watch out for:

o Entrusting operational responsibilities for both development and testing tothe same manager opens the doors to potential conflicts. It is easy to glossover some of the defects to meet the deliveries. Also, when testing anddevelopment are done by the same set of people, it is possible that different


26/28

nt in the Testinperspectives would not come into the picture' Testing degenerates to:plovingthat the prod.uct works rather than finding defects in the product'

o Testing and d.evelopment require somewhat complimentary skill sets' It isnot always possible to make a develoPer do the testing'\ Dedicated Testing Team Model

.ll fig. tZ.e Dedicated testing teamIn this model (Fig. l7.g),the testing team reports directly to the next level ofsenior management]thus the conflicts between-operational derivery responsibili-ties and testing responsibilities are avoided''There are some finer variations possible in this mod'el: The testing team ca1!lr-

ther be broken down into sub-teams and each sub-team assigned the responsibilttyof a particular procluct. Alternatively, the entire testing team can be considered asone set of resources who are assigr,La th" testing of d.ifferent products on a needbasis.

This model has two primary d.isadvantages: First, because of the direct reportingline of the testing team to the senior management, there is a greater chance of thetesting team perceiving the development team as an adversary or vice versa' sec-ond., there is very little opportunity ror diversity in work: testers do the testing-anddevelopers the i".rutoping. This lack of diversity courd eventually end't*rp.;inrreduced motivation.


27/28

Managing Globnl Softutnre Proj ec.ts\,rTesting Teams in Geographically Distributed Product TearnsGeographically distributed product teams present some more opportunities for: or'-ganisation structures and for maximising the quality and optimising the resources..One way to divid"e the product between different locations is to break it into dif-ferent components and give the complete component responsibility on specific com-ponents to each location.

Each component is like a mini-project in itself. Thus the moclel is very similar tothe integrated testing team model presented above; with each location managermanaging a self contained mini-projecf spanning both development and testing.

.I fig. tZ.a Self-contained locationsOne possible variation to the above model is to let one location test the partsthe product developed in another location. This cross-location testing, illustr.atedFig, 17.5, offers some very significant advantages:

Gperational responsibilities in testing and development are distributedBecause both locations get involved in all the product components, there is awider distribution of product knowledge.

ofirr

=+=+

il,bcation 2 Project


28/28

l'rojec't Munugement in the'festing Phltt 295The primary disadvantage of this model is the excessive need for communica-tio1. When the clevelopment and testing teams are co-located, there are more av-

enues for face-to-face interaction and communication. In addition to enhancingteerm work, such interaction leads to speedier resolution of issues and hence abetterthroughput.

. I fig. t Z.S Cross location testingThe finai variation in the case of global teams is the concept of creating testingcompetency centers by designating a particular location for testing. This is verysimilar to the dedicated testing team model discussed above. The added advantagethat i;eographic distribution brings about is the exploitation of time difference. Prod-

ucts clevelopecl during day time in one location can be tested during the subsequenthr>urs in another location, thereby fully exploiting aLL24 hours.\ Hybrid Models'Ihere are various combinations possible from the above basic models. One of themis assigning white box testing to the developers themselves, while the black boxtesting team is a separate te'am under the project manager and the integration test- . ,ing team reports diiectly to the next level of senior management. The advantage of lhvu^pad i, hhz.'| 'd- d^@ .yoh o\c 4 Itu luqw ^Q xIa,u^gd'ibn

I

project management in the testing phase1

Documents