Undergraduate Topics in ComputerScience

Series Editor

Ian Mackie, University of Sussex, Brighton, UK

Advisory Editors

Samson Abramsky, Department of Computer Science, University of Oxford,Oxford, UKChris Hankin, Department of Computing, Imperial College London, London, UKMike Hinchey, Lero – The Irish Software Research Centre, University of Limerick,Limerick, IrelandDexter C. Kozen, Department of Computer Science, Cornell University, Ithaca,NY, USAAndrew Pitts, Department of Computer Science and Technology, University ofCambridge, Cambridge, UKHanne Riis Nielson , Department of Applied Mathematics and Computer Science,Technical University of Denmark, Kongens Lyngby, DenmarkSteven S. Skiena, Department of Computer Science, Stony Brook University, StonyBrook, NY, USAIain Stewart, Department of Computer Science, Durham University, Durham, UK

‘Undergraduate Topics in Computer Science’ (UTiCS) delivers high-qualityinstructional content for undergraduates studying in all areas of computing andinformation science. From core foundational and theoretical material to final-yeartopics and applications, UTiCS books take a fresh, concise, and modern approachand are ideal for self-study or for a one- or two-semester course. The texts are allauthored by established experts in their fields, reviewed by an international advisoryboard, and contain numerous examples and problems, many of which include fullyworked solutions.

More information about this series at http://www.springer.com/series/7592

Gerard O’Regan

Concise Guide to SoftwareTesting

123

Gerard O’ReganSQC ConsultingMallow, Cork, Ireland

ISSN 1863-7310 ISSN 2197-1781 (electronic)Undergraduate Topics in Computer ScienceISBN 978-3-030-28493-0 ISBN 978-3-030-28494-7 (eBook)https://doi.org/10.1007/978-3-030-28494-7

© Springer Nature Switzerland AG 2019This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or partof the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmissionor information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilarmethodology now known or hereafter developed.The use of general descriptive names, registered names, trademarks, service marks, etc. in thispublication does not imply, even in the absence of a specific statement, that such names are exempt fromthe relevant protective laws and regulations and therefore free for general use.The publisher, the authors and the editors are safe to assume that the advice and information in thisbook are believed to be true and accurate at the date of publication. Neither the publisher nor theauthors or the editors give a warranty, expressed or implied, with respect to the material containedherein or for any errors or omissions that may have been made. The publisher remains neutral with regardto jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AGThe registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

To

Noel, Maura and Ger Forde(For friendship)

Preface

Overview

The objective of this book is to provide a concise introduction to the softwaretesting field to students and practitioners. The principles of software testing arediscussed, and the goal is to give the reader a grasp of its fundamentals, as well asguidance on applying the theory in an industrial environment.

Organization and Features

Chapter 1 discusses fundamentals of the software quality field, including a historyof quality and the pioneers who have influenced the field. We discuss the contri-butions of Deming, Juran, and Crosby, as well as the work of Watts Humphrey,who is considered the father of software quality.

Chapter 2 presents the fundamentals of software engineering, andwediscuss varioussoftware lifecycles used in software development. We discuss the various activities inthe waterfall model, including requirements gathering and specification, softwaredesign, implementation, testing, andmaintenance. The lightweight Agile methodologyhas become very popular in industry replacing the traditional waterfall lifecycle model.

Chapter 3 discusses the fundamentals of testing in traditional software engi-neering, and we discuss the various types of testing that may be carried out duringthe project. We discuss test planning, test case definition, test environment set-up,test execution, test tracking, test metrics, test reporting, and testing in ane-commerce environment.

Chapter 4 discusses static testing, which plays an important role in buildingquality into a product. We discuss the well-known Fagan inspection processdeveloped at IBM in the 1970s, as well as lighter review and walkthroughmethodologies. We discuss the static code analysis of software code without exe-cuting the code, which is performed with automated tools.

Chapter 5 discusses software test planning, which involves defining the scopeof the testing to be performed; defining the test environment; estimating the effortrequired to define the test cases and to perform the testing; identifying the resources

vii

needed (including people, hardware, software, and tools); assigning the resources tothe tasks; defining the schedule; and identifying any risks to the testing andmanaging them.

Chapter 6 discusses test analysis and design, which is concerned with deter-mining the test conditions, and designing the test cases (using various techniques)for the testing. The requirements and test conditions are used to specify the testcases, where each test case includes test input, the procedure for carrying out thetest, and the expected results.

Chapter 7 discusses test management, which is concerned with the activitiesinvolved in managing the software testing process. A well-planned and managedtesting process enables teams to deliver high-quality products on time and onbudget. A good test process is repeatable and predictable.

Chapter 8 is concerned with test outsourcing, which involves the selection andmanagement of a testing supplier. It discusses how candidate suppliers are identi-fied, formally evaluated against defined selection criteria, and how the appropriatesupplier is selected. The selected supplier is then managed during the testing.

Chapter 9 is concerned with test metrics and problem-solving, and we discussthe well-known goal, question, metrics (GQM) approach. This enables metricsrelated to the organization goals to be defined. A selection of metrics is presented,and several problem-solving tools such as fishbone diagrams, Pareto charts, trendcharts are discussed.

Chapter 10 discusses tools to support various software testing activities. Thefocus is first to define the process and then to find tools to support the process.Tools to support test management are discussed, as well as tools to support testdesign and execution. Finally, we discuss tools for static testing, performance andmonitoring tools, and defect tracking tools.

Chapter 11 discusses test process improvement and begins with a discussion of asoftware process and the benefits that may be gained from a software processimprovement initiative. We discuss the Capability Maturity Model Integration(CMMI) and dedicated test process improvement models such as the TMM, TPI,TMap, STEP, and CTP.

Chapter 12 discusses testing in the Agile world. Agile is a popular lightweightapproach to software development, and it provides opportunities to assess thedirection of a project regularly throughout the development lifecycle. Ongoingchanges to requirements are considered normal in the Agile world, and Agile has astrong collaborative style of working. It advocates adaptive planning and evolu-tionary development.

Chapter 13 discusses the verification of safety-critical systems, where suchsystems often need an extra level of assurance in their correctness. Formal methodssupport the development and verification of safety-critical systems, and they consistof a set of practical mathematical techniques. They may be employed to rigorouslystate the requirements of the proposed system and to derive a program from itsmathematical specification. Further, they may be employed to provide a rigorousproof that the implemented program satisfies its specification.

viii Preface

Chapter 14 discusses legal and ethical aspects of software testing, and we discussprofessional responsibilities of the professional tester. We discuss legal aspects offailure including lawsuits and the law of tort.

Chapter 15 discusses software configuration management and discusses thefundamental concept of a baseline. Configuration management is concerned withidentifying those deliverables that must be subject to change control and controllingchanges to them. We discuss the importance of configuration management in thetesting field.

Chapter 16 is the concluding chapter in which we summarize the journey that wehave travelled in this book.

Audience

The main audience of this book are computer science students who are interested inlearning about software testing and in learning on how to build high-quality andreliable software on time and on budget. It will also be of interest to industrialistsincluding software engineers, software testers, quality professionals and softwaremanagers, as well as the motivated general reader.

Acknowledgements

I am deeply indebted to family and friends who supported my efforts in thisendeavour and my thanks, as always, to the team at Springer. A special thanks tothe Fordes for their friendship and camaraderie over many years (Noel for leavingme in a cloud of dust during the An Oige cycles; Ger for fun and entertainment; andMaura for being a voice of reason and keeping everything together).

Cork, Ireland Gerard O’Regan

Preface ix

Contents

1 Fundamentals of Software Quality . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 History of Software Failures . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Background to Software Quality . . . . . . . . . . . . . . . . . . . . . . 4

1.3.1 What Is Software Quality? . . . . . . . . . . . . . . . . . . . . 51.3.2 Early Quality Management . . . . . . . . . . . . . . . . . . . . 51.3.3 Total Quality Management . . . . . . . . . . . . . . . . . . . . 61.3.4 Software Quality Control . . . . . . . . . . . . . . . . . . . . . 6

1.4 History of Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.4.1 Shewhart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.4.2 Deming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81.4.3 Juran . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101.4.4 Crosby . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121.4.5 Watts Humphrey . . . . . . . . . . . . . . . . . . . . . . . . . . . 161.4.6 Miscellaneous Quality Gurus . . . . . . . . . . . . . . . . . . 18

1.5 Modern Software Quality Management . . . . . . . . . . . . . . . . . 191.5.1 Software Inspections . . . . . . . . . . . . . . . . . . . . . . . . . 191.5.2 Software Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . 201.5.3 Software Quality Assurance . . . . . . . . . . . . . . . . . . . 201.5.4 Problem-Solving Techniques . . . . . . . . . . . . . . . . . . . 211.5.5 Cost of Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221.5.6 Software Process Improvement . . . . . . . . . . . . . . . . . 231.5.7 Software Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 241.5.8 Customer Satisfaction . . . . . . . . . . . . . . . . . . . . . . . . 251.5.9 Assessments (Appraisals) . . . . . . . . . . . . . . . . . . . . . 261.5.10 Total Quality Management . . . . . . . . . . . . . . . . . . . . 27

1.6 Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281.6.1 Organization Culture and Change . . . . . . . . . . . . . . . 281.6.2 Law of Negligence . . . . . . . . . . . . . . . . . . . . . . . . . . 291.6.3 Quality and the Web . . . . . . . . . . . . . . . . . . . . . . . . 30

1.7 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

xi

2 Fundamentals of Software Engineering . . . . . . . . . . . . . . . . . . . . . 332.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332.2 What Is Software Engineering? . . . . . . . . . . . . . . . . . . . . . . . 362.3 Challenges in Software Engineering . . . . . . . . . . . . . . . . . . . . 392.4 Software Processes and Lifecycles . . . . . . . . . . . . . . . . . . . . . 40

2.4.1 Waterfall Lifecycle . . . . . . . . . . . . . . . . . . . . . . . . . . 412.4.2 Spiral Lifecycles . . . . . . . . . . . . . . . . . . . . . . . . . . . 412.4.3 Rational Unified Process . . . . . . . . . . . . . . . . . . . . . . 432.4.4 Agile Development . . . . . . . . . . . . . . . . . . . . . . . . . 44

2.5 Activities in Waterfall Lifecycle . . . . . . . . . . . . . . . . . . . . . . . 462.5.1 User Requirements Definition . . . . . . . . . . . . . . . . . . 462.5.2 Specification of System Requirements . . . . . . . . . . . . 472.5.3 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482.5.4 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492.5.5 Software Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . 492.5.6 Support and Maintenance . . . . . . . . . . . . . . . . . . . . . 51

2.6 Software Inspections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 522.7 Software Project Management . . . . . . . . . . . . . . . . . . . . . . . . 522.8 CMMI Maturity Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 532.9 Formal Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 542.10 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552.11 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

3 Fundamentals of Software Testing . . . . . . . . . . . . . . . . . . . . . . . . . 593.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 593.2 Software Test Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613.3 Software Test Planning and Scheduling . . . . . . . . . . . . . . . . . 653.4 Test Case Design and Definition . . . . . . . . . . . . . . . . . . . . . . 663.5 Test Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 673.6 Test Reporting and Project Sign-off . . . . . . . . . . . . . . . . . . . . 673.7 Testing and Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

3.7.1 What Is a Software Defect? . . . . . . . . . . . . . . . . . . . 683.7.2 Is Exhaustive Testing Possible? . . . . . . . . . . . . . . . . . 683.7.3 How Much Testing Should Be Done? . . . . . . . . . . . . 693.7.4 Testing and Quality Improvement . . . . . . . . . . . . . . . 69

3.8 Psychology of Software Tester . . . . . . . . . . . . . . . . . . . . . . . 703.9 Test-Driven Development . . . . . . . . . . . . . . . . . . . . . . . . . . . 723.10 E-Commerce Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 723.11 Traceability of Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 743.12 Software Maintenance and Evolution . . . . . . . . . . . . . . . . . . . 743.13 Software Test Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 753.14 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 773.15 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

xii Contents

4 Static Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 794.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 794.2 Economic Benefits of Software Inspections . . . . . . . . . . . . . . 814.3 Informal Reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 824.4 Structured Walk-through . . . . . . . . . . . . . . . . . . . . . . . . . . . . 834.5 Semi-formal Review Meeting . . . . . . . . . . . . . . . . . . . . . . . . 834.6 Fagan Inspections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

4.6.1 Fagan Inspection Guidelines . . . . . . . . . . . . . . . . . . . 884.6.2 Inspectors and Roles . . . . . . . . . . . . . . . . . . . . . . . . 894.6.3 Inspection Entry Criteria . . . . . . . . . . . . . . . . . . . . . . 894.6.4 Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 904.6.5 The Inspection Meeting . . . . . . . . . . . . . . . . . . . . . . 904.6.6 Inspection Exit Criteria . . . . . . . . . . . . . . . . . . . . . . . 914.6.7 Issue Severity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 914.6.8 Defect Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

4.7 Automated Code Inspections . . . . . . . . . . . . . . . . . . . . . . . . . 944.8 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 964.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

5 Software Test Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 995.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 995.2 Test Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

5.2.1 Estimation Techniques . . . . . . . . . . . . . . . . . . . . . . . 1035.2.2 Work Breakdown Structure . . . . . . . . . . . . . . . . . . . . 103

5.3 Test Planning and Scheduling . . . . . . . . . . . . . . . . . . . . . . . . 1035.4 Risk Management in Testing . . . . . . . . . . . . . . . . . . . . . . . . . 1075.5 Dedicated Test Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1095.6 Monitoring and Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

5.6.1 Managing Issues, Change Requests, and Defects . . . . 1115.7 Project Governance During Testing . . . . . . . . . . . . . . . . . . . . 1125.8 Test Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1125.9 Lessons Learned and Project Closure . . . . . . . . . . . . . . . . . . . 1145.10 Configuration Management . . . . . . . . . . . . . . . . . . . . . . . . . . 1155.11 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1155.12 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

6 Test Case Analysis and Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1176.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1176.2 Requirement Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . 1196.3 Test Case Design Techniques . . . . . . . . . . . . . . . . . . . . . . . . 120

6.3.1 Black Box Testing . . . . . . . . . . . . . . . . . . . . . . . . . . 1206.3.2 White Box Testing . . . . . . . . . . . . . . . . . . . . . . . . . . 1256.3.3 Experienced-Based Testing . . . . . . . . . . . . . . . . . . . . 127

Contents xiii

6.4 Test Case Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1286.5 Requirement Traceability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1306.6 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1316.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132

7 Test Execution and Management . . . . . . . . . . . . . . . . . . . . . . . . . . 1337.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1337.2 Test Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

7.2.1 Test Team Organization . . . . . . . . . . . . . . . . . . . . . . 1357.3 Test Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1367.4 Managing Defects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1377.5 Managing Change Requests . . . . . . . . . . . . . . . . . . . . . . . . . . 1397.6 Test Monitoring and Control . . . . . . . . . . . . . . . . . . . . . . . . . 1407.7 Risk Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1417.8 Test Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1417.9 Test Completion Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1437.10 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1437.11 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

8 Test Outsourcing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1458.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1458.2 Planning and Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . 1478.3 Identifying Suppliers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1478.4 Prepare and Issue RFP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1488.5 Evaluate Proposals and Select Supplier . . . . . . . . . . . . . . . . . 1498.6 Formal Agreement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1498.7 Managing the Supplier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1508.8 Acceptance Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1518.9 Rollout and Customer Support . . . . . . . . . . . . . . . . . . . . . . . . 1518.10 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1528.11 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

9 Test Metrics and Problem-Solving . . . . . . . . . . . . . . . . . . . . . . . . . 1539.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1539.2 The Goal, Question, Metric Paradigm . . . . . . . . . . . . . . . . . . 1549.3 Metrics for Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156

9.3.1 Customer Satisfaction Metrics . . . . . . . . . . . . . . . . . . 1579.3.2 Project Management Metrics for Testing . . . . . . . . . . 1589.3.3 Test Execution Metrics . . . . . . . . . . . . . . . . . . . . . . . 1599.3.4 Customer Care Metrics . . . . . . . . . . . . . . . . . . . . . . . 1649.3.5 Miscellaneous Metrics . . . . . . . . . . . . . . . . . . . . . . . 166

9.4 Implementing a Metrics Program . . . . . . . . . . . . . . . . . . . . . . 1679.4.1 Data Gathering for Metrics . . . . . . . . . . . . . . . . . . . . 168

9.5 Problem-Solving Techniques . . . . . . . . . . . . . . . . . . . . . . . . . 1699.5.1 Fishbone Diagram . . . . . . . . . . . . . . . . . . . . . . . . . . 171

xiv Contents

9.5.2 Histograms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1729.5.3 Pareto Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1739.5.4 Trend Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1759.5.5 Scatter Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1769.5.6 Metrics and Statistical Process Control . . . . . . . . . . . 176

9.6 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1779.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

10 Software Testing Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18110.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18110.2 Test Management Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184

10.2.1 Estimation and Scheduling Tools . . . . . . . . . . . . . . . 18610.3 Static Code Analysis Tools . . . . . . . . . . . . . . . . . . . . . . . . . . 18710.4 Requirements and Test Design Tools . . . . . . . . . . . . . . . . . . . 18910.5 Test Execution Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

10.5.1 Tools for Regression Testing . . . . . . . . . . . . . . . . . . 19310.6 Tools for Defect Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . 19410.7 Test Performance and Monitoring Tools . . . . . . . . . . . . . . . . . 19510.8 Tools for Testing in Agile World . . . . . . . . . . . . . . . . . . . . . . 19610.9 Tools for Configuration Management . . . . . . . . . . . . . . . . . . . 19710.10 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19710.11 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

11 Test Process Improvement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19911.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19911.2 Software Process Improvement . . . . . . . . . . . . . . . . . . . . . . . 200

11.2.1 What Is a Software Process? . . . . . . . . . . . . . . . . . . . 20211.2.2 Benefits of Software Process Improvement . . . . . . . . 20311.2.3 Software Process Improvement Models . . . . . . . . . . . 20411.2.4 Process Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . 20511.2.5 Process Improvement Initiatives . . . . . . . . . . . . . . . . 20611.2.6 Barriers to Success . . . . . . . . . . . . . . . . . . . . . . . . . . 20711.2.7 Setting Up an Improvement Initiative . . . . . . . . . . . . 20811.2.8 Appraisals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208

11.3 Test Process Improvement Models . . . . . . . . . . . . . . . . . . . . . 21011.3.1 TMMi Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21111.3.2 TMap Next Model . . . . . . . . . . . . . . . . . . . . . . . . . . 21211.3.3 TPI Next Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21411.3.4 STEP Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21511.3.5 CTP Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21611.3.6 PDCA Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21711.3.7 CMMI Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217

Contents xv

11.4 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21911.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220

12 Testing in the Agile World . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22112.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22112.2 Scrum Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225

12.2.1 User Stories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22612.2.2 Estimation in Agile . . . . . . . . . . . . . . . . . . . . . . . . . 22712.2.3 Pair Programming . . . . . . . . . . . . . . . . . . . . . . . . . . 228

12.3 Software Testing in Agile . . . . . . . . . . . . . . . . . . . . . . . . . . . 22912.3.1 Test-Driven Development . . . . . . . . . . . . . . . . . . . . . 23012.3.2 Agile Test Principles . . . . . . . . . . . . . . . . . . . . . . . . 231

12.4 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23212.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233

13 Verification of Safety-Critical Systems . . . . . . . . . . . . . . . . . . . . . . 23513.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23513.2 Software Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23713.3 Software Dependability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24013.4 Formal Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24213.5 Cleanroom Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24413.6 Formal Methods and Testing . . . . . . . . . . . . . . . . . . . . . . . . . 24513.7 UML and Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246

13.7.1 Model Checking and Testing . . . . . . . . . . . . . . . . . . 24713.8 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24813.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250

14 Legal, Ethical, and Professional Aspects of Testing . . . . . . . . . . . . 25114.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25114.2 Business Ethics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252

14.2.1 What Is Computer Ethics? . . . . . . . . . . . . . . . . . . . . 25414.2.2 The Ethical Software Tester . . . . . . . . . . . . . . . . . . . 255

14.3 Professional Responsibility of Software Engineersand Testers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25614.3.1 ACM Code of Professional Conduct and Ethics . . . . . 257

14.4 Legal Aspects of Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25714.4.1 Legal Impacts of Failure . . . . . . . . . . . . . . . . . . . . . . 25914.4.2 Lawsuits and Professional Negligence . . . . . . . . . . . . 25914.4.3 The Law of Tort and Testing . . . . . . . . . . . . . . . . . . 260

14.5 Legal Aspects of Test Outsourcing . . . . . . . . . . . . . . . . . . . . . 26114.6 Licenses for Test Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

xvi Contents

14.7 Testing and Prevention of Computer Crime . . . . . . . . . . . . . . 26414.7.1 Testing and Hacking . . . . . . . . . . . . . . . . . . . . . . . . 265

14.8 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26814.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270

15 Configuration Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27115.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27115.2 Configuration Management System . . . . . . . . . . . . . . . . . . . . 275

15.2.1 Identify Configuration Items . . . . . . . . . . . . . . . . . . . 27615.2.2 Document Control Management . . . . . . . . . . . . . . . . 27615.2.3 Source Code Control Management . . . . . . . . . . . . . . 27715.2.4 Configuration Management Plan . . . . . . . . . . . . . . . . 277

15.3 Change Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27815.4 Configuration Management Audits . . . . . . . . . . . . . . . . . . . . . 27915.5 Configuration Management in Testing . . . . . . . . . . . . . . . . . . 28015.6 Review Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28115.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282

16 Epilogue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28316.1 The Future of Software Testing . . . . . . . . . . . . . . . . . . . . . . . 285

Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291

Contents xvii

List of Figures

Fig. 1.1 Standish research—project cost estimation accuracyin 1998 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Fig. 1.2 Shewhart’s control chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Fig. 1.3 Shewhart’s PDCA cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8Fig. 1.4 W.E. Deming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9Fig. 1.5 Joseph Juran . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12Fig. 1.6 Cost of poor quality—% of sales . . . . . . . . . . . . . . . . . . . . . . . 13Fig. 1.7 Estimation accuracy—breakthrough and control . . . . . . . . . . . . 14Fig. 1.8 Watts Humphrey. Courtesy of Watts Humphrey. . . . . . . . . . . . 16Fig. 1.9 Cost of quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23Fig. 1.10 Customer satisfaction process . . . . . . . . . . . . . . . . . . . . . . . . . . 25Fig. 1.11 Customer satisfaction metrics . . . . . . . . . . . . . . . . . . . . . . . . . . 27Fig. 2.1 Standish report—results of 1995 and 2009 Survey . . . . . . . . . . 35Fig. 2.2 Waterfall V lifecycle model . . . . . . . . . . . . . . . . . . . . . . . . . . . 41Fig. 2.3 SPIRAL lifecycle model. Public domain . . . . . . . . . . . . . . . . . 42Fig. 2.4 Rational unified process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43Fig. 3.1 Simplified test process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62Fig. 3.2 Number of paths through a trivial program. . . . . . . . . . . . . . . . 69Fig. 3.3 Psychology of software tester. Public domain. . . . . . . . . . . . . . 71Fig. 3.4 Automated testing tools. Creative Commons . . . . . . . . . . . . . . 76Fig. 4.1 Michael Fagan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80Fig. 4.2 Template for semi-formal review . . . . . . . . . . . . . . . . . . . . . . . 85Fig. 4.3 Example of an inspection meeting (public domain) . . . . . . . . . 86Fig. 4.4 Defect types in a project (ODC) . . . . . . . . . . . . . . . . . . . . . . . . 94Fig. 4.5 Template for Fagan inspection . . . . . . . . . . . . . . . . . . . . . . . . . 95Fig. 5.1 Simple process map for test planning . . . . . . . . . . . . . . . . . . . . 105Fig. 5.2 Sample Microsoft Project schedule . . . . . . . . . . . . . . . . . . . . . . 106Fig. 5.3 Risk categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108Fig. 5.4 PRINCE2 project board . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113Fig. 5.5 Project management triangle . . . . . . . . . . . . . . . . . . . . . . . . . . . 114Fig. 6.1 Test case specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120Fig. 6.2 State diagram for PIN authentication . . . . . . . . . . . . . . . . . . . . 124Fig. 6.3 Use-case diagram of ATM . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125Fig. 7.1 Organization of a test team. . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

xix

Fig. 7.2 Computer bug. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138Fig. 7.3 Test monitoring and control process map . . . . . . . . . . . . . . . . . 140Fig. 8.1 Legal agreement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149Fig. 9.1 GQM example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155Fig. 9.2 Customer survey arrivals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157Fig. 9.3 Customer satisfaction measurement . . . . . . . . . . . . . . . . . . . . . 157Fig. 9.4 Schedule timeliness metric for testing. . . . . . . . . . . . . . . . . . . . 158Fig. 9.5 Effort timeliness metric for testing . . . . . . . . . . . . . . . . . . . . . . 159Fig. 9.6 Total number of issues in project . . . . . . . . . . . . . . . . . . . . . . . 160Fig. 9.7 Open issues in project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160Fig. 9.8 Age of open defects in project . . . . . . . . . . . . . . . . . . . . . . . . . 161Fig. 9.9 Problem arrivals per month. . . . . . . . . . . . . . . . . . . . . . . . . . . . 161Fig. 9.10 Phase containment effectiveness . . . . . . . . . . . . . . . . . . . . . . . . 162Fig. 9.11 Test status. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162Fig. 9.12 Cumulative defects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163Fig. 9.13 Problem arrival and closure . . . . . . . . . . . . . . . . . . . . . . . . . . . 163Fig. 9.14 Status of problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163Fig. 9.15 Customer queries (arrivals/closures) . . . . . . . . . . . . . . . . . . . . . 164Fig. 9.16 Outage time per customer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165Fig. 9.17 Availability of system per month . . . . . . . . . . . . . . . . . . . . . . . 166Fig. 9.18 CMMI maturity in current year . . . . . . . . . . . . . . . . . . . . . . . . 166Fig. 9.19 Fishbone cause-and-effect diagram of high number

of defects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171Fig. 9.20 Histogram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173Fig. 9.21 Pareto chart outages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174Fig. 9.22 Trend chart estimation accuracy . . . . . . . . . . . . . . . . . . . . . . . . 175Fig. 9.23 Scatter graph amount inspected rate/error density . . . . . . . . . . . 176Fig. 9.24 Estimation accuracy and control charts . . . . . . . . . . . . . . . . . . . 177Fig. 10.1 HP Quality Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185Fig. 10.2 LDRA Code Coverage Analysis Report . . . . . . . . . . . . . . . . . . 188Fig. 10.3 IBM Rational DOORS Tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190Fig. 10.4 Bugzilla: Creative Commons . . . . . . . . . . . . . . . . . . . . . . . . . . 194Fig. 10.5 Apache JMeter: Creative Commons . . . . . . . . . . . . . . . . . . . . . 196Fig. 11.1 Steps in process improvement . . . . . . . . . . . . . . . . . . . . . . . . . 201Fig. 11.2 Process as glue for people, procedures and tools . . . . . . . . . . . 202Fig. 11.3 Sample process map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203Fig. 11.4 ISO 9001 quality management system . . . . . . . . . . . . . . . . . . . 205Fig. 11.5 Continuous improvement cycle. . . . . . . . . . . . . . . . . . . . . . . . . 207Fig. 11.6 Appraisals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209Fig. 11.7 TMMi maturity levels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212Fig. 11.8 TMap lifecycle model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213Fig. 11.9 TPI model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214Fig. 11.10 Phases of STEP model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215Fig. 12.1 Agile Dog. Creative Commons. . . . . . . . . . . . . . . . . . . . . . . . . 222

xx List of Figures

Fig. 12.2 Scrum framework. Creative Commons . . . . . . . . . . . . . . . . . . . 225Fig. 12.3 User story map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227Fig. 12.4 Pair programming. Creative Commons . . . . . . . . . . . . . . . . . . . 228Fig. 13.1 Grafenrheinfeld Nuclear Power Plant. Germany.

Creative Commons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236Fig. 13.2 Formal signing of the treaty of Versailles in 1919.

Public Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242Fig. 13.3 Cleanroom in semiconductor manufacturing.

Public Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245Fig. 13.4 Deriving tests from abstract model . . . . . . . . . . . . . . . . . . . . . . 246Fig. 13.5 Model checking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247Fig. 14.1 Corrupt legislation. 1896. Public domain . . . . . . . . . . . . . . . . . 253Fig. 14.2 Legal contract. Creative Commons . . . . . . . . . . . . . . . . . . . . . . 262Fig. 14.3 Hacker at work on backlit keyboard. Creative Commons . . . . . 266Fig. 15.1 Simple process map for change requests. . . . . . . . . . . . . . . . . . 279Fig. 15.2 Simple process map for configuration management . . . . . . . . . 280

List of Figures xxi

List of Tables

Table 1.1 ISO 9126 quality characteristics . . . . . . . . . . . . . . . . . . . . . . . . 5Table 1.2 Shewhart cycle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9Table 1.3 Deming 14-step programme . . . . . . . . . . . . . . . . . . . . . . . . . . . 11Table 1.4 Deming—five deadly diseases . . . . . . . . . . . . . . . . . . . . . . . . . 12Table 1.5 Juran’s 10-step programme for quality planning . . . . . . . . . . . 13Table 1.6 Juran’s breakthrough and control . . . . . . . . . . . . . . . . . . . . . . . 14Table 1.7 Crosby’s 14-step programme . . . . . . . . . . . . . . . . . . . . . . . . . . 15Table 1.8 Crosby’s maturity grid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16Table 1.9 Cost of quality categories. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23Table 1.10 Sample customer satisfaction questionnaire . . . . . . . . . . . . . . . 26Table 1.11 Total quality management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28Table 3.1 Types of testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64Table 3.2 Test levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64Table 3.3 Simple test schedule. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66Table 4.1 Informal review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82Table 4.2 Structured walk-throughs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83Table 4.3 Activities for semi-formal review meeting . . . . . . . . . . . . . . . . 84Table 4.4 Overview Fagan inspection process . . . . . . . . . . . . . . . . . . . . . 87Table 4.5 Strict Fagan inspection guidelines . . . . . . . . . . . . . . . . . . . . . . 88Table 4.6 Tailored (relaxed) Fagan inspection guidelines. . . . . . . . . . . . . 88Table 4.7 Inspector roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89Table 4.8 Fagan entry criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90Table 4.9 Inspection meeting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91Table 4.10 Fagan exit criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92Table 4.11 Issue severity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92Table 4.12 Classification of defects in Fagan inspections . . . . . . . . . . . . . 93Table 4.13 Classification of ODC defect types . . . . . . . . . . . . . . . . . . . . . 93Table 5.1 Estimation techniques. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104Table 5.2 Example of work breakdown structure for test estimation . . . . 105Table 5.3 Sample test planning checklist . . . . . . . . . . . . . . . . . . . . . . . . . 107Table 5.4 Risk management activities . . . . . . . . . . . . . . . . . . . . . . . . . . . 109Table 5.5 Project board roles and responsibilities . . . . . . . . . . . . . . . . . . 113Table 6.1 Test design techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121Table 6.2 Decision table with business rules . . . . . . . . . . . . . . . . . . . . . . 123

xxiii

Table 6.3 Planning section in dedicated test plan. . . . . . . . . . . . . . . . . . . 128Table 6.4 Template for test case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129Table 6.5 Sample trace matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130Table 7.1 Test management activities . . . . . . . . . . . . . . . . . . . . . . . . . . . 134Table 7.2 Activities in managing change requests . . . . . . . . . . . . . . . . . . 139Table 7.3 Test status for project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142Table 7.4 Quality status for project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142Table 7.5 Key risks for project key risks . . . . . . . . . . . . . . . . . . . . . . . . . 142Table 8.1 Supplier selection and management . . . . . . . . . . . . . . . . . . . . . 146Table 9.1 Implementing metrics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168Table 9.2 Goals and questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168Table 9.3 Phase containment effectiveness . . . . . . . . . . . . . . . . . . . . . . . . 168Table 10.1 Advantages of test tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182Table 10.2 Tool evaluation table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183Table 10.3 Types of tools for testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183Table 10.4 Tools for requirements development and management . . . . . . . 190Table 11.1 Test process improvement models . . . . . . . . . . . . . . . . . . . . . . 210Table 11.2 TMMi model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211Table 11.3 CMMI requirements for verification. . . . . . . . . . . . . . . . . . . . . 218Table 11.4 CMMI requirements for validation. . . . . . . . . . . . . . . . . . . . . . 218Table 12.1 Agile test principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232Table 13.1 Software reliability testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239Table 13.2 Dimensions of dependability . . . . . . . . . . . . . . . . . . . . . . . . . . 240Table 13.3 Model-checking process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248Table 14.1 Ten commandments on computer ethics. . . . . . . . . . . . . . . . . . 255Table 14.2 Professional responsibilities of software engineers

and testers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258Table 14.3 ACM code of conduct (general obligations). . . . . . . . . . . . . . . 258Table 14.4 Types of lawsuits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260Table 15.1 Features of good configuration management . . . . . . . . . . . . . . 272Table 15.2 Symptoms of poor configuration management . . . . . . . . . . . . . 273Table 15.3 Software configuration management activities . . . . . . . . . . . . . 274Table 15.4 Build plan for project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274Table 15.5 CMMI requirements for configuration management . . . . . . . . . 275Table 15.6 Sample configuration management audit checklist . . . . . . . . . . 281

xxiv List of Tables