![Page 1: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/1.jpg)
Testing 1
![Page 2: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/2.jpg)
Background
Main objectives of a project: High Quality & High Productivity (Q&P)
Quality has many dimensions reliability, maintainability, interoperability
etc. Reliability is perhaps the most important Reliability: The chances of software failing More defects => more chances of failure
=> lesser reliability Hence quality goal: Have as few defects
as possible in the delivered software!
Testing 2
![Page 3: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/3.jpg)
Faults & Failure
Failure: A software failure occurs if the behavior of the s/w is different from expected/specified.
Fault: cause of software failure Fault = bug = defect Failure implies presence of defects
A defect has the potential to cause failure. Definition of a defect is environment and
project specificTesting 3
![Page 4: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/4.jpg)
Role of Testing
Identify defects remaining after the review processes!
Reviews are human processes - cannot catch all defects
There will be requirement defects, design defects and coding defects in code
Testing: Detects defects Plays a critical role in ensuring quality.
Testing 4
![Page 5: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/5.jpg)
Detecting defects in Testing During testing, a program is
executed with a set of test cases Failure during testing => defects are
present No failure => confidence grows, but
can not say “defects are absent” Defects detected through failures To detect defects, must cause
failures during testingTesting 5
![Page 6: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/6.jpg)
2 Basic principles
Test early Test parts as soon as they are implemented Test each method in turn
Test often Run tests at every reasonable opportunity After small additions After changes have been made Re-run prior tests (confirm still working) +
test the new functionality
Testing 6
![Page 7: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/7.jpg)
Retesting: Regression Testing
Retesting software to ensure that its capability has not been compromised
Designed to ensure that the code added since the last test has not compromised the functionality before the change
Usually consists of a repeat or subset of prior tests on the code
Can be difficult to assess whether added/changed code affects a given body of already-tested code
Testing 7
![Page 8: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/8.jpg)
Code dependencies Suppose C is tested code in an application Suppose A has been altered with
new/changed code N If C is known to depend on N
Perform regression testing on C If C is reliably known to be completely
independent of N There is no need to regression test C
Otherwise Regression test C
Testing 8
![Page 9: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/9.jpg)
Test Oracle
To check if a failure has occurred when executed with a test case, we need to know the correct behavior
That is we need a test oracle, which is often a human
Human oracle makes each test case expensive as someone has to check the correctness of its output
Testing 9
![Page 10: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/10.jpg)
Common Test Oracles
specifications and documentation,other products (for instance, an oracle for a
software program might be a second program that uses a different algorithm to evaluate the same mathematical expression as the product under test)
an heuristic oracle that provides approximate results or exact results for a set of a few test inputs,
a statistical oracle that uses statistical characteristics,
a consistency oracle that compares the results of one test execution to another for similarity,
a model-based oracle that uses the same model to generate and verify system behavior,
or a human being's judgment (i.e. does the program "seem" to the user to do the correct thing?).
Testing 10
![Page 11: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/11.jpg)
Role of Test cases
Ideally would like the following for test cases No failure implies “no defects” or “high quality” If defects present, then some test case causes
a failure Psychology of testing is important
should be to ‘reveal’ defects(not to show that it works!)
test cases must be “destructive” Role of test cases is clearly very critical Only if test cases are “good”, does
confidence increases after testing
Testing 11
![Page 12: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/12.jpg)
Test case design
During test planning, have to design a set of test cases that will detect defects present
Some criteria needed to guide test case selection
Two approaches to design test cases functional or black box structural or white box
Both are complementary; we briefly discuss them now and provide details of specific approaches later
Testing 12
![Page 13: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/13.jpg)
Black box testing
Video store application Run it with data like:
Abel rents “The Matrix” on January 24 Barry rents “Star Wars” on January 25 Abel returns “The Matrix” on January 30
Compare the application’s behaviour with its required behaviour
Testing 13
![Page 14: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/14.jpg)
Black box testing
Does not take into account how the application was designed and implement
It can be performed by someone who only needs to know what the application is required to produce
Similar to building an automobile and testing it by driving under various conditions
Testing 14
![Page 15: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/15.jpg)
Also need white box testing Black box testing allows us to
compare actual output with required output
But to uncover as many defects as possible, we need to know how the app has been designed and implemented
With inputs based on our knowledge of design elements, we can validate the expected behaviour
Testing 15
![Page 16: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/16.jpg)
Testing 16
![Page 17: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/17.jpg)
Testing
Testing only reveals the presence of defects
Does not identify nature and location of defects
Identifying & removing the defect => role of debugging and rework
Preparing test cases, performing testing, defects identification & removal all consume effort
Overall testing becomes very expensive : 30-50% development cost
Testing 17
![Page 18: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/18.jpg)
Incremental Testing
Goals of testing: detect as many defects as possible, and keep the cost low
Both frequently conflict - increasing testing can catch more defects, but cost also goes up
Incremental testing - add untested parts incrementally to tested portion
For achieving goals, incremental testing essential helps catch more defects helps in identification and removal
Testing of large systems is always incremental
Testing 18
![Page 19: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/19.jpg)
Integration and Testing
Incremental testing requires incremental ‘building’ I.e. incrementally integrate parts to form system
Integration & testing are related During coding, different modules are
coded separately Integration - the order in which they
should be tested and combined Integration is driven mostly by testing
needsTesting 19
![Page 20: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/20.jpg)
Top-down and Bottom-up
System : Hierarchy of modulesModules coded separatelyIntegration can start from bottom or topBottom-up requires test driversTop-down requires stubsBoth may be used, e.g. for user interfaces
top-down; for services bottom-upDrivers and stubs are code pieces written
only for testing
Testing 20
![Page 21: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/21.jpg)
Levels of Testing
The code contains requirement defects, design defects, and coding defects
Nature of defects is different for different injection stages
One type of testing will be unable to detect the different types of defects
Different levels of testing are used to uncover these defects
Testing 21
![Page 22: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/22.jpg)
Testing 22
User needs Acceptance testing
Requirementspecification
System testing
Design
code
Integration testing
Unit testing
![Page 23: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/23.jpg)
Unit Testing
Different modules tested separately Focus: defects injected during coding Essentially a code verification
technique, covered in previous chapter UT is closely associated with coding Frequently the programmer does UT;
coding phase sometimes called “coding and unit testing”
Testing 23
![Page 24: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/24.jpg)
Integration Testing
Focuses on interaction of modules in a subsystem
Unit tested modules combined to form subsystems
Test cases to “exercise” the interaction of modules in different ways
May be skipped if the system is not too large
Testing 24
![Page 25: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/25.jpg)
System Testing
Entire software system is testedFocus: does the software implement the
requirements?Validation exercise for the system with
respect to the requirementsGenerally the final testing stage before
the software is deliveredMay be done by independent peopleDefects removed by developersMost time consuming test phase
Testing 25
![Page 26: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/26.jpg)
Acceptance Testing
Focus: Does the software satisfy user needs?
Generally done by end users/customer in customer environment, with real data
Only after successful AT software is deployed
Any defects found,are removed by developers
Acceptance test plan is based on the acceptance test criteria in the SRS
Testing 26
![Page 27: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/27.jpg)
Other forms of testing
Performance testing tools needed to “measure” performance
Stress testing load the system to peak, load generation
tools needed Regression testing
test that previous functionality works alright important when changes are made Previous test records are needed for
comparisons Prioritization of testcases needed when
complete test suite cannot be executed for a change
Testing 27
![Page 28: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/28.jpg)
Test Plan
Testing usually starts with test plan and ends with acceptance testing
The test plan is a general document that defines the scope and approach for testing for the whole project
Inputs are SRS, project plan, design Test plan identifies what levels of
testing will be done, what units will be tested, etc., in the project
Testing 28
![Page 29: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/29.jpg)
Test Plan…
Test plan usually contains Test unit specs: what units need to be
tested separately Features to be tested: these may include
functionality, performance, usability,… Approach: criteria to be used, when to
stop, how to evaluate, etc Test deliverables Schedule and task allocation
Testing 29
![Page 30: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/30.jpg)
Typical Steps
1. Define “units” vs non-units for testing2. Determine what types of testing will
be performed3. Determine extent of testing4. Document5. Determine Input Sources6. Decide who will test7. Estimate resources8. Indentify metrics to be collected
Testing 30
![Page 31: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/31.jpg)
1. Unit vs non-unit tests What constitutes a “unit” is defined by
the development team Include or don’t include packages?
Common sequence of unit testing in OO design Test the methods of each class Test the classes of each package Test the package as a whole
Test the basic units first before testing the things that rely on them
Testing 31
![Page 32: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/32.jpg)
2. Determine type of testing Interface testing:
validate functions exposed by modules Integration testing
Validates combinations of modules System testing
Validates whole application Usability testing
Validates user satisfaction
Testing 32
![Page 33: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/33.jpg)
2. Determine type of testing Regression testing Validates changes did not create defects in existing
code Acceptance testing
Customer agreement that contract is satisfied Installation testing
Works as specified once installed on required platform Robustness testing
Validates ability to handle anomalies Performance testing
Is fast enough / uses acceptable amount of memory
Testing 33
![Page 34: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/34.jpg)
3. Determine the extent
Impossible to test for every situation Do not just “test until time expires” Prioritize, so that important tests are
definitely performed Consider legal data, boundary data, illegal
data More thoroughly test sensitive methods
(withdraw/deposit in a bank app) Establish stopping criteria in advance
Concrete conditions upon which testing stops
Testing 34
![Page 35: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/35.jpg)
Stopping conditions
When tester has not been able to find another defect in 5 (10? 30? 100?) minutes of testing
When all nominal, boundary, and out-of-bounds test examples show no defect
When a given checklist of test types has been completed
After completing a series of targeted coverage (e.g., branch coverage for unit testing)
When testing runs out of its scheduled time
Testing 35
![Page 36: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/36.jpg)
4. Decide on test documentation
Documentation consists of test procedures, input data, the code that executes the test, output data, known issues that cannot be fixed yet, efficiency data
Test drivers and utilities are used to execute unit tests, must be document for future use JUnit is a professional test utility to help
developers retain test documentationTesting 36
![Page 37: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/37.jpg)
Documentation questions
Include an individual’s personal document set?
How/when to incorporate all types of testing?
How/when to incorporate testing in formal documents
How/when to use tools/test utilities
Testing 37
![Page 38: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/38.jpg)
5. Determine input sources Applications are developed to solve
problem in specific area May be test data specific to the
application E.g., standard test stock market data for
a brokerage application Output from previous versions of
application Need to plan how to get and use such
domain-specific test inputTesting 38
![Page 39: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/39.jpg)
6. Decide who will test Individual engineer responsible for some
(units)? Testing beyond the unit usually
planned/performed by people other than coders
Unit level tests made available for inspection/incorporation in higher level tests
How/when inspected by QA Typically black box testing only
How/when designed and performed by third parties?
Testing 39
![Page 40: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/40.jpg)
7. Estimate the resources Unit testing often bundles with
development process (not its own budget item) Good process respects that reliability of
units is essential and provides time for developers to develop reliable units
Other testing is either part of project budget or QA’s budget
Use historical data if available to estimate resources needed
Testing 40
![Page 41: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/41.jpg)
8. Identify & track metrics Must specify the form in which
developers record defect counts, defect types, and time spent on testing
Resulting data used: to assess the state of the application To forecast eventual quality and
completion date As historical data for future projects
Testing 41
![Page 42: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/42.jpg)
Testing 42
“More than the act of testing, the act of designing tests is one of the best bug preventers known. The thinking that must be done to create a useful test can discover and eliminate bugs before they are coded – indeed, test-design thinking can discover and eliminate bugs at every stage in the creation of software, from conception to specification, to design, coding and the rest.” – Boris Beizer
![Page 43: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/43.jpg)
Software Testing Templates
http://www.the-software-tester.com/templates.html Software Test Plan Software Test Report
http://softwaretestingfundamentals.com/test-plan/ Software Test Plan
Testing 43
![Page 44: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/44.jpg)
Testing 44
![Page 45: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/45.jpg)
Test case specifications
Test plan focuses on approach; does not deal with details of testing a unit
Test case specification has to be done separately for each unit
Based on the plan (approach, features,..) test cases are determined for a unit
Expected outcome also needs to be specified for each test case
Testing 45
![Page 46: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/46.jpg)
Test case specifications… Together the set of test cases should
detect most of the defects Would like the set of test cases to
detect any defects, if it exists Would also like set of test cases to be
small - each test case consumes effort Determining a reasonable set of test
case is the most challenging task of testing
Testing 46
![Page 47: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/47.jpg)
Test case specifications… The effectiveness and cost of testing depends
on the set of test cases Q: How to determine if a set of test cases is
good? I.e. the set will detect most of the defects, and a smaller set cannot catch these defects
No easy way to determine goodness; usually the set of test cases is reviewed by experts
This requires test cases be specified before testing – a key reason for having test case specs
Test case specs are essentially a tableTesting 47
![Page 48: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/48.jpg)
Test case specifications…
Testing 48
Seq.No Condition to be tested
Test Data Expected result
successful
![Page 49: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/49.jpg)
Test case specifications…So for each testing, test case specs are
developed, reviewed, and executedPreparing test case specifications is
challenging and time consumingTest case criteria can be usedSpecial cases and scenarios may be used
Once specified, the execution and checking of outputs may be automated through scriptsDesired if repeated testing is neededRegularly done in large projects
Testing 49
![Page 50: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/50.jpg)
Test case execution and analysis Executing test cases may require drivers or
stubs to be written; some tests can be auto, others manual A separate test procedure document may be
prepared Test summary report is often an output –
gives a summary of test cases executed, effort, defects found, etc
Monitoring of testing effort is important to ensure that sufficient time is spent
Computer time also is an indicator of how testing is proceeding
Testing 50
![Page 51: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/51.jpg)
Defect logging and tracking A large software may have thousands
of defects, found by many different people
Often person who fixes (usually the coder) is different from who finds
Due to large scope, reporting and fixing of defects cannot be done informally
Defects found are usually logged in a defect tracking system and then tracked to closure
Defect logging and tracking is one of the best practices in industry
Testing 51
![Page 52: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/52.jpg)
Defect logging…
A defect in a software project has a life cycle of its own, like Found by someone, sometime and
logged along with info about it (submitted)
Job of fixing is assigned; person debugs and then fixes (fixed)
The manager or the submitter verifies that the defect is indeed fixed (closed)
More elaborate life cycles possible
Testing 52
![Page 53: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/53.jpg)
Defect logging…
Testing 53
![Page 54: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/54.jpg)
Defect logging…
During the life cycle, info about defect is logged at diff stages to help debug as well as analysis
Defects generally categorized into a few types, and type of defects is recorded Orthogonal Defect Classification (ODC) is
one classification Some standard categories: Logic, standards,
UI, interface, performance, documentation,..
Testing 54
![Page 55: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/55.jpg)
Defect logging…
Severity of defects in terms of its impact on sw is also recorded
Severity useful for prioritization of fixing
One categorization Critical: Show stopper Major: Has a large impact Minor: An isolated defect Cosmetic: No impact on functionality
Testing 55
![Page 56: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/56.jpg)
Defect logging and tracking… Ideally, all defects should be closed Sometimes, organizations release
software with known defects (hopefully of lower severity only)
Organizations have standards for when a product may be released
Defect log may be used to track the trend of how defect arrival and fixing is happening
Testing 56
![Page 57: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/57.jpg)
Defect arrival and closure trend
Testing 57
![Page 58: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/58.jpg)
Defect analysis for prevention Quality control focuses on removing
defects Goal of defect prevention (DP) is to reduce
the defect injection rate in future DP done by analyzing defect log,
identifying causes and then remove them Is an advanced practice, done only in
mature organizations Finally results in actions to be undertaken
by individuals to reduce defects in future
Testing 58
![Page 59: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/59.jpg)
Metrics - Defect removal efficiency
Basic objective of testing is to identify defects present in the programs
Testing is good only if it succeeds in this goal
Defect removal efficiency (DRE) of a QC activity = % of present defects detected by that QC activity
High DRE of a quality control activity means most defects present at the time will be removed
Testing 59
![Page 60: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/60.jpg)
Defect removal efficiency … DRE for a project can be evaluated only when all
defects are know, including delivered defects Delivered defects are approximated as the
number of defects found in some duration after delivery
The injection stage of a defect is the stage in which it was introduced in the software, and detection stage is when it was detected These stages are typically logged for defects
With injection and detection stages of all defects, DRE for a QC activity can be computed
Testing 60
![Page 61: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/61.jpg)
Defect Removal Efficiency … DREs of different QC activities are a
process property - determined from past data
Past DRE can be used as expected value for this project
Process followed by the project must be improved for better DRE
Testing 61
![Page 62: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/62.jpg)
Metrics – Reliability Estimation High reliability is an important goal
being achieved by testing Reliability is usually quantified as a
probability or a failure rate For a system it can be measured by
counting failures over a period of time Measurement often not possible for
software as reliability changes as a result of fixes, and with one-off, not possible to measure
Testing 62
![Page 63: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/63.jpg)
Reliability Estimation…
Sw reliability estimation models are used to model the failure followed by fix model of software
Data about failures and their times during the last stages of testing is used by these model
These models then use this data and some statistical techniques to predict the reliability of the software
Testing 63
![Page 64: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/64.jpg)
Summary
Testing plays a critical role in removing defects, and in generating confidence
Testing should be such that it catches most defects present, i.e. a high DRE
Multiple levels of testing needed for this Incremental testing also helps At each testing, test cases should be
specified, reviewed, and then executed
Testing 64
![Page 65: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/65.jpg)
Summary …
Deciding test cases during planning is the most important aspect of testing
Two approaches – black box and white box Black box testing - test cases derived from
specifications. Coming up: Equivalence class partitioning,
boundary value, cause effect graphing, error guessing
White box - aim is to cover code structures Coming up: statement coverage, branch
coverage
Testing 65
![Page 66: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/66.jpg)
Summary…
In a project both white box & black box testing used at lower levels Test cases initially driven by functional Coverage measured, test cases
enhanced using coverage data At higher levels, mostly functional
testing done; coverage monitored to evaluate the quality of testing
Defect data is logged, and defects are tracked to closure
The defect data can be used to estimate reliability, DRE
Testing 66
![Page 67: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/67.jpg)
Black Box testing
Software tested to be treated as a block box
Specification for the black box is given
The expected behavior of the system is used to design test cases
Test cases are determined solely from specification.
Internal structure of code not used for test case design
Testing 67
![Page 68: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/68.jpg)
Black box testing…
Premise: Expected behavior is specified. Hence just test for specified expected
behavior How it is implemented is not an issue.
For modules: Specifications produced in design detail
expected behavior For system testing,
SRS specifies expected behaviorTesting 68
![Page 69: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/69.jpg)
Black Box Testing…
Most thorough functional testing - exhaustive testing Software is designed to work for an input
space Test the software with all elements in the
input space Infeasible - too high a cost Need better method for selecting test
cases Different approaches have been proposed
Testing 69
![Page 70: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/70.jpg)
White box testing
Black box testing focuses only on functionality What the program does; not how it is
implemented White box testing focuses on implementation
Aim is to exercise different program structures with the intent of uncovering errors
Is also called structural testing Various criteria exist for test case design Test cases have to be selected to satisfy
coverage criteria
Testing 70
![Page 71: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/71.jpg)
Types of structural testing Control flow based criteria
looks at the coverage of the control flow graph
Data flow based testing looks at the coverage in the definition-
use graph Mutation testing
looks at various mutants of the program Later slides discuss control flow based
and data flow based criteria Testing 71
![Page 72: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/72.jpg)
Testing Methods
Black Box White Box
Equivalence partitioning Divide input values into
equivalent groups
Boundary value analysis Test at boundary conditions
Other methods of selecting small input sets: Cause effect graphing Pair-wise testing State-Testing
Statement coverage Test cases cause every
line of code to be executed
Branch coverage Test cases cause every
decision point to execute
Path coverage Test cases cause every
independent code path to be executed
Testing 72
![Page 73: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/73.jpg)
Equivalence Class partitioning Divide the input space into equivalent
classes If the software works for a test case
from a class the it is likely to work for all Can reduce the set of test cases if such
equivalent classes can be identified Getting ideal equivalent classes is
impossible Approximate it by identifying classes for
which different behavior is specified
Testing 73
http://www.testing-world.com/58828/Equivalence-Class-Partitioning
![Page 74: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/74.jpg)
Equivalence Class ExamplesIn a computer store, the computer item can have a quantity between -500 to +500. What are the equivalence classes?
Answer: Valid class: -500 <= QTY <= +500 Invalid class: QTY > +500 Invalid class: QTY < -500
Testing 74
![Page 75: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/75.jpg)
Equivalence Class ExamplesAccount code can be 500 to 1000 or 0 to
499 or 2000 (the field type is integer). What are the equivalence classes?
Answer: Valid class: 0 <= account <= 499Valid class: 500 <= account <= 1000
Valid class: 2000 <= account <= 2000 Invalid class: account < 0 Invalid class: 1000 < account < 2000 Invalid class: account > 2000
Testing 75
![Page 76: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/76.jpg)
Equivalence class partitioning… Rationale: specification requires same
behavior for elements in a class Software likely to be constructed such
that it either fails for all or for none. E.g. if a function was not designed for
negative numbers then it will fail for all the negative numbers
For robustness, should form equivalent classes for invalid as well as valid inputs
Testing 76
![Page 77: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/77.jpg)
Equivalent class partitioning.. Every condition specified as input is
an equivalent class Define invalid equivalent classes also E.g. range 0< value<Max specified
one range is the valid class input < 0 is an invalid class
input > max is an invalid class
Whenever that entire range may not be treated uniformly - split into classes
Testing 77
![Page 78: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/78.jpg)
Equivalence class…
Once equivalence classes selected for each of the inputs, test cases have to be selected Select each test case covering as many
valid equivalence classes as possible Or, have a test case that covers at most
one valid class for each input Plus a separate test case for each invalid
class
Testing 78
![Page 79: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/79.jpg)
Example
Consider a program that takes 2 inputs – a string s and an integer n
Program determines n most frequent characters
Tester believes that programmer may deal with diff types of chars separately
Describe valid and invalid equivalence classes
Testing 79
![Page 80: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/80.jpg)
Example..
Input Valid Eq Class Invalid Eq class
S 1: Contains numbers2: Lower case letters3: upper case letters4: special chars5: str len between 0-N(max)
1: non-ascii char2: str len > N
N 6: Int in valid range 3: Int out of range
Testing 80
![Page 81: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/81.jpg)
Example…
Test cases (i.e. s , N) with first method s : str of len < N that includes lower case, upper
case, numbers, and special chars, and N=5 Plus test cases for each of the invalid eq classes Total test cases: 1 valid+3 invalid= 4 total
With the second approach A separate string for each type of char (i.e. a str
of numbers, one of lower case, …) + invalid cases
Total test cases will be 6 + 3 = 9
Testing 81
![Page 82: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/82.jpg)
Boundary value analysis
Programs often fail on special values These values often lie on boundary of
equivalence classes Test cases that have boundary values
(BVs) have high yield These are also called extreme cases A BV test case is a set of input data that
lies on the edge of an equivalence class of input/output
Testing 82
![Page 83: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/83.jpg)
Boundary value analysis (cont)... For each equivalence class
choose values on the edges of the class choose values just outside the edges
E.g. if 0 <= x <= 1.0 0.0 , 1.0 are edges inside -0.1,1.1 are just outside
E.g. a bounded list - have a null list , a maximum value list
Consider outputs also and have test cases generate outputs on the boundary
Testing 83
![Page 84: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/84.jpg)
Boundary Value Analysis
In BVA we determine the value of vars that should be used
If input is a defined range, then there are 6 boundary values plus 1 normal value (tot: 7)
If multiple inputs, how to combine them into test cases; two strategies possibleTry all possible combination of BV of diff
variables, with n vars this will have 7n test cases!
Select BV for one var; have other vars at normal values + 1 of all normal values
Testing 84
Min Max
![Page 85: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/85.jpg)
BVA.. (test cases for two vars – x and y)
Testing 85
![Page 86: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/86.jpg)
Cause Effect graphing
Equivalence classes and boundary value analysis consider each input separately
To handle multiple inputs, different combinations of equivalent classes of inputs can be tried
Number of combinations can be large – if n diff input conditions such that each condition is valid/invalid, total: 2n
Cause effect graphing helps in selecting combinations as input conditions
Testing 86
![Page 87: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/87.jpg)
CE-graphing
Identify causes and effects in the system Cause: distinct input condition which can
be true or false Effect: distinct output condition (T/F)
Identify which causes can produce which effects; can combine causes
Causes/effects are nodes in the graph and arcs are drawn to capture dependency; and/or are allowed
Testing 87
![Page 88: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/88.jpg)
CE-graphing
From the CE graph, can make a decision table Lists combination of conditions that set
different effects Together they check for various effects
Decision table can be used for forming the test cases
Testing 88
![Page 89: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/89.jpg)
Step 1: Break the specification down into workable pieces.
Testing 89
![Page 90: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/90.jpg)
Step 2: Identify the causes and effects. a) Identify the causes (the distinct or
equivalence classes of input conditions) and assign each one a unique number.
b) Identify the effects or system transformation and assign each one a unique number.
Testing 90
![Page 91: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/91.jpg)
Example
What are the driving input variables? What are the driving output variables?
Can you list the causes and the effects ?
Testing 91
![Page 92: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/92.jpg)
Example: Causes & Effects
Testing 92
![Page 93: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/93.jpg)
Step 3: Construct Cause & Effect Graph
Testing 93
![Page 94: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/94.jpg)
Step 4: Annotate the graph with constraints Annotate the graph with constraints
describing combinations of causes and/or effects that are impossible because of syntactic or environmental constraints or considerations.
Example: Can be both Male and Female? Types of constraints?
Exclusive: Both cannot be true Inclusive: At least one must be true One and only one: Exactly one must be true Requires: If A implies B Mask: If effect X then not effect Y
Testing 94
![Page 95: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/95.jpg)
Types of Constraints
Testing 95
![Page 96: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/96.jpg)
Example: Adding a One-and-only-one Constraint
Why not use an exclusive constraint?
Testing 96
![Page 97: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/97.jpg)
Step 5: Construct limited entry decision table Methodically trace state conditions in the graphs, converting them into a limited-entry decision table.
Each column in the table represents a test case.
Testing 97
Test Case 1 2 3 … n
Cause 1 1 0 …
… 0 1 …
Cause c 0 0 …
Effect 100 … … …
…
Effect e 0
![Page 98: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/98.jpg)
Example: Limited entry decision table
Testing 98
![Page 99: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/99.jpg)
Step 6: Convert into test cases
Columns to rows
Read off the 1’s
Testing 99
![Page 100: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/100.jpg)
Notes
This was a simple example! Good tester could have jumped
straight to the end results Not always the case….
Testing 100
![Page 101: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/101.jpg)
Exercise: You try it!
A bank database which allows two commands Credit acc# amt Debit acc# amt
Requirements If credit and acc# valid, then credit If debit and acc# valid and amt less than
balance, then debit Invalid command – message
Your task… Identify and name causes and effects Draw CE graphs and add constraints Construct limited entry decision table Construct test cases
Testing 101
![Page 102: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/102.jpg)
Example…
Causes C1: command is credit C2: command is debit C3: acc# is valid C4: amt is valid
Effects Print “Invalid command” Print “Invalid acct#” Print “Debit amt not valid” Debit account Credit account
Testing 102
# 1 2 3 4 5
C1 0 1 x x xC2 0 x 1 1 xC3 x 0 1 1 1C4 x x 0 1 1
E1 1E2 1E3 1E4 1E5 1
![Page 103: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/103.jpg)
Pair-wise testing
Often many parmeters determine the behavior of a software system
The parameters may be inputs or settings, and take diff values (or diff value ranges)
Many defects involve one condition (single-mode fault), eg. sw not being able to print on some type of printer Single mode faults can be detected by testing for
different values of diff parms If n parms and each can take m values, we can test
for one diff value for each parm in each test case Total test cases: m
Testing 103
![Page 104: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/104.jpg)
Pair-wise testing…
All faults are not single-mode and sw may fail at some combinations Eg tel billing sw does not compute
correct bill for night time calling (one parm) to a particular country (another parm)
Eg ticketing system fails to book a biz class ticket (a parm) for a child (a parm)
Multi-modal faults can be revealed by testing diff combination of parm values
This is called combinatorial testing
Testing 104
![Page 105: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/105.jpg)
Pair-wise testing…
Full combinatorial testing often not feasible For n parms each with m values, total
combinations are nm For 5 parms, 5 values each (tot: 3125), if
one test is 5 minutes, total time > 1 month! Research suggests that most such faults
are revealed by interaction of a pair of values
I.e. most faults tend to be double-mode For double mode, we need to exercise each
pair – called pair-wise testing
Testing 105
![Page 106: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/106.jpg)
Pair-wise testing…
In pair-wise, all pairs of values have to be exercised in testing
If n parms with m values each, between any 2 parms we have m*m pairs 1st parm will have m*m with n-1 others 2nd parm will have m*m pairs with n-2 3rd parm will have m*m pairs with n-3,
etc. Total no of pairs are m*m*n*(n-1)/2
Testing 106
![Page 107: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/107.jpg)
Pair-wise testing…
A test case consists of some setting of the n parameters
Smallest set of test cases when each pair is covered once only
A test case can cover a maximum of (n-1)+(n-2)+…=n(n-1)/2 pairs
In the best case when each pair is covered exactly once, we will have m2 different test cases providing the full pair-wise coverage
Testing 107
![Page 108: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/108.jpg)
Pair-wise testing…
Generating the smallest set of test cases that will provide pair-wise coverage is non-trivial
Efficient algos exist; efficiently generating these test cases can reduce testing effort considerably In an example with 13 parms each with 3
values pair-wise coverage can be done with 15 testcases
Pair-wise testing is a practical approach that is widely used in industry
Testing 108
![Page 109: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/109.jpg)
Pair-wise testing, Example A sw product for multiple platforms and
uses browser as the interface, and is to work with diff OSs
We have these parms and values OS (parm A): Windows, Solaris, Linux Mem size (B): 128M, 256M, 512M Browser (C): IE, Netscape, Mozilla
Total # of pair wise combinations: 27 # of cases can be less
Testing 109
![Page 110: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/110.jpg)
Pair-wise testing…
Test case Pairs covered
a1, b1, c1a1, b2, c2a1, b3, c3a2, b1, c2a2, b2, c3a2, b3, c1a3, b1, c3a3, b2, c1a3, b3, c2
(a1,b1) (a1, c1) (b1,c1)(a1,b2) (a1,c2) (b2,c2)(a1,b3) (a1,c3) (b3,c3) (a2,b1) (a2,c2) (b1,c2)(a2,b2) (a2,c3) (b2,c3)(a2,b3) (a2,c1) (b3,c1)(a3,b1) (a3,c3) (b1,c3)(a3,b2) (a3,c1) (b2,c1)(a3,b3) (a3,c2) (b3,c2)
Testing 110
![Page 111: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/111.jpg)
Special cases
Programs often fail on special cases These depend on nature of inputs,
types of data structures,etc. No good rules to identify them One way is to guess when the
software might fail and create those test cases
Also called error guessing Play the sadist & hit where it might
hurtTesting 11
1
![Page 112: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/112.jpg)
Error Guessing
Use experience and judgement to guess situations where a programmer might make mistakes
Special cases can arise due to assumptions about inputs, user, operating environment, business, etc.
E.g. A program to count frequency of words file empty, file non existent, file only has blanks,
contains only one word, all words are same, multiple consecutive blank lines, multiple blanks between words, blanks at the start, words in sorted order, blanks at end of file, etc.
Perhaps the most widely used in practiceTesting 11
2
![Page 113: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/113.jpg)
State-based Testing
Some systems are state-less: for same inputs, same behavior is exhibited
Many systems’ behavior depends on the state of the system i.e. for the same input the behavior could be different
I.e. behavior and output depend on the input as well as the system state
System state – represents the cumulative impact of all past inputs
State-based testing is for such systems
Testing 113
![Page 114: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/114.jpg)
State-based Testing…
A system can be modeled as a state machine
The state space may be too large (is a cross product of all domains of vars)
The state space can be partitioned in a few states, each representing a logical state of interest of the system
State model is generally built from such states
Testing 114
![Page 115: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/115.jpg)
State-based Testing…
A state model has four components States: Logical states representing
cumulative impact of past inputs to system
Transitions: How state changes in response to some events
Events: Inputs to the system Actions: The outputs for the events
Testing 115
![Page 116: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/116.jpg)
State-based Testing…
State model shows what transitions occur and what actions are performed
Often state model is built from the specifications or requirements
The key challenge is to identify states from the specs/requirements which capture the key properties but is small enough for modeling
Testing 116
![Page 117: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/117.jpg)
State-based Testing, example… Consider a student survey example
A system to take survey of students Student submits survey and is returned
results of the survey so far The result may be from the cache (if the
database is down) and can be up to 5 surveys old
Testing 117
![Page 118: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/118.jpg)
State-based Testing, example… In a series of requests, first 5 may be
treated differently Hence, we have two states: one for req
no 1-4 (state 1), and other for 5 (2) The db can be up or down, and it can
go down in any of the two states (3-4) Once db is down, the system may get
into failed state (5), from where it may recover
Testing 118
![Page 119: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/119.jpg)
State-based Testing, example…
Testing 119
![Page 120: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/120.jpg)
State-based Testing…
State model can be created from the specs or the design
For objects, state models are often built during the design process
Test cases can be selected from the state model and later used to test an implementation
Many criteria possible for test cases
Testing 120
![Page 121: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/121.jpg)
State-based Testing criteria All transaction coverage (AT): test case
set T must ensure that every transition is exercised
All transitions pair coverage (ATP). T must execute all pairs of adjacent transitions (incoming and outgoing transition in a state)
Transition tree coverage (TT). T must execute all simple paths (i.e. a path from start to end or a state it has visited)
Testing 121
![Page 122: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/122.jpg)
Example, test cases for AT criteriaSNo Transition Test case
12345678
1 -> 21 -> 22 -> 11 -> 33 -> 33 -> 44 -> 55 -> 2
Req()Req(); req(); req(); req();req(); req()Seq for 2; req()Req(); fail()Req(); fail(); req()Req(); fail(); req(); req(); req();req(); req()Seq for 6; req()Seq for 6; req(); recover()
Testing 122
![Page 123: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/123.jpg)
State-based testing…
SB testing focuses on testing the states and transitions to/from them
Different system scenarios get tested; some easy to overlook otherwise
State model is often done after design information is available
Hence it is sometimes called grey box testing (as it not pure black box)
Testing 123
![Page 124: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/124.jpg)
White box testing
Black box testing focuses only on functionality What the program does; not how it is
implemented White box testing focuses on implementation
Aim is to exercise different program structures with the intent of uncovering errors
Is also called structural testing Various criteria exist for test case design Test cases have to be selected to satisfy
coverage criteria
Testing 124
![Page 125: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/125.jpg)
Types of structural testing Control flow based criteria
looks at the coverage of the control flow graph
Data flow based testing looks at the coverage in the definition-
use graph Mutation testing
looks at various mutants of the program We will discuss control flow based and
data flow based criteria Testing 12
5
![Page 126: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/126.jpg)
Control flow based criteria Considers the program as control flow
graph Nodes represent code blocks – i.e. set of
statements always executed together An edge (i,j) represents a possible
transfer of control from i to j Assume a start node and an end node A path is a sequence of nodes from
start to end
Testing 126
![Page 127: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/127.jpg)
Statement Coverage CriterionCriterion: Each statement is executed at least
once during testing i.e., set of paths executed during testing
should include all nodesLimitation: does not require a decision to
evaluate to false if no else clauseE.g. ,: abs (x) : if ( x>=0) x = -x; return(x)
The set of test cases {x = 0} achieves 100% statement coverage, but error not detected
Guaranteeing 100% coverage not always possible due to possibility of unreachable nodes
Testing 127
![Page 128: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/128.jpg)
Branch coverage
Criterion: Each edge should be traversed at least once during testing
i.e. each decision must evaluate to both true and false during testing
Branch coverage implies stmt coverage If multiple conditions in a decision, then
all conditions need not be evaluated to T and F
Testing 128
![Page 129: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/129.jpg)
Control flow based…
There are other criteria too - path coverage, predicate coverage, cyclomatic complexity based, ...
None is sufficient to detect all types of defects (e.g. a program missing some paths cannot be detected)
They provide some quantitative handle on the breadth of testing
More used to evaluate the level of testing rather than selecting test cases
Testing 129
![Page 130: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/130.jpg)
Data flow-based testing
A def-use graph is constructed from the control flow graph
A stmt in the control flow graph (in which each stmt is a node) can be of these types Def: represents definition of a var (i.e.
when var is on the lhs) C-use: computational use of a var P-use: var used in a predicate for control
transfer
Testing 130
![Page 131: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/131.jpg)
Data flow based…
A def-use graph is constructed by associating vars with nodes and edges in the control flow graph For a node I, def(i) is the set of vars for
which there is a global def in I For a node I, C-use(i) is the set of vars
for which there is a global c-use in I For an edge, p-use(I,j) is set of vars whor
which there is a p-use for the edge (I,j) Def clear path from I to j wrt x: if no def
of x in the nodes in the path
Testing 131
![Page 132: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/132.jpg)
Data flow based criteria
all-defs: for every node I, and every x in def(i) there is a def-clear path For def of every var, one of its uses (p-
use or c-use) must be tested all-p-uses: all p-uses of all the
definitions should be tested All p-uses of all the defs must be tested
Some-c-uses, all-c-uses, some-p-uses are some other criteria
Testing 132
![Page 133: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/133.jpg)
Relationship between diff criteria
Testing 133
![Page 134: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/134.jpg)
Tool support and test case selection
Two major issues for using these criteria How to determine the coverage How to select test cases to ensure coverage
For determining coverage - tools are essential
Tools also tell which branches and statements are not executed
Test case selection is mostly manual - test plan is to be augmented based on coverage data
Testing 134
![Page 135: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/135.jpg)
In a Project
Both functional and structural should be used Test plans are usually determined using
functional methods; during testing, for further rounds, based on the coverage, more test cases can be added
Structural testing is useful at lower levels only; at higher levels ensuring coverage is difficult
Hence, a combination of functional and structural at unit testing
Functional testing (but monitoring of coverage) at higher levels
Testing 135
![Page 136: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/136.jpg)
Comparison
Testing 136
Code Review StructuralTesting
FunctionalTesting
Computational M H MLogic M H MI/O H M HData handling H L HInterface H H MData defn. M L MDatabase H M M
![Page 137: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/137.jpg)
Testing 137
![Page 138: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/138.jpg)
Testing
Testing only reveals the presence of defects
Does not identify nature and location of defects
Identifying & removing the defect => role of debugging and rework
Preparing test cases, performing testing, defects identification & removal all consume effort
Overall testing becomes very expensive : 30-50% development cost
Testing 138
![Page 139: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/139.jpg)
Incremental Testing
Goals of testing: detect as many defects as possible, and keep the cost low
Both frequently conflict - increasing testing can catch more defects, but cost also goes up
Incremental testing - add untested parts incrementally to tested portion
For achieving goals, incremental testing essential helps catch more defects helps in identification and removal
Testing of large systems is always incremental
Testing 139
![Page 140: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/140.jpg)
Integration and Testing
Incremental testing requires incremental ‘building’ I.e. incrementally integrate parts to form system
Integration & testing are related During coding, different modules are
coded separately Integration - the order in which they
should be tested and combined Integration is driven mostly by testing
needsTesting 14
0
![Page 141: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/141.jpg)
Top-down and Bottom-up
System : Hierarchy of modulesModules coded separatelyIntegration can start from bottom or topBottom-up requires test driversTop-down requires stubsBoth may be used, e.g. for user interfaces
top-down; for services bottom-upDrivers and stubs are code pieces written
only for testing
Testing 141
![Page 142: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/142.jpg)
Levels of Testing
The code contains requirement defects, design defects, and coding defects
Nature of defects is different for different injection stages
One type of testing will be unable to detect the different types of defects
Different levels of testing are used to uncover these defects
Testing 142
![Page 143: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/143.jpg)
Testing 143
User needs Acceptance testing
Requirementspecification
System testing
Design
code
Integration testing
Unit testing
![Page 144: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/144.jpg)
Unit Testing
Different modules tested separately Focus: defects injected during coding Essentially a code verification
technique, covered in previous chapter UT is closely associated with coding Frequently the programmer does UT;
coding phase sometimes called “coding and unit testing”
Testing 144
![Page 145: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/145.jpg)
Integration Testing
Focuses on interaction of modules in a subsystem
Unit tested modules combined to form subsystems
Test cases to “exercise” the interaction of modules in different ways
May be skipped if the system is not too large
Testing 145
![Page 146: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/146.jpg)
System Testing
Entire software system is testedFocus: does the software implement the
requirements?Validation exercise for the system with
respect to the requirementsGenerally the final testing stage before
the software is deliveredMay be done by independent peopleDefects removed by developersMost time consuming test phase
Testing 146
![Page 147: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/147.jpg)
Acceptance Testing
Focus: Does the software satisfy user needs?
Generally done by end users/customer in customer environment, with real data
Only after successful AT software is deployed
Any defects found,are removed by developers
Acceptance test plan is based on the acceptance test criteria in the SRS
Testing 147
![Page 148: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/148.jpg)
Other forms of testing
Performance testing tools needed to “measure” performance
Stress testing load the system to peak, load generation
tools needed Regression testing
test that previous functionality works alright important when changes are made Previous test records are needed for
comparisons Prioritization of testcases needed when
complete test suite cannot be executed for a change
Testing 148
![Page 149: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/149.jpg)
Test Plan
Testing usually starts with test plan and ends with acceptance testing
Test plan is a general document that defines the scope and approach for testing for the whole project
Inputs are SRS, project plan, design Test plan identifies what levels of
testing will be done, what units will be tested, etc in the project
Testing 149
![Page 150: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/150.jpg)
Test Plan…
Test plan usually contains Test unit specs: what units need to be
tested separately Features to be tested: these may include
functionality, performance, usability,… Approach: criteria to be used, when to
stop, how to evaluate, etc Test deliverables Schedule and task allocation
Testing 150
![Page 151: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/151.jpg)
Test case specifications
Test plan focuses on approach; does not deal with details of testing a unit
Test case specification has to be done separately for each unit
Based on the plan (approach, features,..) test cases are determined for a unit
Expected outcome also needs to be specified for each test case
Testing 151
![Page 152: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/152.jpg)
Test case specifications… Together the set of test cases should
detect most of the defects Would like the set of test cases to
detect any defects, if it exists Would also like set of test cases to be
small - each test case consumes effort Determining a reasonable set of test
case is the most challenging task of testing
Testing 152
![Page 153: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/153.jpg)
Test case specifications… The effectiveness and cost of testing depends
on the set of test cases Q: How to determine if a set of test cases is
good? I.e. the set will detect most of the defects, and a smaller set cannot catch these defects
No easy way to determine goodness; usually the set of test cases is reviewed by experts
This requires test cases be specified before testing – a key reason for having test case specs
Test case specs are essentially a tableTesting 15
3
![Page 154: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/154.jpg)
Test case specifications…
Testing 154
Seq.No Condition to be tested
Test Data Expected result
successful
![Page 155: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/155.jpg)
Test case specifications…So for each testing, test case specs are
developed, reviewed, and executedPreparing test case specifications is
challenging and time consumingTest case criteria can be usedSpecial cases and scenarios may be used
Once specified, the execution and checking of outputs may be automated through scriptsDesired if repeated testing is neededRegularly done in large projects
Testing 155
![Page 156: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/156.jpg)
Test case execution and analysis Executing test cases may require drivers or
stubs to be written; some tests can be auto, others manual A separate test procedure document may be
prepared Test summary report is often an output –
gives a summary of test cases executed, effort, defects found, etc
Monitoring of testing effort is important to ensure that sufficient time is spent
Computer time also is an indicator of how testing is proceeding
Testing 156
![Page 157: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/157.jpg)
Defect logging and tracking A large software may have thousands
of defects, found by many different people
Often person who fixes (usually the coder) is different from who finds
Due to large scope, reporting and fixing of defects cannot be done informally
Defects found are usually logged in a defect tracking system and then tracked to closure
Defect logging and tracking is one of the best practices in industry
Testing 157
![Page 158: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/158.jpg)
Defect logging…
A defect in a software project has a life cycle of its own, like Found by someone, sometime and
logged along with info about it (submitted)
Job of fixing is assigned; person debugs and then fixes (fixed)
The manager or the submitter verifies that the defect is indeed fixed (closed)
More elaborate life cycles possible
Testing 158
![Page 159: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/159.jpg)
Defect logging…
Testing 159
![Page 160: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/160.jpg)
Defect logging…
During the life cycle, info about defect is logged at diff stages to help debug as well as analysis
Defects generally categorized into a few types, and type of defects is recorded ODC is one classification Some std categories: Logic, standards,
UI, interface, performance, documentation,..
Testing 160
![Page 161: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/161.jpg)
Defect logging…
Severity of defects in terms of its impact on sw is also recorded
Severity useful for prioritization of fixing
One categorization Critical: Show stopper Major: Has a large impact Minor: An isolated defect Cosmetic: No impact on functionality
Testing 161
![Page 162: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/162.jpg)
Defect logging and tracking… Ideally, all defects should be closed Sometimes, organizations release
software with known defects (hopefully of lower severity only)
Organizations have standards for when a product may be released
Defect log may be used to track the trend of how defect arrival and fixing is happening
Testing 162
![Page 163: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/163.jpg)
Defect arrival and closure trend
Testing 163
![Page 164: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/164.jpg)
Defect analysis for prevention Quality control focuses on removing
defects Goal of defect prevention is to reduce the
defect injection rate in future DP done by analyzing defect log,
identifying causes and then remove them Is an advanced practice, done only in
mature organizations Finally results in actions to be undertaken
by individuals to reduce defects in future
Testing 164
![Page 165: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/165.jpg)
Metrics - Defect removal efficiency
Basic objective of testing is to identify defects present in the programs
Testing is good only if it succeeds in this goal
Defect removal efficiency of a QC activity = % of present defects detected by that QC activity
High DRE of a quality control activity means most defects present at the time will be removed
Testing 165
![Page 166: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/166.jpg)
Defect removal efficiency … DRE for a project can be evaluated only when all
defects are know, including delivered defects Delivered defects are approximated as the
number of defects found in some duration after delivery
The injection stage of a defect is the stage in which it was introduced in the software, and detection stage is when it was detected These stages are typically logged for defects
With injection and detection stages of all defects, DRE for a QC activity can be computed
Testing 166
![Page 167: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/167.jpg)
Defect Removal Efficiency … DREs of different QC activities are a
process property - determined from past data
Past DRE can be used as expected value for this project
Process followed by the project must be improved for better DRE
Testing 167
![Page 168: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/168.jpg)
Metrics – Reliability Estimation High reliability is an important goal
being achieved by testing Reliability is usually quantified as a
probability or a failure rate For a system it can be measured by
counting failures over a period of time Measurement often not possible for
software as due to fixes reliability changes, and with one-off, not possible to measure
Testing 168
![Page 169: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/169.jpg)
Reliability Estimation…
Sw reliability estimation models are used to model the failure followed by fix model of software
Data about failures and their times during the last stages of testing is used by these model
These models then use this data and some statistical techniques to predict the reliability of the software
A simple reliability model is given in the book
Testing 169
![Page 170: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/170.jpg)
Summary
Testing plays a critical role in removing defects, and in generating confidence
Testing should be such that it catches most defects present, i.e. a high DRE
Multiple levels of testing needed for this Incremental testing also helps At each testing, test cases should be
specified, reviewed, and then executed
Testing 170
![Page 171: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/171.jpg)
Summary …
Deciding test cases during planning is the most important aspect of testing
Two approaches – black box and white box Black box testing - test cases derived from
specifications. Equivalence class partitioning, boundary
value, cause effect graphing, error guessing White box - aim is to cover code structures
statement coverage, branch coverage
Testing 171
![Page 172: Testing 1. Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5518c2c5550346881f8b569f/html5/thumbnails/172.jpg)
Summary…
In a project both used at lower levels Test cases initially driven by functional Coverage measured, test cases
enhanced using coverage data At higher levels, mostly functional
testing done; coverage monitored to evaluate the quality of testing
Defect data is logged, and defects are tracked to closure
The defect data can be used to estimate reliability, DRE
Testing 172