class 9
TRANSCRIPT
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
11
CES 592 Telecommunications CES 592 Telecommunications System Product VerificationSystem Product Verification
Sonoma State UniversitySonoma State University
Class Lecture 9:Class Lecture 9:
Software Testing ProcessesSoftware Testing Processes
Fall 2004Fall 2004
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
22
InstructorsInstructors France AntelmeFrance Antelme Ario BigattiniArio Bigattini Jaseem MasoodJaseem Masood Steven WoodySteven Woody Coordinator: Prof Ravi KumarCoordinator: Prof Ravi Kumar
WebsiteWebsitehttp://www.sonoma.edu/engineering/courses/CES592.shtmlhttp://www.sonoma.edu/engineering/courses/CES592.shtml
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
33
Software Testing ProcessSoftware Testing Process
Test Plans and Test CasesTest Plans and Test Cases Regression versus New Feature Regression versus New Feature
testingtesting Test Entrance / Exit CriteriaTest Entrance / Exit Criteria Version ControlVersion Control Defect TrackingDefect Tracking
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
44
Software Testing ProcessSoftware Testing ProcessTest Plans and Test CasesTest Plans and Test Cases A test plan is a document that describes the A test plan is a document that describes the
planned test activities. For a large software planned test activities. For a large software project, this may be divided into separate feature project, this may be divided into separate feature test plans.test plans.
A test case is a list of steps which test a specific A test case is a list of steps which test a specific feature. Should not be more than one page long. feature. Should not be more than one page long. Must contain a pass/fail criteria. Must contain a pass/fail criteria.
A test case matrix is commonly used in a test A test case matrix is commonly used in a test plan to identify which combinations/permutations plan to identify which combinations/permutations of conditions will be tested.of conditions will be tested.
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
55
Software Testing Process:Test PlansSoftware Testing Process:Test PlansFrom IEEE Standard for Software Test Documentation IEEE 829-1998
• Test plan identifier – Name / Number• Introduction – Brief description of product & test strategy• Test items – Description of item to be tested • Features to be tested - List• Features not to be tested – Very important to list, prevents assumptions• Approach – Describe your test strategy• Item pass/fail criteria – Must have pass/fail criteria for tests• Suspension criteria and resumption requirements – (Entry / Exit criteria)• Test deliverables – Results: Performance charts, bug lists, bug charts • Testing tasks• Environmental needs – Lab setup• Responsibilities – Define & agree• Staffing and training needs – Continuous training, eye on future• Schedule – Time to perform test cases vs. time for software to stabilize• Risks and contingencies – Identify real & meaningful risks• Approvals – Management, Customer Account team
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
66
Software Testing Process: Test PlansSoftware Testing Process: Test Plans
Project Milestones:• Testplan approved• Testbed ready• Scripts complete• Testing Start• Testing Finish• Script transferred to regression
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
77
Software Testing Process: Test PlansSoftware Testing Process: Test Plans
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
88
Software Testing Process: Test PlansSoftware Testing Process: Test Plans
Schedule estimation is difficult!• Have a target schedule (make your best estimate based on prior experience)• When 75% of that schedule has elapsed, nail down a specific date for test completion, taking into account the progress thus far.• Keep track of your project history, so that you can make more accurate schedule estimates for the next project
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
99
Software Testing Process:Test CasesSoftware Testing Process:Test Cases
•Test Case identifier – Name / Number• Test case owner – Who wrote it? Who is responsible for updating it?• Item to be tested – Describe• Input/Output Specification• Item pass/fail criteria – Must have pass/fail criteria for test case• Automation – Can it be automated? Is it automated? Point to file.•Testing tasks – one-page procedure• Environmental needs – Lab setup• Special procedural requirements• Inter-case dependencies• Test case priority – changes constantly, depends on many factors• Track which software version this test case is valid for• Schedule – Time to perform the test case – keep track of it• Bugs found by this test case – update every time a new bug is found
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
1010
Software Testing Process: Test CasesSoftware Testing Process: Test Cases
Start writing test cases as the product requirements are Start writing test cases as the product requirements are being defined.being defined.
Show your test cases to the developer who is doing the Show your test cases to the developer who is doing the coding. Get him/her thinking about how the feature coding. Get him/her thinking about how the feature will be tested. Help prevent the bugs from ever will be tested. Help prevent the bugs from ever getting into the code.getting into the code.
Test cases should give enough information to perform Test cases should give enough information to perform the test, but not so much detail that everyone will the test, but not so much detail that everyone will perform the test in the same way. You want to allow perform the test in the same way. You want to allow some variation and randomness in the test case.some variation and randomness in the test case.
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
1111
Software Testing Process: Test CasesSoftware Testing Process: Test CasesPositive Test Cases: Positive Test Cases:
Does the feature work as required?Does the feature work as required?
Negative Test Cases:Negative Test Cases:Can the feature be broken?Can the feature be broken?
Network: Disconnected, No Ports available…Network: Disconnected, No Ports available…
Disk Storage: File not found, File in use, Disk Full, Invalid Path, CRC errorDisk Storage: File not found, File in use, Disk Full, Invalid Path, CRC error
Memory: Not enough free memory, fragments too small…Memory: Not enough free memory, fragments too small…
Do the Positive testcases first, to verify the feature is Do the Positive testcases first, to verify the feature is working. Then try to break it with the Negative test working. Then try to break it with the Negative test cases.cases.
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
1212
Software Testing ProcessSoftware Testing Process
Test Plans and Test CasesTest Plans and Test Cases Regression versus New Feature Regression versus New Feature
testingtesting Test Entrance / Exit CriteriaTest Entrance / Exit Criteria Version ControlVersion Control Defect TrackingDefect Tracking
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
1313
Software Testing ProcessSoftware Testing Process Regression versus New Feature testingRegression versus New Feature testing
New Feature Testing: Testing the New Feature Testing: Testing the new features that have been added new features that have been added since the previous test cyclesince the previous test cycle
Regression Testing: Re-testing “old” Regression Testing: Re-testing “old” functionality to make sure that no functionality to make sure that no features were inadvertently brokenfeatures were inadvertently broken
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
1414
Software Testing ProcessSoftware Testing Process Regression versus New Feature testingRegression versus New Feature testing
New Feature Testing: New Feature Testing: Usually a small percentage of the overall Usually a small percentage of the overall number of tests to be performednumber of tests to be performed
Regression Testing: Regression Testing: Number of tests grows with each new Number of tests grows with each new feature added; can easily consume all feature added; can easily consume all testing resourcestesting resources
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
1515
Software Testing ProcessSoftware Testing Process Regression test prioritiesRegression test priorities
1.1. Check that the bug was actually fixedCheck that the bug was actually fixed
2.2. Check for related bugsCheck for related bugs3.3. Check that the fix didn’t break something elseCheck that the fix didn’t break something else
Testing Computer Software, Kaner, Falk, NguyenTesting Computer Software, Kaner, Falk, Nguyen
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
1616
Software Testing ProcessSoftware Testing Process
Test Plans and Test CasesTest Plans and Test Cases Regression versus New Feature Regression versus New Feature
testingtesting Test Entrance / Exit CriteriaTest Entrance / Exit Criteria Version ControlVersion Control Defect TrackingDefect Tracking
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
1717
Software Testing ProcessSoftware Testing Process
Software Testing ProcessSoftware Testing Process Entrance / Exit CriteriaEntrance / Exit Criteria
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
1818
Software Testing ProcessSoftware Testing ProcessSoftware Testing Entrance CriteriaSoftware Testing Entrance Criteria Feature list defined & completedFeature list defined & completed Code Review completed, documentedCode Review completed, documented Static Analysis completedStatic Analysis completed Draft user documents readyDraft user documents ready Unit testing plan completedUnit testing plan completed
Does the software meet the entrance criteria? Does the software meet the entrance criteria? If so, then software test will be complete in 12 weeks.If so, then software test will be complete in 12 weeks.If not, then software test will not complete on time.If not, then software test will not complete on time.
Measure the number & severity of bugs found after test start:. Measure the number & severity of bugs found after test start:. If bugs exceeds a limit, then entrance criteria was not met, If bugs exceeds a limit, then entrance criteria was not met, 12 week schedule will not be met.12 week schedule will not be met.
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
1919
Software Testing ProcessSoftware Testing ProcessExit CriteriaExit Criteria Is the software safe for people, property and data? (“First, do Is the software safe for people, property and data? (“First, do
no harm”)no harm”) All test team members agree that it is fit for use?All test team members agree that it is fit for use? Feature testing: At least one full test cycle with >75% pass rateFeature testing: At least one full test cycle with >75% pass rate Regression Testing completed, >90% pass rateRegression Testing completed, >90% pass rate Code Coverage analysis completed, >70% code coverageCode Coverage analysis completed, >70% code coverage No showstopper or catastrophic bugs openNo showstopper or catastrophic bugs open Final review of all open bugsFinal review of all open bugs
DeliverablesDeliverables Automated regression scriptsAutomated regression scripts Updated test plan, test cases, test scripts Updated test plan, test cases, test scripts (constantly re-evaluate)(constantly re-evaluate) Defect Counts, trend summariesDefect Counts, trend summaries Release notes reviewedRelease notes reviewed User Docs reviewedUser Docs reviewed Customer support trainingCustomer support training
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
2020
Software Testing ProcessSoftware Testing ProcessA word about Entrance Criteria…A word about Entrance Criteria…
Don’t “burn your lead time” by waiting for the Don’t “burn your lead time” by waiting for the perfect, 100% complete software build to be perfect, 100% complete software build to be delivered. Start testing as soon as you can.delivered. Start testing as soon as you can.
Coding Start
Coding Finished
TestingStart
TestingFinished
OfficialTestingStart – Must meet entrance criteria
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
2121
Software Testing ProcessSoftware Testing Process
Test Plans and Test CasesTest Plans and Test Cases Regression versus New Feature Regression versus New Feature
testingtesting Test Entrance / Exit CriteriaTest Entrance / Exit Criteria Version ControlVersion Control Defect TrackingDefect Tracking
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
2222
Software Testing Process: Version ControlSoftware Testing Process: Version Control
Version control for software under testVersion control for software under test One central source for software to be One central source for software to be
testedtested Version control for test plans, test cases, Version control for test plans, test cases,
test scriptstest scripts
Keep track of what was tested on what Keep track of what was tested on what version of software & hardwareversion of software & hardware
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
2323
Software Testing Process: Version ControlSoftware Testing Process: Version Control
CVS - Concurrent Versions SystemCVS - Concurrent Versions System RCS - Revision Control SystemRCS - Revision Control System SubversionSubversion
Clearcase - Rational SoftwareClearcase - Rational Software PVCS - MerantPVCS - Merant
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
2424
Software Testing ProcessSoftware Testing Process
Test Plans and Test CasesTest Plans and Test Cases Regression versus New Feature Regression versus New Feature
testingtesting Test Entrance / Exit CriteriaTest Entrance / Exit Criteria Version ControlVersion Control Defect TrackingDefect Tracking
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
2525
Software Testing Process: Defect TrackingSoftware Testing Process: Defect Tracking
Agreed-upon definitions for bug Agreed-upon definitions for bug severity, before the project startsseverity, before the project starts
Quality is everyone’s job – if you see Quality is everyone’s job – if you see a problem, it is your responsibility to a problem, it is your responsibility to enter a bug reportenter a bug report
Weekly bug review meeting - “bug Weekly bug review meeting - “bug scrub”scrub”
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
2626
Software Testing Process: Defect TrackingSoftware Testing Process: Defect Tracking
BugzillaBugzilla GNATSGNATS Debian Bug Tracking SystemDebian Bug Tracking System
Clear DDTS – Rational SoftwareClear DDTS – Rational Software SilkRadar – Segue SoftwareSilkRadar – Segue Software
http://testingfaqs.org/t-track.htmlhttp://testingfaqs.org/t-track.html
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
2727
Software Testing Process: Defect ReportsSoftware Testing Process: Defect Reports TitleTitle Severity categorySeverity category InputsInputs Expected resultsExpected results Actual resultsActual results Anomalies ObservedAnomalies Observed Bug Info: Error logs, debug tracesBug Info: Error logs, debug traces Date and timeDate and time Procedure stepProcedure step EnvironmentEnvironment Attempts to repeat / Steps to repeatAttempts to repeat / Steps to repeat TestersTesters ObserversObservers Last version observed to passLast version observed to pass
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
2828
Software Testing Process: Defect ReportsSoftware Testing Process: Defect Reports The Title is the most important part of the The Title is the most important part of the
Defect Report Defect Report – – make it clear and concise!make it clear and concise!
Find the simplest trigger for the bugFind the simplest trigger for the bug Find the most serious consequences of the Find the most serious consequences of the
bugbug Try to determine if the bug is the tip of an Try to determine if the bug is the tip of an
icecube or the tip of an iceburgicecube or the tip of an iceburg Always give the developer a chance to Always give the developer a chance to
debug it in real time – it may take you debug it in real time – it may take you hours to reproduce it laterhours to reproduce it later
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
2929
Software Testing Process: Exit CriteriaSoftware Testing Process: Exit CriteriaA word about exit criteria:Watch the Defect trend to know if you are ready to release:
Bug Trend
0
100
200
300
400
500
600
700
11/0
7/0
2
11/2
1/0
2
12/0
5/0
2
12/1
9/0
2
01/0
2/0
3
01/1
6/0
3
01/3
0/0
3
02/1
3/0
3
02/2
7/0
3
03/1
3/0
3
03/2
7/0
3
04/1
0/0
3
04/2
4/0
3
05/0
8/0
3
05/2
2/0
3
06/0
5/0
3
06/1
9/0
3
07/0
3/0
3
07/1
7/0
3
07/3
1/0
3
08/1
4/0
3
08/2
8/0
3
09/1
1/0
3
09/2
5/0
3
10/0
9/0
3
10/2
3/0
3
11/0
6/0
3
11/2
0/0
3
12/0
4/0
3
04/11/2304/11/23 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood all righto Bigattini, Jaseem Masood all rights reserveds reserved
3030
ReferencesReferencesTesting Computer Software, Testing Computer Software, by C. Kaner, J. Falk, and H. Nguyenby C. Kaner, J. Falk, and H. Nguyen
Managing the Testing Process, Managing the Testing Process, Rex BlackRex Black
Software Test Automation, Software Test Automation, Fewster & GrahamFewster & Graham
Code Complete, Code Complete, Steve McConnellSteve McConnell
““Software Testing and Quality Assurance”, Software Testing and Quality Assurance”, Ross CollardRoss Collard
IEEE Standard for Software Test Documentation, Std 829 - 1998IEEE Standard for Software Test Documentation, Std 829 - 1998
Black-Box Testing: Techniques for Functional Testing of Software and Systems, Black-Box Testing: Techniques for Functional Testing of Software and Systems, Boris Boris Beizer, Wiley, 1995Beizer, Wiley, 1995
Classic Testing Mistakes, Classic Testing Mistakes, Brian MarickBrian Marickhttp://http://www.testing.com/writings/classic/mistakes.pdfwww.testing.com/writings/classic/mistakes.pdf
Software QA / Test Resource CenterSoftware QA / Test Resource Centerhttphttp://www.softwareqatest.com/index.html://www.softwareqatest.com/index.html