comp8130 and 4130adrian marshall 8130 and 4130 test design & documentation adrian marshall
Post on 21-Dec-2015
224 views
TRANSCRIPT
COMP8130 and 4130 Adrian Marshall
8130 and 4130
Test Design
&
Documentation
Adrian Marshall
COMP8130 and 4130 Adrian Marshall
Test Design
Logical grouping of tests
Test size
Modular test design
Test class checklists
Prioritisation of tests
Test coverage
COMP8130 and 4130 Adrian Marshall
Test Design
Group Tests by• Test classes
White, black, grey box
• Focus areasFunctionalityUsability Reliability Performance
• D, I, A, T (demo, inspection, analysis, test)
COMP8130 and 4130 Adrian Marshall
Test Design
Requirements testing• Group by:
• Similar functions
• Similar set-up conditions
• Minimise verification of a requirement across multiple tests / test cases
• Manage requirements coverage (and traceability) in the design
COMP8130 and 4130 Adrian Marshall
Test Design
Size of tests:• Aim to have self-contained tests that can be
executed within an hour• Where possible have pre-established test data
Ensure tests have set-up and re-set conditions at the end of the test
Use (develop) test class checklists for test design guidance
COMP8130 and 4130 Adrian Marshall
Test Design
Prioritisation of tests• Functional criticality• Build sequence
Test coverage• Criticality• Complexity• Product history
COMP8130 and 4130 Adrian Marshall
Test Design
Test class checklists
• Knowledge base of verification• Environment characteristics• Functional focus• Identifies testing at various levels
COMP8130 and 4130 Adrian Marshall
Test Design
In small groups develop some checklists for the following test classes:
• HMI (Human Machine Interface) functionality
• COTS (Commercial off the Shelf) software• Security (encryption, authentication,
control)
COMP8130 and 4130 Adrian Marshall
Test Documentation
Why do we document?
•Communication
•Repeatability
•Records of what needs to be done and what was done
•To manage test coverage
COMP8130 and 4130 Adrian Marshall
IEEE Test Documentation Set
IEEE 829/ AS 4006
Planning & preparation • Test Plans
• Test Design Specifications
• Test Case Specifications
• Test Procedure Specifications
Execution & reporting• Test Item Transmittal Report
• Test Log
• Test Incident Report
• Test Summary Report
Other useful documents• Test Verification Matrix
• Test Activity Measurement Report….
COMP8130 and 4130 Adrian Marshall
Test Documentation
Other terms for test documents:
Test Specifications
Test Descriptions
COMP8130 and 4130 Adrian Marshall
Test Documentation
Test Plans• Master Test Plan
• Define testing at all levels• Avoid needless overlaps and
dangerous gaps• Defines subordinate test plans• Consists of test concept and summary
test plan• Can be delivered in multiple phases
COMP8130 and 4130 Adrian Marshall
Test Documentation
Other Test PlansPlan tests for:• sub-systems, • major functional groups, and/or • test harnesses
COMP8130 and 4130 Adrian Marshall
Test Documentation
Test Design / Test Case specification
• Defines test design and major test cases• Describes how the tests are to be run• Details the test environment• Describes the test cases (not to a
procedural level), aims and objectives
COMP8130 and 4130 Adrian Marshall
Test Documentation
Test Design and Case specification
1 INTRODUCTION 32 REFERENCES 33 TEST APPROACH 33.1Test design33.2Test Conduct 33.2.1 Test Readiness 33.2.2 Test Execution Strategy 33.2.3 Pass / Fail determination 33.3Test Reporting 3
COMP8130 and 4130 Adrian Marshall
Test Documentation
Test Case spec (contd)4TEST ENVIRONMENT3
5REQUIREMENTS3
6 TEST DESCRIPTION 3
6.1Test Name 1
6.2Test Name 2
7DEFINITIONS, ABBREVIATIONS AND ACRONYMS3
COMP8130 and 4130 Adrian Marshall
Test Documentation
Test Case spec (contd)APPENDIX A: HARDWARE/SOFTWARE CONFIGURATIONS
3
APPENDIX B: SOFTWARE SETUP 3
APPENDIX C: TEST DATA 3
COMP8130 and 4130 Adrian Marshall
Test Documentation
Test Procedures• Include in Test Case (or have as a
separate document)• Can have delivery in multiple phases• Describes the test steps• Can be very prescriptive (beware of
maintenance of such procedures)• High level procedures may not be very
repeatable (good and bad with this)
COMP8130 and 4130 Adrian Marshall
Test Documentation
Test Logs / test record• Identify actual test schedule and test
duration• Describe software and hardware
versions• Identify any deviations from the test
procedures• List who conducted the tests and any
witnesses (eg customer, QA)• Can be included in the case/ procedure
document
COMP8130 and 4130 Adrian Marshall
Test Documentation
Incident/defect reports• Describe the issues raised• To include:
• Title• Adequate description of the issue• Test reference• Configuration info (software/
hardware versions, test data)
COMP8130 and 4130 Adrian Marshall
Test Documentation
Incident/defect reports• To include:
• Priority / severity information• Who assigned to• Status of the issue (open, under
investigation, being fixed, fixed, closed etc.
• Dates (open, closed)
COMP8130 and 4130 Adrian Marshall
Test Documentation
Incident/defect reports
• Should not have restrictions on creation
• Should only be closed the “raiser” or delegate
COMP8130 and 4130 Adrian Marshall
Test Documentation
Test Reports• Identify:
What was testedWhenBy whomOn whatOutcomes (pass / fail)Issues (reference defect number)
COMP8130 and 4130 Adrian Marshall
Test Documentation
Test Reports• Written on conclusion of test
• Aimed to report upwards
• Report recommendations should be documented separately
COMP8130 and 4130 Adrian Marshall
Test Verification Matrix
Maps the test
• Plans/designs
• cases
• procedures
to the project requirements to ensure that all testable requirements are verified and to manage requirement changes….
Test Documentation
COMP8130 and 4130 Adrian Marshall
Activity Measurement Report Quantifies the testing activities and their results :
• e.g. hours per activity• start and end dates• number of defects detected• defect trends• analysis of testing activities against plans….
Test Documentation
COMP8130 and 4130 Adrian Marshall
Summary
DesignDefine methods, priorities, logical groupings, size of tests, test
coverage, and test environment / tools
Use / develop checklists
DocumentationPre execution - Plans, cases, procedures
Execution – logs
Post execution – reports
Documents communicate, provide repeatability, aid in management of test coverage (needless overlaps and dangerous gaps)
Aim for the document is to support test execution not be a distraction to it.