1 time boxed testing bcs sigist 13 th july 1998 graham thomas - osi group
TRANSCRIPT
1
TIME BOXED TESTING
BCS SIGIST
13th July 1998
Graham Thomas - OSI Group
2
Abstract
There is great pressure upon developers today to improve productivity and effectiveness. To achieve this there is a move away from the traditional structured methodologies towards more dynamic, iterative and RAD approaches.
This is being combined with Object and Component based techniques, and delivered with a new generation of IDE’s, to produce thin client, web based, voice and data products.
So how do we test it? Propose Time Boxed Testing Outline an approach Discuss the problems Give a flavour of the fun ?
3
Agenda
Development Lifecycle Testing Lifecycle Testing Helix Testing Stage by Stage Traceability Testing Process How to sell time boxed testing Lessons Learnt
4
The Project
Telephony based project 2 types of proprietary hardware Client Server running Windows NT Interfaces to legacy systems Rational Rose case tool (using UML) Objectory process Visual Studio Was considering Java & thin clients Incremental/Iterative development
5
Incompatible Models?
Sys Test
Classic V Model
Rqmts.
Analysis
Design
Build Unit Test
Int Test
UAT
Breadth
Depth
Iterative (RAD) Model
6
Difficult to Represent Test Stages
Breadth
Depth
unit
unit
unit
ComponentIntegration
ComponentIntegration
System
Acceptance
ComponentIntegration
7
Project Lifecycle
. . .
8
Testing Lifecycle
. . .
9
P
DC
E
A dual spiral life cycle ?
PlanDesignConstructExecute
RqmtsAnalysisDesignBuild
. . .R
AD
B
10
Testing Lifecycle
Unit Testing Component Integration Testing System Testing Acceptance Testing Year 2000 Millenium Compliance Testing Processes and Activities Tools & Traceability Resources Planning
11
Scope of Test Phases
ComponentIntegrationTesting
UnitTesting System
TestingUserAcceptanceTesting
ConicalModel
Iterative Release
12
Tuned Test Lifecycle
System Testing
- Non functional Requirements- Performance Analysis
ComponentIntegrationTesting
UnitTesting
Iterative Release
User Acceptance
Testing- Scenario Testing- Acceptance Criteria
Testing Activities Brought Forward
13
Unit Testing
Unit Testing
StaticAnalysis
DynamicAnalysis
UnitTesting
Iterative Time-Boxed Activity
Specific to delivered components
‘White Box’ testing techniques employed
External Interfaces tested
Use of tools for Static & Dynamic analysis Quality
Assurance
ExternalInterfaceTesting
UnitIntegration
Testing
Sub UnitTesting
14
Unit Testing
SubUnit
Unit
Unit
Unit
Package
Unit Integration Testing
SubUnit
SubUnit
Package
PackageUnit
Unit
Unit
Unit Testing
SubUnit
SubUnit
SubUnitSub
Unit
SubUnit
Sub Unit Testing
SubUnit
Stubs, Drivers, Harnesses
Previously tested & checked-in
15
Component Integration Testing
ComponentIntegration
Non FunctionalRequirements
FunctionalRequirements
Regression Test
ComponentIntegrationTesting
Iterative Time-Boxed Activity
Testing prioritised by Business
End to End process threads where possible
Contains elements of following test phases
Build regression test packs
ScenarioRegression
Test
PerformanceAnalysis
PackageIntegration
16
Package
Component Integration Testing
Package Package Integration must precede Component Integration
Component
Component
Component
Package
Package
Step 1 - Package Integration
17
Component Integration Testing
Component
Component
Component
Component
Component
Drop1
ComponentComponent
Drop2
Component
Component
Component
ComponentComponent
Component
Component
Component
. . .
Step 2 - Component Integration
18
Component Integration Testing
Automated Regression Test Pack
Component
Component
Component
ComponentComponent
Component
ComponentComponent
Component
Component
ComponentComponent
Component
Component
. . .
Component
ComponentComponentComponent
Component
Component
Component
Component
ComponentComponent
Component
Component
ComponentComponent
ComponentComponent
Drop by Drop
19
System Testing
FunctionalRequirements
Regression Test
Non FunctionalRequirements
PerformanceAnalysis
Design Verification
SystemTesting
One off activity Structured testing
techniques End to End process
threads Ties together the
development iterations Regression test
already constructed
20
Acceptance Testing
AcceptanceCriteriaTesting
ScenarioRegressionTesting
BusinessProcessTesting
UsabilityTesting
AcceptanceTesting
Data Take On & Conversion
Testing
One off activity Test that the computer
system integrates with business processes
Structured testing techniques
End to End process threads
Regression test already constructed
21
Y2000 - Millennium Compliance6 Date Scenarios
2 Business Tests
ThisCentury
AcrossCentury
Boundary
Whollynext
CenturyJan 1st Feb 29th Dec 31st
FinancialYear ?
FiscalPeriods ?
Separate test runs Carried out in
sequence Some tests can be
exercised post Y2000
22
Testing Lifecycle TraceabilityHigh Level Function
FunctionalRequirement
Use Case
Scenario
Test Case
Test Script
Non FR
DevelopmentRepository
Test Planning
UserRequirements
23
Testing Processes and Activities
Test Management
Test Reporting Monitor Measure Manage
ExitTest StageSign off
Test Planning Test Design Test Preparation Analysis of Change
Test Execution Test Automation Regression Test
QualityInterface
ProblemMgmt
Conf. Mgmt.Interface
Process
Entry
Activity
24
Processes
Essential Configuration Management
Impact Analysis using a Test Planning repository
Quality Management
Good practice Fault Reporting - on the Intranet - Live
Gathering metric information
25
Tools
Full set of testing tools is required
Static analysis
Dynamic Analysis
Interactive debugging
Test Automation/Capture Replay essential
Test Planning Repository
Performance monitoring (stress/load)
Fault Reporting system
26
Monitoring
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Fault
0
20
40
60
80
100
120
140
Cumulative Error Curve
27
Monitoring
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
0
5
10
15
20
25
30
35
40
45
50
Outstanding
28
Monitoring
1 2 3 4 5 6 7 8 9 10 11 12 13
0
20
40
60
80
100
120
140
Faults by Priority
A
B
C
29
Monitoring
W1 W2 W3 W40
5
10
15
20
25
30
Aged Defect Analysis by Priority
A
B
C
30
Test Resources
UnitTesting
ComponentIntegrationTesting
SystemTesting
UserAcceptanceTesting
TESTTEAM
Testers
BusinessUsersDevelopers
BusinessUsers
Developers
Secondment
31
Test Planning
Definition
Test Strategy
Testing Process
Developm
ent
Non Functional Testing
Test drop
Test drop
Test drop
Decision
Test Approach
Delivery
System Test
Acceptance Test
Y2000 Test
Time-boxed Testing
32
Selling the Approach
Target the following; Test Team
Systems Architect
Development team
Standards & Methods
Business Representatives
Project Sponsor
Via Animated Presentation Test Strategy document
33
So how easy was this?
34
So What went well?
Systems Architect liked it The presentation was well received
35
What was difficult?
Initial cool response from other members of the test team
Development Team were uneasy Dev. Why test anyway, our code is wonderful and our test
harnesses will do all the testing? Business Representatives were unhappy about;
Anything less than 100% testing Bringing their effort forward in the development lifecycle The difference between the ‘Use Case’ modeling they had
done, and the test planning that they would need to carry out
36
What was difficult? (2)
Standards & MethodsSaw nothing wrong with their approach, it was either;
• structured in which case their approach worked fine,
or• it was incremental, in which case their approach worked
fine.• They did not do iterative development, but if they did their
approach would still work.
This took several meetings to gain approval. Project Sponsor
Cancelled the project. (Time Boxed Testing was not cited as the reason !!!)
37
So what fun did we have ?
38
What did I learn from this?
Organisations change their development approach more quickly than their testing approach
There is a reluctance to do anything less than 100% testing
Testing is a key and integral part of any development lifecycle
The more you think about it, the more sense Time Boxed Testing makes
39
Contact Information
Graham Thomas
OSIthe management support company
e-mail : [email protected]
Phone : 0171 323 3353
Copies of slides including animated gifs at
www.osi.co.uk