test-vs-inspection.ppt
TRANSCRIPT
-
7/29/2019 Test-vs-inspection.ppt
1/44
Test vs. inspection
Part 1
Tor Stlhane
-
7/29/2019 Test-vs-inspection.ppt
2/44
What we will cover
Part 1
Introduction
Inspection processes Testing processes
Part 2
Tests and inspections some data Inspection as a social process two experiments
and some conclusions
-
7/29/2019 Test-vs-inspection.ppt
3/44
Introduction
-
7/29/2019 Test-vs-inspection.ppt
4/44
Adams data - 1
Mean Time to Problem Occurrence
yearsProduct 1.6 5 16 50 160 500 1600 5000
1 0.7 1.2 2.1 5.0 10.3 17.8 28.8 34.22 0.7 1.5 3.2 4.3 9.7 18.2 28.0 34.33 0.4 1.4 2.8 6.5 8.7 18.0 28.5 33.74 0.1 0.3 2.0 4.4 11.9 18.7 28.5 34.25 0.7 1.4 2.9 4.4 9.4 18.4 28.5 34.26 0.3 0.8 2.1 5.0 11.5 20.1 28.2 32.07 0.6 1.4 2.7 4.5 9.9 18.5 28.5 34.08 1.1 1.4 2.7 6.5 11.1 18.4 27.1 31.99 0.0 0.5 1.9 5.6 12.8 20.4 27.6 31.2
-
7/29/2019 Test-vs-inspection.ppt
5/44
Adams data 2
The main information that you get from the
table on the previous slide is that
Some defects are important because they will
happen quite often.
Most defects are not important since they will
happen seldom.
How can we tell the difference?
-
7/29/2019 Test-vs-inspection.ppt
6/44
Testing alone is not the solution
As can be seen from the next slide, testing is not
an acceptable solution alone. It will
Take too long time
Cost too much
We can generate tests automatically, but would
never the less have to use large resources tocheck the result the oracle problem
-
7/29/2019 Test-vs-inspection.ppt
7/44
A limit result
The following relation holds under a rather wideset of conditions:
The initial number of defects N0 must be
estimated e.g. based on experience fromearlier projects as number of defects per
KLOC.
tN
e
MTTFt0
-
7/29/2019 Test-vs-inspection.ppt
8/44
An example from telecom
-
7/29/2019 Test-vs-inspection.ppt
9/44
Testing and inspection the V model
-
7/29/2019 Test-vs-inspection.ppt
10/44
Testing and inspection 1
The important message here is that testingcannot always be done.
In the first, important phases, we have nothing
to execute and will thus always have to dosome type of inspection.
This might be considered one of the weaknesses
of traditional software engineering over Agiledevelopment.
-
7/29/2019 Test-vs-inspection.ppt
11/44
Testing and inspection 2
In order to understand the main differences
between testing and inspection, we should
consider Fits list.
Based on this, we will give a short discussion of
the relative merits of testing and inspection.
-
7/29/2019 Test-vs-inspection.ppt
12/44
Area of
competence
Man Machine
Understanding Good at handling variations in
written material
Bad at handling variations in
written materialObserve General observations,
multifunctional
Specialized, good at observing
quantitative data, bad at
pattern recognition
Reasoning Inductive, slow, imprecise but
good at error correction
Deductive, fast, precise but
bad error correction
Memory Innovative, several access
mechanisms
Copying, formal access
Information
handling
Single channel, less than 10
bits per second
Multi channel, several
Megabits per second
Consistency Unreliable, get tired, depends
on learning
Consistent repetition of several
actions
Power Low level, maximum ca. 150
watt
High level over long periods
of time
Speed Slow seconds Fast
-
7/29/2019 Test-vs-inspection.ppt
13/44
Man vs. machine 1
Good when we need the ability to Handle variation
Be innovative and inductive
Recognize and handle patterns
Not so good when we need the ability to
Do the same things over and over again in aconsistent manner
Handle large amount of data
-
7/29/2019 Test-vs-inspection.ppt
14/44
Man vs. machine 2
In order to do the best job possible we needprocesses where we let each part
Do what they are best at:
Man is innovative Machine handles large amounts of data
Support the other with their specialties.
Machine supports man by making large amountsof information available
Man support machine by providing it with
innovative input
-
7/29/2019 Test-vs-inspection.ppt
15/44
General considerations - documents
Architecture, system, sub-system and
component design plus pseudo code. Here we
can only use inspections.
Man will use experience and knowledge to
identify possible problems
Machine can support by identifying information
e.g. find all occurrences of a string.
-
7/29/2019 Test-vs-inspection.ppt
16/44
General considerations code (1)
For executable code, we can use inspection,testing or a combination of both.
The size and complexity degree of dynamism
of the code will, to a large degree, decide ourchoice.
Other important factors are the degree of
experience with The programming language
The algorithms used
-
7/29/2019 Test-vs-inspection.ppt
17/44
General considerations code (2)
Simple code Start with inspection all code
Design and run tests
Complex code Start with inspection focus on algorithm and
logic
Decide test completeness criteria we cannottest everything
Design and run tests
-
7/29/2019 Test-vs-inspection.ppt
18/44
Inspection processes
-
7/29/2019 Test-vs-inspection.ppt
19/44
Inspections 1
The term inspection is often used in a rather
imprecise manner. We will look at three typesof inspection:
Walkthrough
Informal inspection also called informalreview
Formal inspection also called formal review
or just inspectionThe first two types are usually project internal
while the last one is used as a final acceptanceactivity for a document.
-
7/29/2019 Test-vs-inspection.ppt
20/44
Inspections 2
For all types of inspections:
The quality of the results depends on the
experience and knowledge of the participants.
Garbage in Garbage out
It might be a good idea to involve customer
representatives.
-
7/29/2019 Test-vs-inspection.ppt
21/44
The walkthrough process
Walkthrough is a simple process mostly used
for early decisions for an activity. The
document owner:
1. Makes a rough sketch of the solution
architecture, algorithm etc.
2. Presents explain the sketch to whoever
shows up.
3. Registers feedback improvements.
-
7/29/2019 Test-vs-inspection.ppt
22/44
Walkthrough pros and cons
Pros: Easy and inexpensive. Needs no extra
preparation.
Collect ideas at an early stage of development.Cons:
No commitment from the participants
May collect many loose or irrelevant ideas
-
7/29/2019 Test-vs-inspection.ppt
23/44
The informal inspection process
Individual
checking
Planning
Productdocument
Changerequests
Rules,checklists,
procedures
Logging
meeting
-
7/29/2019 Test-vs-inspection.ppt
24/44
Informal inspections pros and cons
Pros: Is simple and inexpensive to perform.
Can be used at all stages of development
Usually has a good cost / benefit ratio
Needs a minimum of planning
Cons:
No participant commitment
No process improvement
-
7/29/2019 Test-vs-inspection.ppt
25/44
The formal inspection process
The formal inspection process described below
is with small variations the most
commonly used. The version shown on the
following slides stem from T. Gilb and D.Graham.
We recommend this process as the final
acceptance process for all importantdocuments
-
7/29/2019 Test-vs-inspection.ppt
26/44
Formal inspection process overview
Walk-
through
Kick-off Individual
checking
Edit
and
follow-
up
Planning
Process
improvements
Product
document
Changerequests
Rules,
checklists,
procedures
Logging
meeting
-
7/29/2019 Test-vs-inspection.ppt
27/44
Distribution of resources
Activity Range %Typical
value %
Planning 3 5 4
Kick-off 4 7 6
Individual checking 20 30 25
Logging 20 30 25
Editing 15 30 20
Process brainstorming 15 30 16
Leader overhead, follow up, entry,
exit
3 5 4
-
7/29/2019 Test-vs-inspection.ppt
28/44
Initiating the inspection process
The inspection process starts with a request
for inspection from the author to the QA
responsible.
The QA responsible appoints an inspection
leader.
First step is always to check that the
document is fit for inspection.
-
7/29/2019 Test-vs-inspection.ppt
29/44
Planning
Important planning points are:
Who should participate in the inspections
Who is interested?
Who have time available for preparation and
meetings?
Who has the necessary knowledge concerning
application, language, tools, methods?
-
7/29/2019 Test-vs-inspection.ppt
30/44
Kick-off
Important activities here are:
Distribution of necessary documents:
Documents that shall be inspected
Requirements
Applicable standards and checklists
Assignment of roles and jobs
Setting targets for resources, deadlines etc.
-
7/29/2019 Test-vs-inspection.ppt
31/44
Individual checking
This is the main activity of the inspection. Each
participant read the document to look for
Potential errors - inconsistencies with
requirements or common application
experience
Lack of adherence to company standards or
good workmanship
-
7/29/2019 Test-vs-inspection.ppt
32/44
Logging meeting
The logging meeting has three purposes:
Log issues already discovered by inspection
participants
Discover new issues based on discussions
and new information that arises during the
logging meeting.
Identify possible improvement to the
inspection or development process.
-
7/29/2019 Test-vs-inspection.ppt
33/44
Improve the product - 1
The author receives the log from the inspection
meeting. All items - issues - in the log are
categorised as one of the following:
Errors in the authors document.
Errors in someone elses document.
Misunderstandings in the inspection team.
-
7/29/2019 Test-vs-inspection.ppt
34/44
Improve the product - 2
Errors in own document:
Make appropriate corrections
Errors in someone elses documents:
Inform the owner of this document.
Misunderstandings in the inspection team:
Improve document to avoid further
misunderstandings.
-
7/29/2019 Test-vs-inspection.ppt
35/44
Checking the changes
This is the responsibility of the inspectionleader. He must assure that all issues raised
in the log are disposed of in a satisfactory
manner: The documents that have been inspected
Related documents - including standards
and checklists Suggested process improvements
-
7/29/2019 Test-vs-inspection.ppt
36/44
Formal inspection pros and cons
Pros:
Can be used to formally accept documents
Includes process improvement
Cons:
Is time consuming and expensive
Needs extensive planning in order to succeed
-
7/29/2019 Test-vs-inspection.ppt
37/44
Testing processes
-
7/29/2019 Test-vs-inspection.ppt
38/44
Testing
We will look at three types of testing:
Unit testing does the code behave as
intended. Usually done by the developer
Function verification testing also called
systems test. Does the component or system
provide the required functionality?
System verification testing also called
acceptance test. Does the hardware and
software work together to give the user the
intended functionality?
-
7/29/2019 Test-vs-inspection.ppt
39/44
The unit testing process
Unit testing is done by the developer one or
more times during development. It is a rather
informal process which mostly run as follows:
1. Implement (part of) a component.
2. Define one or more tests to activate the code
3. Check the results against expectations and
current understanding of the component
-
7/29/2019 Test-vs-inspection.ppt
40/44
Unit testing pros and cons
Pros:
Simple way to check that the code works.
Can be used together with coding in an
iterative manner.
Cons:
Will only test the developers understandingof the spec.
May need stubs or drivers in order to test
-
7/29/2019 Test-vs-inspection.ppt
41/44
The system test process
A systems test has the following steps:
1. Based on the requirements, identify
Test for each requirement, including error handling
Initial state, expected result and final state
2. Identify dependencies between tests
3. Identify acceptance criteria for test suite
4. Run tests and check results against
Acceptance criteria for each test Acceptance criteria for the test suite
-
7/29/2019 Test-vs-inspection.ppt
42/44
Systems test pros and cons
Pros:
Tests systems behavior against customer
requirements.
Cons:
It is a black box test. If we find an error, the
systems test must be followed by extensive
debugging
-
7/29/2019 Test-vs-inspection.ppt
43/44
The acceptance test process
The acceptance test usually has three activitiesall involving the customer or hisrepresentatives:
Rerun the systems test at the customers site. Use the system to solve a set of real-world
tasks.
Try to break the system by stressing it or byfeeding it large amounts of illegal input
-
7/29/2019 Test-vs-inspection.ppt
44/44
Acceptance test pros and cons
Pros:
Creates confidence that the system will beuseful for the customer
Shows the systems ability to operate in thecustomers environment
Cons:
Might force the system to handle input that itwas not designed for, thus creating anunfavorable impression.