integration testing
DESCRIPTION
TRANSCRIPT
Testing of Large Systems 1
Integration Testing
Testing of Large Systems 2
[Abrain and Moore, 2004]
Testing levels– Unit testing
• Verifies isolated functioning of seperately testable softwarepieces
– Integration testing• Verifying the interaction between software components• Architecture-driven
– System testing• Concerned with the behavior of the full system• Functional and ”non-functional”/quality properties of the
system
Testing of Large Systems 3
War Story I
In a project I was developing a component forcreating UML sequence diagrams– The component worked perfectly well (presentation,
editing, undo/redo, load/save, …) when tested byitself
When combined and tested with a class diagramcomponent it failed– The class diagram component might delete/change
packages related to sequence diagram• So if a class was created, sequence diagrams containing
objects instantiated from that class created, and the classdeleted this would lead to a failure
Testing of Large Systems 4
War Story II
I was writing a component that should interfacewith a serial port temperature sensing device(TH-03)– It would read a temperature based on a protocol for
TH-03, do some processing, and eventually give thattemperature away
Worked well… When connected to actual TH-03 device, nothing
happened– Needed to set special serial port options to power
device through serial cable…
Testing of Large Systems 5
Integration defects
Binder list typicalinterface defects…– “2/3 of defects are in
integration”
War stories arecovered by it
Testing of Large Systems 6
Integration testing
Integration testing deals with finding defects in the wayindividual parts work together
We use composition at almost any level in softwareengineering– May argue (as does Binder [2000]) that integration testing occurs
at all levels.
Testing of Large Systems 7
Why integration testing?
Why test component interactions???
If we provide– ‘sufficient’ unit testing
and– ‘sufficient’ system testing
is it not OK then???
Testing of Large Systems 8
Why integration testing?
Binder relates an argument (on axiomatizing test case adequacy):
Antidecomposition axiom:– There exists a program P and component Q [of P] such that T is
adequate for P, T’ is the set of vectors of values that variables canassume on entrance to Q for some t of T, and T’ is not adequate for Q
– System scope coverage does not necessarily achieve componentcoverage
– Why? Examples?
Anticomposition axiom:– There exist programs P and Q, and test set T, such that T is adequate
for P, and the set of vectors of values that variables can assume onentrance to Q for inputs in T is adequate for Q, but T is not adequate forP;Q
– Adequate testing at component level is not equivalent to adequatetesting of a system of components
– Why? Examples?
Testing of Large Systems 9
Premise for integration testing
At any given level: System consisting of components
Testing interactions between componentsrequire that these are stable– Stable = sufficient dependability to allow integration
testing– Threshold reflects practical considerations
In case of unstable component – then what?
Testing of Large Systems 10
Component stability/volatility
An important aspect is of course to preserve ourinvestment in testing
Spending staff hours to test a class that is laterfundamentally changed is a waste of time– In opposition to our wish for early integration testing– Postpone integration of volatile components!
Testing of Large Systems 11
Integration and architecture
Integration testing is highly coupled toarchitecture
Definition by [Bass et al., 2003]
The software architecture of a computing systemis the structures of the system, which comprise
software elements, the externally visibleproperties of those elements, and the
relationships among them.
Testing of Large Systems 12
Integration and architecture
Thus the architecture defines the testing in termsof what are
– the elements• (stable units that have already been unit tested adequately)
– and the relations• interactions that contain defects we wish to reveal
(Binder focuses on the module view of asoftware architecture)
Testing of Large Systems 13
Dependencies
Components may interact in a large number ofways using either explicit or implicit relations– composition and aggregation– delegation to objects / API calls– inheritance– global variables– instance variable access– objects as message parameters– RMI / socket communication
Testing of Large Systems 14
Dependencies
Many are however implicit dependencies thatare much harder to spot– database access / persistent store access– initialization sequences– timing constraints– file formatting / coding of byte streams
Binder states:– Integration testing does not focus on implicit
dependencies– ?
Testing of Large Systems 15
Dependency analysis
(Explicit) dependencies often dictate thesequence of testing– I.e. in which order are component interactions tested?
Thus a very helpful tool is to do a dependencyanalysis– Similar to what build systems do to determine
compilation order– Hard for implicit dependencies
Testing of Large Systems 16
Coupling
A measure of the strength of dependencies between twosubsystems
Which is preferable from a testing perspective?
Cf. testing tactics
high coupling low coupling
Testing of Large Systems 17
Example from Binder
Testing of Large Systems 18
Example from Binder
Remove cycles, do atopological sort on thedirected, acyclical dependencygraph
Root level– unit not used by any other unit
in cluster under test– often there are several roots
Leaf level– units that do not use any other
units in cluster under test. Cycles
– either be tested as a “unit”– or stubs introduced to break
cycle
Here arrows are uses relations
Testing of Large Systems 19
Lessons learned
Key lessons from early 1970s is that incremental integration and testing is most
effective
Integrating all components at the end ofdevelopment is problematic:– Last-minute changes are necessary but no time for
adequate testing– Testing often not systematic
Testing of Large Systems 20
Lessons learned
Advantages in incremental testing
– Interfaces systematically exercised and shown stablebefore new unproven interfaces are exercised
– Observed failures more likely to come from mostrecently added components making debugging moreefficient.
Testing of Large Systems 21
Stubs
So, the doctrine says integration test often buthow do we do integration with components thatare not stable or not developed at all?
Stub: partial or surrogate implementation of acomponent
Testing of Large Systems 22
Reasons for stubs
Stubs are powerful tools for many reasons
Make testing of certain conditions possible– Hardware simulator that outputs data that are seldom occurring
in practice• Example: specific patterns of temperature from temperature sensor
– Replace random behavior with predictable behavior• Example: testing backgammon move validation (calling
getNextMove() or similar to get move to validate)
Make testing economic– Stub for a database that is costly to reset/set up
Testing of Large Systems 23
Reasons for stubs
Often stubs are relevant to decouple interactions– Stub for complex algorithm with ‘simple’ answer
• Ensures that the defect found does not lie in the algorithmimplementation
– Stub for a component in a cycle
There is no free lunch!– Stubs can be costly to produce– … and sometimes you chase defects in production
code – that are actually embedded in the stubs
Morale: Keeps stubs simple!
Testing of Large Systems 24
Integration Strategies
Binder lists nine integration strategies,documented in pattern form, for doing integrationtesting
We focus on the classic strategies:– Big Bang (actually an anti-pattern )– Bottom-up– Top-down
Testing of Large Systems 25
Big Bang
Intent– demonstrate stability by attempting to exercise an
entire system with a few test runs Context
– bring all components together all at once. Allinterfaces tested in one go.
– usually ends in ‘big-bang’ – system dies miserably… Entry criteria
– all components have passed unit testing Exit criteria
– Test suite passes
Testing of Large Systems 26
Big Bang
Consequences– Fails – then what? Failure diagnosis is very difficult.– Even if exit criteria met, many interface faults can still
hide.– On the plus side: if it works, no effort has been spent
on test drivers and writing stubs.
– may be the course of action for• small systems with adequate unit testing• existing system with only minor modifications• system made from certified high quality reusable
components
Testing of Large Systems 27
Bottom Up
Intent– Demonstrate system stability by adding components
to the SUT in uses-dependency order, starting withcomponent having the fewest dependencies
Context– stepwise verification of tightly coupled components
Testing of Large Systems 28
Bottom up
Strategy: 1. stage: leafs
Testing of Large Systems 29
Bottom up
2. stage..
Testing of Large Systems 30
Bottom up
Last stage..
Testing of Large Systems 31
Bottom up
Entry criteria– components pass unit tests
Exit criteria– interface for each subcomponent has been exercised
at least once– complete when all root-level components pass test
suites
Testing of Large Systems 32
Bottom up
Consequences– Actually most unit tests in OO are partly integration tests
• Bottom-up testing implicitly takes place in a bottom up developmentprocess
Disadvantages– Driver development cost significant– Fix in lower level component may require revisions and retest up
the chain– Interfaces only indirectly exercised – only the ones used by the
component under test– Upper levels testing may require stubs in the bottom to test
special conditions– High level testing very late in cycle, i.e., validation only possible
late
Testing of Large Systems 33
Bottom up
Advantages– Parallel implementation and testing possible– Little need for stub writing
Testing of Large Systems 34
Tools: JUnit
Stubs?
Drivers?
Testing of Large Systems 35
Tools: Mock Objects
http://www.mockobjects.com/ /http://www.jmock.org/– Library for testing Java code using mock objects
Mock objects– Given an interface create an advanced stub at
runtime using reflection– May define expected values on mock objects using
constraints
Example...
Testing of Large Systems 36
Tools: Using jMock
Testing of Large Systems 37
Top Down
Intent– Demonstrate stability by
adding components to theSUT in control hierarchyorder, beginning with thetop-level control objects
Strategy– 1 stage:
• test control objects
Testing of Large Systems 38
Top Down
2. stage
Testing of Large Systems 39
Top Down
Final Stage
Testing of Large Systems 40
Top Down
Entry– each component to be integrated passes unit test
Exit– interface of each component has been exercised at
least once– complete when leaf-level components passes system
scope test suite
Testing of Large Systems 41
Top Down
Disadvantages– Large number of stubs necessary which is costly– Stubs are brittle– Fix in lower level component may require revisions up
the chain– Difficult to get lower level components sufficiently
exercised– Lower level components are exercised late
Advantages– Low driver development costs– early demonstration of user-related behavior
Testing of Large Systems 42
Variations
There are many variations of these strategies– Sandwich testing
• Moving from both top and bottom
Bottom line– there is no substitute from being clever and utilize a
combination of techniques that is most cost-efficientfor the project at hand.
Testing of Large Systems 43
High-frequency integration
Binder also describes high-frequency integrationthat is a strategy whose characteristics lies inthe timing, not the ordering, of testing– It is an intrinsic part of the process pattern daily build– At an extreme: continuous integration of eXtreme
Programming fame
Intent– Integrate new code with stabilized baseline frequently
to prevent integration bugs from going undiscoveredfor a long time; and to prevent divergence from thebaseline
Testing of Large Systems 44
High-frequency integration
Context– A stable baseline must be present; increments are the
focus of high-frequency integration– Increments size must match integration frequency
• daily builds + integrations -> increments must be deliverablein a day
– Test code developed in parallel with code– Testing must be automated…– Software Configuration Management must be in place
Testing of Large Systems 45
High-frequency integration
Procedure:
First– Revise code + test code on private branch– Desk check code and test– When all component testing passes, check-in to the integration
branch Second
– Integration tester builds system of increments– Testing using
• smoke tests and as much additional as time permits
Any increments that break HFI are correctedimmediately
Testing of Large Systems 46
High-frequency integration
Disadvantages– Automated tests must be in place– High commitment to maintaining code as well as tests– Be aware of adequacy criteria – the suite that found
the old bugs may not find the new ones Advantages
– Focus on maintaining tests is an effective bugprevention strategy
– Defects found early; debugging easier– Morale high as system works early and keeps doing it
Testing of Large Systems 47
Integration test in the TMM (1/2)
Basic integration test appears in level 2 (as does allbasic test techniques/methods).
Planning of the test process (incl. integration test) is alsoinitiated at level 2.
At level 3, integration test (and all other test activities) isintegrated in the sw life cycle.– In modern iterative/agile development methods, this happens
more or less automatically At level 4, the integration test process gets measured. At level 5 the integration test process is continuously
optimized.
Testing of Large Systems 48
Integration test in the TMM (2/2)
At level 2,At level 2, Managers
– Should ensure that multilevel testing is part of policies,planned for and catered for via project schedules andresources
Developers– Should help PM’s to make multilevel testing work, help
develop use cases/acceptance criteria, implement tests atall levels
Users/clients– Should help provide acceptance criteria, provide ample
and due feedback, participate in tests (accept/beta)
Testing of Large Systems 49
Summary
Testing levels– Unit testing
• Verifies isolated functioning of seperately testable softwarepieces
– Integration testing• Verifying the interaction between software components• Architecture-driven
– System testing• Concerned with the behavior of the full system• Functional and ”non-functional”/quality properties of the
system