l. dillon software engineering & network systems laboratory michigan state university 1 getting...
TRANSCRIPT
Software Engineering & Network Systems Laboratory
Michigan State University1L. Dillon
Getting Results FromTesting
Laura K. DillonSoftware Engineering and Network Systems Lab
Michigan State University
www.cse.msu.edu/sens
Software Engineering & Network Systems Laboratory
Michigan State University2L. Dillon
Overview
Background: Testing
The Problem: Temporal Behavior
Validation
A Solution: Graphical Interval Logic-
Based Oracles
Software Engineering & Network Systems Laboratory
Michigan State University3L. Dillon
PREMISES
Software intensive systems must
be tested extensively
Testing should be done early
and often
Software Engineering & Network Systems Laboratory
Michigan State University4L. Dillon
Testing is more than running tests!
Test results
Reqs/Design/Code
Formal Specifications
Evaluateadequacy
of tests
Evaluatecorrectness
of tests
Create testplans
Test plans
Reqs/Design/Code
Reqs/Design/Code
Runtests
Software Engineering & Network Systems Laboratory
Michigan State University5L. Dillon
A specification-based test oracle
Test results
Formal Specifications
Evaluatecorrectness
of tests
Checks that behaviors exhibitedduring testing satisfy thespecifications
Software Engineering & Network Systems Laboratory
Michigan State University6L. Dillon
Overview
Background: Testing
The Problem: Temporal Behavior
Validation
A Solution: Graphical Interval Logic-
Based Oracles
Software Engineering & Network Systems Laboratory
Michigan State University7L. Dillon
The Oracle Problem
Answers the question:
Were any of the test executions erroneous?
Concurrency adds a new dimension of complexity to this problem
Software Engineering & Network Systems Laboratory
Michigan State University8L. Dillon
A concurrent system …
Consists of asynchronous communicating
threads of control:
Tasks / Processes / Threads
Threads synchronize/communicate by
sending messages or through sharing data
Temporal ordering matters
Software Engineering & Network Systems Laboratory
Michigan State University9L. Dillon
GAS STATION EXAMPLE [Helmbold, Luckham: 1985] Cus(1)
Change
Cus(N)
Change
Operator
Prepay Charg
e
Pump
Start Finish
Act
. . .
Software Engineering & Network Systems Laboratory
Michigan State University10L. Dillon
Example Temporal Properties
Whenever Cus(1) prepays, it
eventually receives change, but only
after first pumping gas
If Cus(1) prepays before Cus(2), then
Cus(1) gets to pump gas before
Cus(2) gets to pump gas
Software Engineering & Network Systems Laboratory
Michigan State University11L. Dillon
Large number of behaviors can be produced
Behaviors can be long Behaviors that violate necessary
temporal constraints are easily overlooked
When validating behaviors of concurrent software systems . . .
Software Engineering & Network Systems Laboratory
Michigan State University12L. Dillon
Overview
Background: Testing
The Oracle Problem: Behavior
Validation
A Solution: Graphical Interval Logic-
Based Oracles
Software Engineering & Network Systems Laboratory
Michigan State University13L. Dillon
Overview of GIL Oracles
Desired temporal properties of execution traces are expressed formally in Graphical Interval Logic (GIL)
Execution traces are generated in testing Oracle checks that execution traces satisfy
GIL specifications Displays anomalies to help user identify
faults
Software Engineering & Network Systems Laboratory
Michigan State University14L. Dillon
Programmer defines “conditions” to trace
Operator
Prepay Charg
e
Pump
Start Finish
Cus(1)
Change
Act
Cus(N)
Change
Pay1
Pay1PayN
PayN
Pump1
Pump1
Software Engineering & Network Systems Laboratory
Michigan State University15L. Dillon
Formal comments define Pay1, . . ., PayN
--% pay$n, for $n in ID_RANGE: Condition is--% triggered by--% begin accept Cus($n):Operator.prepay--% cancelled by--% begin activation Gas_Simulation--% end accept Operator:Cus($n).change
Similarly: Pump1, . . ., PumpN
Software Engineering & Network Systems Laboratory
Michigan State University16L. Dillon
An execution trace induces a state sequence
init
acc C
(1):
Op
.pre
acc C
(2):
Op
.pre
acc C
(3):
Op
.pre
acc C
(1):
Pp
.sta
rt
en
d C
(1):
Pp
.fin
i
acc C
(2):
Pp
.sta
rt
acc O
p:C
(1).
ch
an
en
d C
(2):
Pp
.fin
acc C
(1):
Op
.pre
acc C
(3):
Pp
.sta
rt
acc O
p:C
(2).
ch
an
en
d C
(3):
Pp
.fin
acc C
(2):
Op
.pre
acc C
(3):
Op
.pre
acc C
(1):
Pp
.sta
rt
Pay1Pump1Pay2Pump2
. . .
. . .
. . .
. . .
. . .
Software Engineering & Network Systems Laboratory
Michigan State University17L. Dillon
DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas
Software Engineering & Network Systems Laboratory
Michigan State University18L. Dillon
DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas
[ )
Pay1
| |
|
[ )
Pay1
Pay1
Pump1
Software Engineering & Network Systems Laboratory
Michigan State University19L. Dillon
DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas
[ )
Pay1
| |
|
[ )
Pay1
Pay1
Pump1
Software Engineering & Network Systems Laboratory
Michigan State University20L. Dillon
DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas
[ )
Pay1
| |
|
[ )
Pay1
Pay1
Pump1
Software Engineering & Network Systems Laboratory
Michigan State University21L. Dillon
|
DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas
[ )
Pay1
| |
[ )
Pay1
Pay1
Pump1
Software Engineering & Network Systems Laboratory
Michigan State University22L. Dillon
|
DoesPump1: Whenever Cus(1) prepays, it eventually receives change, but only after first pumping gas
[ )
Pay1
| |
[ )
Pay1
Pay1
Pump1
Software Engineering & Network Systems Laboratory
Michigan State University23L. Dillon
Fair1.2: If Cus(1) prepays before Cus(2), then Cus(1) gets to pump before Cus(2) gets to pump
Software Engineering & Network Systems Laboratory
Michigan State University24L. Dillon
Fair1.2:
|
[ )Pump1
Pump2
[ )
Pay1
| |
Pay1
Pay2
Pay1
| |
Pay1
Software Engineering & Network Systems Laboratory
Michigan State University25L. Dillon
Fair1.2:
[ )
Pay1
| |
Pay1
Pay2
| |
|
[ )Pump1
Pump2
Pay1 Pay1
Software Engineering & Network Systems Laboratory
Michigan State University26L. Dillon
Fair1.2:
[ )
Pay1
| |
Pay1
Pay2
| |
|
[ )Pump1
Pump2
Pay1 Pay1
Software Engineering & Network Systems Laboratory
Michigan State University27L. Dillon
Fair1.2:
[ )
Pay1
| |
Pay1
Pay2
| |
|
[ )Pump1
Pump2
Pay1 Pay1
Software Engineering & Network Systems Laboratory
Michigan State University28L. Dillon
Fair1.2:
[ )
Pay1
| |
Pay1
Pay2
| |
|
[ )Pump1
Pump2
Pay1 Pay1
Software Engineering & Network Systems Laboratory
Michigan State University29L. Dillon
init
acc C
(1):
Op
.pre
acc C
(2):
Op
.pre
acc C
(3):
Op
.pre
acc C
(1):
Pp
.sta
rt
en
d C
(1):
Pp
.fin
acc C
(2):
Pp
.sta
rt
acc O
p:C
(1).
ch
an
en
d C
(2):
Pp
.fin
acc C
(1):
Op
.pre
acc C
(3):
Pp
.sta
rt
acc O
p:C
(2).
ch
an
en
d C
(3):
Pp
.fin
acc C
(2):
Op
.pre
acc C
(3):
Op
.pre
acc C
(1):
Pp
.sta
rt
Pay1Pump1Pay2Pump2[ )
Pay1
| |
Pay1
[ )Pay1
Violates
DoesPump1
Software Engineering & Network Systems Laboratory
Michigan State University30L. Dillon
A GIL formula, f, determines a deterministic finite state automaton, Df , such that, for every
finite state sequence, s, . . .
s f
if and only if
Accept(s, Df )
Software Engineering & Network Systems Laboratory
Michigan State University31L. Dillon
5
DFA for DoesPump1 0 1
2
4 3
P1
P1
P1
P1
P1
P1
P1 M1
P1 M1
P1 M1
P1 M1
P1
P1 = Pay1M1 = Pump1
trueP1
Software Engineering & Network Systems Laboratory
Michigan State University32L. Dillon
Checking an execution trace
init
acc C
(1):
Op
.pre
acc C
(2):
Op
.pre
acc C
(3):
Op
.pre
acc C
(1):
Pp
.sta
rt
en
d C
(1):
Pp
.fin
i
acc C
(2):
Pp
.sta
rt
acc O
p:C
(1).
ch
an
en
d C
(2):
Pp
.fin
acc C
(1):
Op
.pre
acc C
(3):
Pp
.sta
rt
acc O
p:C
(2).
ch
an
en
d C
(3):
Pp
.fin
acc C
(2):
Op
.pre
acc C
(3):
Op
.pre
acc C
(1):
Pp
.sta
rt
Pay1Pump1Pay2Pump2
. . .
. . .
. . .
. . .
. . .
0 2 3 3 4 4 4 4 2 2 3 3 3 3 3 3 . . .
Software Engineering & Network Systems Laboratory
Michigan State University33L. Dillon
Summary
Testing concurrent systems is complicated by the need to check temporal properties
Temporal oracle can be produced from GIL specifications Checks that execution traces satisfy GIL
specifications Displays anomalies to help user identify faults
Software Engineering & Network Systems Laboratory
Michigan State University34L. Dillon
Yet to do
Refine heuristics for displaying failures
Conduct case studies/user studies
Integrate oracles into a proactive debugger
Software Engineering & Network Systems Laboratory
Michigan State University35L. Dillon
Acknowledgements
Thanks!
This work was supported in part by NSF grants EIA-0000433, CDA-9617310,
and CCR-9896190 Department of the Navy, Office of Naval
Research under grant N00014-01-1-0744
Software Engineering & Network Systems Laboratory
Michigan State University36L. Dillon
Meridian: An Integrated Toolkit for Developing Interactive Distributed
Applications NSF Experimental Partnership Program
Betty Cheng, Laura Dillon,
Phillip McKinley, Kurt Stirewalt
Software Engineering & Network Systems Laboratory
Michigan State University37L. Dillon
Interactive Distributed Applications (IDAs)
Examples: On-board driver/pilot navigation
systems. Computer-supported collaborative
work environments. Distributed interactive simulation.
Interact with users.Processing/data distributed across a network.
Software Engineering & Network Systems Laboratory
Michigan State University38L. Dillon
Characteristics of IDAs
Interactivity: Must interact with one or more human users. Design requires prototyping and experimentation.
Concurrency: Comprise levels of communicating, concurrent components. Analysis requires formal reasoning.
Reuse: IDAs built primarily from reusable components. E.g., comm. protocols, resource managers, data displays. Design involves selecting/specializing components.
Software Engineering & Network Systems Laboratory
Michigan State University39L. Dillon
Meridian goals Advance development technology for IDAs.
Formalize development artifacts and relationships between them
Libraries of reusable components
Provide end-to-end automation techniques.
Impact practice: Must complement existing design methods and
notations, e.g. UML
Software Engineering & Network Systems Laboratory
Michigan State University40L. Dillon
Meridian Vision
Model Editing
SpecificationAnalysis
DesignProcessing
Testing/Simulation
IDA Models IDA Constraints IDA Interface Requirements
IDA ReuseRepository
IDA ExternalParameters
SpecificationsRefinedSpecifications
Code
FeedbackUser
Req
uire
men
ts
Test Cases, Oracles