claus brabrand, itu, denmark feb 26, 2008white-box testing white-box testing claus brabrand [...
TRANSCRIPT
Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
White-Box Testing
Claus Brabrand[ [email protected] ]
( “FÅP”: First-year Project Course, ITU, Denmark )
[ 2 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Outline
Motivation Why bother with testing?
What is Testing? Relation to other programming-related tasks?
White-box Testing Methodology: ”coverage testing”
Automation How to automate testing
Exercises Training for the exam
[ 3 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Learning & Exam Goals
”Product”:
”Oral Exam”:
Today, we’ll ”train for the exam”… :-)
[ 4 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Software Errors
Mobile Phones ’00-…
Freeze and odd behaviors (really annoying)!
Cruise Control System Model ’86 (Grady Booch)
Accellerated after car ignition car crashes
Baggage Handling System ’94-’95 (at Denver Int’l Airport)
$ 360,000,000 USD
[ 5 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
zzzz
Software Errors (cont’d)
Train Control System ’98 (Berlin)
Train cancellations
Mars Pathfinder July ’97
Periodic resets
Win95/98 w/ 3rd-Party Device Drivers late ’90es
Dysfunction (“blue screen of death”)!
...on mars!
[ 6 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Software Errors (cont’d2)
Therac-25 Radiation Therapy ’85-’87
Massive overdoses (6 deaths / amputations)!
Patriot Missile Guidance System ’91 (Gulf War 1.0)
Accumulating rounding errors deaths
Ariane V ’96 (one of the most expensive bugs, ever)
Conversion from 64-bit float to 16-bit signed int
[ 7 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
…and what about?!
Surgical Laser Control System
Oops…!
Air Plane Control System
Dysfunction (plane crash)!
Nuclear Powerplant Control System
Core melt-down (“China-syndrome”)!
[ 8 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Outline
Motivation Why bother with testing?
What is Testing? Relation to other programming-related tasks?
White-box Testing Methodology: ”coverage testing”
Automation How to automate testing
Exercises Training for the exam
[ 9 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Errors! (different kinds)
Syntactic errors: Mal-formed program:
Semantic errors: Symbol errors Type errors Other semantic errors:
(e.g. uninitialized vars)
Logical errors: Compiler: ”no errors”
int square(int x) { return x*x} *** syntax error at line 2
’;’ expected
int square(int x) { return n*n;} *** symbol error at line 2
undefined variable ”n”
int square(float x) { return x*x;}*** type error at line 2 function returns float, not int
int square(int x) { return x+x;} no errors found!!!
[ 10 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
ISO9126
Maintainability
ReliabilityUsability
Portability
How robust is the SW wrt. incorrect inputs, ’^C’, external netwk failures, ...?
How easy is the SW to understand?…and use?
How easy is it to transfer and adapt SW to new environment / platform?How easy is it to modify the SW?
And fix errors?
Int’l standard for evaluation of: Software Quality
Functionality Efficiency
ISO 9126
Does the SW do what it’s supposed to?
Does it work as intended?
How much time/memory/space/- bandwidth/… does the SW consume?
[ 11 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Testing vs. Debugging?
Testing vs. Debugging:
Functionality: Efficiency:
Quality Assurance:(Functionality)
Testing(Performance)
Testing
Diagnosis: Profiling
Purpose:
Regarding:
Debugging
[ 12 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Testing vs. Debugging (cont’d)
Evaluate test resultsFix problem(reprogram)
FunctionalityTesting
Quality assurance:
Is program ok?
Debugging
Diagnosis:
Determine problem?
01101021Program:
01101011
(greater confidence!)
SYSTEMATIC
Document test results
(confidence?!?)
Re-
01101011
[ 13 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Performance Testing vs. Profiling
Evaluate test resultsImprove program(reprogram)
Performance Testing
Quality assurance:
Efficient enough?
Profiling
Diagnosis:
Determine problem?
01101021Program:
01101011
(greater confidence!)Document test results
(confidence?!?)
Re-
SYSTEMATIC
01101011
[ 14 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Testing…:
”Testing is easy” (e.g., ”random experimentation”)
”Testing well is not easy” Requires SYSTEMATIC approach; test-case
production evaluation documentation
”Testing can never prove error absence” (i.e., testing is an ”incomplete process”)
[ 15 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Representative?
Comprehensive?
Quality?
Quantity?
…?
Appropriate Test Sampling?
[ 16 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
White-box vs. Black-box Test
White-box Testing: (aka., ”structural testing”) (aka., ”internal testing”)
Test focus: source code
Black-box Testing: (aka., ”behavioral testing”) (aka., ”external testing”)
Test focus: specification (manual)
Complementary Approaches!!!
n = in();
n = n/2;
odd(n)
n = 3*n+1;
out(n);
tt ff ~?program spec
[ 17 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
”Software Testing” (R. Patton)
Background reading: ”Software Testing”, Ron Patton, Sams Publishing, 2006 Part II (pp. 53 – 123); ”Testing Fundamentals”:
”Examining the Code”(chapter 6)
”Examiningthe Spec.”(chapter 4)
”Testing w/Blinders On”
(chapter 5)
”Testing w/X-ray Glasses”
(chapter 7)
Dynamic(at runtime)
Static(before runtime)
-testing -testingTime:
Type:
[ 18 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Outline
Motivation Why bother with testing?
What is Testing? Relation to other programming-related tasks?
White-box Testing Methodology: ”coverage testing”
Automation How to automate testing
Exercises Training for the exam
[ 19 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Test Coverage?
Method coverage: Does every method run (at least once)?
Statement coverage: Does every statement run (at least once)?
Branch coverage: Does every branch run (at least once)?
Path coverage: Does every path run (at least once)?
[ 20 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Statement coverage
Branch coverage: Does every branch run (at least once)?
-Box ”Branch Coverage Testing” is: Efficient (fast) ! Effective (thorough) !
Good for complicated program logic(esp. ”initialization errors”)
[ 21 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Control Structures
Control Structures: Statements (or Expr’s) that affect ”flow of control”:
if-else:
if:
if ( Exp ) Stm1 else Stm2
if ( Exp ) Stm
Stm1
Exptrue false
Stm2
confluence
Stm
Exptrue false
confluenceThe expression must be of type boolean; if it evaluates to true, the given statement is executed, otherwise not.
The expression must be of type boolean; if it evaluates to true, Statement-1 is executed, otherwise Statement-2 is executed.
[syntax]
[semantics]
[syntax]
[semantics]
[ 22 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Control Structures (cont’d)
while:
for:
while ( Exp ) Stm
for (Exp1 ; Exp2 ; Exp3) Stm
Equivalent to:
The expression must be of type boolean; if it evaluates to false, the given statement is skipped, otherwise it is executed and afterwards the expression is evaluated again. If it is still true, the statement is executed again. This is continued until the expression evaluates to false.
{ Exp1; while ( Exp2 ) { Stm Exp3; }}
Stm
Exptrue false
confluence
Exp1;
Exp2true false
Stm
confluence
Exp3;
[syntax]
[semantics]
[syntax]
[semantics]
[ 23 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Stm/Branch Coverage Testing
if: TEST condition true and false
if-else: TEST condition true and false
while: TEST zero, one, more-than-one iterations in loop
for: TEST zero, one, more-than-one iterations in loop
[ 24 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Example 1
public static void main ( String[] args ) { int mi, ma; if (args.length == 0) System.out.println("No numbers"); else { mi = ma = Integer.parseInt(args[0]); for (int i=1; i < args.length; i++) { int obs = Integer.parseInt(args[i]); if (obs > ma) ma = obs; else if (mi < obs) mi = obs; } System.out.println(”min=" + mi + "," + "max=" + ma);}}
/* 1if-else */
/* 2for */
/* 3if-else */
/* 4if */
if
else
for
if
elseif
Choice points?
[ 25 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Control-Flow Graph
CFG:int mi, ma;
System.out.println("No numbers");
args.length == 0
mi = ma = Integer.parseInt(args[0]);
int i=1;
i < args.length
i++;
System.out.println(”min=" + mi + "," + "max=" + ma);
int obs = Integer.parseInt(args[i]);
obs > ma
ma = obs; mi < obs
mi = obs;
true false
true false
true false
true false
2
34
1
[ 26 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
”Coverage Table”:
Coverage Table
Input property
No numbers
At least one number
Exactly one number
Exactly two numbers
At least three numbers
N > current maxN current maxN cur max & N > cur minN cur max & N cur min
Data set
A
B
B
C
E
C
D
E (3rd num)
E (2nd num)
Choice
1ife true
false
2for zero-times
once
more-than-once
3ife true
false
4if true
false
[ 27 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
”Expectancy Table”:
Expectancy Table
Data set
A
B
C
D
E
Advice:Avoid expected 0’s (i.e., zeroes)
(Default value in many languages.)Advice:Avoid reusing same numbers in tests
(Data layout sometimes reuse old memory.)
Input
[17]
[27,29]
[39,37]
[49,47,48]
Expected output
”no numbers”
”min=17,max=17”
”min=27,max=29”
”min=37,max=39”
”min=47,max=49”
Actual output
”no numbers”
”min=17,max=17”
”min=27,max=29”
”min=39,max=39”
”min=49,max=49”
=
[ 28 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
public static void main ( String[] args ) { int mi, ma; if (args.length == 0) System.out.println("No numbers"); else { mi = ma = Integer.parseInt(args[0]); for (int i=1; i < args.length; i++) { int obs = Integer.parseInt(args[i]); if (obs > ma) ma = obs; else if (mi < obs) mi = obs; } System.out.println(”min=" + mi + "," + "max=" + ma);}}
Debugging ’ D ’ then reveals…
/* 1if-else */
/* 2for */
/* 3if-else */
/* 4if */
if
else
for
if
elseif
Should have been:
(obs < mi)
[ 29 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Re-Test !
…as debugging oftenintroduces new errors !
Fixed Program:Coverage Table:
Expectancy Table:
Recall: no guarantee!
[ 30 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Example 2public static void main ( String[] args ) { int mi1 = 0, mi2 = 0; if (args.length == 0) /* 1if-else */ System.out.println("No numbers"); else { mi1 = Integer.parseInt(args[0]); if (args.length == 1) /* 2if-else */ System.out.println("Smallest = " + mi1); else { int obs = Integer.parseInt(args[1]); if (obs < mi1) /* 3if */ { mi2 = mi1; mi1 = obs; } for (int i = 2; i < args.length; i++) { /* 4for */ obs = Integer.parseInt(args[i]); if (obs < mi1) /* 5if-else */ { mi2 = mi1; mi1 = obs; } else if (obs < mi2) /* 6if */ mi2 = obs; } System.out.println("The two smallest are: " + mi1 + " and " + mi2);} } }
[ 31 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Coverage Table (Ex. 2)
Choice Input property Data set
1ife true No numbers
1ife false At least one number
2ife true Exactly one number
2ife false At least two numbers
3if true 2nd number ≥ 1st number
3if false 2nd number < 1st number
4for zero-times Exactly two numbers
4for once Exactly three numbers
4for more-than-once At least four numbers
5ife true 3rd number < current min
5ife false 3rd number ≥ current min
6if true 3rd ≥ cur min & 3rd < 2nd least
6if false 3rd ≥ cur min & 3rd ≥ 2nd least
A
B
B
C
C
D
D
E
H
E
F
F
G
[ 32 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Expectancy Table (Ex. 2)
Data set Input Expected output = Actual output
A ”no numbers” ”no numbers” B [17] ”17” ”17”
C [27,29] ”27 and 29” ”27 and 0”
D [39,37] ”37 and 39” ”37 and 39”
E [49,48,47] ”47 and 48” ”47 and 48”
F [59,57,58] ”57 and 58” ”57 and 58”
G [67,68,69] ”67 and 68” ”67 and 0”
H [77,78,79,76] ”76 and 77” ”76 and 77”
Debugging reveals that variable
”mi2” erroneously retains initialization (0).
[ 33 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Debugging (Ex. 2)public static void main ( String[] args ) { int mi1 = 0, mi2 = 0; if (args.length == 0) /* 1if-else */ System.out.println("No numbers"); else { mi1 = Integer.parseInt(args[0]); if (args.length == 1) /* 2if-else */ System.out.println("Smallest = " + mi1); else { int obs = Integer.parseInt(args[1]); if (obs < mi1) /* 3if */ { mi2 = mi1; mi1 = obs; } for (int i = 2; i < args.length; i++) { /* 4for */ obs = Integer.parseInt(args[i]); if (obs < mi1) /* 5if-else */ { mi2 = mi1; mi1 = obs; } else if (obs < mi2) /* 6if */ mi2 = obs; } System.out.println("The two smallest are: " + mi1 + " and " + mi2);} } }
mi2 = obs;
Re-Test:
[ 34 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Control Structures (cont’d2)
switch:
do-while:
”?:”; ”conditional expression”:
”&&”; ”lazy conjunction” (aka., ”short-cut ”):
”||”; ”lazy disjunction” (aka., ”short-cut ”):
switch ( Exp ) { Swb* }
case Exp : Stm* break;
do Stm while ( Exp );
default : Stm* break;
Swb:
Exp1 ? Exp2 : Exp3
Exp1 && Exp2
Exp1 || Exp2
Choice points?
[ 35 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Control Structures (cont’d3)
try-catch-finally (exceptions):
return / break / continue:
”method invocation”: e.g.;
”recursive method invocation”: e.g.;
”virtual dispatching”: e.g.;
try Stm1 catch ( Exp ) Stm2 finally Stm3
f(x)
return ; return Exp ; break ; continue ;
f(x)
f(x)
[ 36 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Outline
Motivation Why bother with testing?
What is Testing? Relation to other programming-related tasks?
White-box Testing Methodology: ”coverage testing”
Automation How to automate testing
Exercises Training for the exam
[ 37 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Test Automation
(Re-)Running tests is boooring (& error prone) Thus, automate them ”once-and-for-all”
JUnit:
Can be run from Eclipse/JUnit: (if appropriately ”subclassing” TestCase)
public class MyTestCase extends TestCase {
/* ...other tests... */}
@Test // Testing if 3*2=6: public void testMultiplication() { assertEquals("Multiplication", 6, 3*2); }
[ 38 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Outline
Motivation Why bother with testing?
What is Testing? Relation to other programming-related tasks?
White-box Testing Methodology: ”coverage testing”
Automation How to automate testing
Exercises Training for the exam
[ 39 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Exercise: Part I:
Part II:
Program merge (in Java):
Test your merge method:a) Label choice pointsb) Build ”coverage table” & make data set (test suite)c) Build ”expectancy table”d) Run test suite (upon failure: fix and retest program)
Introduce subtle bug Run test to document presence of bug Submit “erroneous program” to class program pool
List<Integer> merge(List<Integer> list1, List<Integer> list2);
Pick “erroneous program” from class program pool Test merge program (and debug to find bug) Re-Test fixed merge program Write report (and send it to the teaching assistant)
(produce)
(consume)
Warm up exercise:Draw a control-flow diagram for the ”do-while” construction
[ 40 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Specification (merge)
Interface (for ”merge”):
I/O assumptions: Input: both lists are sorted:
…and, numbers occur maximum once (in each list)
Output: list must be sorted: …and, numbers occur maximum once
Programming constraints: no recursion ! no java.util.Iterator’s !
List<Integer> merge(List<Integer> list1, List<Integer> list2);
[ 41 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Report (ca. 3 pages): Must explain how you tested, (debugged), & re-tested
the erroneous ”merge” (it must include, at least): i) The erroneous program ii) a ”Control-Flow Graph” for the program [hand-drawn ok]
(incl. labelled ”choice points”) iii) ”Coverage Tables” iv) ”Expectancy Tables”
Submit the testing report to the T.A. (Anders) (deadline: Tuesday; March, 11 at 09:00 CET) Note: the report is only on Part II
(i.e., the program you didn’t write)
The Testing Report
[ 42 ]Claus Brabrand, ITU, Denmark Feb 26, 2008WHITE-BOX TESTING
Example 1
public static void main ( String[] args ) { int mi, ma; if (args.length == 0) System.out.println("No numbers"); else { mi = ma = Integer.parseInt(args[0]); for (int i=1; i < args.length; i++) { int obs = Integer.parseInt(args[i]); if (obs > ma) ma = obs; else if (mi < obs) mi = obs; } System.out.println(”min=" + mi + "," + "max=" + ma);}}