presentation slides: "how to get 100% code coverage"
TRANSCRIPT
![Page 1: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/1.jpg)
Intelligent Testing 2015
Bristol, 18th June 2015
Dr Guillem Bernat
How to get to 100% code
coverage
![Page 2: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/2.jpg)
Introduction
Code coverage is important
Sometimes 100% is difficult
What do we do if we can't get 100%
![Page 3: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/3.jpg)
Code coverage defined
![Page 4: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/4.jpg)
What does it mean to get 100% code coverage?
![Page 5: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/5.jpg)
“Testing shows the presence of bugs, not their absence”
Edsger Dijkstra (1969)
100% code coverage does not mean:
your code is correct
![Page 6: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/6.jpg)
100% code coverage does not mean:
your software requirements are correct
![Page 7: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/7.jpg)
100% code coverage does not mean:
the compiler translated your code correctly
const volatile int x;
volatile int y;
void foo(void){
for(y=0; y>10; y++)
{
int z = x;
}
}
foo:
mov1 $0, y
mov1 x, %eax
jmp .L2
.L3:
mov1 y, %eax
incl %eax
mov1 %eax, y
.L2:
mov1 y, %eax
cmp1 $10, %eax
jg .L3
ret
At the -0s optimization level, GCC 4.3.0 for IA32
compiles the C code on the left to the assembly on
the right.
The compiler output is wrong: the access to volatile-
qualified variable x should not be hoisted out of the
loop.
![Page 8: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/8.jpg)
100% code coverage does not mean:
you have covered 100% of your object code
Source code Object code Compiled to
![Page 9: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/9.jpg)
Does it matter which code coverage criteria I'm
using?
Coverage type DO-178B/C ISO 26262
(Software architecture)
ISO 26262
(unit test)
Function Used with MC/DC
ASIL C, D Highly
Recommended
ASIL A, B Recommended
-
Call Coverage Not required
ASIL C, D Highly
Recommended
ASIL A, B Recommended
-
Statement Level A, B, C
Required -
ASIL A, B Highly
Recommended
ASIL C, D Recommended
Decision Level A, B
Required - -
Branch Not required -
ASIL B, C, D Highly
Recommended
ASIL A Recommended
MC/DC Level A Required - ASIL D Highly Recommended
ASIL A, B, C Recommended
![Page 10: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/10.jpg)
What are the barriers to 100% code coverage?
Missing requirements.
Missing or incorrect tests.
Dead code.
Deactivated code.
Defensive programming.
Impossible combinations of events.
Compiler-introduced errors.
![Page 11: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/11.jpg)
Barrier #1: Missing Requirements
Add new requirements
Develop tests
Run the tests
What to do about it
![Page 12: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/12.jpg)
Barrier #2: Missing/incorrect tests
What to do about it Implement additional tests
Ensure that they trace to
the correct requirement(s)
Run the new tests
Implement additional tests
Ensure that they trace to
the correct requirement(s)
Run the new tests
What to do about it
![Page 13: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/13.jpg)
Barrier #3: Dead code
foo(a,c){
int b;
mode = MOVING;
if(mode == STATIONARY){
b = 1 + a;
}
c = a;
return c;
}
Remove the dead code if
possible or
Justify why the code cannot
be run
What to do about it
DO-178C recommends "The [dead] code should be removed and
an analysis performed to assess the effect and the need for
reverification." [6.4.4.3c]
For ISO 26262, the general guidance for "insufficient" coverage
applies, namely to provide a rationale for why the dead code is
acceptable [6:9.4.5].
What to do about it…
![Page 14: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/14.jpg)
Barrier #4: Deactivated code
settings = read_input(CONFIG_PINS);
if ((settings & CONFIG_MASK_A) == CONFIG_A) {
/* code to execute in configuration A */
...
} else if ((settings & CONFIG_MASK_B) == CONFIG_B) {
/* code to execute in configuration B */
...
}
Justify why code won’t be
executed
Never executed during normal operation
Will not affect the execution of the system
What to do about it
![Page 15: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/15.jpg)
Barrier #5: Defensive Programming
switch(a % 4){
case 0:
..........
break;
case 1:
..........
break;
case 2:
..........
break;
case 3:
..........
break;
default:
..........
break;
}
Justify why defensive
programming cannot be
executed
Provide evidence that if
triggered, the defensive
programming works
correctly
What to do about it
![Page 16: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/16.jpg)
Barrier #6: Impossible Combinations of Events
-- “enabled” might already be set at this point
if speed < 1700 then
enabled := true;
end if;
if speed < 1500 and enabled then
...
end if;
Simplify the condition?
or
Justify why 100% could not
be achieved
What to do about it
![Page 17: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/17.jpg)
Barrier #7: Compiler Introduced Errors
int a,x;
if a < 10{ /*Compiler generates “a <=10”*/
result = x / 5;
} else {
result = x / 10;
}
a x
Test case 1 3 4
Test case 2 10 4
Review existing code base
Add to coding guidelines /
code review guidelines
What to do about it
![Page 18: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/18.jpg)
Justifying code: in-line justifications
void fatal_error (void)
{
#pragma RVS justification( "COV_FUNCTIONS", "The fatal_error
function cannot be executed during normal execution")
/* some code */
}
...
void check_status (int status)
{
if ( (check_enabled) && (FATAL_ERROR == status) ) {
#pragma RVS justification( "COV_DECISIONS", "Variable 'status'
cannot take value FATAL_ERROR during normal execution")
fatal_error();
}
}
...
![Page 19: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/19.jpg)
Justifying code: justifications in a separate file
#pragma RVS [selftest.c:34] justification("COV_MCDC", "2: (RAM_ERROR ==
status)", "RAM_ERROR cannot be triggered during normal test");
#pragma RVS [selftest.c:439] justification("COV_FUNCTIONS", "Cannot trigger
emergency_shutdown() during normal test");
![Page 20: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/20.jpg)
Justifications editor
![Page 21: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/21.jpg)
![Page 22: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/22.jpg)
Review – establish missing coverage
Identify cause Identify action to take
Dealing with less than 100% code coverage
![Page 23: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/23.jpg)
Review – establish missing coverage
Identify cause Identify action to take
Dealing with less than 100% code coverage
Missing requirements.
Missing or incorrect tests.
Dead code.
Deactivated code.
Defensive programming.
Impossible combinations of events.
Compiler-introduced errors.
![Page 24: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/24.jpg)
Review – establish missing coverage
Identify cause Identify action to take
Dealing with less than 100% code coverage
Remove code Add (or fix)
requirements Justify why code
won't be executed Justify why it isn't
possible to test
![Page 25: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/25.jpg)
reduce testing timescales by running fewer on-target tests
reduce risk through greater tool flexibility
reduced effort for certification activities
Structural coverage analysis to DO-178B/C Level A and ISO 26262
![Page 26: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/26.jpg)
Logos and images can be positioned within this area.
Change layout to blank (without position box) before use
![Page 27: Presentation slides: "How to get 100% code coverage"](https://reader030.vdocuments.mx/reader030/viewer/2022032513/55d15946bb61eb664a8b46a3/html5/thumbnails/27.jpg)
Q A