Code Coverage in Theory and in practice form the DO178B perspective

Daniel Liezrowice – Engineering Software Lab (The Israeli Center for Static Code Analysis )

Presentation is Supported by ParasoftMakers of Automated DO178B Certified Analysis Tools &

A Wind River Certified partner

Testing of Code Coverage is essential part of Safety-Critical

Software Certification

A little bit of Baselining

• Why Certify?

• What is DO-178B?

• DO-178B Criticality Levels

• Effect of Criticality Levels on Testing

Why Certify?

• Software flying on a commercial aircraft and sometimes military aircraft must adhere to DO-178B.

• Meant to ensure software safety, provide high software quality, and high reliability.

What is DO-178B?

• Developed by the International Commercial Aviation Community.

• Technically an established set of guidelines for developing safety-critical software.

• In reality, it’s a set of requirements for safety-critical software development.

• Part of DO-178B identifies “levels” of software criticality and what is required to certify each of the different levels.

Software Criticality Levels

• DO-178B categorizes five levels of software criticality

• Levels A through E, with Level A being the highest level of criticality.

• Application of DO-178B to Level E systems is not required

Software Criticality Levels

• Level A: anomalous software behavior would cause or contribute to a catastrophic failure condition resulting in a total loss of life.

• Level B: anomalous software behavior would cause or contribute to a hazardous/sever-major failure condition, resulting in some loss of life.

Software Criticality Levels

• Level C: anomalous behavior would cause or contribute to a major failure condition, resulting in serious injuries.

• Level D: anomalous behavior would cause or contribute to a minor failure condition, resulting in minor injuries.

• Level E: anomalous behavior would not effect aircraft operation or pilot workload, resulting in no impact on passenger or aircraft safety.

Testing Criticality Levels

• Level D– Must have some verification process in

place, but no structural code testing is required.

• Level C– Must perform everything required by Level D

– Additionally, must perform “Statement Coverage”

– Statement Coverage - Every statement of source code must be executed by a formal test case

Testing Criticality Levels

• Level B– Must perform everything required by

Level C– Additionally must perform

“Decision/Condition” coverage– Decision/Condition Coverage - Every

code branch must be executed by a formal test case.

Testing Criticality Levels

• Level A– Must perform everything required by

Level B– Additionally must perform “Modified

Condition/Decision Coverage”– Modified Condition/Decision Coverage

- Every condition within each decision statement needs to be independently verified for its effect on the statement.

Sub summery

• Avionics software must adhere to the DO-178B guidelines before it can fly.

• The DO-178B guidelines specify five levels of software criticality.

• The level of testing required varies by the software criticality level.

• The structural coverage testing required by DO-178B is complex and is a primary cost driver on avionics software projects.

Code Coverage Agenda

•Introduction to structural test case investigation techniques•Why do we measure code coverage?•Method Coverage•Statement Coverage•Branch Coverage•Condition Coverage•Path Coverage•Limitations of structural techniques and coverage measurement

Code Coverage classified as White Box Testing•The basis for designing white box tests is the source code of the software under test •Program source must be available•Basic idea: Try to execute all parts of the source at least once

Working with Code Coverage

•Code coverage analysis is the process of:– Writing test cases and execute them– Finding areas of code not covered by a set of test cases.– Creating additional test cases to increase coverage.– Determining a quantitative measure of code coverage

•Code Coverage is an indirect measure of quality. •A code coverage analyser tool supports and automates this process.

•Code coverage is not fool-proof– There is still the possibility of errors, even with 100% code coverage

Code Coverage Tools

Visualization of code not coveredin the code editor

Visualization of coverage rate achieved

Warnings generated by the tool to point tocode not covered

Coverage rate achieved broken downby packages and classes

Method Coverage (Function Coverage)

•Reports whether a method (function) was invoked while testing the application.

– Identifies untested classes/methods.– Is the weakest of all code coverage models.– May help identifying where to focus testing at the beginning of a

testing phase.– No meaningful statement about the quality of test (coverage)

possible.

Statement Coverage (Line Coverage)

•Reports whether each executable statement was executed.– Identifies statements that are never called (Dead Code)– High statement coverage is still far from good test coverage.

• Lot’s of bugs may still be undetected!

public void doIfShort(condition){ if(condition){ statement(); } statement(); }

true covers all statement

Statement Coverage (Line Coverage)

The main disadvantage of statement coverage is that it is insensitive to some control structuresint* p = NULL; if (condition) p = &variable; *p = 123;Without a test case that causes condition to evaluate false, statement coverage rates this code fully covered. In fact, if condition ever evaluates false, this code fails.

Branch Coverage (Decision Coverage)•Reports whether boolean expressions evaluate to true AND false.

– Ignores branches within boolean expressions which occur due to short-circuit operators

– High branch coverage shows that some substantial unit testing was done

– May still miss errors in complex (compound) boolean expressions because of its focus of the two possible outcomes (true or false).

public void doIfShort(condition){ if(condition){ statement(); } statement(); }

true

false

Branch Coverage (Decision Coverage)

• This metric has the advantage of simplicity without the problems of statement coverage.

• A disadvantage is that this metric ignores branches within Boolean expressions which occur due to short-circuit operators. For example

if (condition1 && (condition2 || function1())) statement1; else statement2;

This metric could consider the control structure completely exercised without a call to function1.

The test expression is true when condition1 is true and condition2 is true,

and the test expression is false when condition1 is false. In this instance, the short-circuit operators preclude a call to function1.

public void doCondition(a,b){ if( a||b ){ statement(); } statement(); }

Condition Coverage•Reports the true or false outcome of each boolean sub-expression

– similar to branch coverage but has better sensitivity to the control flow

– But (simple) condition coverage does not guarantee full branch coverage.

•Variations– (Simple) Condition Coverage– Multiple Condition Coverage– ...

true, truetrue, falsefalse, true

false, false

Condition Coverage

• Condition coverage reports the true or false outcome of each condition. • A condition is an operand of a logical operator that does not contain logical

operators. • Condition coverage measures the conditions independently of each other.• This metric is similar to decision coverage but has better sensitivity to the

control flow.• However, full condition coverage does not guarantee full decision coverage.

For example, consider the following C++/Java fragment.

bool f(bool e) { return false; } bool a[2] = { false, false }; if (f(a && b)) ... if (a[int(a && b)]) ... if ((a && b) ? false : false) ...

All three of the if-statements above branch false regardless of the values of a and b. However if you exercise this code with a and b having all possible combinations of values, condition coverage reports full coverage.

Path Coverage (Predicate Coverage)•Reports whether each of the possible paths in each method have been followed

– A path is a unique sequence of branches from the method entry to the exit.

– 100% Path coverage may not be achieved because: • Many paths are impossible to exercise (dynamically) due to

relationships of data.• The number of paths is exponential to the number of

branches. (in some cases we can even reach the “halting problem") • Considers only a limited number of looping possibilities.

Variations of Path Coverage•data flow coverage

(By Static code Analysis)

public void doPath(condition){ if(condition){ statement(); } statement(); if(condition){ statement(); }}

10-Mar-05 28

False Positive Problem: False ErrorsSolution By Data Flow Analysis

• false error: reported by analyzer but not in fact a latent error in the program

1 int f(int x) {2 int y;3 if (x > 0) y = x;4 if (x > 3) x = y;5 return x;6 }

3

4

6

(x ≤ 0)(x > 0)

y = x

5

(x ≤ 3)(x > 3)

x = y

return x

WarningVariable 'y' (line 2) may not have been initialized

(x ≤ 0)

(x > 3)

Code Coverage: Brief Summary

public void method(a, b, c){ if( a && b ){ call(); } call(); if( c || call()){ call(); } }

Method coverage

Statement coverage

Branch coverage

Condition coverage

Path coverage

Condition/Decision Coverage

Condition/Decision Coverage is a hybrid metric composed by the union of condition coverage and decision coverage.

Modified Condition/Decision CoverageDO178B Level A

Every point of entry and exit in the program has been invoked at least once, every condition in a decision has taken all possible outcomes at least once, every decision in the program has taken all possible outcomes at least once, and each condition in a decision has been shown to independently affect that decisions outcome. A condition is shown to independently affect a decisions outcome by varying just that condition while holding fixed all other possible conditions 

This metric is specified for safety critical aviation software by RCTA/DO-178B

How It is Done? - Instrumentation

1 void foo()   2 {   3   found=false;   4   for (i=0;(i<100) && ( ! found );i++)   5   {   6     if (i==50) break;   7     if (i==20) found=true;   8     if (i==30) found=true;   9   }  10   printf("foo\n");  11 }

How It is Done? - Automatic Instrumentation.

Instrumentation for statement coverage

1 char inst[5];   2 void foo()   3 {   4   found=false;   5   for (i=0;(i<100) && (! found);i++)   6   {   7     if (i==50 ) { inst[0]=1;break;}   8     if (i==20 ) { inst[1]=1;found=true;}   9     if (i==30 ) { inst[2]=1;found=true;}  10    inst[3]=1; }  11   printf("foo\n");  12  inst[4]=1; }

How It is Done? - Instrumentation

   1 char inst[15];   2 void foo()   3 {   4   found=false;   5   for (i=0;((i<100)?inst[0]=1:inst[1]=1,0) && ((! found)?inst[2]=1:inst[3]=1,0);i++)   6   {   7     if ((i==50?inst[4]=1:inst[5]=1,0) ) { inst[6]=1;break;}   8     if ((i==20?inst[7]=1:inst[8]=1,0) ) { inst[9]=1;found=true;}   9     if ((i==30?inst[10]=1:inst[11]=1,0) ) { inst[12]=1;found=true;}  10   inst[13]=1; }  11   printf("foo\n");  12 inst[14]=1; }

Full code coverage instrumentation at condition level

Inserting the full instrumentation code for the condition coverage in this example will produce the following code:

References

• Cornett, Steve. “Code Coverage Analysis” <http://www.bullseye.com/coverage.html#basic_statement>

• Enea Teksci. “DO-178B Training”. March 2007• High Rely, Reliable Embedded Solutions. “FAA

Compliance Questions and Answers” <http://www.highrely.com/do178b_questions.php>

• Nilsen, Kelvin. “Certification Requirements for Safety-Critical Software” <http://www.rtcmagazine.com/home/article.php?id=100010&pg=1>

We now moving to the Demo but before…

Questions?

Thank You

Daniel Liezrowice – Engineering Software Lab (The Israeli Center for Static Code Analysis & Dynamic testing)

Daniel.l@eswlab.com 09-8855803 www.eswlab.com