coverage analysis from execution traces

Theoretical foundation of source coverage analysis from execution traces!

Thomas Quinot!

Summary!

•  Original Needs & Goals!

•  Challenges along the way!

•  Main Results!

Original Needs!

•  Structural Coverage Analysis is required by certification standards:!•  Open source Coverage Tools exist but are not usable in a HI context!

•  Proprietary Tools exist but do not support all versions of Ada!

•  Complete the GNAT Pro Toolset for the High Integrity Market!

•  Better support for the rapidly evolving versions of Ada (83 … 95 … 2005 … 2012 …)!

Original Goals!

•  Provide an High Quality Open Source alternative to existing proprietary tools!

•  Provide Support for Agile/Lean Development!•  In particular: Continuous Integration/Certification!

•  Open-DO initiative!

•  Find the best compromise between Source and Object Coverage!

The Couverture Project (2008-2010)!

•  One of the first FUI projects from the GTLL at System@atic!

•  4 partners (AdaCore, Openwide, Telecom PT, Paris 6)!

•  Effort of 160 man-month (2,23 M€) over 2 years

•  45% Financed by the city of Paris, IdF region, DGE !

This project gave us the capability to meet the unexpected challenges we were facing.

Object Coverage vs Source Coverage!

•  Big debate in the Certification Community!

•  Which one is the most Accurate / Appropriate ?!

•  Which one is the most efficient ?!

Source

-  Statement/Decision are source concepts

-  usually works by instrumenting the code

-  can be done on fast native platforms

-  requires double testing strategy

Object

-  on final code (no instrumentation)

-  on final hardware

-  not language specific

-  more precise

Object Coverage vs Source Coverage!

•  Object coverage metrics:!

•  Instruction Coverage!

•  Object Branch Coverage (OBC)!

•  Source coverage metrics:!

•  Statement Coverage!

•  Decision Coverage (DC)!

•  Modified Condition/Decision Coverage (MC/DC)Independent influence of each condition within a decision!

Challenge 1!

!It is difficult to provide accurate source coverage info from execution traces:!

! !- no trace of “statement” / “condition” / “decision” at ! binary level!

! !- optimization can change significantly the control flow

Accurate Source Coverage Info!

Sources Sources Sources Sources GNAT Pro

Exec traces Exec traces Exec traces GNATcoverage

Executable

decorated sources decorated sources decorated sources

Accurate Source Coverage Info!

Sources Sources Sources Sources GNAT Pro

Exec traces Exec traces Exec traces GNATcoverage

Executable

decorated sources decorated sources decorated sources

Debug info

Not sufficient to locate precise statements, decisions, or conditions boundaries

Accurate Source Coverage Info!

Sources Sources Sources Sources

Exec traces Exec traces Exec traces GNATcoverage

decorated sources decorated sources decorated sources

Source Coverage Information

(Static analysis)

Enhanced GNAT Pro

SCOs

Executable

Debug info

-fpreserve-control-flow

Challenge 2!

OBC does not imply MC/DC!

We need better theoretical foundations !

Initial ideas!

•  General Belief at beginning of project :!

 Object Coverage => Statement Coverage!

 Object Branch Coverage => Decision Coverage!

 Object Branch Coverage => MC/DC (when using short circuit operators)!

•  But a FAA study arrived after the beginning showing unexplainable differences between OBC and MC/DC DOT/FAA/AR-07/17!

Elementary counter-example!

function P (A, B, C : Boolean) return Boolean is begin

if ( A and then B ) or else C then

return True;

end if; end P;

A B C if statement

T T F T

F T F F

F T T T

T F F F

Decision

Conditions

At least n+1 tests n = number of conditions

A

C B

MC / DC

A B C if statement

T T x T

T F T T

F x F F

OBC

3 tests are sufficient

Counter-measures!

•  Definition of a formal model to express coverage metrics based on BDD (Binary Decision Diagram)!

•  Express OBC and MC/DC in this model!

•  Find counter-examples!

•  Find precise perimeter where the equivalence can be proven!

•  Formally prove this result!

Use Open Source

Model Checker Alloy

Evaluation of short circuit boolean expressions!

•  Evaluating a short circuit decision is a traversalof its Reduced Ordered Binary Decision Diagram!

•  Each ROBDD node is a test for a condition!

•  Evaluate conditions left to right!

•  Do not evaluate RHS if LHS is sufficient!

•  A condition vector denotes a path trough the ROBDD!

A

B C

T

T T

F

F

F

T T F

( A and then B ) or else C

More counter-examples (Alloy)!

function P (A, B, C : Boolean) return Boolean is begin

if ( A and then B ) or else C then

return True;

end if; end P;

A

B

C

T

T

T

F

F

F

BDD

function P (C0, C1, C2, C3, C4 … : Boolean) return Boolean is begin

if ((((…(C0 and then C1) or else C2) and then C3) or else C4 …

then

return True;

end if; end P;

C0

T

T

T

F

F

C2

C1

C3

C4

F T F

F

T

F

T

Pathological case!

3 tests sufficient instead of N+1, for any N

(((C0 AND THEN C1) OR ELSE C2) AND THEN C3) OR ELSE C4...

•  N conditions •  MC/DC requires at least N + 1 tests •  OBC can be achieved in 3 tests!

T

T

T

T

T T

T F

F

F

F

F

F

C0

C4

C3

C2

C1

What does that mean?!

•  For a given test campaign!

•  OBC (BDDBC) are local properties of each BDD node: stateless (union of all paths are covering the BDD)!

•  MC/DC is a property of trajectories taken through the ROBDD: stateful (all paths through the BDD are taken)!

•  In general MC/DC requires complete history of each conditional branch instruction (each BDD node)!

•  Are there specific cases where we can do better?!

Equivalence can be proven when !

•  There are no diamonds in the BDD (nodes that can be reached through multiple paths)!

•  How does this translate in “User Terms” ?!

•  No easy formulation… the best we found is!

•  Transform Boolean expression in “Negative Normal Form”!

•  No “and then” in left operand of a “or else”!

•  No “or else” in left operand of a “and then”!

Proof sketch!

•  In the no-diamond case, each path covers a distinct terminal edge ⇒ all terminal edges covered implies all paths covered, MC/DC is achieved!

•  If thereʼs a diamond, we construct a covering path set that fails to show independent influence of one condition (all paths through that condition have the same outcome)!

A

B

C

T

T

T

F

F

F

F

T

Main Results!

•  Emulation is key to Agile cross development!

•  GNATcoverage takes advantage of the theoretical results to:!

•  Implement properly MC/DC in the complex case!

•  Optimize the simple case by using OBC!

•  Definition of specific compilation artefacts (SCOs) and of a certification-friendly code generation mode in GCC (-fpreserve-control-flow)!

•  Creation of “open source” qualification material as part of Open-DO!

Conclusion!

•  The Couverture project allowed us to concentrate on solving properly the unexpected challenges!

•  Existing Open-Source technologies have played a key role:!

•  Qemu is the base of GNATemulator!

•  Alloy helped a lot for the mathematical proofs!

•  As a result, new industrial-ready Open Source tools are now available for the HI developersʼ community !