coverage analysis from execution traces
DESCRIPTION
Theoretical foundation of source coverage analysis from execution traces!TRANSCRIPT
![Page 1: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/1.jpg)
Theoretical foundation of source coverage analysis from execution traces!
Thomas Quinot!
![Page 2: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/2.jpg)
Summary!
• Original Needs & Goals!
• Challenges along the way!
• Main Results!
![Page 3: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/3.jpg)
Original Needs!
• Structural Coverage Analysis is required by certification standards:!• Open source Coverage Tools exist but are not usable in a HI context!
• Proprietary Tools exist but do not support all versions of Ada!
• Complete the GNAT Pro Toolset for the High Integrity Market!
• Better support for the rapidly evolving versions of Ada (83 … 95 … 2005 … 2012 …)!
![Page 4: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/4.jpg)
Original Goals!
• Provide an High Quality Open Source alternative to existing proprietary tools!
• Provide Support for Agile/Lean Development!• In particular: Continuous Integration/Certification!
• Open-DO initiative!
• Find the best compromise between Source and Object Coverage!
![Page 5: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/5.jpg)
The Couverture Project (2008-2010)!
• One of the first FUI projects from the GTLL at System@atic!
• 4 partners (AdaCore, Openwide, Telecom PT, Paris 6)!
• Effort of 160 man-month (2,23 M€) over 2 years
• 45% Financed by the city of Paris, IdF region, DGE !
This project gave us the capability to meet the unexpected challenges we were facing.
![Page 6: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/6.jpg)
Object Coverage vs Source Coverage!
• Big debate in the Certification Community!
• Which one is the most Accurate / Appropriate ?!
• Which one is the most efficient ?!
Source
- Statement/Decision are source concepts
- usually works by instrumenting the code
- can be done on fast native platforms
- requires double testing strategy
Object
- on final code (no instrumentation)
- on final hardware
- not language specific
- more precise
![Page 7: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/7.jpg)
Object Coverage vs Source Coverage!
• Object coverage metrics:!
• Instruction Coverage!
• Object Branch Coverage (OBC)!
• Source coverage metrics:!
• Statement Coverage!
• Decision Coverage (DC)!
• Modified Condition/Decision Coverage (MC/DC)Independent influence of each condition within a decision!
![Page 8: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/8.jpg)
Challenge 1!
!It is difficult to provide accurate source coverage info from execution traces:!
! !- no trace of “statement” / “condition” / “decision” at ! binary level!
! !- optimization can change significantly the control flow
![Page 9: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/9.jpg)
Accurate Source Coverage Info!
Sources Sources Sources Sources GNAT Pro
Exec traces Exec traces Exec traces GNATcoverage
Executable
decorated sources decorated sources decorated sources
![Page 10: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/10.jpg)
Accurate Source Coverage Info!
Sources Sources Sources Sources GNAT Pro
Exec traces Exec traces Exec traces GNATcoverage
Executable
decorated sources decorated sources decorated sources
Debug info
Not sufficient to locate precise statements, decisions, or conditions boundaries
![Page 11: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/11.jpg)
Accurate Source Coverage Info!
Sources Sources Sources Sources
Exec traces Exec traces Exec traces GNATcoverage
decorated sources decorated sources decorated sources
Source Coverage Information
(Static analysis)
Enhanced GNAT Pro
SCOs
Executable
Debug info
-fpreserve-control-flow
![Page 12: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/12.jpg)
Challenge 2!
OBC does not imply MC/DC!
We need better theoretical foundations !
![Page 13: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/13.jpg)
Initial ideas!
• General Belief at beginning of project :!
Object Coverage => Statement Coverage!
Object Branch Coverage => Decision Coverage!
Object Branch Coverage => MC/DC (when using short circuit operators)!
• But a FAA study arrived after the beginning showing unexplainable differences between OBC and MC/DC DOT/FAA/AR-07/17!
![Page 14: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/14.jpg)
Elementary counter-example!
function P (A, B, C : Boolean) return Boolean is begin
if ( A and then B ) or else C then
return True;
end if; end P;
A B C if statement
T T F T
F T F F
F T T T
T F F F
Decision
Conditions
At least n+1 tests n = number of conditions
A
C B
MC / DC
A B C if statement
T T x T
T F T T
F x F F
OBC
3 tests are sufficient
![Page 15: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/15.jpg)
Counter-measures!
• Definition of a formal model to express coverage metrics based on BDD (Binary Decision Diagram)!
• Express OBC and MC/DC in this model!
• Find counter-examples!
• Find precise perimeter where the equivalence can be proven!
• Formally prove this result!
Use Open Source
Model Checker Alloy
![Page 16: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/16.jpg)
Evaluation of short circuit boolean expressions!
• Evaluating a short circuit decision is a traversalof its Reduced Ordered Binary Decision Diagram!
• Each ROBDD node is a test for a condition!
• Evaluate conditions left to right!
• Do not evaluate RHS if LHS is sufficient!
• A condition vector denotes a path trough the ROBDD!
A
B C
T
T T
F
F
F
T T F
( A and then B ) or else C
![Page 17: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/17.jpg)
More counter-examples (Alloy)!
function P (A, B, C : Boolean) return Boolean is begin
if ( A and then B ) or else C then
return True;
end if; end P;
A
B
C
T
T
T
F
F
F
BDD
function P (C0, C1, C2, C3, C4 … : Boolean) return Boolean is begin
if ((((…(C0 and then C1) or else C2) and then C3) or else C4 …
then
return True;
end if; end P;
C0
T
T
T
F
F
C2
C1
C3
C4
F T F
F
T
F
T
![Page 18: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/18.jpg)
Pathological case!
3 tests sufficient instead of N+1, for any N
(((C0 AND THEN C1) OR ELSE C2) AND THEN C3) OR ELSE C4...
• N conditions • MC/DC requires at least N + 1 tests • OBC can be achieved in 3 tests!
T
T
T
T
T T
T F
F
F
F
F
F
C0
C4
C3
C2
C1
![Page 19: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/19.jpg)
What does that mean?!
• For a given test campaign!
• OBC (BDDBC) are local properties of each BDD node: stateless (union of all paths are covering the BDD)!
• MC/DC is a property of trajectories taken through the ROBDD: stateful (all paths through the BDD are taken)!
• In general MC/DC requires complete history of each conditional branch instruction (each BDD node)!
• Are there specific cases where we can do better?!
![Page 20: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/20.jpg)
Equivalence can be proven when !
• There are no diamonds in the BDD (nodes that can be reached through multiple paths)!
• How does this translate in “User Terms” ?!
• No easy formulation… the best we found is!
• Transform Boolean expression in “Negative Normal Form”!
• No “and then” in left operand of a “or else”!
• No “or else” in left operand of a “and then”!
![Page 21: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/21.jpg)
Proof sketch!
• In the no-diamond case, each path covers a distinct terminal edge ⇒ all terminal edges covered implies all paths covered, MC/DC is achieved!
• If thereʼs a diamond, we construct a covering path set that fails to show independent influence of one condition (all paths through that condition have the same outcome)!
A
B
C
T
T
T
F
F
F
F
T
![Page 22: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/22.jpg)
Main Results!
• Emulation is key to Agile cross development!
• GNATcoverage takes advantage of the theoretical results to:!
• Implement properly MC/DC in the complex case!
• Optimize the simple case by using OBC!
• Definition of specific compilation artefacts (SCOs) and of a certification-friendly code generation mode in GCC (-fpreserve-control-flow)!
• Creation of “open source” qualification material as part of Open-DO!
![Page 23: Coverage analysis from execution traces](https://reader034.vdocuments.mx/reader034/viewer/2022051609/547eaa4a5906b5dd718b46b5/html5/thumbnails/23.jpg)
Conclusion!
• The Couverture project allowed us to concentrate on solving properly the unexpected challenges!
• Existing Open-Source technologies have played a key role:!
• Qemu is the base of GNATemulator!
• Alloy helped a lot for the mathematical proofs!
• As a result, new industrial-ready Open Source tools are now available for the HI developersʼ community !