formal methods tool qualification for do178b & do178c
DESCRIPTION
Formal Methods Tool Qualification for DO178B & DO178C. Nick Tudor tel: +44 1684 894489 email: [email protected]. Agenda. Tool Overview and Qualification Approach The Current Guidance - DO178B Qualification Considerations – DO178B Applicant’s Claims – DO178B - PowerPoint PPT PresentationTRANSCRIPT
www.QinetiQ.com© Copyright QinetiQ limited 2010
Formal Methods Tool Qualification for DO178B & DO178C
Nick Tudortel: +44 1684 894489email: [email protected]
www.QinetiQ.com© Copyright QinetiQ limited 2010
Agenda
• Tool Overview and Qualification Approach
• The Current Guidance - DO178B
• Qualification Considerations – DO178B
• Applicant’s Claims – DO178B
• Qualification Considerations – DO178C
• Applicant’s Claims – DO178C and Supplements
• Summary
www.QinetiQ.com© Copyright QinetiQ limited 2010
Tool Overview and Qualification Approach
www.QinetiQ.com© Copyright QinetiQ limited 2010
Overview of the tool • The tool is a Formal Methods based software verification tool
−Automates the verification of automatically generated code
−Constraints include a subset of Simulink and a subset of the Ada programming language
• Exploits the Z formal specification language−Automatic generation of formal specification of the Simulink in Z
−Gives required independence
• Uses Carroll Morgan’s Theory of Refinement, also automatic
• Built in automated use of an off-the-shelf Theorem Prover (ProofPower)
• Having used the tool manually for independent code verification, we aimed to:−Automatically prove automatically generated code for productivity reasons
−Understand the tool qualification issues
www.QinetiQ.com© Copyright QinetiQ limited 2010
Overview of the CLawZ Process
Z
Discharge proof
Ada
Refinement Script Generator
Supertac
Z Produce
r Compliance Notation Tool
ProofPower
Verification Conditions
B4SSimulink
Development
VerificationUser Interface
www.QinetiQ.com© Copyright QinetiQ limited 2010
The GUI and some functions (PID example)
Importing Simulink & Ada files & defining units
Creating Z specification, doing refinement
and proof
Creating Interface for each unitDefining variables
Proof results
www.QinetiQ.com© Copyright QinetiQ limited 2010
The ‘pre-qualification’ approach• As part of the final development steps, we took advice from CAA
• This is new because:
−No-one has previously approached them to do any form of “pre-qualification” of a tool
−NB This does not constitute certification authority pre-approval of this tool
− It’s a formal methods tool
−A formal methods tool which is intended to only automate code verification [and will be qualified as such]
• We needed to examine the guidance from DO178B to ensure that we knew what we were aiming at and to check likely impact of DO178C
−Particularly in light of both the Formal Methods and Tool Qualification Supplements
www.QinetiQ.com© Copyright QinetiQ limited 2010
RTCA DO178C/EUROCAE ED12C
Tool Qualification Supplement
How high is the bar?
RTCA DO178B/EUROCAE ED12B
Minimum
QinetiQ Internal Processes
RTCA DO178B/EUROCAE ED12B
“Plus”
www.QinetiQ.com© Copyright QinetiQ limited 2010
A virtual, parallel approach
Tool Development Theoretical Applicant
DO178B Theoretical DO178C
Includes (theoretically):Core text, Tool Qualification, FM Supplement MBD Supplement
Theoretical System
www.QinetiQ.com© Copyright QinetiQ limited 2010
The Current Guidance - DO178B
www.QinetiQ.com© Copyright QinetiQ limited 2010
Not Shooting for the Minimum
• RTCA DO178B states that for a Verification tool:− “The qualification criteria for software verification tools should be achieved by demonstration
that the tool complies with its Tool Operational Requirements under normal operational conditions”
− Production of Tool Operational Requirements, the Verification Plan, the Verification Results and the Tool Accomplishment Summary
• NB FAA-8110-49 says you could also have a TQP and TAS, but specifically:− “Tool Operational Requirements satisfies the same objectives as the Software Requirements
Data of the airborne software”. …ie (from 12.2.3.2):− “Tool Operational Requirements describe the tool's operational functionality. This data should
include:− a. A description of the tool's functions and technical features. [For software
development tools, it includes the software development process activities performed by the tool].
− b. User information, such as installation guides and user manuals.− c. A description of the tool's operational environment.− d. [For software development tools, the expected responses of the tool under abnormal
operating conditions.] ”
www.QinetiQ.com© Copyright QinetiQ limited 2010
Adapted from FAA-8110-49 Fig 9-2
Data Applicability Available/ Submit/Note RTCA/DO-178B Ref.
Plan for Software Aspects of Certification (PSAC)
Verification & Development Submit 12.2, 12.2.3.a, & 12.2.4
Tool Qualification Plan Development Submit 12.2.3.a(1), 12.2.3.1, & 12.2.4
Tool Operational Requirements
Verification & Development Available 12.2.3.c(2) & 12.2.3.2
Software Accomplishment Summary (SAS)
Verification & Development Submit 12.2.4
Tool Accomplishment Summary
Development Submit 12.2.3.c(3) & 12.2.4
Tool Verification Records (for example, test cases, procedures, and results)
Verification & Development Available 12.2.3
Tool Qualification Development data (for example, requirements, design, and code)
Development Available 12.2.3
www.QinetiQ.com© Copyright QinetiQ limited 2010
Qualification considerations – DO178B
www.QinetiQ.com© Copyright QinetiQ limited 2010
An Applicant’s perspective
• We needed to consider how an Applicant would use the tool and subsequently be able to claim credit
− Despite confused guidance, what artefacts would need to be made available
− Given that this is a tool based upon FM, what “language” to use (this is non-trivial!)
• We needed to be cognisant of context, ie the overall software development life cycle
• We have defined CLawZ Simulink Subset (CSS) used for Independent Verification and used a well known subset of code (Ada)
− This constrains the possible input space to something that is well defined, but also very flexible
− Also constrained the output space (code)
• By doing this we set the Tool Operational Requirements and hence the scope of the qualification
www.QinetiQ.com© Copyright QinetiQ limited 2010
Proposed approachHistor
y
Upgrade/production
of Plans
Configuration Control
Tool Accomplishment
Summary
Tool Qualification
Plan
Applicants PSAC
Verification Plan
Verification Results
Tool Operational
Requirements
Applicants SAS
Configuration Index
Verification Cases and Procedures
QA and Technical Audit
Available information includes:Development PlanConfiguration Management PlanQA Plan/records (not for 178B)Design DescriptionConfiguration RecordsDevelopment Environment IndexUser Guide
www.QinetiQ.com© Copyright QinetiQ limited 2010
Applicant’s Claims – DO178B
www.QinetiQ.com© Copyright QinetiQ limited 2010
Verification Objectives – DO178B
High-LevelRequirements
(A-2: 3, 4, 5)
ExecutableObject Code
Source Code
(A-2: 7)
(A-2: 6)
SystemRequirements
(A-2: 1, 2)
A-6.1 Compliance A-6.2 Robustness
A-5. 7 Complete & Correct
A-3.1 Compliance A-3.6 Traceability
A-3.2 Accuracy & Consistency A-3.3 HW Compatibility A-3.4 VerifiabilityA-3.5 Conformance A-3.7 Algorithm Accuracy
A-4. 8 Architecture Compatibility A-4.1 Compliance A-4.6 Traceability
A-4.9 ConsistencyA-4.10 HW Compatibility A-4.11 Verifiability A-4.12 Conformance A-4.13 Partition Integrity
SoftwareArchitecture
Low-LevelRequirements
A-4.2 Accuracy & ConsistencyA-4.3 HW Compatibility A-4.4 Verifiability A-4.5 Conformance A-4.7 Algorithm Accuracy
A-5.2 Compliance A-5.1 Compliance A-5.5 Traceability
A-5.3 Verifiability A-5.4 ConformanceA-5.6 Accuracy & Consistency
A-6.3 Compliance A-6.4 Robustness
A-6.5 Compatible With TargetCompliance: with requirementsConformance: with standards
NB Verification of the Verification,QA, DER liaison and other documentation alsorequired
www.QinetiQ.com© Copyright QinetiQ limited 2010
Claims relating to source code verification Table A-5
Objective Applicability bySoftware Level
Description Ref. A B C D1 Source Code complies with low-level
requirements.6.3.4a
2 Source Code complies with software architecture.
6.3.4b
3 Source Code is verifiable. 6.3.4c
4 Source Code conforms to standards. 6.3.4d
5 Source Code is traceable to low-level requirements.
6.3.4e
6 Source Code is accurate and consistent.
6.3.4f
7 Output of software integration process is complete and correct.
6.3.5
Partial
We can show that eg there are no uninitialised or unused variables or constants. There are no claims re stack usage, resource contention, WCET, etc
www.QinetiQ.com© Copyright QinetiQ limited 2010
Verification ObjectivesSystem
Requirements
High-LevelRequirements
Source Code
ExecutableObject Code
(A-2: 3, 4, 5)
(A-2: 7)
(A-2: 6)
(A-2: 1, 2) Simulink [Continuous]
Simulink [Discrete]
Simulink for CLawZ
Autocode, proof
Major Contribution
SoftwareArchitecture
Low-LevelRequirements
A-5.3 Verifiability A-5.4 ConformanceA-5.6 Accuracy & Consistency
A-5.1 Compliance A-5.5 Traceability
A-5.2 Compliance
A-5. 7 Complete & Correct
www.QinetiQ.com© Copyright QinetiQ limited 2010
Qualification considerations – DO178C
www.QinetiQ.com© Copyright QinetiQ limited 2010
DO178C introduces 3 categories of tools (see section12.2.2)
Development tool Criteria 1
Criteria 3Verification tool
DO178B DO178CDAL Criteria
1Criteria
2Criteria
3A TQL1 TQL4 TQL5B TQL2 TQL4 TQL5C TQL3 TQL5 TQL5D TQL4 TQL5 TQL5
Criteria 2?
Criteria 1: A tool whose output is part of the airborne software and thus could insert an error.Criteria 3: A tool that, within the scope of its intended use, could fail to detect an error.
Criteria 2: A tool that automates verification process(es) and thus could fail to detect an error, and whose output is used to justify the elimination or reduction of:1. Verification process(es) other than that automated by the tool, or2. Development process(es) that could have an impact on the airborne software.
www.QinetiQ.com© Copyright QinetiQ limited 2010
DO178C Tool Qualification Supplement
• We wanted to know what we might be under ED12C/DO178C we we’re either
−A verification tool (Criteria 3) and hence TQL5 would be required, or
−A super verification tool (Criteria 2) and hence TQL5 or even TQL4
• We believe CLawZ would be a TQL5 tool, but this depends largely on what the Applicant will wish claim as a result of using the tool
• And whether this is acceptable for the certifier
• More on this at the end…
• We will also examine TQL4 …just in case!
www.QinetiQ.com© Copyright QinetiQ limited 2010
• We needed to keep and eye on what TQL4 required at the same time
• Referring to Table T-1 and T2, there is considerable extra material required over DO178B
• No requirement for Tool Operational Requirements, but is implicit
• Interesting that QA Plan appears to be required, but not QA records
• Also, that Tool Plans do not have to comply with the Supplement (T-1-6)
DO178C TQL 4 – Tables T-1 & T-2
Document 178B TQL4 Note
The Tool Qualification Plan (TQP) Y Y
Tool Development Plan (TDP) N Y Not strictly needed, but certifier will look for it
Tool Verification Plan (TVP) Y Y Not entirely clear if this is needed, but…
Tool Configuration Management Plan (TCMP)
N Y Not entirely clear if this is needed, but the Configuration Index is required
Tool Quality Assurance Plan (TQAP) N Y Not needed, but needed for eg TickIT?
www.QinetiQ.com© Copyright QinetiQ limited 2010
TQL4 – DO178C – Tables T-3, T-4 and T-5
• There is large concentration of effort in Table T-3 (requirements) which is not “needed” under DO178B− It is implied that the Tool Operational Requirements are the “requirements”.
• Suspect that in practice, for all but trivial tools under qualified under DO178B, there is a more clearly defined set of requirements contained within the TOR
• It was not immediately clear why the only requirement for Table T4 is T-4-10 (protection mechanisms). Sought guidance from FAQ #2 Tool Qualification Supplement :−Seeks to show evidence for separation of functions, essentially through
architecture – fine, but…
−There are no other architecture requirements for TQL 4 (ie compatible with requirements, consistency, standards), so does this objective help safety?
• No objectives for Table T-5 verification of the coding process…
www.QinetiQ.com© Copyright QinetiQ limited 2010
TQL4 - Tables T-4 and T-5 - A closer examination
• From DP #2: Rationale for Tool Qualification Criteria Definition, para 2.2.3.2 Rationale for introducing Criteria 2“These additional uses of tools raise some concerns about the rigor required for qualification. For example:
− From a safety perspective, the impact of these tools on the final software can vary.
− The required confidence may not be able to be assessed by considering only the Tool Operational Requirements.
− The maturity and soundness of a mathematical theory and its implementation may be necessary to provide the required confidence
• The major concern would appear to be the implementation, so no examination of low level requirements or the source code?
• Reliance is therefore left to Table T-6…• NB No objectives are required to be met for TQL5
www.QinetiQ.com© Copyright QinetiQ limited 2010
TQL4 – ED12C/DO178C – Tables T-6, T-7, T-8 and T-9
• Focus here is on how the tool functions when actually undertaking verification
• Examines object code with respect to the Tool Requirements
−Checks compliance and robustness
• Also looks at verification results
−Checks that results were correct/discrepancies explained and
−Coverage of Tool Requirements
• Ensures that the configuration was identified
www.QinetiQ.com© Copyright QinetiQ limited 2010
Applicant’s Claims – DO178C and Supplements
www.QinetiQ.com© Copyright QinetiQ limited 2010
Possible claims relating to source code verification – DO178C FM Supplement
• Specific claims for formal verification could be made in accordance with the Formal Methods Supplement (see table FM A-5)
−Specific claims can be made FM A5-1 to 6 (FM7 refers to core text as above)
− Also need to examine possible to claims to Extra Objectives FM8-11
Objective Applicability bySW Level
Description Ref. A B C DFM 8 Formal analysis cases and
procedures are correct.FM.6.3.6aFM.6.3.6b
FM 9 Formal analysis results are correct and discrepancies explained.
FM.6.3.6c
FM 10 Requirements formalization is correct.
FM.6.3i
FM 11 Formal method is correctly defined and sound.
FM.6.2
www.QinetiQ.com© Copyright QinetiQ limited 2010
Perspective of OTS tools for DO178C vs in-house tool development for DO178B
• Number of views from Tool Qualification Supplement
−Re-use – see section 11.2
−OTS - see section 11.3
−Service History – see section 11.4 (not applicable)
−Alternative methods – see section 11.5 (not applicable)
• Section 11.3.2 shows activities required by the tool developer
−Which are to produce: TOR, TQP, TCI, TAS
−Also outlines which objectives are modified
• Section 11.3.3 shows activities for the tool user
−To assess the TOR, TQP, TCI and TAS and plan extra activities such as division of responsibilities
www.QinetiQ.com© Copyright QinetiQ limited 2010
Summary
• We believe we can support an Applicant in system certification to DO178B
• We have gone further than was required because this tool was “pre-developed” and is a formal methods tool
• We believe that it would be a TQL5 tool under DO178C Tool Qualification Supplement, but…
• Could be TQL4, depending on development processes of the Applicant
• Interesting that our approach has mirrored the COTS advice to tool qualification
• So does TQL4 achieve anything extra in terms of system safety?