www.qinetiq.com © copyright qinetiq limited 2010 formal methods tool qualification for do178b &...

www.QinetiQ.com© Copyright QinetiQ limited 2010

Formal Methods Tool Qualification for DO178B & DO178C

Nick Tudortel: +44 1684 894489email: [email protected]


Agenda

• Tool Overview and Qualification Approach

• The Current Guidance - DO178B

• Qualification Considerations – DO178B

• Applicant’s Claims – DO178B

• Qualification Considerations – DO178C

• Applicant’s Claims – DO178C and Supplements

• Summary


Tool Overview and Qualification Approach


Overview of the tool • The tool is a Formal Methods based software verification tool

−Automates the verification of automatically generated code

−Constraints include a subset of Simulink and a subset of the Ada programming language

• Exploits the Z formal specification language

−Automatic generation of formal specification of the Simulink in Z

−Gives required independence

• Uses Carroll Morgan’s Theory of Refinement, also automatic

• Built in automated use of an off-the-shelf Theorem Prover (ProofPower)

• Having used the tool manually for independent code verification, we aimed to:

−Automatically prove automatically generated code for productivity reasons

−Understand the tool qualification issues


Overview of the CLawZ Process

Z

Discharge proof

Ada

Refinement Script Generator

Supertac

Z Produce

r Compliance Notation Tool

ProofPower

Verification Conditions

B4S

SimulinkDevelopment

VerificationUser Interface


The GUI and some functions (PID example)

Importing Simulink & Ada files & defining units

Creating Z specification, doing refinement

and proof

Creating Interface for each unitDefining variables

Proof results


The ‘pre-qualification’ approach

• As part of the final development steps, we took advice from CAA

• This is new because:

−No-one has previously approached them to do any form of “pre-qualification” of a tool

−NB This does not constitute certification authority pre-approval of this tool

− It’s a formal methods tool

−A formal methods tool which is intended to only automate code verification [and will be qualified as such]

• We needed to examine the guidance from DO178B to ensure that we knew what we were aiming at and to check likely impact of DO178C

−Particularly in light of both the Formal Methods and Tool Qualification Supplements


RTCA DO178C/EUROCAE ED12C

Tool Qualification Supplement

How high is the bar?

RTCA DO178B/EUROCAE ED12B

Minimum

QinetiQ Internal Processes

RTCA DO178B/EUROCAE ED12B

“Plus”


A virtual, parallel approach

Tool Development Theoretical Applicant

DO178B Theoretical DO178C

Includes (theoretically):Core text, Tool Qualification, FM Supplement MBD Supplement

Theoretical System


The Current Guidance - DO178B


Not Shooting for the Minimum

• RTCA DO178B states that for a Verification tool:

− “The qualification criteria for software verification tools should be achieved by demonstration that the tool complies with its Tool Operational Requirements under normal operational conditions”

− Production of Tool Operational Requirements, the Verification Plan, the Verification Results and the Tool Accomplishment Summary

• NB FAA-8110-49 says you could also have a TQP and TAS, but specifically:

− “Tool Operational Requirements satisfies the same objectives as the Software Requirements Data of the airborne software”. …ie (from 12.2.3.2):

− “Tool Operational Requirements describe the tool's operational functionality. This data should include:

− a. A description of the tool's functions and technical features. [For software development tools, it includes the software development process activities performed by the tool].

− b. User information, such as installation guides and user manuals.

− c. A description of the tool's operational environment.

− d. [For software development tools, the expected responses of the tool under abnormal operating conditions.] ”


Adapted from FAA-8110-49 Fig 9-2

Data Applicability Available/ Submit/Note RTCA/DO-178B Ref.

Plan for Software Aspects of Certification (PSAC)

Verification & Development Submit 12.2, 12.2.3.a, & 12.2.4

Tool Qualification Plan Development Submit 12.2.3.a(1), 12.2.3.1, & 12.2.4

Tool Operational Requirements

Verification & Development Available 12.2.3.c(2) & 12.2.3.2

Software Accomplishment Summary (SAS)

Verification & Development Submit 12.2.4

Tool Accomplishment Summary

Development Submit 12.2.3.c(3) & 12.2.4

Tool Verification Records (for example, test cases, procedures, and results)

Verification & Development Available 12.2.3

Tool Qualification Development data (for example, requirements, design, and code)

Development Available 12.2.3


Qualification considerations – DO178B


An Applicant’s perspective

• We needed to consider how an Applicant would use the tool and subsequently be able to claim credit

− Despite confused guidance, what artefacts would need to be made available

− Given that this is a tool based upon FM, what “language” to use (this is non-trivial!)

• We needed to be cognisant of context, ie the overall software development life cycle

• We have defined CLawZ Simulink Subset (CSS) used for Independent Verification and used a well known subset of code (Ada)

− This constrains the possible input space to something that is well defined, but also very flexible

− Also constrained the output space (code)

• By doing this we set the Tool Operational Requirements and hence the scope of the qualification


Proposed approachHistor

y

Upgrade/production

of Plans

Configuration Control

Tool Accomplishment

Summary

Tool Qualification

Plan

Applicants PSAC

Verification Plan

Verification Results

Tool Operational

Requirements

Applicants SAS

Configuration Index

Verification Cases and Procedures

QA and Technical Audit

Available information includes:Development PlanConfiguration Management PlanQA Plan/records (not for 178B)Design DescriptionConfiguration RecordsDevelopment Environment IndexUser Guide


Applicant’s Claims – DO178B


Verification Objectives – DO178B

High-LevelRequirements

(A-2: 3, 4, 5)

ExecutableObject Code

Source Code

(A-2: 7)

(A-2: 6)

SystemRequirements

(A-2: 1, 2)

A-6.1 Compliance A-6.2 Robustness

A-5. 7 Complete & Correct

A-3.1 Compliance A-3.6 Traceability

A-3.2 Accuracy & Consistency A-3.3 HW Compatibility A-3.4 VerifiabilityA-3.5 Conformance A-3.7 Algorithm Accuracy

A-4. 8 Architecture Compatibility A-4.1 Compliance A-4.6 Traceability

A-4.9 ConsistencyA-4.10 HW Compatibility A-4.11 Verifiability A-4.12 Conformance A-4.13 Partition Integrity

SoftwareArchitecture

Low-LevelRequirements

A-4.2 Accuracy & ConsistencyA-4.3 HW Compatibility A-4.4 Verifiability A-4.5 Conformance A-4.7 Algorithm Accuracy

A-5.2 Compliance A-5.1 Compliance A-5.5 Traceability

A-5.3 Verifiability A-5.4 ConformanceA-5.6 Accuracy & Consistency

A-6.3 Compliance A-6.4 Robustness

A-6.5 Compatible With TargetCompliance: with requirementsConformance: with standards

NB Verification of the Verification,QA, DER liaison and other documentation alsorequired


Claims relating to source code verification Table A-5

Objective Applicability bySoftware Level

Description Ref. A B C D

1 Source Code complies with low-level requirements.

6.3.4a

2 Source Code complies with software architecture.

6.3.4b

3 Source Code is verifiable. 6.3.4c

4 Source Code conforms to standards. 6.3.4d

5 Source Code is traceable to low-level requirements.

6.3.4e

6 Source Code is accurate and consistent.

6.3.4f

7 Output of software integration process is complete and correct.

6.3.5

Partial

We can show that eg there are no uninitialised or unused variables or constants. There are no claims re stack usage, resource contention, WCET, etc


Verification ObjectivesSystem

Requirements

High-LevelRequirements

Source Code

ExecutableObject Code

(A-2: 3, 4, 5)

(A-2: 7)

(A-2: 6)

(A-2: 1, 2) Simulink [Continuous]

Simulink [Discrete]

Simulink for CLawZ

Autocode, proof

Major Contribution

SoftwareArchitecture

Low-LevelRequirements

A-5.3 Verifiability A-5.4 ConformanceA-5.6 Accuracy & Consistency

A-5.1 Compliance A-5.5 Traceability

A-5.2 Compliance

A-5. 7 Complete & Correct


Qualification considerations – DO178C


DO178C introduces 3 categories of tools (see section12.2.2)

Development tool Criteria 1

Criteria 3Verification tool

DO178B DO178C

DAL Criteria 1

Criteria 2

Criteria 3

A TQL1 TQL4 TQL5

B TQL2 TQL4 TQL5

C TQL3 TQL5 TQL5

D TQL4 TQL5 TQL5

Criteria 2?

Criteria 1: A tool whose output is part of the airborne software and thus could insert an error.

Criteria 3: A tool that, within the scope of its intended use, could fail to detect an error.

Criteria 2: A tool that automates verification process(es) and thus could fail to detect an error, and whose output is used to justify the elimination or reduction of:1. Verification process(es) other than that automated by the tool, or2. Development process(es) that could have an impact on the airborne software.


DO178C Tool Qualification Supplement

• We wanted to know what we might be under ED12C/DO178C we we’re either

−A verification tool (Criteria 3) and hence TQL5 would be required, or

−A super verification tool (Criteria 2) and hence TQL5 or even TQL4

• We believe CLawZ would be a TQL5 tool, but this depends largely on what the Applicant will wish claim as a result of using the tool

• And whether this is acceptable for the certifier

• More on this at the end…

• We will also examine TQL4 …just in case!


• We needed to keep and eye on what TQL4 required at the same time

• Referring to Table T-1 and T2, there is considerable extra material required over DO178B

• No requirement for Tool Operational Requirements, but is implicit

• Interesting that QA Plan appears to be required, but not QA records

• Also, that Tool Plans do not have to comply with the Supplement (T-1-6)

DO178C TQL 4 – Tables T-1 & T-2

Document 178B TQL4 Note

The Tool Qualification Plan (TQP) Y Y

Tool Development Plan (TDP) N Y Not strictly needed, but certifier will look for it

Tool Verification Plan (TVP) Y Y Not entirely clear if this is needed, but…

Tool Configuration Management Plan (TCMP)

N Y Not entirely clear if this is needed, but the Configuration Index is required

Tool Quality Assurance Plan (TQAP) N Y Not needed, but needed for eg TickIT?


TQL4 – DO178C – Tables T-3, T-4 and T-5

• There is large concentration of effort in Table T-3 (requirements) which is not “needed” under DO178B

− It is implied that the Tool Operational Requirements are the “requirements”.

• Suspect that in practice, for all but trivial tools under qualified under DO178B, there is a more clearly defined set of requirements contained within the TOR

• It was not immediately clear why the only requirement for Table T4 is T-4-10 (protection mechanisms). Sought guidance from FAQ #2 Tool Qualification Supplement :

−Seeks to show evidence for separation of functions, essentially through architecture – fine, but…

−There are no other architecture requirements for TQL 4 (ie compatible with requirements, consistency, standards), so does this objective help safety?

• No objectives for Table T-5 verification of the coding process…


TQL4 - Tables T-4 and T-5 - A closer examination

• From DP #2: Rationale for Tool Qualification Criteria Definition, para 2.2.3.2 Rationale for introducing Criteria 2

“These additional uses of tools raise some concerns about the rigor required for qualification. For example:

− From a safety perspective, the impact of these tools on the final software can vary.

− The required confidence may not be able to be assessed by considering only the Tool Operational Requirements.

− The maturity and soundness of a mathematical theory and its implementation may be necessary to provide the required confidence

• The major concern would appear to be the implementation, so no examination of low level requirements or the source code?

• Reliance is therefore left to Table T-6…

• NB No objectives are required to be met for TQL5


TQL4 – ED12C/DO178C – Tables T-6, T-7, T-8 and T-9

• Focus here is on how the tool functions when actually undertaking verification

• Examines object code with respect to the Tool Requirements

−Checks compliance and robustness

• Also looks at verification results

−Checks that results were correct/discrepancies explained and

−Coverage of Tool Requirements

• Ensures that the configuration was identified


Applicant’s Claims – DO178C and Supplements


Possible claims relating to source code verification – DO178C FM Supplement

• Specific claims for formal verification could be made in accordance with the Formal Methods Supplement (see table FM A-5)

−Specific claims can be made FM A5-1 to 6 (FM7 refers to core text as above)

− Also need to examine possible to claims to Extra Objectives FM8-11

Objective Applicability bySW Level

Description Ref. A B C D

FM 8 Formal analysis cases and procedures are correct.

FM.6.3.6aFM.6.3.6b

FM 9 Formal analysis results are correct and discrepancies explained.

FM.6.3.6c

FM 10 Requirements formalization is correct.

FM.6.3i

FM 11 Formal method is correctly defined and sound.

FM.6.2


Perspective of OTS tools for DO178C vs in-house tool development for DO178B

• Number of views from Tool Qualification Supplement

−Re-use – see section 11.2

−OTS - see section 11.3

−Service History – see section 11.4 (not applicable)

−Alternative methods – see section 11.5 (not applicable)

• Section 11.3.2 shows activities required by the tool developer

−Which are to produce: TOR, TQP, TCI, TAS

−Also outlines which objectives are modified

• Section 11.3.3 shows activities for the tool user

−To assess the TOR, TQP, TCI and TAS and plan extra activities such as division of responsibilities


Summary

• We believe we can support an Applicant in system certification to DO178B

• We have gone further than was required because this tool was “pre-developed” and is a formal methods tool

• We believe that it would be a TQL5 tool under DO178C Tool Qualification Supplement, but…

• Could be TQL4, depending on development processes of the Applicant

• Interesting that our approach has mirrored the COTS advice to tool qualification

• So does TQL4 achieve anything extra in terms of system safety?

www.qinetiq.com © copyright qinetiq limited 2010 formal methods tool qualification for do178b &...

Documents

copyright qinetiq

tool qualification supplements

tool qualification issues

tool nb

qualification approach

software verification

agenda tool overview

qualification crit