Verification Planning and Compliance Matrices

Brian SelvyWednesday, August 13, 2014

3:30 – 5:00 p.m.Phoenix West Conference Room

Agenda• LSST V&V Process (LSE-160) review with

respect to Verification Planning and Compliance Matrices

• V&V Implementation methodology in SysML• Overview of Requirements and Verification

Planning Spreadsheets

References / Support Docs

• LSST Verification and Validation Process (LSE-160) – Link to Docushare Released Version

• SPIE SysML Implementation Paper – Link• SPIE SysML Implementation Presentation –

Link• Requirements eaXL spreadsheet template –

Link• Verification Planning eaXL spreadsheet

template - Link

LSE-160 - LSST V&V Process

Verification Planning (Step 3)

• Verification Planning is performed on each requirement to define how each will be verified

• A verification plan consists of:– Verification Owner: LSST subsystem responsible for verification– Verification Method(s): Test, Demonstration, Analysis,

Inspection– Verification Level: verified at Same Level as requirement , or at

a Lower Level or Higher Level– Verification Requirement: Statement that defines what will be

done to verify the requirement. – Success Criteria: statement that defines explicit pass/fail

criteria• Each specific instance of a Verification Method is referred

to as a Verification Activity– An individual requirement’s Verification Plan may have more

than one Verification Activity

Identify Task Interdependency (Step 4) & Schedule Verification Events (Step 5)

• Identify Task Interdependency (Step 4)– Some Verification Activities can be naturally

grouped and conducted at the same time– These Verification Activities are then grouped into

a single Verification Event.• Can result in cost and schedule savings from eliminating

redundant or nearly redundant V&V activities

• Schedule Verification Events (Step 5)– Events are scheduled, identifying predecessor/

successor relationships and other schedule constraints

The Need for Compliance Assessments

• In principle, a system’s physical design can be managed two ways:– Design Definition Documents

• Formal change controlled documents that define the functional and physical definition of a designed system. They document the design response in the synthesis phase to the provided requirements.

• These must be produced and kept up to date for each level of system definition (system, subsystem, element, component, etc.)– Can be cumbersome

– Verification Compliance Matrices• Each system provides updated compliance matrices at regular intervals

(design reviews, periodically scheduled updates, etc).• Design teams write short statements in response to each requirement to

justify if their design complies with each levied requirement or not.• Design teams can reference additional documents with analyses, test results,

design details, etc. that substantiate the compliance statements.• Ties in seamlessly with overall Verification Planning activities (I&T planning

and Commissioning Planning)• This is the approach LSST is implementing.

– Should be less of a burden on teams than the alternative

Extending the SysML Stereotype for Verification Planning

• SysML does not have a predefined element capable of capturing LSST’s Verification Planning information

• SysML is extensible, allowing for the definition of additional stereotypes

• LSST created a VerificationPlanning stereotype as an extension of the SysML1.3::requirement metaclass

Creation of Verification Plans & Test Cases in the Model

Sequencing Test Cases (Verification Events)

• Test Cases (Verification Events) are sequenced on Activity Diagrams to show:– Predecessor/ successor

relationships– Events that are conducted

in parallel/ series– Outside constraints that

must be met before a Verification Event can be executed

• Results can be used to validate or update the project’s schedule for the Commissioning period.

Refinement of Individual Test Cases (Verification Events)

• As plans mature, individual Verification Events can be further detailed via association with its own detailed behavior diagram

• Serves as refined and more detailed input to the commissioning planning effort– Can be used directly as

inputs to writing detailed test & analysis procedures

Mapping Individual Test Case Steps to LSST’s PMCS

• Refined Test Case Actions mapped to associated Project Management Control System (PMCS) activity steps.

• Ensures Verification Activities are included in EVMS

V&V Planning & Compliance Spreadsheets

• If you do not want to work in the EA SysML directly, you can work in Excel.

• LSST has two licenses to an Enterprise Architect add-on called DocX that can be used to create documents from the model and import and export data via Excel.– eaXL is the sub-tool that supports imports and

exports from Excel worksheets• Procedures are available for:– Exporting requirements from specification

packages– Filling in compliance matrices

Extracting Requirements to Excel (1 of 2)

• Click on package that contains the requirements you want to extract

• Click Extensions -> eaDocX -> eaXL -> Create new workbook

Extracting Requirements to Excel (2 of 2)

1. On Elements tab, click Package and Requirement2. On Columns tab, click Stereotype, TV: LSSTRequirements, Name, and Description

– Order as shown– Make sure Output Child Elements is clicked

3. Click “EA “ to export requirements to Excel Workbook4. Click File -> Save As to save the file to a desired location.

1 23

4

Create Verification Package in SysML

• Verification Packages Mirror Requirements Specification and ICD Packages• ICD Packages have subsystem and system level verification packages

– Allows subsystems to specify which verifications they can do at their level with emulators, simulators, mockups, etc.

– Allows PSE to define final interface verification activities during system I&T

Requirements Specifications

ICDs

Create Verification Spreadsheet (1 of 2)

• Click on the Verification Package you previously created• Click Extensions -> eaDocX -> eaXL -> Create new

workbook

Create Verification Spreadsheet (2 of 2)

1. On Elements tab, click Package and Requirement2. On Columns tab, click all of the Tagged Value Types shown

– Order as shown– Make sure Output Child Elements is clicked

3. Click “EA “ to export requirements to Excel Workbook4. Click File -> Save As to save the file to a desired location.

1 23

4

Transfer Relevant Data from Req to Verif SpreadsheetRequirements Spreadsheet

Verification Spreadsheet

• Copy over ElementType and Name in same order to create basic Verification Spreadsheet structure

• Replace “requirement” stereotype with “VerificationPlanning” wherever it occurs

Fill out Verif Planning & Compliance

• Teams can then fill out all of the Verification Planning and Compliance columns in Excel

• When done, please pass the Spreadsheet back to me (Brian Selvy)– I will import back into SysML

• I need to ensure Excel row groupings are correct, etc., before importing• I need to check for compliance with items in enumerated lists• I need to check that spreadsheet metadata is correct• I need to ensure that the package is under Version Control so that roll

backs are possible if issues arise during import

– I will assign unique IDs to all items