Verification and Validation on LSST

Brian Selvy

LSST All-Hands MeetingBremerton

August 17, 2015

LSST Systems Engineering 2

Agenda

8/17/2015

• Project Systems Engineering’s Responsibilities• Review V&V Process

– Verification Plan Requirements– Compliance Assessment Requirements– Final Verification Matrix Requirements

• Verification as Part of AIV• Non-Conformances and the Relationship between V&V and QA• PSE & Subsystem Collaboration on Verification

PSE’s Verification Responsibilities

• Project Systems Engineering is responsible for the overall, system-level integration, verification, and validation of the LSST observatory. PSE has responsibility for the Commissioning phase of the LSST project.

• Verification is one critical aspect of the broader manufacturing, assembly, integration, and verification (AIV) set of activities

• In order for PSE to properly plan and understand the inputs to the early integration, integration, and commissioning phases, PSE needs to understand the activities being conducted by the subsystems that directly impact system level requirements, interfaces, assemblies, and verification activities.

– This leads to the need for collaboration and input from the subsystem teams on developing and reviewing verification plans, observing verification activities, reviewing verification results, and collaborating on the response to deviations/waivers for NCRs affecting project-controlled requirements (via the CCB process)

LSST Systems Engineering 4

Verification & Validation on LSST

8/18/2015

• LSE-160 Verification and Validation Process is the governing document for V&V on LSST

– Establishes a consistent, project-wide process for the development of V&V plans, compliance assessments, V&V reporting, and deliverables

• LSE-160 is currently undergoing a revision to:

– Harmonize with LPM-55 LSST QA Plan (also in revision)

• Define interactions and dependencies between V&V and QA

– Further define the Assembly, Integration, and Verification (AIV) Framework during Early I&T and Commissioning

– Document the SysML Implementation methodologies for bottoms up and top down modeling

LSE-160 Applicability

• Applies to all Project-Controlled requirements:– Specifications– Requirements Documents– Interface Control Documents (ICDs)

• Each “shall” statement in each of these documents must be formally verified

Project-Controlled Specifications Project-Controlled ICDs

LSST Verification & Validation Process

Verification Planning Requirements

• For each requirement (“shall statement”) a Verification Plan will be created that includes the following:

– Verification Owner – the subsystem team that is responsible for verification– Responsible Technical Authority (RTA) – the point-of-contact assigned

responsibility for the verification of the requirement from the responsible subsystem. The RTA, along with the responsible QA individual, has responsibility for overseeing all associated verification events.

– Verification Method(s) – Test, Analysis, Inspection, Demonstration– Verification Level – Same Level, Higher Level, Lower Level– Verification Requirement – A statement that defines precisely what will be

done to verify the requirement. If there is any vagueness in the requirement, the Verification Requirement should clearly address the noted issues and define what precisely will be verified and any limitations. The statement should define what will be done, where it will be done, what special test equipment (SPE) is needed, and what project hardware/software is needed.

– Success Criteria - A statement that defines the explicit pass/fail criteria. This statement should be clear enough that an independent third party observer should be able to determine if the verification event was successful or not.

Verification Planning Requirements (Cont)

• Documented in a Verification Matrix for Planning (VM-P)– One required for each Specification– One required for each subsystem involved in an ICD

Compliance Assessment Requirements

• Compliance is defined as the ability of the current (any point in time) “as-designed” system to meet its associated requirements.

• The difference between compliance and verification is that verification is conducted on the final designed and built system, whereas compliance can be done at any earlier time and is an early step in the overall verification process.

• Compliance Assessments are required at each major subsystem and component design review.

• Required documentation:– Compliance Method(s) – Analysis, Test, Demonstration, Inspection– Verification Requirement– Success Criteria– Compliance Status (Y/N)– References to any additional documentation that further justifies the

assessment, if available.

Compliance Assessment Rqmts (Cont)

• For any non-compliances:– A risk must be documented in the Risk & Opportunity Register– The risk Number should be cross-referenced in the compliance table– The Risk entry should identify mitigation activities to correct the non-

compliance

Final Verification Reporting Rqmts

• A final Verification Record is compiled after all requirements within a specification / ICD have been verified.

• A Verification Matrix for Final Verification (VM-V) serves as the final record and summary of the verification process.

• For each requirement, summary information from the Verification Plan is included along with:

– Responsible Technical Authority (RTA)– Quality Assurance Officer (QAO) or designee’s name– Verification Successful (Y/N)– Verification Result Summary – a concise summary narrative explaining why

the verification activities were successful or not.– Verification Report – A reference to the Verification Report that contains the

details of the results of the verification activities.

• For each requirement, an RTA and QAO (or designee) will be identified. These individuals are responsible for vouching that the requirement has been verified and generating initial responses to Non-Conformances (more on later slides)

Verification as Part of AIV

• Verification is one critical aspect of the broader manufacturing, assembly, integration, and verification set of activities

• Project Systems Engineering needs to understand the early integration and verification activities being conducted by the subsystems that impact system level requirements, interfaces, assemblies, and verification activities.

• PSE is assembling a more detailed Project-level AIV plan. It is crucial that subsystem inputs are provided to support this activity.

• A general pattern has been defined that PSE will use to document these activities in Enterprise Architect using the SysML language (next slide)

Verification Process Steps 4 and 5

• After Verification Plans are generated, PSE uses this information, along with additional input from the subsystem teams, to develop a comprehensive system-level Verification model, :

• Identify Task Interdependency (Step 4)– Some Verification Activities can be naturally grouped and conducted

at the same time– These Verification Activities are then grouped into a single

Verification Event.• Can result in cost and schedule savings from eliminating redundant or nearly

redundant V&V activities

• Schedule Verification Events (Step 5)– Events are scheduled, identifying predecessor/ successor relationships

and other schedule constraints

AIV Pattern

Hardware Component(s)

Software Component(s)

Manufacturing /Value Added Process

Assembly

Integration Activity

To Verification & Acceprance

From Verification & Acceprance

Hardware Component(s)

Software Component(s)

Manufacturing /Value Added Process

Assembly

To Verification & Acceprance

From Verification & Acceprance

To Verification & Acceprance

From Verification & Acceprance

To Next Manufacturing

Step

In Process QA & Verification (as

needed)

In Process QA & Verification (as

needed)

From Previous Step

From Previous Step

Verify Form, Fit, and Function

Verify Form, Fit, and Function

Verify Interface Requirements

Input Objects Output Objects

TransformationProcesses

Verification(Next Slide)

IntegrationProcesses

Verification(Next Slide)

VERIFICATION

Creation of Verification Plans & Test Cases in the Model

Sequencing Test Cases (Verification Events)

• Test Cases (Verification Events) are sequenced on Activity Diagrams to show:

– Predecessor/ successor relationships

– Events that are conducted in parallel/ series

– Outside constraints that must be met before a Verification Event can be executed

• Results can be used to validate or update the project’s schedule for the Commissioning period.

Refinement of Individual Test Cases (Verification Events)

• As plans mature, individual Verification Events can be further detailed via association with its own detailed behavior diagram

• Serves as refined and more detailed input to the commissioning planning effort

–Can be used directly as inputs to writing detailed test & analysis procedures

Non-Conformances

• The are two categories of non-conformances:– Incidents – short-term anomalies or observations that require immediate

attention• Examples:

– Failure of a Verification Activity due to operator error– Need to redline a verification procedure due to realized constraints of the verification

environment that were not anticipated by the author of the procedure. These redlines only affect the details of how the activity is conducted and do not impact the validity of the results

– Problems – confirmed nonconformities that would cause the project to fail to meet requirements.

• Examples:– Repeated Failure of a verification activity that was executed to the procedure. The results

indicate a real problem with the form, fit, and/or function of the device.

• The RTA and QAO should directly respond to and disposition Incidents. Responses should be documented, and then the team may proceed immediately.

• Problems must be documented in a Non Compliance Report (NCR) and submitted as soon as possible to the Change Control Board (CCB) for communication purposes.

Relationship between V&V and QA

• The Verification Team (with RTA lead) is responsible for technical execution of the verification activities, which includes proper planning, execution of the activities, and reporting.

• Quality Assurance is responsible for surveillance and auditing of verification processes, including witnessing selected verification activities, to ensure that all steps are executed according to plan and that deviations and non-compliances are handled in a methodical way. QA provides confidence to stakeholders that products and services conform to requirements

• An RTA and QA Individual are jointly responsible for the verification of each requirement

– RTA is responsible for the technical execution of the verification events– QA is responsible that the verification events are carried out according to plan

Acceptance and Verification Process

PSE Working with the Subsystems

• In order for the PSE team to meet its responsibility of overall verification of the LSST Observatory system, the PSE will work collaboratively with the subsystems during subsystem component verification.

– Enhances the PSE team’s understanding of the performance and functional capabilities of the as-built hardware and software

– Ensures the PSE understands more comprehensively how lower level verification activities support subsystem level verification activities, which ultimately feed into project-level commissioning activities.

• The PSE will work with each subsystem to iterate and refine a list of subsystem milestones that the PSE requests visibility into

– PSE will work with the subsystems to refine these lists and generate PMCS milestones, which will act as the notification mechanisms to PSE.

– Work to be completed by Oct. 30, 2015

Subsystem Level Milestones

Review, Verification, and Acceptance Milestones to be identified for each

Component:

Requirements Review

Final Design Review

Procurement Review

Manufacturing Readiness Review

Verification Plan Review

Start of Verification Activities (i.e. Tests)

Subsystem Pre-shipment Review (if applicable)

Subsystem Acceptance Review

…..

Lists to be iterated and Milestones finalized by Oct. 30, 2015

Review, Verification, and Acceptance Milestones to be identified for each

Component:

Release Objectives Review

Verification Plan Review

Unit Test

Low Level Integration Test

End to End Test

Acceptance Test

Acceptance Test Review

…..

Hardware Centric Software Centric

Conclusions

• LSST Systems Engineering Process has been in place for several years

– It is now time to start using it!– The process has been expanded to deal with non-conformances and integrate

Verification with Commissioning planning in more detail

• Initial interactions between V&V and QA have been proposed• The modeling methodology has been defined• Next steps:

– PSE and the subsystems to work collaboratively to generate the Verification Planning Matrices for the subsystem specifications and project-level ICDs.

– PSE to generate AIV Activity Diagrams for early integration, system level integration, and commissioning

Backup Slides

Modeling Methodology for V&V using SysML

SysML Implementation – Defining Requirements

• All project-controlled requirements are captured as elements in the EA SysML model

– Each specification from the LSST Specification Tree is modeled as a version-controlled package

– Requirements are modeled as Requirement elements under the applicable package.

SysML Implementation – Definition of Rqmts Hierarchy

• Requirements Diagrams used to show:

– Model hierarchy (using Containment relationship)

– Requirements traceability via decomposition and allocation (using Derived relationship)

Extending the SysML Stereotype for Verification Planning

• SysML does not have a predefined element capable of capturing LSST’s Verification Planning information

• SysML is extensible, allowing for the definition of additional stereotypes

• LSST created a VerificationPlanning stereotype as an extension of the SysML1.3::requirement metaclass