1 iv&v program software independent verification and validation (iv&v): an agency overview...
DESCRIPTION
Independent Verification and Validation: The NASA Approach 3 IV&V Program Setting the stage: A History /99: Senior Management Council: IV&V mandate for all NASA software 06/99: Senior Management Council: IV&V mandate for all NASA software 10/91: Grant provided to WVU to build IV&V Facility assigned to HQ-OSMA 10/91: Grant provided to WVU to build IV&V Facility assigned to HQ-OSMA 04/94: Space Station Program Implements IV&V through Facility 04/94: Space Station Program Implements IV&V through Facility IV&V, IA Projects 04/96: Facility transitioned to AMES Research Center 04/96: Facility transitioned to AMES Research Center focus=research 00 07/00: Facility transitioned to Goddard Space Flight Center 07/00: Facility transitioned to Goddard Space Flight Center focus=IV&V 1996: Facility Omnibus contract: Enabled IV&V across all NASA Projects 1996: Facility Omnibus contract: Enabled IV&V across all NASA Projects 08/01 NPD : Software IV&V Policy 08/01 NPD : Software IV&V Policy /03 NASA Executive Council makes IV&V an Agency OSMA Program 05/03 NASA Executive Council makes IV&V an Agency OSMA Program 05/88: Space Shuttle Program Implements IV&V 05/88: Space Shuttle Program Implements IV&V /03 IV&V Funding changed to Corporate G&A 10/03 IV&V Funding changed to Corporate G&A 08/05 NPD : Software Policy 08/05 NPD : Software Policy 24 26TRANSCRIPT
1
IV&V ProgramSoftware Independent Verification and
Validation (IV&V): An Agency OverviewKenneth A Costello
IV&V Program Lead Engineer
GSFC Systems Engineering Seminar Series12 Sep 2006
Independent Verification and Validation: The NASA Approach
2
IV&V ProgramIV&V Program
Agenda• A Quick IV&V Facility/Program History
• The “Software Crisis”
• IV&V/NASA IV&V
• Forming an IV&V Project
• IV&V Relationships
• Closing
Independent Verification and Validation: The NASA Approach
3
IV&V ProgramIV&V Program
Setting the stage: A History
1
1991 20041996
06/99:Senior
ManagementCouncil: IV&V mandate for all NASA software
10/91:Grant provided to WVU to build IV&V Facility assigned to HQ-OSMA
04/94:Space Station
Program Implements
IV&V through Facility
1993 1999
2-3
3642
2015
12 IV&V, IA Projects
04/96:Facility
transitioned to AMES Research
Center
focu
s=re
sear
ch
00
07/00:Facility
transitioned to Goddard Space
Flight Center
focu
s=IV
&V
1996:Facility Omnibus
contract: Enabled IV&V
across all NASA Projects
08/01 NPD 8730.4:
Software IV&V Policy
37
9196
05/03NASA Executive Council makes IV&V an Agency OSMA Program
05/88:Space Shuttle
Program Implements
IV&V
1988
10/03IV&V
Funding changed to Corporate
G&A
08/05 NPD 2820.1:
SoftwarePolicy
2426
Independent Verification and Validation: The NASA Approach
4
IV&V ProgramIV&V Program
Setting the Stage: An Agency Requirement• NPD 8730.4 SW IV&V Policy
– Cancelled on 08/30/05
• Current Requirements
– NPD 2820.1C Software Policy
– NPR 7150.2 Software Engineering Requirements
– NASA-STD-8739.8 Software Assurance
Independent Verification and Validation: The NASA Approach
5
IV&V ProgramIV&V Program
NPD 2820.1C Software Policy• NASA policy regarding software activities for each project is to accomplish the following:
(5) Projects shall ensure software providers allow access to software and associated artifacts to enable insight/oversight by software engineering and software assurance which includes Independent Verification and Validation (IV&V) and NASA's Safety and Mission Assurance organizations.
c. Use the NASA IV&V Facility as the sole provider of IV&V services when software created by or for NASA is selected for IV&V by the NASA Chief Safety and Mission Assurance Officer.
• Responsibilitiesc. The NASA Chief Safety and Mission Assurance Officer shall: (1) … (6) Oversee the functional management of the NASA IV&V Program and assure the performance of all
of IV&V processes, services, and activities. (7) Establish and manage processes for the selection of software to which to apply IV&V. (8) Charter the IV&V Board of Directors (IBD) which makes prioritized recommendations for allocating
IV&V services to projects based on the annual Software Inventory (maintained by the Chief Engineer) and the Office of Safety and Mission Assurance(OSMA) defined process.
(9) Select and maintain the list of software projects to which IV&V is to be applied. (10)…d. The IV&V Program Manager shall 1) establish and manage the Agency's software IV&V services and
procedures; 2) establish, maintain, and report on the results of IV&V services and findings; and 3) support NASA's program for improving software assurance and other trusted verifications (e.g., independent assessments, peer reviews, and research). The IV&V Facility shall determine and document the services provided by the Facility on projects selected for IV&V by the NASA Chief Safety and Mission Assurance Officer.
Independent Verification and Validation: The NASA Approach
6
IV&V ProgramIV&V Program
NPR 7150.2 Software Engineering Requirements• Section 5.1.1.1 states required content for SW Development
Plans. – “The Software Development or Mgmt Plan shall contain: [SWE-102]
• a. Project organizational structure showing authority and responsibility of each organizational unit, including external organizations (i.e., Safety and Mission Assurance, Independent Verification and Validation (IV&V), Independent Technical Authority (ITA), NASA Engineering and Safety Center (NESC))".
• Additionally, within section 5.1.5 which addresses SW Assurance, it states: "The SW Assurance Plan details the procedures, reviews, and audits required to accomplish software assurance. The project office should coordinate, document, and gain concurrence with the Office of Safety and Mission Assurance as to the extent and responsibilities of the assurance and safety of the project. This will be documented into the project plans and reflected in the assurance process.
• Section 5.1.5.1 states “The SW Assurance Plan(s) shall be written per NASA-STD-8739.8, NASA SW Assurance Standard. [SWE-106]".
Independent Verification and Validation: The NASA Approach
7
IV&V ProgramIV&V Program
NASA-STD-8739.8 Software Assurance• Std states the following:
– Section 6.1.4 When IV&V has been selected for a project, the provider shall coordinate with IV&V personnel to share data and information.
– Section 7.5.3 When the IV&V function is required, the provider shall provide all required information to NASA IV&V Facility personnel. (This requirement includes specifying on the contracts and subcontracts, IV&V’s access to system and software products and personnel.)
8
IV&V Program
A Software Crisis
Independent Verification and Validation: The NASA Approach
Independent Verification and Validation: The NASA Approach
9
IV&V ProgramIV&V Program
Growing Software Importance• Fundamental Concern:
– First NASA robotic mission with actual software launched in 1969 (Mariner 6)
– Software size has grown over time• 128 words of assembly; equivalent 30 lines of C code• MER has about 600,000 lines of equivalent C code
– More functionality is being placed within software and software constructed devices (Programmable Logic Devices)
– With increased processing power and memory, more tasks are running concurrently• Control software increasing in complexity and size• Software used to monitor and react to hardware faults
Independent Verification and Validation: The NASA Approach
10
IV&V ProgramIV&V Program
Software is still hard to get right• The Carnegie Mellon Software Engineering
Institute reports(1) that at least 42-50 percent of software defects originate in the requirements phase.
• The Defense Acquisition University Program Manager Magazine(2) reports in a Department of Defense study that over 50 percent of all software errors originate in the requirements phase.
1 – Carnegie Mellon Software Engineering Institute, The Business Case for Requirements Engineering, RE’ 2003, 12 September 20032 - Defense Acquisition University Program Manager Magazine, Nov-Dec 1999, Curing the Software Requirements and Cost Estimating Blues
Independent Verification and Validation: The NASA Approach
11
IV&V ProgramIV&V Program
Fixing errors early can conserve resources• Early error detection and correction are vital to development success
– The cost to correct software errors multiplies during the software development lifecycle
– Early error detection and correction reduces cost and saves time• IV&V assurance vital to mission success
– Independent evaluation of critical software is value-needed– Agency goal
Average relative costs for finding errors late:
"Software Engineering Economics" by Barry Boehm
Independent Verification and Validation: The NASA Approach
12
IV&V ProgramIV&V Program
Overview of Defects found by IV&V Teams
11
435
129
282 271
2
133
2576
410 1 0 1 1770
1859
1054
874
275
364 361
74
1513
447
782
684
0
200
400
600
800
1000
1200
1400
1600
Concept Phase Requirements Phase Design Phase Implementation Phase Test Phase
WBS Activity
Num
ber o
f TIM
s
Severity 1Severity 2Severity 3Severity 4Severity 5Severity Total
13
IV&V Program
Independent Verification and Validation
Independent Verification and Validation: The NASA Approach
Independent Verification and Validation: The NASA Approach
14
IV&V ProgramIV&V Program
What is Verification and Validation?• Simply put, assuring that a software system
meets the user’s needs
• Verifying that the software is accurate and representative of its specification
• Validating that the software will do what the user really wants it to do
Independent Verification and Validation: The NASA Approach
15
IV&V ProgramIV&V Program
What is up with that I?• I = Independent
• Financially: Funded from Corporate G&A for Agency identified high priority Projects
– Customer Project may also fund the effort
• Technical: IV&V program defines scope and tasks, tailored by an IV&V criticality assessment
– Uses a predefined work breakdown structure
• Managerial: Functional management supplied by OSMA
– Project management supplied from the IV&V program
Independent Verification and Validation: The NASA Approach
16
IV&V ProgramIV&V Program
So what is IV&V?• An engineering discipline employing rigorous
methods for evaluating the correctness and quality of the software product throughout the software life cycle from a system level viewpoint.
• The NASA Software IV&V approach covers not only expected operating conditions but the full spectrum of the system and its interfaces in the face of unexpected operating conditions or inputs.
Independent Verification and Validation: The NASA Approach
17
IV&V ProgramIV&V Program
So what else is IV&V?• Testing at the end of the life cycle?
• NoNo• IV&V is testing, but it is whole life cycle testing
• The IV&V team “tests” artifacts ranging from system and software requirements to source code and test results
• Each task in the IV&V WBS is designed to “test” a development artifact or process
Independent Verification and Validation: The NASA Approach
18
IV&V ProgramIV&V Program
What are the objectives of IV&V?• Find defects within the system with a focus on
software and its interactions with the system
• Make an assessment of whether or not the system is usable in an operational environment, again with a focus on the software within the system
• Identify any latent risks associated with the software
Independent Verification and Validation: The NASA Approach
19
IV&V ProgramIV&V Program
What is the goal of IV&V?
• Note that the software may not be free from defects– Rarely the case and difficult to prove
• The software must be good enough for its intended use– As described by the requirements– Correct requirements
• The type of use will determine the level of confidence that is needed– Consequence of software defect/failure
Establish confidence that the software is fit for its purpose within the context of the system
Independent Verification and Validation: The NASA Approach
20
IV&V ProgramIV&V Program
Are there any other benefits to IV&V?• Primary purpose to provide confidence to OSMA,
however...
• Development projects receive all findings
– The good, the bad, the ugly
– Allows PM to have unbiased view of software development effort
– Provides knowledge resource for software developers
– In phase IV&V work provides early error detection and may save the project money in error correction
Independent Verification and Validation: The NASA Approach
21
IV&V ProgramIV&V Program
IV&V is process as well as product oriented
Software schedules, development tracking,critical path analysis, configuration mgmt
Program processes
Ancillary developmentsSimulations, trainers, test environments
Increased probability of success- Good processes allow early error identification and correction- Quality documentation enhances software maintenance
Independent Verification and Validation: The NASA Approach
22
IV&V ProgramIV&V Program
IV&V Increases Project Awareness
Program
Iden
tific
atio
n of
top
risk
sE
val o
f Pro
gram
Dev
el st
atus
Eva
l of P
rogr
am S
ched
ule
stat
us
Week Week Week Week MonthIV&V
Stat
us R
evie
ws
Stat
us R
evie
ws
Stat
us R
evie
ws
Stat
us R
evie
ws
Reqts Design
Phas
e co
mpl
ete
anal
ysis
rep
ort
IV&V is a program level “tool” to efficiently and effectivelymanage software development risk.
Independent Verification and Validation: The NASA Approach
23
IV&V ProgramIV&V Program
IV&V Interfaces and Reporting• Formal and informal interface with developers
– The formal interface with an IV&V Program project manager
– Informal interface between the IV&V analysts and the developers
– Helps to get identified problems and issues into the appropriate hands quickly
• Results of the effort thoroughly documented
– Issues identified to the developers in a timely manner
– Status reports to the Project Management
– Monthly/Quarterly reviews to GPMCs/Directorates/HQs
• Project close out report
– All inputs and outputs archived
– Final report delivered to project for its own internal records
– Lessons learned documented throughout
Independent Verification and Validation: The NASA Approach
24
IV&V ProgramIV&V Program
NASA IV&V
Independent Verification and Validation: The NASA Approach
25
IV&V ProgramIV&V Program
IV&V Scoping and Costing Flow for Project “A”
PDRSRR Project EndFor Years 1–5 Before SRR
IV&V
Sco
ping
and
Cos
ting
Proc
ess
IV&V Scopes andCosts Projects
IV&V
Impl
emen
tatio
n Pr
oces
sH
Q
ApprovedPrioritized List
of Projects
ProjectA
IV&V Scopes andCosts Projects
Project A iscosted using
MissionModel
Approach
POPSubmit
ApprovedPrioritized List
of Projects
ProjectA
IV&V Scopes andCosts Projects
Project A iscosted using
high-levelProject
particulars(e.g.,
proposalbasis)
Relationshipbetween
Project andIV&V Facility
Initiated(IVVP)
POPSubmit
ApprovedPrioritized List
of Projects
ProjectA
POPSubmit
IV&V Scopes andCosts Projects
IV&V CostBaseline for
Project Abased on
bottoms-upestimate from
criticalityanalysis and
SRR info
Perform CriticalityAnalysis, Update
IVVP
ApprovedPrioritized List
of Projects POPSubmitProject
A
Modificationsmade toProject A
costs basedon changes in
Project
IV&V Tasking/Reporting
Agency Generic IV&V Scoping and Costing Flow
Independent Verification and Validation: The NASA Approach
26
IV&V ProgramIV&V Program
The IV&V Life Cycle• An IV&V Project follows a life cycle similar
to most Projects
– Formulation
– Execution
– Close-out
Independent Verification and Validation: The NASA Approach
27
IV&V ProgramIV&V Program
Formulation Phase• The Formulation phase is used to plan and scope the
work to be performed
• Starts usually prior to System Requirements Review (SRR) with initial planning and contact with the Project
• Generally between SRR and Preliminary Design Review (PDR) planning and scoping process is executed
– Criticality analysis developed as foundation for the IV&V effort on the project
– The effort addresses all of the software on a Project
– The process generates a tailored approach based on the results of the assessment
Independent Verification and Validation: The NASA Approach
28
IV&V ProgramIV&V Program
Execution Phase• Majority of the IV&V effort is performed• Documented in an IV&V Plan (IVVP) that is an
output of the Formulation work– The IVVP is provided to the Project for review and
applicable concurrence
• Approach taken from the WBS and tailored based on the results of the Formulation work
• The Execution phase generally ends somewhere around or shortly after launch– In some cases, work may extend beyond launch when
software is still being developed (MER)
Independent Verification and Validation: The NASA Approach
29
IV&V ProgramIV&V Program
IV&V WBS for NASA Missions• The purposes of the IV&V Work Breakdown Structure are to
– Provide a consistent approach to IV&V across the Agency
– Provide a consistent and comprehensive basis for collection and reporting of metrics
– Help Projects anticipate and understand what IV&V will do
– The IV&V WBS was developed using industry standards and IV&V history on NASA missions as reference
• IEEE Std. 1012-2004 IEEE Standard for Software Verification and Validation• IEEE/EIA 12207.0-1996 Standard for Information Technology-Software life cycle
processes
• WBS Tasks for NASA Missions– Task selection is based on an algorithm using software development risk
– Risk is generated based various on Project characteristics (size, complexity, reuse, risk, etc.) as part of IV&V planning and criticality analysis tasks
– The full WBS can be found at http://ims.ivv.nasa.gov/isodocs/IVV_09-1.pdf
Independent Verification and Validation: The NASA Approach
30
IV&V ProgramIV&V Program
IV&V Activities Fit within the Project Schedule
• Designed to mesh with the Project schedule and provide timely inputs to mitigate risk
• Dialog between the IV&V Facility and the Project begins before SRR
System Requirements Review
Preliminary DesignReview
CriticalDesignReview
System TestS/W
FQT
Initial IVVPSigned
Mission Readiness Review
Concept Phase
2.0
Requirements Phase
3.0
Design Phase
4.0
Implementation Phase
5.0
Test Phase
6.0
Operations &Maintenance Phase
7.0
Baseline IVVPSigned
- IV&V provides support and reports for Project milestones - Technical Analysis Reports document major phases- IVVP is updated to match changes in Project
IV&VProvidesCoFR
IV&V Final Report
IV&V Phase Independent Support1.0
SystemRetirement
Launch
Note: numbers correspond to IV&V WBS
Independent Verification and Validation: The NASA Approach
31
IV&V ProgramIV&V Program
Close Out Phase• The Close Out phase concludes the IV&V effort
• All of the work performed is summarized in a final technical report
• Additionally, Lessons Learned are captured and either documented separately or incorporated into the final technical report
• In some cases, the IV&V Team is retained to provide mission support during critical phases of the Project which may occur after Close Out of the primary effort
Independent Verification and Validation: The NASA Approach
32
IV&V ProgramIV&V Program
The IV&V Life Cycle FlowConcept Phase
Software Requirements
Design
Implementation
Maintenance
System Requirements
Simulator/Environment/
Hardware
Software Planning
ValidationTesting
Verification
Verification
Verification
Verification
Verification
Verification
Focused activity at the earliest pointSystem requirements and software role importantIssues are introduced at lowest level
IV&V in phasewith
development
Later life cycle activity also importantIssues are still introduced at lowest levelFocused more on individual componentsIV&V support continues
over initial operational phase and beyond based on mission profile
Covers all levels of testing.Ensures that system meets the needs of the mission
Independent Verification and Validation: The NASA Approach
33
IV&V ProgramIV&V Program
IV&V Testing Philosophy
• Most testing is designed to show the software works within the envelope of the mission (Test what you fly, fly what you test)
• The IV&V approach is to focus more on off-nominal and unexpected situations in the software
• The higher the level of confidence needed the deeper the analysis
• The guiding goal is not necessarily to perform additional testing
– The goal is to improve the Project's test planning and execution– In some cases, IV&V may independently test highly critical software
Unit Test (CSC, CSCI)
S/W Integration
S/W Functional Qualification Testing
Acceptance Testing
System Integration and Test
[ Component Based Testing ][ Integration ] [ System Testing ][ Acceptance Testing ]
34
IV&V Program
Forming an IV&V Project
Independent Verification and Validation: The NASA Approach
Independent Verification and Validation: The NASA Approach
35
IV&V ProgramIV&V Program
IV&V Project Requirements: Background• Critical first steps is to develop the requirements for
the IV&V project– A set of engineering/management tasks that are determined
through a criticality analysis process • Previously accomplished individually by different
NASA contractors using different processes– This sometimes led to confusion with the NASA
development projects as there was little consistency– There was also a mixture of terminology used that was
sometimes in conflict with other NASA terminology and industry standard terminology
– There was also a perception among some parts of NASA that the IV&V contractors were determining their own work
Independent Verification and Validation: The NASA Approach
36
IV&V ProgramIV&V Program
Software Integrity Level Assessment Process• To help mitigate or eliminate some of these issues the
IV&V Program undertook an initiative to develop a new process
• Examined the best of current criticality analysis processes from industry and academia
• The primary objective of the process is to develop the requirements for an IV&V project
Independent Verification and Validation: The NASA Approach
37
IV&V ProgramIV&V Program
SILAP GoalsScalable Reasonably applicable from a mission-level, down to a
function levelRisk-based Ranking
A combination of Consequence (impact if the software component fails) and Error Potential (likelihood an error exists)
Minimal Complexity
Relatively simple such that it can be executed across a broad range of experience levels
Minimal Impact Minimize the level of participation from the project we are assessing
Objective Criteria
Minimize the use of engineering judgment and maximize the use of measurable criteria
Disjoint Tasking Produce tasking that is different for each software integrity level
Applicable Applicable throughout the life cycleUnderstandable The process and reasons for the results can be completely
described and should make sense to a general engineer/project manager
Independent Verification and Validation: The NASA Approach
38
IV&V ProgramIV&V Program
Software Integrity Level (SIL)• Software Integrity Levels
– Want to define, for a software component, the required level of integrity in terms of its role in the system• Understand how the component fits within the system • Understand what is required of that component to be able
to maintain the functionality of the system
Independent Verification and Validation: The NASA Approach
39
IV&V ProgramIV&V Program
Software Integrity Level: Definition• Definition of Software Integrity Level
– A range of values that represent software complexity, criticality, risk, safety level, security level, desired performance, reliability, or other project-unique characteristics that define the importance of the software to the user and acquirer
– The characteristics used to determine software integrity level vary depending on the intended application and use of the system. • A software component can be associated with risk because
– a failure (or defect) can lead to a threat, or – its functionality includes mitigation of consequences of initiating events in the system’s
environment that can lead to a threat
• Developed using not only software but also system level integrity as a basis (ISO/IEC 15026, 6)
Independent Verification and Validation: The NASA Approach
40
IV&V ProgramIV&V Program
Risk: A Common Denominator• Previously development projects (IV&V stakeholders)
could not easily link risk with the scoring that was performed
• Prime requirement for this new process is that it clearly defined the system risk and is linked to the software
• Process was built around two project factors the combination of which would define some level of system risk linked to the software
• The factors are Consequence and Error Potential
Independent Verification and Validation: The NASA Approach
41
IV&V ProgramIV&V Program
Consequence vs. Error Potential• Consequence is a measure of the system level impact of an error
in a software component
– Generally, take the worse case error (at the software component level) that has a reasonable or credible fault/failure scenario
– Then consider the system architecture and try to understand how that software fault/failure scenario may affect the system
• Error Potential is a measure of the probability that the developer may insert an error into the software component– An error is a defect in the human thought process
– A fault is a concrete manifestation of errors within the software
– A failure is a departure of the system behavior from the requirements
– With these definitions in mind, the approach is not to assess faults or failures, but to assess errors
• Scoring
Independent Verification and Validation: The NASA Approach
42
IV&V ProgramIV&V Program
Consequence• Consequence consists of the following items
– Human Safety – This is a measure of the impact that a failure of this component would have on human life
– Asset Safety – This is a measure of the impact that a failure would have on hardware
– Performance – This is a measure of the impact that a failure would have on a mission being able to meet its goals
Independent Verification and Validation: The NASA Approach
43
IV&V ProgramIV&V Program
Error Potential• Error Potential consists of the following items
– Developer Characteristics• Experience – This is a measure of the system developer’s experience in
developing similar systems• Organization – This is a measure of the complexity of the organization
developing the system (distance and number of organizations involved tend to increase the probability of errors being introduced into the system)
– Software/System Characteristics• Complexity – This is a measure of the complexity of the software being
developed• Degree of Innovation – This is a measure of the level of innovation
needed in order to develop this system/software• System Size – This is a measurement of the size of the system in terms
of the software (i.e., Source Lines of Code)
Independent Verification and Validation: The NASA Approach
44
IV&V ProgramIV&V Program
Error Potential (2)– Development Process Characteristics
• Formality of the Process – This is a measure of how maturity of the developer’s processes
• Re-use Approach – This is a measure of the level of re-use for the system/software
• Artifact Maturity – This is a measure of the current state of the development documentation in relation to the state of the overall development project (i.e., the is past critical design review but the requirements documents are still full of TBDs and incompletes)
Independent Verification and Validation: The NASA Approach
45
IV&V ProgramIV&V Program
Determining the Scores• Using the criteria, each software component is assessed and a score generated
• The scores are then processed through an algorithm to create a final score for Consequence and Error Potential
• The algorithm takes into account a weight for each of the characteristics
Consequence Factors Weight
Human Safety 0.0%
Asset Safety 35.0%
Performance 65.0%
Error Potential Factors
Sub-Factor Weight
Factor Weight
Developer 57.9%
Experience 82.8%
Development Organization 17.2%
Development Process 24.9%
Formality of Process 53.2%
Re-use Approach 22.6%
Artifact Maturity 24.2%
System/Software Characteristic 17.2%
Complexity 54.7%
Degree of Innovation 35.1%
Size of System 10.2%
Note that the Human Safety score carries no weight. Rather it is treated in a special manner as shown on the next slide
Independent Verification and Validation: The NASA Approach
46
IV&V ProgramIV&V Program
Calculating Consequence• The following algorithm is used to determine
the final Consequence scoreIf a component has no human safety impact then Human Safety = 0elsescore the Human Safety (hs) 1-5 using the criteria
Score the Asset Safety (as) 1-5 using the criteria
Score the Performance (pf) 1-5 using the criteria
If hs > (.35as + .65pf) thenFinal score = hs
elseFinal score = (.35as + .65pf)
This last step is important as it places emphasis human safety by using it as an overriding score if it is larger than the sum of the weighted asset safety and
performance score}
} This step defines the Human Safety score (hs)
Independent Verification and Validation: The NASA Approach
47
IV&V ProgramIV&V Program
Calculating Error Potential• The algorithm for the Error
Potential calculation has no special provisions
• It is simply a sum of the weighted scores
These attributes have:
- Values (vi)
- generated during the assessment
- Weights (wi)
- pre-defined
Error Potential =
8
1
3
1
2
1
sDev_Proces
eristicsSW_Charact
Developer
iiiw
iiiw
iiiw
wv
wv
wv
Note that all scores are rounded to the next whole integer
The first three terms represent the high level weights
Independent Verification and Validation: The NASA Approach
48
IV&V ProgramIV&V Program
Developing Tasking• A tasking set based on each individual score
– Tasking associated with a given Consequence score
– Tasking associated with a given Error Potential score
• One set of tasks per component
– The tasks are not exclusive to a given score
• This results in a matrix of software components and scores that provides the starting set of requirements for IV&V on that project
• The current matrix of score and tasks is provided on the next slide
Independent Verification and Validation: The NASA Approach
49
IV&V ProgramIV&V Program
IV&V Tasking MatrixFactors Consequences
Error Potential
Factor Scores 1 2 3 4 5 1 2 3 4 5
1.0 Phase Independent Support
1.1Management and Planning of Independent Verification and Validation X X X X X X X X X X
1.2 Issue and Risk Tracking X X X X X X X X
1.3 Final Report Generation X X X X X X X X
1.4 IV&V Tool Support X X X X X X X X
1.5 Management and Technical Review Support X X X X X X X X X X
1.6 Criticality Analysis X X X X X X X X X X
1.7Identify Process Improvement Opportunities in the Conduct of IV&V X X X X X X X X
Items with a carat (^) next to them are only invoked when human safety is involved
Independent Verification and Validation: The NASA Approach
50
IV&V ProgramIV&V Program
IV&V Tasking Matrix (2)Factors Consequences Error Potential
Factor Scores 1 2 3 4 5 1 2 3 4 5
2.0 Concept Phase
2.1 Reuse Analysis X X X
2.2 Software Architecture Assessment X X X
2.3 System Requirements Review X X X X X
2.4 Concept Document Evaluation X^ X^ X^
2.5 Software/User Requirements Allocation Analysis X^ X^ X^
2.6 Traceability Analysis X^ X^ X^
3.0 Requirements Phase
3.1 Traceability Analysis – Requirements X X X X X X
3.2 Software Requirements Evaluation X X X X X
3.3 Interface Analysis – Requirements X X X X X
3.4 System Test Plan Analysis X X X
3.5 Acceptance Test Plan Analysis X
3.6 Timing and Sizing Analysis X^ X^
Independent Verification and Validation: The NASA Approach
51
IV&V ProgramIV&V Program
IV&V Tasking Matrix (3)
Factors Consequences Error Potential
Factor Scores 1 2 3 4 5 1 2 3 4 5
4.0 Design Phase
4.1 Traceability Analysis – Design X X X X
4.2 Software Design Evaluation X X X^ X
4.3 Interface Analysis – Design X X X
4.4 Software FQT Plan Analysis X X X X
4.5 Software Integration Test Plan Analysis X X
4.6 Database Analysis X X X
4.7 Component Test Plan Analysis X
Independent Verification and Validation: The NASA Approach
52
IV&V ProgramIV&V Program
IV&V Tasking Matrix (4)Factors Consequences Error Potential
Factor Scores 1 2 3 4 5 1 2 3 4 5
5.0 Implementation Phase
5.1 Traceability Analysis - Code X X X X X
5.2 Source Code and Documentation Evaluation X X X X X
5.3 Interface Analysis - Code X X X X X
5.4 System Test Case Analysis X X
5.5 Software FQT Case Analysis X X
5.6 Software Integration Test Case Analysis X
5.7 Acceptance Test Case Analysis X
5.8 Software Integration Test Procedure Analysis X
5.9 Software Integration Test Results Analysis X X
5.10 Component Test Case Analysis X
5.11 System Test Procedure Analysis X^
5.12 Software FQT Procedure Analysis X^
Independent Verification and Validation: The NASA Approach
53
IV&V ProgramIV&V Program
IV&V Tasking Matrix (5)Factors Consequences Error PotentialFactor Scores 1 2 3 4 5 1 2 3 4 5
6.0 Test Phase
6.1 Traceability Analysis - Test X X X X X
6.2 Regression Test Analysis X^ X^
6.3 Simulation Analysis X^
6.4 System Test Results Analysis X X
6.5 Software FQT Results Analysis X X
7.0 Operations and Maintenance Phase
7.1 Operating Procedure Evaluation X^
7.2 Anomaly Evaluation X^
7.3 Migration Assessment X^
7.4 Retirement Assessment X^
Independent Verification and Validation: The NASA Approach
54
IV&V ProgramIV&V Program
IV&V Relationships
Independent Verification and Validation: The NASA Approach
55
IV&V ProgramIV&V Program
IV&V Facility Relationship to HQ• IV&V reports annual performance and receives
approved budget from IBD (Chaired by OSMA)
• AA/OSMA delegates Program to GSFC Center Director
• IV&V Facility Director is Program Manager
• Facility works with OSMA IV&V Liaison to coordinate IBD budget inputs and performance reporting
• OSMA works with IBD to identify and prioritize Projects annually
Independent Verification and Validation: The NASA Approach
56
IV&V ProgramIV&V Program
IV&V/Center/Project Relationships• IV&V-Project Relationship:
– IV&V still reports issues to Project first and treats Project as primary “customer” for technical findings and risks
– As a Code Q Program, IV&V will keep Center S&MA personnel informed of IV&V technical issues so that S&MA has a complete mission assurance picture
• IV&V-Center Relationship:
– Center Liaison facilitates the startup of IV&V on new Projects
– Center Liaison and IV&V Facility Leads facilitate technical issue resolution
– Center Liaison promotes consistent approaches to IV&V on Projects, and promotes awareness of IV&V Center-wide
– S&MA, Projects, and IV&V provide technical status and issues to the GPMC
• IV&V reports to GSFC PMC as a Program Office
57
IV&V Program
Closing
Software IV&V, as practiced by the NASA Software IV&V Facility, is a well-defined, proven,
systems engineering discipline designed to reduce the risk in major software developments
Independent Verification and Validation: The NASA Approach
Independent Verification and Validation: The NASA Approach
58
IV&V ProgramIV&V Program
Points of Contact• Bill Jackson
Acting Director
304-367-8202
• Ken Costello
Lead Engineer
304-367-8343