Nancy S. Eickelmann, PhDMotorola Labs1303 E. Algonquin Rd.Annex-2Schaumburg, IL 60196Phone: (847) 310-0785Fax: (847) [email protected]
2
FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL (CSIP)for the
NASA Independent Verification and Validation Facility COTR: Ken McGill
PI: Nancy EickelmannS-54493-G
September 5, 2001
Developing Risk-Based Financial Analysis Tools and Techniques to Aid IV&V Decision-
Making
3
PROBLEM STATEMENTPROBLEM STATEMENT
• This research addresses NASA’s need to evaluate the ROI and
cost/benefit of applying IV&V technologies. • A prototype is to be developed that will provide financial
valuation of IV&V for a given program. • The prototype will be developed using an iterative process that
will incrementally implement the models and methodology researched and developed during prior years of this effort.
• The tool will be evaluated for usability, accuracy, and consistency through limited use scenarios with NASA program managers.
4
Return on Investment - StatusReturn on Investment - Status
• This project was funded July 20, 2001• Evaluation of data sets is in progress• Benchmarking for key factor target value ranges in
progress• Model integration and interface to existing
programs in progress, Ask Pete, ARRT
5
RESEARCH APPROACHRESEARCH APPROACH
Phase 1:• Reduce the models we developed earlier to actionable
guidelines for practice
Phase 2:• Introduce these models, processes and support tools to a small
group of carefully selected pilot projects• Evaluate the results of applying the tools and methods
Phase 3:• Use the feedback from step 3 to adapt the tools and methods
for widespread dissemination, if warranted within the software project decision-making community at NASA.
6
HYPOTHESES/OBJECTIVEHYPOTHESES/OBJECTIVE • The IV&V valuation methodology will be iteratively refined based on
feedback from NASA program managers and statistical evaluation of the methodology and results.
• Specific factors to be evaluated:
Hypothesis 1: The cost relative to the potential benefits of IV&V is inversely proportional to key organizational factors, such as the capability maturity of the development organization.
Hypothesis 2: The realization of potential IV&V benefits is directly related to the development organizations’ acceptance of IV&V.
Hypothesis 3: The cost/benefit ratio for IV&V is directly related to the criticality of the application (and its individual subsystems).
7
IV&V YIELDIV&V YIELD
• Ultimately, the yield of an IV&V program is based upon the difference between the net resource flow with IV&V and without IV&V.
• If the resources saved (e.g., reduced rework) or returns gained (e.g., improved customer satisfaction or increased safety) are greater than the resources consumed to save/gain these resources, we have a net benefit.
• Should the resources saved be less than the resources consumed, we have a net cost.
9
What we already know…3 issues of empirical studies...
June 5-6, 1986 the 1st Workshop on Empirical Studies of Programmers, Washington, D.C.
• Need scientific rigor…“A Plan for Empirical Studies” Victor Basili
• Need to look at real world variable values…“By the Way, Did Anyone Study Real Programmers” Bill Curtis
• Need to study PITL…“Meeting the Challenge of Programming in the Large (PITL)” Elliot Soloway
10
Why is it Difficult to Apply Quantitative Management
Principles for Software Engineering?
• SE domain has a large number of key variables that have different degrees of significance depending on the environment
• SE domain has key variables that have extreme variance within the same environment (i.e., programmer productivity 10:1)
• SE domain variables in combination may create a “critical mass” not present when variables are studied in isolation
1986 IEEE TSE, Basili, Selby and Hutchins, Surveyed software engineering empirical studies published to date. Cited 116 published studies.
11
Software Requirements Analysis
Software Interface Analysis
Software Code Analysis
Developer Test Analysis
Software Design Analysis
Iterative IV&V MethodologyIterative IV&V Methodology
IV&V Planning- Activities - Organization - CARA- Schedules - Tools - WBS
Inputs Activities Outputs
SoftwareIV&V Plan
Critical/High RiskFunctions List
IV&V Technical Reports
Software ProblemReports
IV&V Traceability
Matrix
Findings and Recommendations
IV&V Metrics
MonthlyProgress/Status
Reports
Iter
ativ
e P
er S
oftw
are
Rel
ease
Source Code
SoftwareDevelopment Folders
Software Test Plans & Procedures
Problem Reports
RequirementsRepositories
ProgramMilestonesand Schedules
Phase Dependent IV&V Tasks Phase Independent IV&V Tasks
DeveloperDocumentation
Software IV&V SOW
- Objectives- Requirements
TR
AC
EA
BIL
ITY
AN
AL
YSIS
CH
AN
GE
IMP
AC
T A
NA
LY
SIS
DE
LIV
ER
AB
LE
S VA
LID
AT
ION
TE
CH
NIC
AL
RE
VIE
WS A
ND
AU
DIT
SSP
EC
IAL
STU
DIE
S
12
IV&V Technologies - COQIV&V Technologies - COQ
Algorithm analysis Analytic modeling Back-to-back testingBoundary value
analysisCode reading Control flow analysis
Coverage analysis Critical time analysis Database analysisDataflow analysis Desk checking Error seeding
Event tree analysis Functional testing InspectionsInterface analysis Mutation testing Performance testing
Proof ofcorrectness
Prototyping Regression testing
Requirementsanalysis
Requirementstraceability
Reviews
Sensitivity analysis Simulation Size analysisSlicing Software failure mode
anal.Stress testing
Structural testing Symbolic execution Test certificationWalkthroughs
13
Empirical Research SummaryEmpirical Research Summary
• Experimental Simulation Qualitative and quantitative results based on non-deterministic or hybrid simulation model
• Math Modeling quantitative results based on a deterministic model
• Mirrors a segment of the real world, control of variables is high, supports testing of causal hypothesis, results can be replicated, high internal validity and generalizability
• Captures real world context in which to isolate and control variables
• Researcher bias can be introduced through selection of variables, parameters and assumptions concerning the model. Modeling requires high degree of analytical skill, and interdisciplinary knowledge
• Results are not typically generalizable to other populations or environmental contexts, researcher bias is common,
14
Process Modeling and SimulationProcess Modeling and Simulation
Process Characterization
Syste mR equi rements
Syste m
Archit ectureR equi rements
Syste mDetai led
Design
B oxR qmts
B ox
Design
Syste mFunc tional
R equireme nts
B ox(auto)Coding
B oxUnit Tes t
B oxSyste m
Te st
Syste mEa rly SDL
Integrat iont esting
Syste mEa rly
Int egrat ion
t esting
Syste m
Integrat ionTe st
Syste m
Syste mTe st
SubsystemIntegrat ion
Te st
SubsystemSyste m Te st
(host /t arge t)
B ox
Int egrat ionTe st
Syste mB eta
Te st
T extT ext
T extT ext
eMSCeMSC
SL + SDLSL + SDL
T extT ext
eMSCeMSC
SDL(pr oc ess), CSDL(pr oc ess), CBox Validation Activities
System Val idation Activi ti es
eMSCeMSCExtended Message Sequence ChartsSDLSDLSpeci fi cation and Design LanguageSLSL “SL”, data language
eMSC, SDL(bloc s)eMSC, SDL(bloc s)
SDL (blocs)SDL (blocs)
Technology Evaluation
ImplementationImplementation
Refine
Feedback
Plan for ChangesPlan for Changes
Feedback
Objective Decision Criteria
Project Cost Estimating, Planning ,& Tracking
tsNoManScripptsNoAutoScri
Durationxoductivity
DurationxtsNoManScripxptsNoAutoScri
×+==
=×´+´
3Pr
)3()(
SDL Models
Managed, measured, productivity gains through:
• process improvement• data driven decision-making• technological innovation• Quantitative valuation of COQ vs COPQ
15
COQ versus COQ versus COPQCOPQTest
Activities Total Number of Days Opportunities for
Savings Proposed Change
Test Execution 132 Days 34 Days Process Control
Defect Removal 164 Days 67 Days Process Control
296 Days 101 Days
Total Number Hours Opportunities for Savings
Test Automation 5426 Hours 2662 Hours SDL Modeling
Test Automation 296 Days 25 Days SDL Modeling
16
Process Simulation ModelsProcess Simulation Models
• Experimental Simulation Qualitative and quantitative results based on non-deterministic or hybrid simulation model
• mirrors a segment of the real world • control of variables is high • supports testing of causal hypothesis• results can be replicated• high internal validity • high external validity, generalizability
18
Independent Verification and ValidationIndependent Verification and Validation
• An organization independent from the developers study the artifacts of software production.
• This requires:- Technical independence. Members of the IV&V team may
not be personnel involved in the development of the software.
- .Managerial independence. The responsibility for IV&V belongs to an organization outside the contractor and program organizations that develop the software.
- Financial independence. Control of the IV&V budget is retained in an organization outside the contractor and program organization that develop the software.
• IV&V is often perceived as testing the code after the development is completed NASA IV&V is full life cycle activities
20
Measuring IV&V EffectivenessMeasuring IV&V Effectiveness
InformationManagement
Other
InformationAnalysis
X33
NOAAISSEOS
Shuttle
CLCSAATT x x x
x x x x xx x x x xx x x x x x x x
xx x x x xx x x x x x x xx x x x x x x x
IV&VDevelopedTools
xx
x
Objectives Measures Targets Initiatives
Minimize testeffort
# test casestest pass/failrates
Minimum testset99% Interval
TestEffectivenessIV&V Toolset
Track issues todisposition
# Total IssuesOpen/Closed
2:1 rate ofclosure
Web-basedtracking systemPITS
InternalBusinessProcesses
Risk Mgmt # HazardsIdentified
No IFAs100%Traceable Req
ODC StudyRMS – ARM
21
US Data ?
AverageBest inClass
Current Level
% Improve-ment
Productivity(KAELOC per staff month) 3.23 7.14 4.03
Cost(dev. cost per KAELOC)
$4,334 $1,962 $1,008
Defect Content(defects per KAELOC)
15.6 8.1 18.8
Defect Removal Efficiency (by Technology) 95% 99.5% 92.7%
Cost of Delivered Defects(post-release defects per KAELOC, Domain)
$1.5M
IndustryIndustryBenchmarkingBenchmarking
Source for US Data: Capers Jones (2000) Software Assessments, Benchmarks, and Best Practice,
Addison-Wesley, p 339, System Software Baseline.
23
Prior Empirical ROI StudiesPrior Empirical ROI Studies
Savings As % of IV&V $Study Phase IV&VApplied Development Life-Cycle
ESC 82 Requirement +485% NA
Rad 81 Requirement +80% +102%
Rad 81 Requirement -8% +5%
Sap 83 Design NA +620%
Rad 81 Coding -72% -66%
Rad 81 Coding -80% -74%
ROI: Independent V&V Benefits
IV&V applied early in the lifecycle has the greatest ROI. Source Jet Propulsion Laboratory TR.
24 02/09/01
IMPACT of Major Air & Space Software ProblemsIMPACT of Major Air & Space Software Problems
Aggregate Cost:
Loss of Life:
‘93 ‘96 ‘97 ‘98 ‘99
Airbus A320
Ariane 5
3
Flight 965
160
$640 million
Loss of Data:
[Poseidon]
[Galileo]
USAF STEP
[Pathfinder]
[Lewis]
$116.8 million
Zenit 2
Delta 3
[NEAR]
$255 million
Titan 4B
[DS-1]
Orion 3
[Galileo]
$1.6 billion
[‘99] – NASA IV&V presentation
25
Tracing Impacts to Causes… Cause-Effect Graphing
Mission Success at Reduced Cost
Reliability Objective
Safety Objective Cost Objective
Process ImprovementAvoid Rework
Eliminate RedundancyEfficient Resource Allocation
Skilled Workforce
Domain ExpertsEngineers
V&V Experts
Skills training program
IT Infrastructure, Web-based reporting, DSS, ARM, PITS, RMS, Ask
Pete, ARRT
Communication Channels & Reporting
PL Reuse TechnologiesDomain Engineering
Knowledge MaintenanceV&V Models and Methods
Quality Objective
Defect Prevention
Defect Detection
Identify and Eliminate Hazards
Identify and Manage Risks
Information Analysis& Information Management,
Product Certification
26
Strategic and Financial Goals
Reliability Objective
Competitive Objective Cost Objective
Process ImprovementAvoid Rework
Eliminate RedundancyEfficient Resource Allocation
Skilled Workforce Black Belts
EngineersTelecom Experts
Skills training program - Motorola University
IT Infrastructure, Web-based reporting, COMPASS, TIGERS,
TeamPlay, Communication Channels & Reporting
SIX SIGMAPerformance ExcellenceKnowledge Maintenance
Communications Models and Methods
Quality Objective
Defect Prevention
Defect Detection
Optimize resource allocation & utilization
Identify and Manage Risks
Information Analysis& Information Management
Product Certification
BSC Cause and Effect GraphingBSC Cause and Effect Graphing
32
BENEFITSBENEFITS
• The benefits of this proposed Center Initiative would be applicable to all NASA software development organizations for whom IV&V is an option. The formalization of an objective decision-making process, along with enabling support tools would provide key capabilities to make rational budgetary decisions that impact safety and mission critical aspects of all NASA software systems. This is significant in enabling NASA to engage in effective administrative and managerial control based on objective, quantified information.
• The techniques proposed under this initiative will also provide NASA participants increased visibility into their process improvement efforts. The ISO-9001 certification requires that managers be able to document the benefits contributed to the organization by specific processes and process improvement effort [8]. A formalized, well-defined decision-making process would therefore make a significant contribution to NASA’s overall quality strategy.
33
MILESTONESMILESTONES
Start=July 20, 2001 + 3 mo
IV&V Process Description – Product Characterization
Based on prior CSIP results
Start=July 20, 2001 + 6 mo
Information Analysis
Data gathering for methodology
Start=July 20, 2001 + 6 mo
Initial Prototype Demonstration(s) & Iteration(s) Delivered
GSFC IV&V interface required
Nancy S. Eickelmann, PhDMotorola Labs1303 E. Algonquin Rd.Annex-2Schaumburg, IL 60196Phone: (847) 310-0785Fax: (847) [email protected]