Introduction for SMC

QUELCE: Quantifying Uncertainty for DoD Acquisition ProgramsJim McCurley

Senior Member of the Technical StaffSoftware Engineering InstituteCarnegie Mellon University October 2014QUELCE - October 2014 2014 Carnegie Mellon University

#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorSoftware Engineering Institute5/15/2015Copyright 2014 Carnegie Mellon University

This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.

Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense.

References herein to any specific commercial product, process, or service by trade name, trade mark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by Carnegie Mellon University or its Software Engineering Institute.

NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN AS-IS BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.

This material was prepared for the exclusive use of JMS Program at AF SMC and may not be used for any other purpose without the written consent of permission@sei.cmu.edu.

DM-0001364

#QUELCE October 2014 2014 Carnegie Mellon UniversityThe Uncertainty Problem: DoD Context..programs that breach appear to have the strongest relationship with three factors: the total dollar size of a project, the quantity change cost category, and the estimating cost changes....Much of the data collected now does not help decision-makers determine why a breach or unit-cost-growth has occurred or what programmatic changes would improve performance....The available information makes it difficult to assert any conclusions definitively because all factors appear interrelated, which means that an unconsidered exogenous variable may be confounding all conclusions.The Effect Of The NunnMcCurdy Amendment On Unitcost- Growth Of Defense Acquisition Projects, By Jacques S. Gansler, William Lucyshyn, and Adam Spiers , Univ of MD Center for Public Policy and Private Enterprise, July 2010_____________________

Unrealistic estimates are caused by the invalidity of major cost-estimating assumptions, not methodological errors... PARCA deems an estimate to be unrealistic if it is based on an uncertain assumption. Such assumptions might concern technical issues, related programs, organizational relationships, threats, policy matters or the industrial base.Inside the Pentagon, Vol. 27, No. 46, November 17, 2011Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE)#QUELCE October 2014 2014 Carnegie Mellon UniversityThe data needed to do rigorous modeling and robust prediction do not exist. Cost Estimating Relationships (regression models) are generally used to estimate cost and schedules on a system component basis, but they are analogies and the data used for the modeling is extremely noisy.3QUELCE addresses the assumptions and decisions made during system development.Account for change and uncertainty during the DoD acquisition life cycle.Synthesis of Expert Elicitation, Dependency Structure Matrix techniques, Bayesian Belief Network (BBN) modeling, Scenario Planning and Monte Carlo simulation into a method that models uncertainties among program change drivers as inputs to cost modelsUse of domain expert judgment, program artifacts, and and data-based inputs

DoD domain-specific method for improving expert judgment regarding uncertainty in program change drivers, their relationships, and impacts on cost drivers.Expert judgment is optimistic and over-confidentExpert calibration training improves estimates.Repository of program change histories inform experts.#QUELCE October 2014 2014 Carnegie Mellon UniversityEarly Estimation Lacks Dataand Change is ComplexInformation available at the start is not in a form typically used in preparing an estimate.Program does not yet have detailed scope and specifications.Can we model the uncertainties not captured by the estimate?Visual depiction of influential relationships, scenarios and outputs to aid team-based model development, and explicit description and documentation underlying an estimate.

Interdependencies cause problems to cascade.When a project goes off the rails there is often a cascade of problems before the magnitude of the problem becomes clear.Scenario modeling and simulation makes impact of changes visible.

#QUELCE October 2014 2014 Carnegie Mellon UniversityChanges in DODs 2011 Portfolio of Major Defense Acquisition Programs over TimeSource: DEFENSE ACQUISITIONS: Assessments of Selected Weapon Programs, GAO-12-400SP, March 2012

#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorProgram5/15/2015DoD often underestimates development costsdue in part to a lack of knowledge and optimistic assumptions about requirements and critical technologies. - A Knowledge-Based Funding Approach Could Improve Major Weapon System Program Outcomes, GAO Report to the Committee on Armed Services, U.S. Senate s, U.S. Senate, July, 2008 GAO-08-619

Sources Cost and time overruns for Major Defense Acquisition Programs, Joachim Hofbauer Gregory Sanders Jesse Ellman David Morrow, Center for Strategic and International Studies, April 201140% of accumulated cost overrunsEstimates are a Root Cause of Cost Overruns#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorSoftware Engineering Institute5/15/2015

DoD Programs Re-visit their Estimate Many TimesDoD 5000 with Naval POPS Gate Reviews #QUELCE October 2014 2014 Carnegie Mellon UniversityMotivation for QUELCEKnown-UnknownsEach decision and especially the assumptions have the potential to be wrong or at least subject to change. Thus each represents a known-unknown. Capture the probability of change and the change effectsWe can get some data about probability of change from past experience and reference data. Adjusting an estimate by probability of changeWe should be able to estimate the probability of change and cascade effects to develop a formal probabilistic method to add a range to the estimate.Include assessment of riskThe scenarios devised to assess change effects tell us which risk-based scenarios have the potential to require formal mitigation strategies, because they will create risks that exceed the boundaries set by contingency budgets. #QUELCE October 2014 2014 Carnegie Mellon UniversityUncertainty in Product DevelopmentAssumption: We can identify most sources of uncertainty and causal effects of change in product development projects.Challenge: Can this knowledge (awareness) be exploited in developing better estimates and plans for the product development work?Examples:Technology introduction: Introduction of 1-10mm wavelength radio frequency allows much higher volume traffic but development of a modem is more complicated.Change in scope: Antenna that fits a 747 will not fit in fighter jet.Change in interfacing external systemChange in sponsorship: Mission capability of proposed system fits a need for a new user community provided that certain performance can be improved.#QUELCE October 2014 2014 Carnegie Mellon UniversityMORS*: Affordability Analysis Workshop*Military Operations Research SocietyWorkshop recommends an approach similar to QUELCE

Drivers of un-affordable solutionsMistakes and poor decisionsFocus on development costs and minimizing sustainment investmentLimiting (performance budgets) or failing to understand capability trade spaceRecommendationsUse a distribution when estimating (not point estimates)Document ground rules and assumptions for estimatesInclude a sensitivity analysis for scenarios of changeIdentify drivers of uncertainty using past program performance historyUse analytic methods such as Monte Carlo and probability theory*From Affordability Analysis Workshop Report Oct 2012#QUELCE October 2014 2014 Carnegie Mellon UniversityChallenges to ImplementationCalibrating subject matter experts for probability judgment Subject matter experts are (almost always) over-confident.Reduce scale and complexity of the decision spaceChange space is very large.Development is often iterative so the same item may be a multiple driver of change.Multiple cost estimation relationships Many inputs, most of them are not directly represented in the decision space.Multiple scenarios of riskExperts must reason about potential changes with simultaneous drivers.

Access to subject matter experts and other stakeholders is required.Estimation is not simply a cost concern!A facilitated method is most valuable.#QUELCE October 2014 2014 Carnegie Mellon UniversityMission / CONOPSCapability Based Analysis ...KPP selectionSystems DesignSustainment issues ...Production QuantityAcquisition MgtScope definition/responsibilityContract AwardTechnology Development StrategyOperational CapabilityTrade-offsSystem CharacteristicsTrade-offsInformation from Analogous Programs/SystemsProposed Material Solution & Analysis of AlternativesProgram Execution Change Drivers (with Driver States & Probabilities)Probabilistic Modeling (BBN) & Monte Carlo SimulationCalibrated Expert JudgmentQUELCE Inserts New Activity in Information FlowsPlans, Specifications, AssessmentsanalogyparametricCost EstimatesProgram Execution Scenarios EvaluatedengineeringCERs#QUELCE October 2014 2014 Carnegie Mellon UniversityWhat is the likelihood of change...not what is the range of cost?

Blue is where SEI address uncertainty thru workshop with expertsWere informing the inputs that will be used in cost estimation.AuthorProgram5/15/2015

Modeling UncertaintyComplexity Reduction1. Elicit Change Drivers and Alternatives3. Assign Conditional Probabilities Build BBN Model4. Apply Uncertainty to Cost Formula Inputs for Basis and Scenarios5. Monte Carlo Simulation to Compute Cost Distribution2. Cause and Effect Analysis.Reduce ComplexityOverview of the QUELCE MethodLegend:QUELCE Change Repository

Queries of Historical MDAP Experience and Context

1. Driver State Matrix

2. Dependency Structure Matrix3. BBN Model4. Cost Factor Distributions by Scenario of Change

5. Monte Carlo with Cost Estimation Tools (e.g., COCOMO, SEER-SEM

ScopeDAES, etc.#QUELCE October 2014 2014 Carnegie Mellon UniversityUse of Expert JudgmentBroad and diverse literature on eliciting expert judgment exists, especially in Psychology and Statistics (see Selected References).

In this context, a great deal of information is generated by experts to propose a major new system but only a small portion of that information is utilized in the cost estimation process.

Estimates for budgeting, including lifecycle costs, are initially made when the system is only a concept.

The minimum life cycle for major systems is several years and often spans decades.

Contextual uncertainties, particularly assumptions which can change over time, are not captured by cost models.

Methods exist to help sharpen expert judgment, which also allow us to generate data to incorporate the uncertainty associated with that judgment.

[expert] judgments of probability, however elicited, are just that judgments ... made in response to the facilitators questions, not pre-formed quantifications of pre-analyzed beliefs. - Uncertain Judgements: Eliciting Expert Probabilities, Anthony OHagan, et.al., Wiley, 2006.

#QUELCE October 2014 2014 Carnegie Mellon University

16Most people are significantly overconfident about their estimates, especially educated professionalsExperts Tend to Be Over-Confident#QUELCE October 2014 2014 Carnegie Mellon University16

Reference Points to Help Calibrate Judgment for EstimationSolution

CalibratedUn-CalibratedEstimate of SW SizeDoD Domain-Specific reference pointsSize of ground combat vehicle targeting feature xyz in 2002 consisted of 25 KSLOC AdaSize of Army artillery firing capability feature abc in 2007 consisted of 18 KSLOC C++

Step 1: Virtual training using reference pointsStep 2: Iterate through a series of domain specific testsStep 3: Feedback on test performanceOutcome: Expert renders calibrated estimate of sizeCalibrated = more realistic size and wider range to reflect true expert uncertaintyUsed with permission from Douglas Hubbard Copyright HDR 2008 dwhubbard@hubbardresearch.com#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorProgram5/15/2015Calibration Training of ExpertsA series of training exercisesTypically day of 3 or 4 modules in sequenceWeb-enabled version availableEach exercise includes:A battery of factual questionsMost likely value; upper and lower bounds within which people are 90 percent certain the correct answer liesTrue false questions where people provide their confidence in their answersBrief reviews of the correct answersGroup discussions of why the participants answered as they didGuidance with heuristics about ways to explicitly consider interdependencies among related factors ... that might affect the basis of ones best judgments under uncertain circumstances

Group estimation experiments to be conducted this coming year.

#QUELCE October 2014 2014 Carnegie Mellon UniversityTraining Leads to Better Recognition of Uncertainty

Generic TestsDomain Specific Tests#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorProgram5/15/2015Experts Improved with Training

Test 1: Inaccurate & impreciseTest 2: Accurate & impreciseTest 3: Accurate & Precise#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorProgram5/15/20151: Experts Identify Change Drivers and States

1. Identify Change Drivers & States3. Assign Conditional Probabilities to BBN Model4. Calculate Cost Factor Distributions for Program Execution Scenarios5. Monte Carlo Simulation to Compute Cost Distribution2.Reduce complexityof Cause and Effect relationships via matrix techniques#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorProgram5/15/2015From: Obligations & Expenditures: An Investigation into the Factors that Affect OSD Goals, DoD Approved Survey: Report Control Symbol DD-AT&L(OT)2513 Rob Tremaine, Donna Seligman, Shandy Arwood, John Higbee

Briefing Presented to: Honorable Katrina McFarland 05 Feb 2013

Example:

Expert Data#QUELCE October 2014 2014 Carnegie Mellon University

Examples of Expert data on frequency of change,#QUELCE October 2014 2014 Carnegie Mellon UniversityAddressing the ComplexityExperts specify the concepts we need to address, then judge the conceptual interdependencies. We elicit likelihood ratings of cause and effect for change, which are used to construct the Bayesian Belief Network and to assign conditional probabilities.

Factor reductionSums of strength and error calculations help us to identify factors that contribute little to the analysis. Experts decide on appropriateness. Identify cyclesCycles requires some detailed analysis or discussion with experts. The tool identifies the cycles (typically 2-4) so that a manageable number remain. If retained, the cycles can be recast as different variables. In the end, we capture the experts judgments as to the likelihood of change and the cascading consequences. #QUELCE October 2014 2014 Carnegie Mellon UniversityRestate one of the change drivers so precedence is clearly stated.Consolidate two change drivers into a single one.Mission/CONOPS and Strategic Vision could easily be lumped together.Introduce an additional step to clarify the sequencing.It is possible that two different scenarios are involved that can be distinguished by the extra step.Review scenarios to see if cycle exists because of alternative scenarios.Is one of the scenarios quite unlikely? Perhaps it can be ignored in the analysis.

24

Indicates remaining cycles that must be addressed1. Identify Change Drivers & States3. Assign Conditional Probabilities to BBN Model4. Calculate Cost Factor Distributions for Program Execution Scenarios5. Monte Carlo Simulation to Compute Cost Distribution2. Reduce complexityof Cause and Effect relationships via matrix techniques2: Reduce Complexity via Dependency Structure Matrix #QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorProgram5/15/2015SEI DSM Tool Control PanelAll operations are done via the DSM tools control panel. Excel is used as a front end to systematically bring in your data, build a matrix, and provide a selection of options for matrix manipulation.

2: Reduce Complexity via Dependency Structure Matrix #QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorSoftware Engineering Institute5/15/2015

Reduction of the complexity of the network from 57 to 29 Program Change Drivers#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorProgram5/15/2015This example shows the reduction of need change drivers based on the experts judgements through the use of DSM.DSM -> BBNThe DSM matrix captured the experts change probabilities for one change driver affecting another change driver (all possible pairings).

The BBN models the probabilistic relationships for the nominal (expected) state, but experts then adjust the probabilities to construct a set of critical scenarios to provide the basis for program assessment based on cascading change.

For each scenario, the BBN produces probability distributions for the output nodes which will then be used to assign probability distributions to the input factors of the cost estimation models.#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorProgram5/15/20153: Assign Conditional Probabilities to BBN Model

#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorProgram5/15/2015This is an MDAP BBN visualization.

3: Assign Conditional Probabilities to BBN Model1. Identify Change Drivers & States3. Assign Conditional Probabilities to BBN Model4. Calculate Cost Factor Distributions for Program Execution Scenarios5. Monte Carlo Simulation to Compute Cost Distribution2.Reduce complexityof Cause and Effect relationships via matrix techniquesTruth table methodCapability Definition is affected by CONOPS and Strategic Vision#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorProgram5/15/20151. Identify Change Drivers & States3. Assign Conditional Probabilities to BBN Model4. Calculate Cost Factor Distributions for Program Execution Scenarios5. Monte Carlo Simulation to Compute Cost Distribution2.Reduce complexityof Cause and Effect relationships via matrix techniques

Engineering Change Proposal schedule delay = .56 + .0323 (waveform) + .0323 (Program Planning) + 0323 (Acquisition Contracting) +.0215 (technology in motion) + .0323 (Contractor Program Mgt)

Equationmethoda standard deviation of 0.08828 (8.9%)=3: Assign Conditional Probabilities to BBN ModelDerived from calibration exercise#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorProgram5/15/2015

3: Assign Conditional Probabilities to BBN Model#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorProgram5/15/2015This is the same visualization but with the conditional probabilities shown.

4: Calculate Cost Factor Distributions for Program Execution ScenariosAn example scenario with 4 drivers in nominal state1. Identify Change Drivers & States3. Assign Conditional Probabilities to BBN Model4. Calculate Cost Factor Distributions for Program Execution Scenarios5. Monte Carlo Simulation to Compute Cost Distribution2. Reduce Cause and Effect Relationships via Dependency Structure Matrix techniques#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorProgram5/15/2015Illustrates the glue node concept and also demonstrates the scenario ability of the BBN.Example: BBN Connection to the COCOMO estimation model.

#QUELCE October 2014 2014 Carnegie Mellon UniversityUnderstandand analyze costmodel input factorsUse empirical analysis from Repository as basis to map scale(XL EH) of original cost model input factors to scale (15) of BBN output factors

Group similar input factors based on empirical analysis in task 3.Connecting BBNs to Cost Estimation Models1. Identify Change Drivers & States3. Assign Conditional Probabilities to BBN Model4. Calculate Cost Factor Distributions for Program Execution Scenarios5. Monte Carlo Simulation to Compute Cost Distribution2.Reduce complexityof Cause and Effect relationships via matrix techniques#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorSoftware Engineering Institute5/15/2015

Monte Carlo simulation for program change factor distributions uses uncertainty on the input side to determine the cost estimate distributionMappedCOCOMO value45BBN Outputs45: Monte Carlo Simulation to Compute Cost Distribution1. Identify Change Drivers & States3. Assign Conditional Probabilities to BBN Model4. Calculate Cost Factor Distributions for Program Execution Scenarios5. Monte Carlo Simulation to Compute Cost Distribution2.Reduce complexityof Cause and Effect relationships via matrix techniques#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorProgram5/15/2015SummaryQUELCE includes the effects of uncertainty in the resulting estimate by: Making visible the quantified uncertainties that exist in basic assumptions.Calculating uncertainty of the input factors to the model rather than adjusting the output factors.Using scenario planning to calculate how specific changes might affect outcomes.The method utilizes subjective and objective data as inputExpert judgments are documented and made explicit.Information typically not used for estimation purposes can be leveraged.

The method explicitly includes factors that have been documented as sources of program failure in the past but are not typically captured by cost models.

Visibility of potential change drivers to program management enables quicker mitigation of emerging problems during the system lifecycle; impacts can be quickly calculated.

#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorProgram5/15/2015Selected ReferencesAir Force Cost Risk and Uncertainty Metrics Manual (CRUAMM) 2013 https://www.ncca.navy.mil/tools/csruh/CRUAMM_Printable_Version_16Nov2011_Edited_for_The_Joint_CSRUH_05Apr2013.pdf

Design structure matrix methods and applications, Steven D. Eppinger and Tyson R. Browning, The MIT Press May 2012.

Enhanced Scenario-Based Method for Cost Risk Analysis: Theory, Application, and Implementation, Paul R. Garvey, Brian Flynn, Peter Braxton, Richard Lee, Journal of Cost Analysis and Parametrics, 5:98142, 2012

GAO Cost Estimating and Assessment Guide: http://www.gao.gov/assets/80/77175.pdf

Investigation into Risk and Uncertainty: Identifying Coefficient of Variation Benchmarks for Air Force ACAT I Programs, Shaun T. Carney, Captain, USAF, AFIT-ENV-13-M-05, Air Force Institute Of Technology, Wright-Patterson Air Force Base, Ohio, Mar 2013. http://www.dtic.mil/dtic/tr/fulltext/u2/a579314.pdf

Joint Cost Schedule Risk and Uncertainty Handbook https://www.ncca.navy.mil/tools/csruh/CSRUH_Printable_Version_16Jul2013.pdf

#QUELCE October 2014 2014 Carnegie Mellon UniversitySelected ReferencesPrinciples of Uncertainty, Joseph B. Kadane Chapman & Hall/CRC Texts in Statistical Science, 2011

Risk Assessment and Decision Analysis with Bayesian Networks, Norman Fenton, Martin Neil, CRC Press, Nov 7, 2012

Software Development Cost Estimating Handbook Volume I, Naval Center for Cost Analysis and Air Force Cost Analysis Agency, Sept 2008 https://acc.dau.mil/adl/en-US/323892/file/46968/SW%20Cost%20Est%20Manual%20Vol%20I%20rev%2010.pdf

Statistical Methods for Eliciting Probability Distributions, Paul H Garthwaite, Joseph B Kadane & Anthony O'Hagan, J American Statistical AssociationVolume 100, Issue 470, 2005 pp. 680-701.

The Correct Use of Subject Matter Experts in Cost Risk Analysis, Coleman, Richard L. ; Braxton, Peter J. ; Druker, Eric R.,, 7th Annual Acquisition Research Symposium, May 2010 http://www.dtic.mil/cgi-bin/GetTRDoc?Location=U2&doc=GetTRDoc.pdf&AD=ADA530603

The Design Structure Matrix (DSM), www.dsmweb.org

Uncertain Judgments: Eliciting Expert Probabilities, Anthony OHagan, et.al., Wiley, 2006.

#QUELCE October 2014 2014 Carnegie Mellon UniversityExtra#QUELCE October 2014 2014 Carnegie Mellon UniversityBuilding a Repository of ChangeTwo challengesCollect the history and classify itAssist experts to search and identify potential changeCollect and classify:Initiate taxonomy from Naval POPS Guidebook v2Expand to cover more detailAssist the expertsAnalyze group effects of calibration (probabilities)Develop search tools so experts can identify possible assumptions and decisions

Experiment to Test Inter-Rater Reliability & Value of RepositoryCoded 200+ Artifacts against Change Taxonomy using Text Analytic Tool

Experiment to Test Inter-Rater Reliability & Value of Repository

#QUELCE October 2014 2014 Carnegie Mellon UniversitySources of Data to Populate the Reference Point RepositoryMDAP DataSourcesInformation CloudProgram Rpts:SARS, DAESProgram Artifacts:AoAs, ISPs, CBAsDoDRepositoriesARJArticlesDoDExpertsCAPE and Service Cost CentersSubject Matter Experts need DoD MDAP data about uncertainty to quantify relationships of program change drivers and their impact on program execution.

Why Hard? Empirical data need to be identified, accessed, extracted and analyzed from a myriad of sources. Data about program change is not structured nor quantified for use in estimation.

DoD Need: Quantified information about cost driver uncertainty should inform estimates.1. Identify Change Drivers & States3. Assign Conditional Probabilities to BBN Model4. Calculate Cost Factor Distributions for Program Execution Scenarios5. Monte Carlo Simulation to Compute Cost Distribution2. Reduce Cause and Effect Relationships via Design Structure Matrix techniques#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorSoftware Engineering Institute5/15/2015Why is this problem hard? Pertinent historical data do exist. Cite Marty Roper, McCrillis, SRDRs. However such data remains incomplete and inaccessible to support quantitative analyses of change and uncertainty across DoD domains & MDAPs.

Why does the DoD need it to be solved? Insufficient consideration of factors leading to change and uncertainty often leads to under-estimates and cost overruns. More and better quantitative data about likely change drivers provides a basis for considering alternative scenarios and their likely impact on program performance and cost.Example Use of the Repository to Support Cost Estimation - 1Materiel Solution Analysis Phase Pre Milestone EstimateAProgram Change Repository

For C2 systems, how often does Strategic Vision change?Records show that Strategic Vision changed in 45% of the MDAPS

Driver State MatrixThe Materiel Solution of a global network command and control system anticipates a possible change in Strategic Vision which will include allied participation.

Sharing information with allies creates new encryption requirements (a change in Mission/CONOPs).

These changes lead to changes in Capability Definition.Repository identifies probability of change in MDAP cost drivers.1. Identify Change Drivers & States3. Assign Conditional Probabilities to BBN Model4. Calculate Cost Factor Distributions for Program Execution Scenarios5. Monte Carlo Simulation to Compute Cost Distribution2. Reduce Cause and Effect Relationships via Dependency Structure Matrix techniquesProgStateDriverDDG51cond 1CONOPScond 2System Decond 3CapDefJTRScond 1InterOperacond 2Prod uctioF22cond 1Contractcond 2Functional cond 3CONOPS#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorSoftware Engineering Institute5/15/2015Note that the factors likely states and conditional probabilities will be characterized explicitly by such data for use in QUELCE steps 1 & 2 as seen in the next two slides. Others may be used as direct analogies from similar MDAPs as input to the BBNs or directly as inputs to existing cost models as described in task 4.Materiel Solution Analysis Phase Pre Milestone EstimateA

If Strategic Vision changes, what else changes?70% of the time the Mission/CONOPS changes

Driver State Matrix

DSM Cause-Effect MatrixProgram Change RepositoryProgStateDriverDDG51cond 1CONOPScond 2System Decond 3CapDefJTRScond 1InterOperacond 2Prod uctioF22cond 1Contractcond 2Functional cond 3CONOPSThe Materiel Solution of a global network command and control system anticipates a possible change in Strategic Vision which will include allied participation.

Sharing information with allies creates new encryption requirements (a change in Mission/CONOPs).

These changes lead to changes in Capability Definition.Repository identifies cascading effects of change in MDAP cost drivers.1. Identify Change Drivers & States3. Assign Conditional Probabilities to BBN Model4. Calculate Cost Factor Distributions for Program Execution Scenarios5. Monte Carlo Simulation to Compute Cost Distribution2. Reduce Cause and Effect Relationships via Dependency Structure Matrix techniquesExample Use of the Repository to Support Cost Estimation - 2#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorSoftware Engineering Institute5/15/2015Note that the factors likely states and conditional probabilities will be characterized explicitly by such data for use in QUELCE steps 1 & 2 as seen in the next two slides. Others may be used as direct analogies from similar MDAPs as input to the BBNs or directly as inputs to existing cost models as described in task 4.Materiel Solution Analysis Phase Pre Milestone EstimateA

Driver State Matrix

DSM Cause-Effect Matrix

BBN Model When both Strategic Vision & Mission/CONOPsexperience change, the BBN calculates thatCapability Definition will also change95% of the time.

Joint conditional probabilities can be calculated for downstream changes.The Materiel Solution of a global network command and control system anticipates a possible change in Strategic Vision which will include allied participation.

Sharing information with allies creates new encryption requirements (a change in Mission/CONOPs).

These changes lead to changes in Capability Definition.1. Identify Change Drivers & States3. Assign Conditional Probabilities to BBN Model4. Calculate Cost Factor Distributions for Program Execution Scenarios5. Monte Carlo Simulation to Compute Cost Distribution2. Reduce Cause and Effect Relationships via Dependency Structure Matrix techniquesExample Use of the Repository to Support Cost Estimation - 3#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorSoftware Engineering Institute5/15/2015Value of Change Driver Taxonomy

Step 3. BBN Model

Step 2. Dependency Structure MatrixStep 1. Driver State Matrix

Enables data mining sources of cost changeEnables common language to query repository for applicability to current programEnables common language to assess probability of occurrence and cascading effects#QUELCE October 2014 2014 Carnegie Mellon UniversityQUELCE Technical Reports:http://www.sei.cmu.edu/library/abstracts/reports/11tr026.cfm

http://www.sei.cmu.edu/library/abstracts/reports/13tr001.cfmSEI Webinar (recorded Oct 31, 2012)http://www.sei.cmu.edu/library/abstracts/webinars/Quantifying-Uncertainty-in-Early-Lifecycle-Cost-Estimation.cfmSEI Blog http://blog.sei.cmu.eduImproving the Accuracy of Early Cost Estimates for Software-Reliant Systems, First in a Two-Part SeriesA New Approach for Developing Cost Estimates in Software Reliant Systems, Second in a Two-Part SeriesQuantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE): An Update Journal of Software Technology. http://journal.thedacs.com/issue/64/207An Innovative Approach to Quantifying Uncertainty in Early Lifecycle Cost Estimation, Proceedings,10th Acquisition Research Symposium, 2013 http://www.acquisitionresearch.net/files/FY2013/NPS-CE-13-C10P04R05-058.pdf

For More Information #QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorProgram5/15/2015Contact InformationPresenters / Points of ContactSEMA Cost Estimation Research Group

Robert Fergusonrwf@sei.cmu.edu

Dennis Goldensondg@sei.cmu.edu

Jim McCurleyjmccurle@sei.cmu.edu

Robert Stoddardrws@sei.cmu.edu

Dave Zubrowdz@sei.cmu.edu U.S. MailSoftware Engineering InstituteCustomer Relations4500 Fifth AvenuePittsburgh, PA 15213-2612, USA

Webwww.sei.cmu.eduwww.sei.cmu.edu/contact.cfm

Customer RelationsEmail: info@sei.cmu.eduTelephone: +1 412-268-5800SEI Phone: +1 412-268-5800SEI Fax: +1 412-268-6257

#QUELCE October 2014 2014 Carnegie Mellon UniversityAuthorSoftware Engineering Institute5/15/2015Change DriverNominal StateAlternative States

Scope DefinitionStableUsers addedAdditional (foreign) customerAdditional deliverable (e.g. training & manuals)Production downsizedScope Reduction (funding reduction)

Mission / CONOPSAs definedNew conditionNew missionNew echelonProgram becomes Joint

Capability DefinitionStableAdditionSubtractionVarianceTrade-offs [performance vs affordaility, etc.]

Funding ScheduleEstablished Funding delays tie up resources {e.g. operational test}FFRDC ceiling issueFunding change for end of yearFunding spread outObligated vs. allocated funds shifted

Advocacy ChangeStableJoint service program loses particpantSenator did not get re-electedChange in senior pentagon staffAdvocate requires change in mission scopeService owner different than CONOPS users

Closing Technical Gaps (CBA)Selected Trade studies are sufficientTechnology does not achieve satisfactory performanceTechnology is too expensiveSelected solution cannot achieve desired outcomeTechnology not performing as expectedNew technology not testing well

~~~~~~~~~~~~

~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~

Input TablesEffectProduct Challenge5Project Challenge4Estimated Size (KSLOC)50Product Challenge factors5XLVLLNHVHEHProdProjScale FactorsPREC54321XHow familiar is this type product? (Highly=1)Val6.24.963.722.481.24XFLEX5431XHow much flexibility in req.? (Lots=1, little=5)5.074.053.042.03XIf product challenge is High Flex is LowRESLXAlways Nominal2.53XTEAMJointNormXIn project challenge3.950.99XPMAT54321XProcess maturity (high maturity is low multiplier)6.244.683.121.560XEffort MultipliersPERS5321X?If product challenge is High, PERS capability is lower1.261.000.830.630.49X?RCPX12345XProduct complexity, including safety, etc0.490.60.8311.331.912.72XPDIF11.5345XPlatform difficulty (product)0.8711.291.812.61XPREXXPeople (proj)1.120.74XFCILXFacility (proj)1.10.87XRUSE12345XDesign for reuse is expensive0.9511.071.151.24XSCED135Xproject factor11.141.43XProject Challenge factors4COCOMO ParameterXLVLLNHVHEHScale FactorsPRECUsed in productValFLEXUsed in product

RESLAlways Nominal2.53TEAMJointNormDirectly fed3.950.99Joint program or multiple lines of authority get higher multiplierPMAT54321Process maturity (high maturity is low multiplier)6.244.683.121.560Reversed values from COCOMOEffort MultipliersPERS0.831.62Fixed range not dependent on our modelRCPXProduct complexity inc safety and securityin Product challengePDIFin Product challenge

PREXFixed range depends on contractor1.120.74FCILFixed range depends on contractor1.10.87RUSEIn product challenge

SCED1.0035Runs reverse of COCOMO 11.141.43High challenge is schedule compression

FormulasCalculate Effort Multiplier and Scale FactorEMis the product of PERSRCPXPDIFPREXFCILRUSESCED10.071.002.4962.368751.021.051.231.30randLFLFrandrandLFLFSF is the sum ofPRECFLEXRESLTEAMPMAT19.2646.24.862.530.994.68LFLFFixed2-pointLFCOCOMO EffortPerson-MonthsSizeE=B+0.01*SUM(SF)AEM2211.4252097441501.102642.9410.0774.7054499049219.6340227204

Sheet3Calculate Effort and ScheduleXLVLLNHVHEHProdProjScale FactorsPREC54321XHow familiar is this type product? (Highly=1)Val6.24.963.722.481.24XFLEX5431XHow much flexibility in req.? (Lots=1, little=5)5.074.053.042.03XIf product challenge is High Flex is LowRESLXAlways Nominal2.83XTEAMJointNormXIn project challenge3.950.99XPMAT54321XProcess maturity (high maturity is low multiplier)6.244.683.121.560XEffort MultipliersPERS5321X?If product challenge is High, PERS capability is lower1.261.000.830.630.49X?RCPX12345XProduct complexity, including safety, etc0.490.60.8311.331.912.72XPDIF11.5345XPlatform difficulty (product)0.8711.291.812.61XPREXXPeople (proj)1.120.74XFCILXFacility (proj)1.10.87XRUSE12345XDesign for reuse is expensive0.9511.071.151.24XSCED135Xproject factor11.141.43XDriversXLVLLNHVHXHScale FactorsPREC6.204.963.722.481.240.00FLEX5.074.053.042.031.010.00RESL7.075.654.242.831.410.00TEAM5.484.383.292.191.100.00PMAT7.806.244.683.121.560.00Effort MultipliersRCPX0.490.602.721.001.331.912.72RUSE1.241.001.071.151.24PDIF0.871.001.291.812.61PERS2.121.621.261.000.830.630.50PREX1.591.331.121.000.870.740.62FCIL1.431.301.101.000.870.730.62SCED1.431.141.001.001.00DriversXLVLLNHVHXHProductProjectScale FactorsPREC6.204.963.722.481.240.00FLEX5.074.053.042.031.010.00RESL7.075.654.242.831.410.00TEAM5.484.383.292.191.100.00PMAT7.806.244.683.121.560.00Effort MultipliersRCPX0.490.600.831.001.331.912.72XRUSE0.951.001.071.151.24XPDIF0.871.001.291.812.61XPERS2.121.621.261.000.830.630.50PREX1.591.331.121.000.870.740.62FCIL1.431.301.101.000.870.730.62SCED1.431.141.001.001.00

Sheet1PRECFLEXRESLTEAMPMAT6.25.077.075.487.08