Towards COSYSMO 2.0Future Directions and Priorities

CSSE Annual Research ReviewLos Angeles, CA

March 17, 2008

Garry Roedler

Gan Wang

Jared Fortune

Ricardo Valerdi

Agenda

• Context setting

• Discussion on COSYSMO 2.0 improvements

• Prioritization exercise

8:15 – 9:00 am Introductions overview of the model summary of COSYSMO 2.0 improvements

Garry Roedler

9:00 – 10:00 am Reuse (overview only) Integration between SwE & SysE

Jared Fortune,Gan Wang

10:00 – 10:30 am Break

10:30 – 11:00 am Assumption of linearity in cost drivers Cost drivers vs. scale factors

Gan Wang

11:00 – 12:00 pm Recursive levels in the design Risk modeling (overview only)

Ricardo Valerdi, Garry Roedler

12:00 – 1:00 pm Lunch

1:00 – 1:45 pm Best practice guidance Garry Roedler

1:45 – 2:30 pm Modeling organizational factors in space systems

Darryl Webb

2:30 – 3:00 pm Break

3:00 – 4:00 pm Cost driver impact survey results from Oct ‘07 Discussion & wrap-up

Gan Wang,Garry Roedler

4:00 – 5:00 pm Joint meeting w/ SoS cost estimation group Jo Ann Lane

Context setting

How is Systems Engineering Defined?• Acquisition and Supply

– Supply Process– Acquisition Process

• Technical Management– Planning Process– Assessment Process– Control Process

• System Design– Requirements Definition Process– Solution Definition Process

• Product Realization– Implementation Process– Transition to Use Process

• Technical Evaluation

– Systems Analysis Process

– Requirements Validation Process

– System Verification Process

– End Products Validation Process

EIA/ANSI 632, Processes for Engineering a System, 1999. Note: The requirements of EIA/ANSI 632 are addressed in ISO/IEC 15288, which was also used as a Source for consistent definition in COSYSMO.

COSYSMO Origins

COSYSMO

Systems Engineering (SE)

1950

Software Cost Modeling

1980

CMMI®

1990

*CMM and CMMI are registered trademarks of Carnegie Mellon UniversityWarfield, J. N., Systems Engineering, United States Department of Commerce PB111801, 1956. Boehm, B. W., Software Engineering Economics, Prentice Hall, 1981.Humphrey, W. Managing the Software Process. Addison-Wesley, 1989.EIA/ANSI 632, Processes for Engineering a System, 1999ISO/IEC 15288, System Life Cycle Processes, 2002.

(Humphrey 1989)

(Boehm 1981)

(Warfield 1956, EIA 1999,

ISO/IEC 2002)

SW-CMM®

SE-CMM ®

SECM

2000

2000

Current SE StandardsEIA-632 ISO/IEC 15288

COSYSMO Data SourcesBoeing Integrated Defense Systems (Seal Beach, CA)

Raytheon Intelligence & Information Systems (Garland, TX)

Northrop Grumman Mission Systems (Redondo Beach, CA)

Lockheed Martin Transportation & Security Solutions (Rockville, MD)

Integrated Systems & Solutions (Valley Forge, PA)

Systems Integration (Owego, NY)

Aeronautics (Marietta, GA)

Maritime Systems & Sensors (Manassas, VA; Baltimore, MD; Syracuse, NY)

General Dynamics Maritime Digital Systems/AIS (Pittsfield, MA)Surveillance & Reconnaissance Systems/AIS (Bloomington, MN)

BAE Systems National Security Solutions/ISS (San Diego, CA)

Information & Electronic Warfare Systems (Nashua, NH)

SAIC Army Transformation (Orlando, FL)

Integrated Data Solutions & Analysis (McLean, VA)

L-3 Communications Greenville, TX

Modeling Methodology

3 rounds; > 60 experts

62 data points; 8 organizations

COSYSMO Scope• Addresses first four phases of the system

lifecycle (adapted from ISO/IEC 15288)

• Considers standard Systems Engineering Work Breakdown Structure tasks (per EIA/ANSI 632)

Conceptualize DevelopOper Test & Eval

Transition to Operation

Operate, Maintain, or Enhance

Replace orDismantle

COSYSMO

SizeDrivers

EffortMultipliers

Effort

Calibration

# Requirements# Interfaces# Scenarios# Algorithms

+3 Adj. Factors

- Application factors-8 factors

- Team factors-6 factors

COSYSMO Operational Concept

x

COSYSMO Model Form

14

1,,,,,, )(

jj

E

kkdkdknknkekeNS EMwwwAPM

Where: PMNS = effort in Person Months (Nominal Schedule)

A = calibration constant derived from historical project data k = {REQ, IF, ALG, SCN}wx = weight for “easy”, “nominal”, or “difficult” size driver

= quantity of “k” size driverE = represents diseconomies of scaleEM = effort multiplier for the jth cost driver. The geometric product results in an

overall effort adjustment factor to the nominal effort.

x

Size Drivers vs. Effort Multipliers

• Size Drivers: Additive, Incremental– Impact of adding a new item inversely proportional to

current size• 10 -> 11 rqts = 10% increase• 100 -> 101 rqts = 1% increase

• Effort Multipliers: Multiplicative, system-wide– Impact of adding a new item independent of current

size• 10 rqts + high security = 40% increase• 100 rqts + high security = 40% increase

Easy Nominal Difficult

# of System Requirements 0.5 1.00 5.0

# of Interfaces 1.7 4.3 9.8

# of Critical Algorithms 3.4 6.5 18.2

# of Operational Scenarios 9.8 22.8 47.4

Size Driver Weights

UNDERSTANDING FACTORS– Requirements understanding – Architecture understanding– Stakeholder team cohesion – Personnel experience/continuity

COMPLEXITY FACTORS– Level of service requirements– Technology Risk– # of Recursive Levels in the Design– Documentation Match to Life Cycle Needs

OPERATIONS FACTORS– # and Diversity of Installations/Platforms– Migration complexity

PEOPLE FACTORS

– Personnel/team capability

– Process capability

ENVIRONMENT FACTORS

– Multisite coordination

– Tool support

Cost Driver Clusters

Criteria + Matched driver polarity + Grouped by theme + Combined moderately correlated parameters

Cost Driver Rating ScalesVery Low Low Nominal High Very High

Extra High EMR

Requirements Understanding 1.87 1.37 1.00 0.77 0.60   3.12

Architecture Understanding 1.64 1.28 1.00 0.81 0.65   2.52

Level of Service Requirements 0.62 0.79 1.00 1.36 1.85   2.98

Migration Complexity     1.00 1.25 1.55 1.93 1.93

Technology Risk 0.67 0.82 1.00 1.32 1.75   2.61

Documentation 0.78 0.88 1.00 1.13 1.28   1.64

# and diversity of installations/platforms     1.00 1.23 1.52 1.87 1.87

# of recursive levels in the design 0.76 0.87 1.00 1.21 1.47   1.93

Stakeholder team cohesion 1.50 1.22 1.00 0.81 0.65   2.31

Personnel/team capability 1.50 1.22 1.00 0.81 0.65   2.31

Personnel experience/continuity 1.48 1.22 1.00 0.82 0.67   2.21

Process capability 1.47 1.21 1.00 0.88 0.77 0.68 2.16

Multisite coordination 1.39 1.18 1.00 0.90 0.80 0.72 1.93

Tool support 1.39 1.18 1.00 0.85 0.72   1.93

EMR = Effort Multiplier Ratio

Cost Drivers Ordered by Effort Multiplier Ratio (EMR)

Life Cycle Phases/Stages

Conceptualize DevelopTransition to

Operation

Operate,Maintain,

or Enhance

Replace or Dismantle

EIA

/AN

SI 6

32

Acquisition & Supply

Technical Management

System Design

Product Realization

Technical Evaluation

Operational Test &

Evaluation

Effort Profiling

Academic prototype

Commercial Implementations

Proprietary Implementations

COSYSMO-R

SECOST

SEEMaP

ImpactAcademic Curricula

Intelligence CommunitySheppard Mullin, LLC

Policy & Contracts

Model

14

1,,,,,, )(

jj

E

kkdkdknknkekeNS EMwwwAPM

10 theses

COSYSMO 2.0 Improvements

Recommended Improvements(from user community)

1. Reuse

2. Integration of SwE & SysE estimation

3. Assumption of linearity in COSYSMO cost drivers

4. Effect of cost drivers and scale factors

5. Number of recursive levels of design

6. Risk modeling

7. Establishing best practice guidance

8. Consideration of SoS scope in COSYSMO

9. Estimation in Operation & Maintenance Phase

10. Requirements volatility

Joint meeting

Deferred

1. Reuse

• Central question: What is the effect of reuse in estimating systems engineering size/effort?

• Hypothesis: A COSYSMO reuse submodel will improve the model’s estimation accuracy

• POC: Jared Fortune• References

– Valerdi, R., Wang, G., Roedler, G., Rieff, J., Fortune, J., “COSYSMO Reuse Extension,” 22nd International Forum on COCOMO and Systems/Software Cost Modeling, 2007.

2. Integration of SwE & SysE estimation

• Central question: What is the overlap between COCOMO II and COSYSMO?

• Hypothesis: By identifying the WBS elements in COSYSMO that overlap with the WBS in COCOMO II, the systems engineering resource estimation accuracy increases

• POC: Ricardo Valerdi• References

– Valerdi, R., The Architect and the Builder: Overlaps Between Software and Systems Engineering. (working paper)

3. Linearity in COSYSMO cost drivers

• Central question: How do we characterize the non-linearity of cost drivers across the system life cycle?

• Hypothesis: Not all cost drivers have a constant impact on systems engineering effort throughout the life cycle.

• POC: Gan Wang• References

– Wang, G., Valerdi, R., Boehm, B., Shernoff, A., “Proposed Modification to COSYSMO Estimating Relationship,” 18th INCOSE Symposium, June 2008.

4. Effect of cost drivers and scale factors

• Central question: Can some of the cost drivers become scale factors in the cost estimating relationship calibrated by the new data set?

• Hypothesis: The current set of size and cost drivers are too sensitive to small variations in rating levels.

• POC: Gan Wang• References

– Wang, G., Valerdi, R., Boehm, B., Shernoff, A., “Proposed Modification to COSYSMO Estimating Relationship,” 18th INCOSE Symposium, June 2008.

Note from GJR: Potential relationship between this and improvement #7. Also, in reality, number of recursive levels has a different impact on the effort than other cost drivers. This may be able to be addressed by this improvement, by improvement #5, or could be a separate improvement depnding on scope of each. Should be an item for discussion as we go through these.

5. Number of recursive levels of design

• Central question: How can the integration complexity of system elements one layer below the system-of-interest be operationalized?

• Hypothesis: The integration complexity of system elements is a predictor of systems engineering effort.

• POC: John Rieff• References

– Marksteiner, B., “Recursive Levels and COSYSMO”, October 2007. (working paper)

6. Risk Modeling

• Central question: How can risk associated with the COSYSMO estimate be quantified?

• Hypothesis: The output generated by COSYSMO can be quantified using probability distributions for better assessment of the likelihood of meeting the estimate

• POC: John Gaffney (developer of COSYSMO-R)• References

– Valerdi, R., Gaffney, J., “Reducing Risk and Uncertainty in COSYSMO Size and Cost Drivers: Some Techniques for Enhancing Accuracy,” 5th Conference on Systems Engineering Research, March 2007, Hoboken, NJ.

7. Best practice guidance for use of Cost Drivers

• Central question: How can misuse of the COSYSMO cost drivers be avoided?

• Hypothesis: By developing a best practice guide that describes common pitfalls associated with COSYSMO cost drivers, over-estimation can be reduced or avoided

• POC: Garry Roedler• References

– COSYSMO User Manual

8. Consideration of SoS scope in COSYSMO

• Central question: How can COSYSMO be updated to address system of systems effort estimation?

• Hypothesis: To be discussed in joint session• POC: Jo Ann Lane

9. Estimation in Operation & Maintenance Phase

• Central question: How can we estimate systems engineering effort in the Operate & Maintain phase?

• Hypothesis: Coverage of the Operate & Maintenance phases will broaden to model’s life cycle coverage

• POC: Ricardo Valerdi

10. Requirements volatility

• Central question: How do we quantify the effects of requirements volatility on systems engineering effort throughout the life cycle?

• Hypothesis: Requirements volatility is a significant factor for predicting systems engineering effort and can serve as a leading indicator for project success

• POC: Ricardo Valerdi• Feb 15, 2007 Workshop led by Rick Selby

– Identified critical success factors in: technical, product, process, people

– http://sunset.usc.edu/events/2007/ARR/presentations/RequirementsVolatilityWorkshopSummaryARR2007.ppt

– Loconsole, A., Borstler, J., “An industrial case study on requirements volatility measures,” 12th Asia-Pacific Software Engineering Conference, 2005.

Prioritization Exercise

• Factors to Consider– Availability of data– Impact on total cost of ownership– Frequency of use– Compatibility with other models (i.e., COCOMO

family, PRICE-H, etc.)– Addressal of future trends (Volatility, Uncertainty,

Scalability)– Factor interactions

Recommended Improvements(from user community)

1. Reuse (completed and approved for V2.0 baseline – see minutes from workshop at PSM User Conference)

2. Integration of SwE & SysE estimation3. Assumption of linearity in COSYSMO cost drivers4. Effect of cost drivers and scale factors5. Number of recursive levels of design6. Risk modeling (completed and approved for V2.0 baseline –

see minutes from workshop at PSM User Conference)

7. Establishing best practice guidance8. Consideration of SoS scope in COSYSMO9. Estimation in Operation & Maintenance Phase10. Requirements volatility

Joint meeting

Deferred

PriorityImprove-ment

1 2 3 4 5 6 7 8 9 10Availability of Data

M L L L H L

Impact on TOC

H H H H M M

Frequency of Use

H H H H H L

Compatible with Models

H L L M L M

Address Trends

L L L M M M

Factor Interactions

L H H L H L

Priority H M+ M M M H M+ L+ L L