why johnny can’t make good decisions and what we can do...

Why Johnny Can’t Make Good Decisions and What We Can

Do About It C. Robert Kenley

22nd Annual INCOSE International Symposium - Rome, Italy - July 9-12, 2012

Introduction •  The purpose of this panel is to illuminate

the importance and impact of early program decisions

•  My purpose is to illuminate the – Loss of focus in our community on making

good decisions early in a program – Evidence that we are poised to re-focus our

community on this important aspect of systems engineering

22nd Annual INCOSE International Symposium - Rome, Italy - July 9-12, 2012 2

A Digression: The Original Johnny and His Problem

1955: A problem is identified and a solution is proposed 1981: The problem persists


WHAT HAPPENED WHEN JOHNNY GREW UP AND BECAME A SYSTEMS ENGINEER?


1990: US DoD SE Process Includes Decision Making as a Key Step


DSMC, Systems Engineering Management Guide, January 1990

1991: Forsberg and Mooz Innocently Introduce the Vee-Model

The Original Vee-Model Accompanying SE Process Included Decision Making


Forsberg and Mooz, Proceedings of the First Annual Symposium of NCOSE, Chattanooga, TN, October 1991

Decision-Making Step

1991: Forsberg and Mooz Describe How the Vee-Model and the SE Process (with Decision Making) Fit Together


Forsberg and Mooz, Proceedings of the First Annual Symposium of NCOSE, Chattanooga, TN, October 1991

2000: Sheard Identifies Different Types Of Systems Engineering


Sheard, S. A., Proceedings of the Tenth International Symposium of INCOSE , Minneapolis, MN, July 2000.

Com

plex

ity

Life Cycle

Type 3 Approach Any kind of engineering

Type 2 Program SE Unprecedented Solutions

Type 1 Discovery Unprece- dented Problems

My Hypothesis •  The Program SE cadre implicitly adopted the Vee-Model as their

systems engineering process –  Sophisticated decision-making methods were not always needed and

were not highly valued –  Program SE developed and integrated components into unprecedented

solutions after the system requirements and solutions were defined –  Johnny had lots of fun doing other technical work that kept him busy

•  The Discovery cadre found employment elsewhere –  Sophisticated decision making for unprecedented problems was the

main thing –  Discovery SE produced the tradeoffs between problem solutions,

performance, costs, and schedule, and justified the selection of an solution to be developed by Program SEs

–  Jane (Johnny’s cousin) worked as a decision analyst for the Discovery SE industry and had lots of fun doing applied mathematics that kept her busy

9

Johnny’s Framework in One Chart


Occasional Decision Making When We Must

From Haskins, C., ed.. 2011. Systems Engineering Handbook: A Guide for System Life Cycle Processes and Activities. Version 3.2.1 Revised by M. Krueger, D. Walden, and R. D. Hamelin. San Diego: INCOSE

Jane’s Framework in One Chart (All Decision-Making, All the Time)

Known alternatives

Alternative Design

Multiple known alternatives

Multiple alternative designs

Strategy

National Policy

International Policy

None

Decision

Decision Maker & Stakeholder Interaction

Representatives

Single Decision Maker

External Decision Reviews

Decision Conference

Dialog Decision Process

Single objective

Multiple objective (Direct Assessment)

Multiple Objective Value Functions

Value-Focused Thinking

Value Preference

Importance Weights

Swing Weights

Swing Weight Matrix

Uncertainty

New Scenarios

Existing Scenarios

None Probability (Multiple Experts)

Probability (Single Expert)

None

No Weights

Risk Preference

Time Preference

None

Discounting

Single objective Utility

Expected Value

Multiple Objective Utility

Monte Carlo Simulation Decision Trees Influence Diagrams Bayesian Nets

Not recommended.

Strategy Generation Table

Decision Opportunity

Parnell, G. S., Decision Analysis in One Chart, Decision Line, Newsletter of the Decision Sciences Institute, May 2009 © Copyright 2011 Gregory S. Parnell. All rights reserved.

WHAT HAVE WE BEEN DOING ABOUT JOHNNY’S ABILITY TO MAKE DECISIONS?


2002: Plowman Promotes A Systems Engineering Process with Decision Making

Existing Technologies

Existing Designs

Existing Facilities

Customer/ User Needs

External Influences

• Product • Process • System •  Facility

Know What the Customer Wants and How the System Must Perform

(Mission / Requirements Analysis)

Program Requirements Rules, Orders,

Regulations, Standards Technical

Requirements

Plan and Integrate Your Work

(Systems Engineering Planning, Baseline Management,

Tradeoff Studies System Integration)

Verify the Preferred Solution

(System Verification)

Detailed Specifications

System Boundary SE Checklist

Customer Needs/Expectations List

System Requirements List (with Customer Approval)

Top-level Functional Architecture Alternatives Designs (e.g., sketches, designs, etc.)

Decision Criteria (“System has to ...” list with

measurable units)

Concept Matrix (Decision Criteria vs. Alternatives)

System Engineering Management Plan (SEMP) Document Control Log

Customer Requirements and System Requirements Documents

System Performance Measures Initial Program and Technical Interface Requirements

Formalized Functional Hierarchy Detailed Alternative Concepts

Decision Criteria System Functional Architecture Technical, Cost Baselines

Trade Studies/Analysis Plans and Results

System Requirements Review

System Requirements Baseline

Preliminary Design Review

System Baselines Revisions

Conceptual Design Review

System Baselines Revisions

Verification Matrix

Design Reviews

Systems Engineers at the Idaho National Engineering and Environmental Laboperform the following roles in support of Lockheed Martin Idaho Technologies Company and U.S. Department of Energy programs and projects:

¼ assess specific program and customer needs and expectations through interaction and facilitation of groups meetings and discussions

¼ implement an appropriate graded SE approach to meeting program and customer needs

¼ identify and manage system requirements, including the use of requirements management tools

¼ identify and develop system alternatives

¼ analyze system alternatives to ensure program requirements are met, including evaluating system risk, cost, and other performance measures using computer simulation models and other techniques

¼ provide “what-if” scenario modeling for sensitivity analysis of changes in program requirements and/or system variables

¼ document and validate analysis results

¼ support the decision making process

Test Plans

Systems Engineers at the Idaho National Engineering and Environmental Laboperform the following roles in support of Lockheed Martin Idaho Technologies Company and U.S. Department of Energy programs and projects:

¼ assess specific program and customer needs and expectations through interaction and facilitation of groups meetings and discussions

¼ implement an appropriate graded SE approach to meeting program and customer needs

¼ identify and manage system requirements, including the use of requirements management

tools

¼ identify and develop system alternatives

¼ analyze system alternatives to ensure program requirements are met, including evaluating system risk, cost, and other performance measures using computer simulation models and other techniques

¼ provide “what-if” scenario modeling for sensitivity analysis of changes in program

requirements and/or system variables

¼ document and validate analysis results

¼ support the decision making process

¼ support product development and implementation do

Determine Desired Functionality

to Meet Customer Needs (Functional Analysis)

Functional Hierarchy

Functional System Architecture

Study Various Options and

Determine a Preferred Solution (Alternatives Analyses / System Synthesis)

Analytical Data

Models & Simulations

Parametric Data

Ranked Decision Criteria

Trade Studies

Physical System Architecture

13 Buede, Forsberg, Mooz, Plowman, and Tufts. “Systems Engineering Body of Knowledge, Section II: Systems Engineering Processes”, INCOSE INSIGHT, April 2002

2007: Jain, Squires, Verma, & Chandrasekaran Indicate We Might Have A Problem in SE Education

“The research revealed that the following three core courses had a weak relationship or absence of any relationship with the other topical areas:

–  Quality, safety, and systems suitability –  Modeling, simulation, and optimization –  Decisions, risks and uncertainty

The most serious gaps were noticed between the above three core courses and the three specialized, elective courses below:

–  General project management –  Finance, economics, and cost estimation –  Organizational leadership”


Jain, Squires, Verma, and Chandrasekaran. A Reference Curriculum for a Graduate Program in Systems Engineering, INCOSE INSIGHT, July 2007

Mission Analysis / Strategic

Planning

Needs Assessment /

Functional Analysis

Concept Selection /

System Architecture

Project Planning

SE / VE Planning

Customer Concept of Operations

Requirements Development

Alternatives Evaluation / Synthesis

Detailed Design / Specifications

Monitor Development / Prepare for IV&V

Subsystem & Component Integration

System Verification

System Validation

Process Improvement

Modification Management

System Retirement or Replacement

Life Cycle Time Line

Phase -‐1

Preconceptual / Mission Analysis

Phase 0

Concept Explora:on, Benefits Analysis, and Project Planning

Phase 1

Tech. Planning / Concept of Opera:ons

Phase 2

System Defini:on, Design, and Development

Phase 3

System Integra:on, Implementa:on, Verifica:on &

Valida:on

Phase 4

Opera:ons, Maintenance, and Modifica:on Management

Phase 5

System Re:rement / Replacement

Dep

loym

ent

Rev. 5, May 6, 2009 INL/MIS-09-16067

Alternatives Development / Trade Studies

CD-0 CD-1 CD-2 CD-3 CD-4

Typi

cal

Pro

ject

Li

fe-c

ycle

Typi

cal

Dec

isio

n P

oint

s Sy

stem

s Eng

inee

ring

Fun

ctio

ns

Subsystem & Component Verification

Cross-Cutting Activities (Alphabetical)

Modeling & Simula:on

Human Systems Integra:on

Configura:on Management

Decision Analysis (e.g., Metrics)

Program / Project Integra:on (e.g., Lead SE)

Stakeholder Analysis / Elicita:on

Risk Management

System Life Cycle Management

Requirements Management

Facilita:on & Consulta:on / VE

Reliability, Availability, Maintainability, Inspectability

Decision Planning (e.g., Roadmapping)

2009: Idaho National Lab Reinvents the Vee-Model to Include Decision Making

15

2009: De Weck Invents a New Vee-Model for His Course Syllabus


Olivier de Weck, 16.842 Fundamentals of Systems Engineering, Fall 2009. (Massachusetts Institute of Technology: MIT OpenCourseWare), http://ocw.mit.edu (Accessed April 5, 2012). License: Creative Commons BY-NC-SA

2010: GRCSE Survey of Systems-Centric Master’s Programs Shows Decision Analysis as Visible But on the Fringe


Top$10$Courses*$Identified$within$the$Program!Introduction$to$SE! 56%!Modeling$and$Simulation! 53%!Project$Management! 38%!Systems$Architecture$and$Design! 38%!Systems$Integration! 34%!Systems$Analysis! 25%!Systems$Management! 25%!Systems$Requirements$Analysis! 25%!Risk$and$Decision$Analysis! 25%!Probability$and$Statistical$Analysis! 19%!

!Graduate Reference Curriculum for Systems Engineering (GRCSE), version 0.5, December 2011

2011: INCOSE Decision Analysis Working Group Chartered

INCOSE�Decision�Analysis�Working�Group�(DAWG)�Charter�

3 of 3

9 RESOURCE�REQUIREMENTS��TPP�will�be�submitted�for�specific�projects�as�required.�

10 DURATION�This�Charter�will�remain�in�effect�until�rescinded�by�the�signatory.��

11 SIGNATURES�

Chair:�Frank Salvatore� � � � � � Date:�2011Ͳ03Ͳ12�

� � � � � � Date� � August�23,�2011�

Technical�Director,�INCOSE��

Chairman,�INCOSE�Board�of�Directors�� Date�

�

Revision�History��

Date� Revision� Description� Author�

2001Ͳ03Ͳ12� 1.0� Initial�Draft.� Frank�Salvatore�

� � � �

� � � �

� � � �

�


2012: 31 Papers Self-Identified as Relating to Decision Analysis / Management at this Symposium


0" 5" 10" 15" 20" 25" 30" 35" 40" 45" 50"

SEE09"Transi1on"/"Opera1on"/"Maintenance"

SEE18"Environmental"Compa1bility"

SEE17"Logis1cs"/"Supportability"

SEE12"Cause"Analysis"

SEE16"Reliability"/"Availability/Maintainability"

SEE20"Resource"Management"

SEE21"Safety"/"Security"

SEE02"Complexity"Science"

SEE03"Systems"Dynamics"

SEE22"LifeKcycle"Cos1ng"/"Economic"Evalua1on"

SEE23"Acquisi1on"/"Supply"

SEE19"Human"Factors"/"Human"System"Interface"

SEE15"Measurement"

SEE14"Configura1on"/"Informa1on"Management"

SEE29"Other"SE"Enablers"

SEE08"Verifica1on"/"Valida1on"

SEE04"Systems"Science"

SEE13"Risk"/"Opportunity"Management"

SEE07"System"Integra1on"

SEE25"Systems"of"Systems"(SOS)"

SEE05"Requirements"Elicita1on"and"Management"

SEE26"Modelling"and"Simula1on"

SEE27"Teaching"and"Training"

SEE11"Decision"Analysis"/"Management"

SEE10"Project"Planning"/"Assessment"/"Control"

SEE24"MBSE"

SEE01"Systems"Thinking"

SEE06"Architectural"Design"

SEE28"Processes"

Bergsjö. Papers and Posters for the 2012 International Symposium: A Great Variety of Topics, INCOSE INSIGHT, April 2012

Number 6 on the Pareto chart!

Conclusion •  “Industry will always be ahead of academia in systems engineering”* •  Industry is sending signals that there is a need for better decision

making •  Academia needs to step up their game

–  Recognize the need in curriculum and course development –  Provide training that is applicable for both Discovery and Program SE

environments •  System acquirers and producers need to step up their game

–  Recognize that Discovery SE and Program SE both exist –  Support better decision-making in their both environments –  Incentivize better decision making


* Gerard Voland, Former Dean, College of Engineering, Technology, and Computer Science, Indiana-Purdue Fort Wayne, in a talk given at the INCOSE Crossroads of America Mini-Conference, October 2004