benefit study on digital transformation for project …...benefit study on digital transformation...

National Aeronautics and Space Administration

Benefit Study on Digital Transformation for Project Planning &

Control ProcessNipa Phojanamongkolkij and Rob Moreland

August 2018

1

National Aeronautics and Space Administration 22

Background and Motivation (1 of 2)Current collaborative network

• Functional data system (FDS, the disc) for each function. • The project’s centralized Technical Data Management System

(TDMS, the yellow disc in the center) used mainly for an archival purpose.

• Coordinative activities (a.k.a. the data collection and preparation activities)

• Collaborative and innovative activities (the solid linkages) for problem solving, brainstorming, or creating alternative solutions.

EPIIC’s Vision(Evolving Project Integration, Innovation & Collaboration)

• FDS for each function for uncontrolled and explorative-in-nature models or documents uniquely pertaining to its function.

• The project’s centralized Digital Data Management and Analytics (DDMA) containing a true source of project data and enabling collaborative and innovative activities.

• The DDMA includes embedded intelligence, relying upon statistical engineering, machine learning and data analytics, etc. to identify and anticipate trends within the project.

2Figure 1. Current collaborative functional network for project management Figure 2. Future collaborative functional network for project management


Background and Motivation (2 of 2)• This Discrete Event Simulation (DES) model is to characterize

the potential benefits of the EPIIC’s vision with respect to the current collaborative network.

– Specifically, the model will characterize the % time reduction for collecting data, requesting data, reconcile inconsistent data.

• Currently, PP&C process is far-well developed than other functions. As such, the DES model will be demonstrated with the PP&C process.

Current collaborative network EPIIC’s Vision


Modeling Process Flow (1 of 3)

Inputs and Outputs by each activity

Inputs (Rows) by Activities (Columns)

Outputs (Rows) by Activities (Columns)

4

92 data (combined inputs and outputs) and 48 activities

See Appendix A for an example.

Link to the PP&C Handbook:https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20160013229.pdf

https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20160013229.pdf



Derive the sequence of activities for DSM analysis

5

See Appendix B for examples of inconsistent data in the

handbook.



Outputs (Rows) by Activities (Columns)

Inputs (Rows) by Activities (Columns)

VBA script (MakeDSM.xlsm)

Rearrange Sequence of Activities based on

PP&C Handbook

Planning Activities

Control Activities

Simulation Model2

1

3

4

4

6


The Sequence of Planning ActivitiesNetwork Diagram View

7

17

Start

14

19

11,12

5,6,7

20

3,4

21,22,23,24

1

2

1516

9

13

End

18

18

18

18

1810

Activity Name Phase ACT_ID Function_AliasDevelop Acquisition Plan Planning ACT_01 ACM

Support Establishment of Contracts Planning ACT_02 ACMDevelop CM/DM plans Planning ACT_03 CMIdentify items to be under configuration control Planning ACT_04 CM

Prepare cost analysis strategy Planning ACT_05 Cost

Execute cost assessment tasksPlanning ACT_06 Cost

Execute cost estimating and supplementary analytical tasks Planning ACT_07 CostConduct a Joint Confidence Level (JCL) analysis Planning ACT_08 CostPresent analyses to stakeholders Planning ACT_09 Cost

Capture Project Scope of Work Planning ACT_10 PP&C Int.Define Approach/Strategy to Executing Scope of Work (includes Acquisition Strategy) Planning ACT_11 PP&C Int.Provide Guidance, Decisions, Adjustments Planning ACT_12 PP&C Int.Refine Plan Planning ACT_13 PP&C Int.

Activity Name Phase ACT_ID Function_AliasDevelop Plans for Resource Management Planning ACT_14 ResourceDevelop Project Cost Budget based on Available Obligation Authority Planning ACT_15 ResourceImplement EVM System Planning ACT_16 ResourceIdentify Initial Risk Planning ACT_17 RiskExecute an initial Risk Informed Decision Making (RIDM) iteration as a part of project formulation Planning ACT_18 RiskDevelop a Risk Management Plan (RMP) that includes a definition of a Continuous Risk Management (CRM) process Planning ACT_19 RiskDevelop a strategy for schedule estimation and assessment Planning ACT_20 Scheduling

Develop schedule Planning ACT_21 SchedulingAssess & Analyze Schedule Integrity Planning ACT_22 SchedulingValidate schedule consistency with cost & labor plans (if not RLS) Planning ACT_23 Scheduling

Baseline schedule estimate Planning ACT_24 SchedulingProvide JCL Schedule Planning ACT_25 Scheduling

8

25


Discrete Event Simulation View

8

IN (Inputs) 92 data by 25 activitiesOUT (Outputs) 92 data by 25 activitiesDataOwner 92 data ownershipActivityOwner 25 activity ownershipRequest data time duration

Triangular with (min, mode, max) (95%, 1, 110%)

Revise data time duration (95%, 10, 110%)

Execute activity time duration (95%, 20, 110%)

Minimum = 95% of modeMaximum = 110% of mode

More details in modeling in Appendix C


Lower Level, Re-Useable Model

9

Execute Activity

Request Input Data

Generate Output Data

EPIIC


Preliminary Results

• Primary Objective:– To quantify benefit of the EPIIC’s w.r.t. the Current Collaborative

Network by means of:• Benefit in terms of % improvement in total time duration• Sensitivity analysis of the benefit on model assumptions

Ø Stakeholder: OCE PPM Strategic InitiativeØ Lower fidelity model may be sufficient for “relative benefit” of

strategic planning.

• Secondary Objectives: – To gain understanding of process dynamics– To identify process improvement opportunities for the Current

Collaborative Network Ø Stakeholders: PP&C Community, OCE PPM Policy, Practice, and

Development DivisionØ Higher fidelity model is preferred for operational/tactical planning.

10


Primary Objective (1 of 8)Boxplot to represent Monte Carlo Simulation result for total time duration for single-event-driven activities (e.g., gate review), and not including any monthly re-planning activities.

11

IN (Inputs) 92 data by 25 activitiesOUT (Outputs) 92 data by 25 activitiesDataOwner 92 data ownershipActivityOwner 25 activity ownershipRequest data time duration

Triangular with (min, mode, max) (95%, 1, 110%)

Revise data time duration (95%, 10, 110%)

Execute activity time duration (95%, 20, 110%)

95% Confidence Interval (CI) on the average.


Primary Objective (2 of 8)

• Determine number of replications– Curve shows relatively the half-width of the Confidence Interval (CI)

converging as number of replications increases. At some points, there is no Return-On-Investment on the half-width for increasing replications.

12


Primary Objective (3 of 8)• Low-fidelity estimate of the EPIIC’s vs. Current Network

shows ~ 10% total time improvement** from (1-10-20)* to (0-10-20).

13

** Based on single-event-driven activities (e.g., gate review), and not including any monthly re-planning activities.* (Request-Revise-Execute)

(1-10-20) (0-10-20)

95% Confidence interval on the average total duration, based on 1,000 replications.



• Sensitivity analysis on modeling assumptions

• Total of 8 simulation scenarios:– 4 Current Network scenarios (each with 1,000 replications) for all

assumption combinations (2 distributions x 2 ProcTimeSets)– 4 EPIIC’s scenarios

• Computed (post-processed) % Improvement14

Assumption (Variable) Case 1 Case 2Distribution Triangular Uniform

Time Durations (or ProcTimeSet)(Request-Revise-Execute)

(1-10-20) (1-5-40)

Minimum = 95% of modeMaximum = 110% of mode

MaximumMinimum



• Sensitivity of % Improvement to modeling assumptions

15

Statistically sensitive* (P-value <<< 0) to the time durations used.

Practically insensitive to either uniform or triangular distribution (P-value ~ 0.02).

* Overlapping the CI indicates statistically insensitive.

** No statistical evidence that % improvement depends on the combination of both duration and distribution (a.k.a., interaction effect).



• Sensitivity analysis to time durations (ProcTimeSetvariable in the previous analysis)

• Total of 40 simulation scenarios:– 20 Current Network scenarios (each with 1,000 replications) for all

assumption combinations (4 Revise x 5 Execute)– 20 EPIIC’s scenarios

• Computed (post-processed) % Improvement

16

Variable Case 1 Case 2 Case 3 Case 4 Case 5Request Time 1Revise Time(ReviseTime)

1 (R1)

3 (R3)

5 (R5)

10 (R10)

Execute Time(ProcTime)

10 (P10)

20 (P20)

40 (P40)

80 (P80)

100 (P100)

Distribution Triangular



17** Based on single-event-driven activities (e.g., gate review), and not including any monthly re-planning activities.

• Sensitivity of % Improvement** to Time Duration Assumptions

• % Improvement decreases as both revise and execute times increase and eventually converges to a minimum region.

• With the request time of 1 unit, an operating region with 6-8% improvement** (the gray rectangular region) can be expected for revise time is in the range of (5, 8) and execute time is (40, 100). Other regions can also be determined from the 2D contour plot on the left.


Primary Objective (8 of 8)• Theoretical Minimum % Improvement**


• Determine the theoretical minimum % improvement by increasing the Execute Time from 100 (P1e2) to 10,000,000 (P1e7) units with Revise Time set at 20% and 100% of the Execute Time.

• % Improvement decreases as both revise and execute times increase and eventually convergesto a minimum region.

• As the Revise and Execute Times approach a significantly large number (w.r.t Request Time), there is still an average benefit (~3.7%) of the EPIC’s over the Current Network.

• There is a benefit (> 0%) with 98% confidence (in other words, 98% of all 12,000 replications show > 0% improvement), with the worst case of -1% benefit (Current is better than EPIC’s).


Secondary Objectives (A preliminary example)

• Identify function’s level of effort (LoE) • Probable opportunities to

– reassign activity owners to balance workload;– have 1 FTE performing multiple functions;– or have 1 FTE performing the same function for multiple projects.

19

PPC Int. has the highest workload than other functions.


Total Level of Effort (LoE) Cost• Low-fidelity estimate of the EPIIC’s vs. Current Network shows ~ 7%

total cost improvement** from (1-10-20)* to (0-10-20).

20

** Based on single-event-driven activities (e.g., gate review), and not including any monthly re-planning activities.* (Request-Revise-Execute)

Staffing Cost Assumptions:Function Annual Cost % Fringe

BenefitPP&C $117K

(GS-14 Step 5)30%

Cost Analyst $99K (GS-13 Step 5)

30%

Risk Manager $180K (WYE) -

Scheduler $160K (WYE) -

Resource Mgmt(EVM)

$160K (WYE) -

CDM $160K (WYE) -

Acquisition & Contract Manager

$160K (WYE) -

(1-10-20) (0-10-20)


Repeat RSM for Cost Improvement


• Sensitivity of % Improvement** to Time Duration Assumptions

• % Improvement decreases as both revise and execute times increase and eventually converges to a minimum region.

• With the request time of 1 unit, an operating region with 4-6% improvement** (the gray rectangular region) can be expected for revise time is in the range of (5, 8) and execute time is (55, 75). Other regions can also be determined from the 2D contour plot on the left.

Execute Time

Revise

Time

% Im

provement

4

6

8

10

12

14

16

Execute Time

Rev

ise

Tim

e

4

6

8

10

12

14

20 40 60 80 100

24

68

10

2

4

6

8

10

12

14

16

18

20

22

24


Theoretical Minimum Cost Improvement

• Theoretical Minimum % Improvement**


• Determine the theoretical minimum % improvement by increasing the Execute Time from 100 (P1e2) to 10,000,000 (P1e7) units with Revise Time set at 20% and 100% of the Execute Time.

• % Improvement decreases as both revise and execute times increase and eventually converges to zero.

0.0

0.5

1.0

1.5

2.0

2.5

E1e2 E1e3 E1e4 E1e5 E1e6 E1e7ExecuteTime

% Im

prov

emen

t

ReviseTime 20% 100%


Future Works

• Increase model fidelity• Secondary Objectives

• Refine modeling assumptions • Add iterative monthly re-planning cycles• Perform process improvement through DSM analysis

and simulate the proposed process to quantify any benefit

• Primary Objective can be repeated with the higher fidelity model

• Extend beyond PP&C (e.g., Systems Engineering and Safety and Mission Assurance functions)

23


Acknowledgments

• Donald E. Shick, Manager, Program Planning & Control Capability Office (PCO), OCFO, NASA Langley Research Center

• Nani Tosoc, Cost Lead, Proposal Development Office, NASA Langley Research Center

24


BACK-UP

25


How sensitive the benefit is to % fringe benefits? Both the delta and the current cost increase proportionally causing the % to be ~7%.

26

6.0

6.5

7.0

7.5

8.0

30% 40% 50% 60% 70% 80% 90% 100%Fringe Benefit

Perc

ent I

mpr

ovem

ent

Percent LoE Cost Benefit vs. Fringe Benefit


The cost benefit decreases as fringe benefit increases due to complex cost benefit relationship.

• Only PP&C and Cost Analyst functions drive the change in % benefit as fringe changes.

• Let:a = delta (between the current and EPIIC) in LoE in hours for PP&C functionb = delta (between the current and EPIIC) in LoE in hours for Cost Analyst functionc = the current LoE in hours for PP&C functiond = the current LoE in hours for Cost Analyst functionC1 = constant delta cost (between the current and EPIIC) for all other 5 functionsC2 = constant current network’s cost for all other 5 functions$_PPC_f = LoE hourly cost at f % fringe for PP&C function$_Cost_f = LoE hourly cost at f % fringe for Cost Analyst function

Function Annual Cost % Fringe Benefit

PP&C $117K (GS-14 Step 5)

30%

Cost Analyst $99K (GS-13 Step 5)

30%

Risk Manager $180K (WYE) -

Scheduler $160K (WYE) -

Resource Mgmt(EVM)

$160K (WYE) -

CDM $160K (WYE) -

Acquisition & Contract Manager

$160K (WYE) -

%Benefit at 30% fringe = !"#$%&'((")$ *%+"#,)" =

%∗$_11&_2345∗$_&6+$_234&78∗$_11&_2349∗$_&6+$_234&:

%Benefit at 100% fringe = %∗$_11&_;3345∗$_&6+$_;334&78∗$_11&_;3349∗$_&6+$_;334&:

benefit study on digital transformation for project …...benefit study on digital transformation...

Documents