cots integration estimation-erp.pdf

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Headquarters U.S. Air Force

COTS Integration Estimation: Enterprise Resource Planning Systems

1

Dr. Wilson Rosa

James Bilbro (Cirrus/JBCI)

USC CSSE Annual Research Review 2012

March 6, 2012

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Outline

ERP Overview

Data Collection and Analysis

Estimating Models

The Way Forward

2

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Headquarters U.S. Air Force

3

ERP Overview

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

What is ERP?

4

Enterprise Resource Planning (ERP) systems are typically

Commercial-Off-The-Shelf (COTS) software applications

that integrate an organization’s core business functions

around a unified data base.

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

How complex is an ERP

Implementation?

5

While an ERP system can fall under the category of

Major Automated Information System (MAIS) it is NOT

simply just another IT system!

Treating an ERP system as though it were just

another IT system is guaranteed to result in significant

cost and schedule overruns!

Implementing an ERP system involves a complex mixture of

activities that are impacted by:

business processes

organizational makeup and culture

hardware, and software development and modification

And much, much more ---

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

How complex is an ERP Implementation?

6

23

A Spectrum of COTS-Based Systems

Multiple products from multiple suppliers integrated to

collectively provide system functionality

• probably more flexible in supporting end user/business processes

• project maintained• integration, engineering focus

­products/parts are “black boxes”­COTS, NDI, legacy

COTS-Intensive

SystemsCOTS-Solution

Systems

One substantial product (suite) tailored to provide significant

system functionality

• generic solutions; tightly coupled to end user/business processes

• vendor maintained • tailoring, parameterization focus

Slide Presented at SERC COTS/NDI T&E Workshop, 15 January 1999 by

Software Engineering Institute (SEI)

ERPs are

COTS-Solution

Systems….

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

How is ERP implemented?

Business processes are automated via an integrated

COTS software application:

7

Integration is typically

done by a 3rd Party Vendor

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

What is the problem?

8

Exceeds Schedule 61.10%

Exceeds Budget 74.10%

2010 Results of Industry Survey of 185 Implementers

Industry as well as DoD has significant cost and

schedule problems in implementing ERP systems!

All Major DoD ERP Programs

Have Exceeded Original Cost

and Schedule Estimates by

more than 30%!

As of Dec. 2009, DoD had

invested $5.8B in their ERP

programs!

DoD Industry-wide

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Reasons for Cost Growth

AFCAA Root Cause Analysis

1. Estimation: A lack of understanding of the new technology

and business environment led to the use of inadequate cost

models and dubious estimating methods

2. Schedule: limited budgets have forced decision makers to

extend the period of performance of “Level of Effort” related

tasks – Civilian, Contractor, and Military FTEs

3. Engineering: Inexperience with Oracle/SAP Customization

has led to underestimation of requirements. Difficulty

changing business processes to match ERP processes

4. Quantity: war-fighter need has led some program offices to

reassess user and implementation requirements

9

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

System Developer

38%

Deployment Hardware and

Software7%

Program Office

9%

Site Activation, User Training,

Data Migration18%

Interim Logistics Support

28%

Impact of Schedule Overruns? Extends “Standing Army” Costs!!!

10

Impact of “Standing Army” Costs (Major DoD ERPs)

~55% of Total Development

~1-10$M / Month Overrun

LEGEND

“Standing Army” Costs

Final Project Product

Standing Army -- Level of Effort activities… not itself a work item directly associated with accomplishing the final project

product, service or result, but rather one that supports such work, its duration is based on the duration of the discrete work

activity it is supporting

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Headquarters U.S. Air Force

Data Collection and Analysis

11

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Instrumentation and Dataset

Dataset: Collected from 22

projects government-wide

including 19 programs

from DoD

Instrument: A modified

version of the Software

Resources Data Report

(SRDR) questionnaire* New fields were added to collect

data on infrastructure -- operational

sites, users, software licenses,

servers, tailored objects, CPUs, etc.

12

*Defense Cost and Resource Center (DCARC)

A.

1. System/Element Name (version/release): 2. Report As Of:

3. Authorizing Vehicle (MOU, contract/amendment, etc.): 4. Reporting Event:

Submission # _1____

(Supersedes # __N/A__, if applicable)

5. Name of Development Organization: 8. Lead Evaluator of CMM Certification:

5a. Address of Development Organization: 9. Affiliation to Development Organization:

5b. POC Name:

POC Phone Number:

POC Email:

10. Precedents (list up to ten similar systems completed by the same organization or team):

11. Comments on Part A responses:

B.

13. Peak staff (maximum team size in FTE) that worked on and charged to this project:

14. Percent personnel that was: Highly experienced in domain:X% Nominally experienced:X% Entry level, no experience:X%

15. Comments on Part B responses:

C.

24. Comments on Part C responses:

DD Form 2630-3 Page 1 of 4

22. Legacy System Phase-Out: Number of external legacy systems (i.e., not under project control) expected to be shutdown

whenever the ERP system becomes fully operational. Not planned for migration.

23. Legacy System Migration: Number of external legacy systems (i.e., not under project control) expected to migrate into the

new ERP system.

17. ERP Modules: Standard business modules required by the customer. These standard modules are captured in the ERP

Tool (e.g., SAP, ORACLE 11i, PeopleSoft). Examples of these include Accounts Receivable, Accounts Payable, General

Ledger, etc. Please also provide the list of modules used by the product.

18. Business Processes: Number of processes required by the customer. Processes fall under Business Modules --

System's Process Flow Diagram Level 2. The following example shows three processes required for the Accounts Receivable

business module:

19. Business Sub-processes: Number of sub-processes required by the customer. Sub-processes fall under business

processes -- System's Process Flow Diagram Level 3. The following example shows three sub-processes required for the

________ business process:

20. Functional Requirements: Number of software functional requirements needed by the customer. Functional

Requirements fall under business sub-processes -- System's Process Flow Diagram Level 4. This information is also found

in the System Requirement Specification (SRS). The following example shows a functional requirement:

21. Legacy System Interfaces: Number of external legacy systems (i.e., not under project control) expected to interface with

the delivered ERP system

12. Project Functional Description:

Project RequirementsProvide Actuals

at Final Delivery

16. Business Modules: Number of business modules required by the customer. This information is found in the System's

Process Flow Diagram Level 1 (PFD Level 1). Examples of these include Accounts Receivable, Accounts Payable, General

Ledger, etc. Please also provide the list of business areas required by the product.

Description of Actual Development Organization

6. Certified CMM Level

(or equivalent):

7. Certification Date:

Product and Team Description

ERP Resources Data Report: Final Developer Report

Page 1: Report Context, Product Description and Project Requirements

Report Context

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Cost Drivers (Functional)

Name Appropriateness Definition

RICE+ Inception or

Elaboration Phase

(Milestone B)

Sum of Reports (R), Conversions (C), Interfaces (I), and Extensions

(E+). Those objects within the ERP Software requiring customization,

as these are not compliant with user requirements

Reports (R): Number of formatted and organized presentations of data

Interfaces (I): Number of boundaries across which two independent

systems meet communicate with each other.

Conversion (C): Number of processes that transfer or copy data from an

existing system to load production systems.

Extension (E+): Number of programs that are in addition to an existing

standard program. Extensions are additional programming functionality.

Sometimes referred to as “Enhancements

Functional

Requirements

Early

Inception Phase

(Milestone A)

Functional requirements are defined as a detailed breakdown of business solution

outcomes in terms of the intended functional behaviors of the software

application.

System Interfaces Product Start

(Pre-Milestone A)

Number of external legacy systems (i.e., not under project control) expected to

interface with the delivered ERP system.

Migrated Systems Migration of a legacy or existing application to a new operating environment. It

could also mean to replace a legacy system, moving the data from the legacy

application to the new application while preserving data integrity.

13

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Cost Drivers (Non-Functional)

14

Infrastructure: Servers, CPUs, etc.

Operating Locations

Business: users, processes

Software: applications, licenses

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Cost Driver Selection:

Pair-wise Correlation Results:

RICE is strongly correlated to COTS Integration effort

Requirements is correlated to COTS Integration Effort

However, System Interfaces and Migrated Systems are poorly

correlated to COTS Integration Effort

Recommendation:

Use RICE as software size variable at Elaboration Phase

Use Requirements as software size variable at Inception Phase

15

EFFORT

EFFORT 1

Requirements 0.718

System Interfaces 0.296

Migrated Systems 0.471

RICE 0.918

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Distribution of Total ERP

Implementation Cost:

16

COTS Integration

Support21%

COTS Integration

Effort17%

Interim Logistics Support

28%

Site Activation, User Training,

Data Migration18%

Program Office9%

Deployment Hardware &

Software7%

Influenced by Non-Functional Cost Drivers

Influenced by Functional Cost Drivers

LEGEND

Only 17-38% of Total Cost is captured by Parametric Cost Models

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

COTS Integration Productivity Hours/Tailored Object (“RICE”)

Productivity influenced by Business Area (Finance vs. Supply Chain)

Productivity NOT influenced by ERP COTS Tool (SAP vs. Oracle)

0

200

400

600

800

1000

1200

1400

1600

1800

Supply Chain Finance

Ho

urs

/RIC

E

Business Area

Productivity Comparison by Business Area

17

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

COTS Maintenance Staff (FTEs): System Developer Only

18

43%

34%

21%

16%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

Post Go-Live Year 1

Post Go-Live Year 2

Post Go-Live Year 3

Post Go-Live Year 4

Staf

fin

g Le

vel

Post Go-Live System Developer Staff (As Percent of Go-Live Year Staff)

Predicts the Annual Maintenance Staff as a Percent of Go-Live Year Staff

Sustaining Engineering

System Operations

Software Maintenance

Data collected from 5 programs

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Headquarters U.S. Air Force

19

Estimation Models

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

COTS Integration:

COTS Integration Effort

Glue Code Assessment Tailoring

20

Scope:

1. effort for assessing and tailoring COTS software applications or modules …

2. all necessary labor and material …for analyzing, designing/building/configuring,

and testing the required business objects [within the ERP COTS application] –

reports, interfaces, conversions, extensions,…, scripts, enhancements…

**MIL-STD-881 C Reference: K.4.2

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

COTS Integration Effort: As a Function of Software Requirements

21

UNIT

EQUATION

METRICS

R2 PRED

(25)

CV

(%) N

Hours 218.9 * Requirements 65% 23% 90.4 17

Application

Suitable for programs at “Product Start” or “Early Inception” Phase

Includes Financial and Supply Chain ERPs (150 – 5500 Requirements)

Limitation

Dataset only captures Government Sector ERPs, SAP/ORACLE/MOMENTUM Tools

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

COTS Integration Effort: As a Function of Tailored Objects (“RICE”)

22

UNIT

EQUATION

METRICS

R2 PRED

(25)

CV

(%) N

Hours 204.4*Report + 2021*Interface + 2713*Conversion + 496.8*Extension 82% 32% 47.8 20

Hours 922.2 * RICE 83% 58% 63.8 22

Application

Includes supply chain and Finance ERPs

Limitation

Dataset only captures Government Sector ERPs, SAP/ORACLE/MOMENTUM Tools

Note: Reports may include Forms, and Extensions may include Workflow and Bolt-ons

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

COTS Integration Schedule: As a Function of Size and FTEs

23

UNIT

EQUATION

METRICS

R2 PRED

(25) CV(%) N

Months

5.995*RICE0.4621 / FTE0.3332 79% 71% 32.0% 14

Months REQ0.6369 / FTE0.3844 98% 60% 43.2% 15

FTE = Full Time Equivalent Staff; RICE = Report, Interface, Conversion, Extension; REQ = number of software

requirements

Application

Dataset Range (15-1500 RICE; 150 – 5500 Requirements)

Limitation

Dataset captures System Developer and Government Staff

Dataset only captures Government Sector ERPs

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

COTS Integration Support:

COTS Integration Support

Systems Engineering

Program Management

Change Management

System Test & Evaluation

Training Development

24

**MIL-STD-881 C Reference: L.6

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Systems Engineering…Cost As a Function of COTS Integration Cost

25

WBS Element(s) UNIT

EQUATION

METRICS

R2

PRED

(25)

CV

(%) N

System Engineering

Program Management

Change Management

BY10$K 1.347 * COTS Integration Cost 82% 50 55.0 16

Application

Dollars in Thousands, Base Year 2010 (BY10$K)

COTS Integration Range: $2,000K (LOW), $200,000K (HIGH)

Limitation

Dataset only captures System Developer Cost

Dataset only captures Government Sector ERPs

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

System Test & Evaluation Effort As a Function of Software Requirements

26

WBS Element UNIT EQUATION (MEAN)

Descriptive Statistics

Median StdDev CV(%) Lower

Bound

Upper

Bound N

Development Test

& Evaluation Hours 137.5 * REQ 93.6 48.5 71.7 72.1 202.9 18

REQ = number of software requirements

Application

Dataset Range (150-5500 Requirements)

Only captures Development Test & Evaluation

Limitation

Dataset captures System Developer Staff

Dataset only captures Government Sector ERPs, Supply Chain and Finance

Excludes Operational Test & Evaluation

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

System Test & Evaluation Effort As a Function of Tailored Objects (“RICE”)

27

WBS Element

UNIT

EQUATION

METRICS

R2 PRED

(25)

CV

(%) N

Development Test &

Evaluation Hours 437.1 * RICE 72% 6% 77.9 18

RICE = sum of reports, interfaces, extensions, conversions

Application

Dataset Range (15-1500 Requirements)

Only captures Development Test & Evaluation

Limitation

Dataset captures System Developer Staff

Dataset only captures Government Sector ERPs

Excludes Operational Test & Evaluation

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

System Test & Evaluation Schedule As a Function of Size and FTEs

28

WBS Element UNIT EQUATION

METRICS

R2 PRED

(25)

CV

(%) N

Development Test &

Evaluation Months REQ0.6315 / FTE0.4945 96% 55 47.4 12

FTE = Full Time Equivalent Staff; REQ = number of software requirements

Application

Dataset Range (150-5500 Requirements)

Only captures Development Test & Evaluation

Limitation

Dataset captures System Developer Staff

Dataset only captures Government Sector ERPs

Excludes Operational Test & Evaluation

I n t e g r i t y - S e r v i c e - E x c e l l e n c e 29

The Way Forward

I n t e g r i t y - S e r v i c e - E x c e l l e n c e

Proposed Research Initiatives

Calibrate COCOTS to address

ERP Systems using AFCAA Data

Add research results into the

TruePlanning® framework

Add dataset to Data-Manager

and calibrate SLIM-Estimate

30

SLIM-Estimate™