War-winning Capabilities…On Time, On Cost

Acquisition Cost Acquisition Cost

Estimating UpdateEstimating Update

28 Feb 0828 Feb 08

Ken Kennedy, AAC/FMC Kristy Golden, AFCAA/FMAE

Air Armament CenterAir Armament Center

DISTRIBUTION STATEMENT A: Approved for public release; distribution is unlimited.1

2

War-winning Capabilities…On Time, On Cost

Outline

• Estimating Trends• Leadership Concern• Congressional Reaction• Root Cause Analysis• 2007/2008 Initiatives • AAC Cost Demand• AFCAA Standup• Industry Help• Summary

3

War-winning Capabilities…On Time, On Cost

3

Estimating Trends

• GAO-07-406SP Defense Acquisitions Assessments of Selected Weapon Programs– GAO assessed 62 weapon systems with a total

investment of over $950 billion, some two-thirds of the $1.5 trillion DOD plans for weapons acquisition

– “Fully mature technologies were present in 16 % of the systems at development start” - the point at which best practices indicate mature levels should be present.

– Programs that began development with immature technologies (84%) experienced a 32.3 percent cost increase, whereas

– Those that began with mature technologies (16%) increased just 2.6 percent.”

4

War-winning Capabilities…On Time, On Cost

4

Leadership Concern

• Sue Payton: Jane's Defence [sic] Weekly, 07/17/2007– Reduce pressure on budget by improving methods for

calculating the overall cost of acquisition programme [sic]

– Stop taking contractors' one-off cost estimates at face value

– Develop own methods for calculating the true, long-term cost of a weapon system

– Tendency to highlight the basic price tag of a weapon system and play down the much bigger 'lifecycle cost

– Wait until a preliminary design review has been completed, rather than trying to guess a weapon's price before the final requirements are defined

– Push contractors to build more prototypes … important insight into future development costs

5

War-winning Capabilities…On Time, On Cost

5

Congressional Reaction

Nunn-McCurdy Breach Thresholds• 2006 Defense Appropriations Act (Public Law No:

109-148)

• Requires Increased emphasis on – Accurate TRL assessments– Quantification of risk and uncertainty – Optimal decisions within trade space constraints– Better cost and schedule forecasts– Budgeting to Independent Cost Estimates

Type Breach Notification CertificationInitial APB 30% 50%Current APB 15% 25%

Growth Over Baseline

6

War-winning Capabilities…On Time, On Cost

Root Cause

LowMed-LowMedMed-HighHigh

Cost risk levels

Extremely High

Range of Missile Program Cost Outcomes

TRL Level

AOA Risk Reduction SDD LRIP FRP

1 2 3 4 5 6 7 8 9

Risk is retired as a program progresses and technology matures

LowMed-LowMedMed-HighHighExtremely

High

We are too optimistic about risk/uncertainty

Ris

k/U

ncer

tain

ty

High

High

Med

Med

Low

7

War-winning Capabilities…On Time, On Cost

2007/2008 Initiatives

• Better Risk ID and Modeling

• Continued Sufficiency Reviews

• New Source Selection Focus

• More Contract Risk Sharing

8

War-winning Capabilities…On Time, On Cost

• Integrate Probability of Consequence Screening (P/CS) “risk cube” and Probability of Program Success (PoPS) assessment into cost estimating process – Existing infrastructure tools– Realistic and consistent approach– Results in a “time now” WBS risk rating

• Use Monte Carlo simulation with WBS correlation to define total program cost – Present decision maker with better picture of

uncertainty based on knowledge today

Better Risk ID and Modeling

9

War-winning Capabilities…On Time, On Cost

• Cost “gray beard” review – Reinforce estimating best practices– Verify data sources and methodology– Suggest alternate benchmarks– Request appropriate SME inputs

• Independent technical assessment• Independent schedule assessment

Continued Sufficiency Reviews

10

War-winning Capabilities…On Time, On Cost

• Cost risk # 1 evaluation priority– High, medium, or low rating based on

percent difference between proposed contractor cost and government estimate of offeror’s approach

• Requires clear communication– Full data source ID and extract for element

basis of estimates (BOEs)– Full data source adjustment justification in

element BOEs– Full discussion of risk and uncertainty in

element BOEs

New Source Selection Focus

11

War-winning Capabilities…On Time, On Cost

More Contract Risk Sharing

• Solicitation and proposal– Request WBS element uncertainty/risk and $

magnitude as part of each narrative basis of estimate worksheet

– Request sum of element risk $ to be included as a separate proposal line

• Evaluation– Description of element uncertainty used to

model risk for independent estimate• Execution

– Proposal risk line becomes earned value MR– SOO requirement for discussion before use

Air Force Cost Analysis Agency (AFCAA) Standup

Organization Headquarters

• Deputy Assistant Secretary (Cost & Economics), AFCAA– Policy and functional oversight

– Major program milestone decisions -- Independent Cost Estimates (ICE) for delegated programs (required by law); Component Cost Analysis (CCA); AF CAIG service cost position (SCP)

– Headquarters/functional support (analysis, review, etc.)

THE ASSISTANT SECRETARYOF THE AIR FORCE

(Financial Management & Comptroller)

THE ASSISTANT SECRETARYOF THE AIR FORCE

(Financial Management & Comptroller)

DEPUTY ASSISTANTSECRETARY

(Plans, Systems & Analysis)

DEPUTY ASSISTANTSECRETARY

(Plans, Systems & Analysis)

DEPUTY ASSISTANTSECRETARY

(Cost & Economics)

DEPUTY ASSISTANTSECRETARY

(Cost & Economics)

DEPUTY ASSISTANTSECRETARY

(Budget)

DEPUTY ASSISTANTSECRETARY

(Budget)

Deputy Assistant Secretary(Cost & Economics) SAF/FMC

*The Deputy Assistant Secretary holds two titles as SAF/FMC and AFCAA/FM

Deputy Assistant Secretary for Cost & Economics (SAF/FMC)Executive Director (AFCAA/FM)

Mr Hartley, SES*Associate Deputy Assistant Secretary for Cost & Economics

Ms Woods, SES

Deputy Assistant Secretary for Cost & Economics (SAF/FMC)Executive Director (AFCAA/FM)

Mr Hartley, SES*Associate Deputy Assistant Secretary for Cost & Economics

Ms Woods, SES

AFCAA Technical Director

Mr. Jordan

AFCAA Technical Director

Mr. Jordan

Space ProgramsDivision

(AFCAA/FMS)Mr. Seeman

Space ProgramsDivision

(AFCAA/FMS)Mr. Seeman

Info TechDivision

(AFCAA/FMI)Mr. Moul

Info TechDivision

(AFCAA/FMI)Mr. Moul

ResourcesDivision

(AFCAA/FMR)Ms. Cann

ResourcesDivision

(AFCAA/FMR)Ms. Cann

Force Analysis Division

(AFCAA/FMF)Lt Col Hurley

Force Analysis Division

(AFCAA/FMF)Lt Col Hurley

A/C & Weapons Division

(AFCAA/FMA)Mr. McVicker

A/C & Weapons Division

(AFCAA/FMA)Mr. McVicker

IntegrationDivision

(AFCAA/FMC)Col DuPre

IntegrationDivision

(AFCAA/FMC)Col DuPre

Economics & Business Mgmt

(SAF/FMCE)Mr. Connair

Economics & Business Mgmt

(SAF/FMCE)Mr. Connair

Principal Economic & Financial Advisor

Mr. Hosey

Principal Economic & Financial Advisor

Mr. Hosey

ASC/AACOL

ASC/AACOL

AFSCP/SMCOL

AFSCP/SMCOL

ESCOL

ESCOL

AFCAA Eglin OL

Kristy GoldenKristy GoldenChief, AFCAA Eglin OL

YC-501-03

Paul PinkowskiPaul PinkowskiCost AnalystYA-501-02

VacantVacantCost AnalystYD-1515-02

John CargillJohn CargillTechnical Director

YD-1515-03

Justin NeeceJustin NeeceCost AnalystYD-1515-01

Beth FloresBeth FloresCost Analyst (PAQ)

YD-1515-01

Operating Locations

Los Angeles AFB CA

Peterson AFB CO

Eglin AFB FL

AFCAA - DC

Wright-Patterson AFB OH

Hanscom AFB MA

Air Force Concern

• Current AF-wide policy does not consistently require adequate, objective cost analysis or review– Limited to infrequent milestone reviews

– Inconsistent leadership value placed on cost analysis (contrast DAB to AFIPT)

• Cost resources down 60% since 1992 -- approximately 400 positions!– Staffs hit hardest, no career path/pipeline, compare poorly with

Army, Navy, others

• Little “functional” governance – Inexperienced analysts “isolated” in programs with little functional

support -- fractured estimating methods, databases, models -- inconsistent data/estimate quality, analyst qualifications

There is a Cost Growth Problem.

Cost Estimating is a Significant, but not Lone, Contributing Factor.

However, it is a factor we can manage!

Acquisition Cost Capability Initiative

• Better cost estimating processes– Policy and standards

– Re-writing AFPD 65-5 and accompanying AFIs and AFMANs– Building standards and templates– Performance metrics

– Collaboration– Encourage team efforts– Establish operating locations at Product Centers and MAJCOMs

– Transparency – Share data / research / methods

• Repopulate personnel pipeline • Upgrade education, training, and certification programs

– Provide Cost Estimators with the tools to succeed

Improve Air Force Cost Estimating

Current Status

• Policy AFD 65-5– The acquiring organization shall build program cost estimates*

• Annually

• Milestone decisions

• Nunn-McCurdy Cost Breaches

• Source Selections

• POM initiatives, disconnects, and offsets – Headquarters shall provide non-advocate cost / risk assessments to decision makers– Collaboratively develop and execute training, data collection/methods and model

development programs – Headquarters shall review cost, economic, or business case analyses to be presented

to SECAF, USECAF, CSAF, VCSAF• Personnel

– Hired 20+ positions• Operating Locations

– Completed beddown agreements at Peterson AFB, Wright-Patterson AFB, Eglin AFB, Hanscom AFB, and LA AFB

– Working Host Tenet Support Agreement at each location• Training and Certification

– Partnered with SCEA and DAU to improve certification program– Developed Training OI for new analysts

* Led by government employees

* SAF/FMC defined standards and procedures

AFCAA OL Mission

Shorten “Kill Chain” (Cost Budget)

• AFCAA-OL will function as the liaison with the Center

• Work collaboratively with the program office estimators

• Non Advocate Cost Assessments (NACAs)

• Independent Cost Estimates (ICEs)

How the AFCAA-OLs will Help

• Collaborative program cost estimates– Build cost models– Cost estimating relationships– Risk analysis– Data / research / methods

• Potential roles– Team lead– Team member– Sufficiency reviewer– Consultant– Researcher– Facilitator between Program Office and DCARC (OSD CAIG)

22

War-winning Capabilities…On Time, On Cost

Industry Help

• Don’t over promise capability with unrealistic resources

• Define and quantify uncertainty for your proposed concept

• Provide true risk now versus what it could be in a future year

• Identify the right level of MR to neutralize problems before they overwhelm the program

23

War-winning Capabilities…On Time, On Cost

Summary

• Cost estimating community needs to provide more realistic resource projections for Congress and DoD

• Additional AFCAA resources are in place now with more FMC positions coming to help programs

• New initiatives are in work to improve estimate quality, quantify uncertainty, and fund appropriate risk efforts