Forecasting costs and completion dates for defense research and development contracts
Post on 25-Feb-2017
Embed Size (px)
IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. EM-19, NO. 4, NOVEMBER 1972
ment for automatic malfunction detection. It will further contribute to determination of the cost-effective number of spares and support equipment.
The general requirements document could incorporate any requirements that either are system oriented or apply to more than one subsystem. In addition to containing the require-' iients for reliability and quality assurance (R&QA), safety, testing, configuration management, materials, design reviews, etc., it could also include the results of studies defining the allowable budgets per subsystem of electrical power, space, weight, etc. Utilization of a general requirements document in any program will preclude duplication of system requirements in each specification, thereby eliminating the possibility of inadvertent differences. It further enhances the probability of having design and documentation uniformity.
Scheduling and all the related program control techniques serve the same function in this approach to system definition, oc rn i m r n r A n r o m P Y ^ f l n t t h a t a l l tVlA QnalvClQ p f T n r t **.w v " ~"-r r*~er r - J -' included in the scheduling. This will provide management with the potential for a more comprehensive view of program
progress. It will also provide a means of identifying problems in all phases of program development, and facilitate management decisions based on a more complete compilation of related facts. This will be especially significant following the critical design review (CDR); when by joint utilization of the formal analysis, schedules, and cost, the total impact of proposed changes can be evaluated.
The SRA approach will provide a disciplined medium for effective program management, in addition to complete definition of all system hardware, software, and logistics. The Minuteman Program, with its inherent complexities, demonstrates the extreme effectiveness of this technique. The available procedures (^8>499, AFSCM 375-5, etc.) may, however, be more comprehensive than is necessary for less complex programs. The author therefore recommends that these procedures, as modified by this paper, be adapted to the individual needs of each program.
Because this analysis procedure is adaptable to any program, the potential benefits should not be overlooked by private industry. A cost savings for a governmental agency can readily be equated to an increased profit margin for a private concern.
Forecasting Costs and Completion Dates for Defense Research and
Development Contracts ARNOLD M. RUSKIN AND ROBERT LERNER
Abstract-It is hypothesized that the actual final cost and time required to develop a new technological system can be forecast from initially negotiated costs and times and other administrative details that are available when the contract is negotiated. This hypothesis was tested by examining multiple regressions of total cost and time as functions of data available at the end of negotiations for 73 contracts administered by one division of the Air Force.
Factors correlating with over 70 percent of the cost growth were identified: 1) the initial cost and period of performance, 2) whether or not the entire system is investigated, 3) whether or not a significant change in scope of ihc effort is anticipated, 4) whhc i the effort is classified as a study (in contrast to hardware development), and 5) who is the procurement officer.
Manuscript received December 14, 1971; revised March 31,1972. A. M Rusk in is with Haxvey Mudd College, Claremont, Calif. R. Lerner was with the Claremont Graduate School, Claremont,
Calif. He is now a Financial Consultant in San Bernardino, Calif.
Factors correlating with one-third of the growth in the time required to perform the contract were identified: 1) what type of performance incentives and limitations are specified, if any, 2) whether or not the study is classified as "level of effort," 3) the initial cost and period of performance, 4) the project branch, and 5) whether or not a significant change of scope is anticipated.
Regression equations and coefficients are reported for the division studied.
FORECASTING completion dates and costs in the defense industry is a complex art  - , Nevertheless, it is hypothesized that the final cost and the time required to develop a system can be estimated with some accuracy on the basis of past experience, although the intrinsic reasons for changes may not be fully understood. This hypothesis is tested
RUSKIN AND LERNER: FORECASTING COSTS AND COMPLETION DATES 129
by examining multiple regressions of total cost and time as functions of certain administrative details, such as scope of the project and incentives and limitations placed on the project, for a population of 73 contracts administered by a division of the Air Force.
The division studied deals only in research and development of advanced ballistic reentry systems (ABRES) and is the only agency in the Air Force working in its field. A different agency is responsible for applying weapon system concepts, and the division studied has no weapon system commitments to meet. Once a program is approved, the division is fairly well isolated from direction by and commitments to other agencies, and the group manages its programs as it sees fit. The group's annual budget is firmly limited, but it can divert money from one contract to another within its total budget.
The sample of 73 contracts includes all available completed contracts managed by the division for its own mission objectives during an 8-year period.1 During this time, the mission of the division did not change.
The organization has two parallel sections: a projects section and a procurement section. The projects section is
development contracts that ABRES places; the procurement section is responsible for legal, fiscal, and other administrative aspects of ABRES's contracts. All decision-making personnel in the projects section are military; they sometimes consult, however, with civilian personnel of the Aerospace Corporation, with whom they maintain a continuing relationship. In the procurement section there are buyers and contract officers, both civil-service career positions. The buyers report to the contract officers, and the latter report eventually to a military officer in charge of the procurement section. The minimum assignment of the military members of ABRES, whose tenure is shortest, was three years.
BACKGROUND Several models have been proposed in recent years for
predicting costs and scheduling for research and development tasks. Doering, for example, has proposed a model for risks in scheduling ; Dean et al. have proposed a model of cost distributions for a collection of projects ; Souder has proposed a project control model [5 ] ,  ; and Roberts has proposed a general theory of research and development .
Doering studied the regression of 11 variables or dimensions with differences between planned and actual completion dates . His work is based on 16 historical research and development tasks. The results of the analysis are used to structure a predictive relation. The variables whose effect Doering examined are basically characteristics of the task, e.g., design complexity, flexibility of systems specifications, flexibility of quality assurance requirements, similarity to previous performance requirements, and so forth. Doering employed Delphi techniques to quantize each variable for each project. He found that four variables, i.e., similarity to previous performance requirements, the extent of precontractual effort,
1 Sixty-four other contracts were issued during this time that could not be included; 4 were prematurely terminated and 60 were lodged in a permanent storage area that was not accessible to the authors.
the extent of systems knowledge, and the extent of the group's experience, were significantly more important in their correlations and for predictive purposes than the other seven variables.
x ean s niOuCi tocuses on oig-range uUuget forecasting [i j . Individual projects are classified by categories, program elements, projects, tasks, and work units. Actual research costs are studied for the various classifications and their probability distributions and related parameters are determined for the various classes. The distributions are then used to predict overall budgets for future periods in which projects will be undertaken. In the cases considered, the data fit the log-normal distribution with a type I risk of less than one percent.
Souder reported experience with a dynamic model for controlling cost and achievement in research and development projects ,  . His approach was sk project personnel periodically to evaluate their project progress in detail. He thereby obtained early warnings of impending project failures, a conceptual understanding of forces affecting these project failures, and an analysis of achievement per dollars spent. This evaluative information can be used, among other things, to nrwr&ni H r i f t frr\rn / \ririna1 r1aro
Roberts' general theory is an analytic representation of project dynamics . It directs attention to the sequential and iterative or cyclic aspects of projects, including their interactions with resource and output markets. The theory thus suggests critical points in project progress and control that may be examined in detail to learn the details of project activity.
Roberts' work and the work of Doering, Dean, and Souder tend to complement each other. Roberts' model is conceptual and general, while the work of the others might be characterized as experiential and specific. At various times, Roberts points out the lack of data that are required to fully exploit his approach. Doering, Dean, and Souder, whose works were reported after Roberts published his general theory, are getting data, although they often seem not to be measuring the parameters that Roberts' model calls for. In time, no doubt, models that take advantage of the accumulated data will be formulated and new empirical investigations will be made to provide data specifically needed by the models.
Doering offers a description of how adequate models are developed in the introduction to his work . In Doering's terms, Roberts' work represents the first phase: a macro examination. Detailed investigations of specific relations constitute a second phase: micro studies. Finally, when sufficient understanding exists at the micro level, attention is again focused at the macro level as attempts are made to fit all the separate parts into a comprehensive model. At the moment, it seems as if modeling research and development is in its second phase and will be there for some time.
This study was made to determine if final costs and completion dates for research and development contracts correlate reliably with certain administrative details and other factors that are known when the contract is negotiated. There are two possible uses for such a study if a positive outcome is
IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, NOVEMBER 1972
TABLE I INDEPENDENT VARIABLES USED IN COST AND
PERFORMANCE PERIOD REGRESSIONS
Cn nth contractor: n = 1 refers to a misceUaneous group of contractors with only one contract each in the sample, n - 2, , 8 each refer to a single contractor. Cn = 1 for the contractor of the particular project and Cn = 0 for all other contractors.
E contractor's prior experience in the field, in months. 5 study effort; S = 1 for mere study efforts; S = 0 for all other
projects. G ground test effort: G = 1 for projects involving ground tests;
G = 0 otherwise. F flight test effort: F=l for projects involving flight tests;
f = 0 otherwise. An nth project branch', n ranges from 1 to 4. An = 1 when the
project is administered by branch Ap\ An = 0 for ali other branches.
K system effort: K - 1 for projects involving entire reentry systems; K = 0 otherwise.
Bn nth contracting officer: Bn = 1 when the project contracting officer is Bn\ Bn - 0 for all other contracting officers.
Tn *>' of contract: e.g., cost plus fixed fee, cost plus incentive fee, ?tc. (see Table H) T~ - 1 for a proiect with contract type Tn; Tn = 0 for all other contract types,
/ performance incentives: / = 1 when a project contract includes performance incentives;/ = 0 otherwise.
P originally negotiated period of performance, in months/10. H letter contract: H = 1 for projects under a letter contract;
H = 0 otherwise, D function of the initially negotiated cost (in thousands of
dollars): Dx is the common logarithm of the cost; D2 is the square root of the cost.
L level of effort limitations: L = 1 for projects with level of effort limitations; L = 0 otherwise.
V change of scope anticipated: V - 1 for projects with a significant change of scope anticipated at the outset; V- 0 otherwise.
reached: l ) t o enable* the organization studied to forecast costs and completion dates for its own research and development contracts, and 2) to provide insight at the micro level that may be useful in a more general model of research and development.
The study was formulated as a hypothesis to be tested, namely that multiple regressions of costs and completion dates as functions of certain independent variables are statistically significant. The dependent variables, i.e., costs and completion dates, were cast in the following forms:
final cost to the Government Yi =
initially negotiated cost
actual performance period initially negotiated performance period
The independent variables were drawn from five categories: 1) the degree of project complexity, 2) the type of contract, 3) the characteristics of the contractor, 4) the key personnel in the procurement section of ABRES, and 5) the project branch. Each category is described below. Altogether, these 5 categories yielded 15 independent variables, which are shown
in Table !. Most of the independent variables were incorporated into the regressions by the technique of dummy variables.
Five measures of project complexity were developed from data available to ABRES when the contracts were negotiated.
1) Initially negotiated period of performance. (Longer contract periods were believed necessary for more complex projects.)
2) A function of the initially negotiated cost; the cost, its square root, and its log were tried. (More complex projects were believed to cost more, if only to provide for more experiments before reaching success.)
3) Whether hardware was required for ground or flight test or only a study was involved. (Studies were believed to be inherently simpler than projects involving ground or flight testina of hardware.^
4) Whether the entire reentry system was investigated or only a subsystem. (The contracts were initially divided into ten areas being handled by the d...