international cost estimating and analysis association

UNCLASSIFIED QinetiQ Proprietary

Copyright QinetiQ Ltd 2017

International Cost Estimating and Analysis Association (ICEAA) Canberra chapter Cost Engineering Health Check (CEHC)

Report Number: QINETIQ/17/00688/1.0 Author: D Shermon Publication Date: 17/02/2017 Issue: 1.0


QINETIQ/17/00688/1.0 Copyright QinetiQ Ltd 2017

Administration page

Customer information

Customer reference number n/a

Project title ICEAA Canberra Cost Engineering Heath Check

Customer contact Tracey Clavell, ICEAA President

Milestone number n/a

Date due n/a

Principal author

Dale Shermon 07785 522847

QinetiQ Bristol, Building 240, The Close, Bristol Business Park, Coldharbour Lane, Bristol, BS16 1FJ, United Kingdom

Email: [email protected]

Release authority

Name Daryl Brown

Post Head of Service Delivery, Advisory Services

Date of issue 14th February 2017

Record of changes

Issue Date Detail of Changes

0.1 14/02/17 Draft release for IDR

1.0 17/02/17 Final version



Executive summary

Introduction

QinetiQ’s Cost Engineering Health Check (CEHC) is a standardised competency assessment framework that is used to assess the cost engineering capability of an organisation. It provides an objective evidence-based audit against both best practice and industry standards, thus allowing areas of strength and weakness in the organisation’s costing capability to be identified and understood. The CEHC is assessed against the six pillars of an effective cost engineering capability: People, Process, Tools, Data, Culture and Stakeholders.

QinetiQ offered to undertake a CEHC survey for the International Cost Estimating and Analysis Association (ICEAA) Canberra chapter [1] to enable them to establish themes for future workshops. The output of the CEHC will enable current capabilities to be properly understood as well as allowing ICEAA Canberra chapter to focus resources towards costing areas with identified gaps needing attention when developing new guidance and tools, and improving the skills and knowledge of its people.

This report can provide the basis of a capability improvement programme of workshops for the ICEAA Canberra chapter and its members.

The CEHC workshop was conducted on 8th February 2017. The workshop involved the delegates using electronic voting to score the maturity of their organisations against a sub-set of our 38 maturity criteria under the six pillars of the CEHC. A facilitated discussion was then held to capture the reasons for the scores. QinetiQ has used the results of the workshop to quantify the cost engineering maturity of the ICEAA members present at the workshop and to develop an outline improvement plan for the ICEAA Canberra chapter.

Maturity assessment results

The CEHC assessment has demonstrated that while some benchmarks are comparable with previous workshops (e.g. Figure 2-2: Principal supplier engagement results) providing confidence in the CEHC technique; some have identified significant gaps (e.g. Figure 6-1: Results for capture of historical financial cost data) indicating a need for action.

The electronic voting was used successfully to stimulate comments from the attendees. The live display of the voting acted as a catalyst for the attendees to reflect upon the scores and suggest why the voting had a particular pattern.

These comments have been recorded in the body of this report and have led to the improvement initiatives that have been identified in the recommendations section.

Improvement plan

QinetiQ has developed an outline improvement plan in terms of a set of actions that should be implemented by the ICEAA Canberra chapter in order to improve the cost engineering capability across their members. These actions are detailed in the report and summarised in Table 1-1 below.



Pillar Workshop themes

Stakeholders Senior management workshop

Cost engineering with supplier workshop

Culture Corporate guidance workshop

People Risk management workshop

Process Cost engineering practitioner workshop

Cost estimating methodologies workshop

Data Data warehouse and knowledge management workshop

Tools Risk analysis workshop

Software estimating workshop

Table 1-1: Recommended workshop themes



List of contents

1 Introduction 7

1.1 Background 7

1.2 The CEHC process 8

1.3 Undertaking a CEHC assessment 8

1.4 A CEHC assessment of ICEAA Canberra chapter 9

1.5 This document 9

2 Stakeholders assessment 11

2.1 Introduction 11

2.2 Effectiveness of engagement with senior management 11

2.3 Effectiveness of engagement with principal suppliers 12

3 Culture assessment 13

3.1 Introduction 13

3.2 Integration with corporate processes and procedures 13

4 People assessment 14

4.1 Introduction 14

4.2 Staff understanding of projects 14

4.3 Staff understanding of risk management processes 14

4.4 Staff understanding of relevant engineering practices 15

5 Process assessment 17

5.1 Introduction 17

5.2 Management of roles and responsibilities 17

5.3 The fundamental generation of cost estimates 18

6 Data assessment 19

6.1 Introduction 19

6.2 Capture of historical financial cost data 19

6.3 Capture of historical technical and contextual data 20

7 Tools assessment 21

7.1 Introduction 21

7.2 Risk analysis tools 21

7.3 Software costing tools 22

8 Improvement plan 23

8.1 Introduction 23

8.2 Stakeholders 23

8.2.1 Senior management workshop 23

8.2.2 Cost engineering with supplier workshop 23

8.3 Culture 23

8.3.1 Corporate guidance workshop 23

8.4 People 23

8.4.1 Risk management workshop 23

8.5 Process 24

8.5.1 Cost engineering practitioner workshop 24

8.5.2 Cost estimating methodologies workshop 24



8.6 Data 24

8.6.1 Data warehouse and knowledge management workshop 24

8.7 Tools 24

8.7.1 Risk analysis workshop 24

8.7.2 Software estimating workshop 24

8.8 Improvement roadmap 24

8.8.1 Improvement criteria 25

8.8.2 Benefit/Time Analysis 25

8.8.3 Prioritisation of implementation actions 25

8.8.4 Timeline of implementation actions 26

9 Conclusions and recommendations 28

9.1 Conclusions 28

9.2 Recommendations 28

9.3 Barriers to Change 27

10 List of references 29

11 List of abbreviations 30



1 Introduction

1.1 Background

Cost estimation is an important capability underpinning the operation of any project-based organisation since it supports activities such as business planning and the preparation of costed bids and business cases. If cost estimation is not undertaken in an appropriate manner then this can lead to decisions being made on the basis of cost information which may not be accurate.

QinetiQ’s cost engineering capability is delivered through an approach called knowledge based estimating (KBE). This is a philosophy that builds upon the essential building blocks of any good estimate; Data, Tools, People and Processes, as shown in Figure 1-1 below.

Figure 1-1: The building blocks of KBE

It is widely acknowledged in the estimating and forecasting community that a good estimate cannot be created in a vacuum. It requires the normalisation of historical data gathered in a rigorous process and the application of tools for the statistical analysis of this historical data and the prediction of the future forecasts. This then needs to be combined with suitably qualified staff to apply these techniques and tools. The application of appropriate skills and knowledge then results in robust, justified, auditable predictions of cost. The application of a consistent process will enable a third party or scrutineers with the same skills, tools and data to reproduce the same credible and justified cost forecast.

QinetiQ’s experience and expertise means that it is well-placed to assist other organisations in understanding their own cost modelling capability and supporting development in areas where weaknesses are identified. To this end, QinetiQ has developed the Cost Engineering Health Check (CEHC).

This report will provide the basis of a programme of ICEAA future workshops.



1.2 The CEHC process

The CEHC is a standardised competency assessment framework that is used to assess the cost engineering capability of an organisation. It provides an objective evidence-based audit against both customer and industry standards, thus allowing areas of strength and weakness in the organisation’s costing capability to be identified and understood.

The CEHC process allows an organisation to answer the following questions:

How good is our estimating capability? This is achieved through benchmarking capability against best practice.

How do we compare to our peers? This is achieved through benchmarking capability against UK government and industry standards by comparing the CEHC results with those undertaken for other organisations. These benchmarks are treated generically, identified in terms of domain rather than specific company or organisation so as to avoid any issues of commercial sensitivity.

How can we improve? The results of the assessment enable resources to be directed towards areas that have been identified as offering the most potential for improvement and optimisation.

The output of the CEHC process is an assessment of the maturity of the organisation’s cost engineering capability against six pillars. These pillars are drawn from the four building blocks of the KBE approach (Data, Tools, People and Processes) plus two critical enablers (Stakeholders and Culture). Maturity against each pillar is measured using a five point scale that is defined in Table 1-1.

Level Title Definition

Level 1 Naïve Process design or application flawed and probably not adding value

Level 2 Novice Some value-add, but weakness in process design or implementation

Level 3 Normalised Newly formalised process, implemented systematically and adding value

Level 4 Natural Formalised process, implemented systematically and adding value for a reasonable duration

Level 5 Optimised Applied at strategic level in driving objectives and optimising outcomes for a considerable length of time.

Table 1-1: CEHC maturity definitions

1.3 Undertaking a CEHC assessment

The CEHC is delivered through a structured process that involves the following activities:

An initial review of documentary evidence by QinetiQ’s CEHC Team to enable them to gather information concerning the type of cost engineering that is undertaken and to gain an initial understanding of capability through the quality of these materials.



A CEHC workshop which is attended by appropriate personnel from the organisation that is being assessed. The attendees are asked to assess the organisation’s cost engineering capability against 38 indicators within the six pillars, providing a score using a five point scale. This information is gathered using an electronic voting process underpinned by the Delphi method. This involves two votes being taken for each question; the first is an initial view which is used as the basis for a discussion to obtain evidence and reasons for the scores. A second vote then takes place to record a consensus view. It is the data from this second vote that is used in the subsequent analysis.

Analyse and report results. The data from the workshop and the review of documentary evidence is used to derive overall scores for each of the 38 questions and then for each of the six pillars as a measure of overall capability that can also be compared with other organisation. These scores are underpinned by the rationale for the assessments, thus allowing recommendations to be developed for specific improvement actions that can be undertaken that will lead to an improvement in capability.

A periodic CEHC re-assessment can then be undertaken to enable the effectiveness of the improvements to be assessed.

1.4 A CEHC assessment of ICEAA Canberra chapter

ICEAA Canberra chapter took the opportunity to invite QinetiQ to present their CEHC workshop. They recognised that cost management across their membership could be strengthened in order to deliver improved capability. As a consequence Tracey Clavell contacted QinetiQ to undertake a CEHC assessment while Dale Shermon was in the country from the UK.

ICEAA Canberra chapter was seeking an understanding of the maturity of cost community within its membership to enable them to identify areas where improvements are required. The CEHC assessment will therefore allow this ICEAA chapter to set an action plan for building stronger costing capability within its membership

The CEHC followed a reduced version of the standard process to accommodate a 2 hour workshop. The CEHC workshop was held on the 8 February 2017 in the Department Of Defence, Russell 1 building lecture theatre. This was attended by 25 representatives from across the ICEAA membership both public and private.

The CEHC has been deployed internationally as well as in the UK. It contains benchmarks for UK defence, European space, UK non-defence, US defence, UK government, Canada government and Consultancies.

A benchmark is created by taking the average of all the electronic votes recorded in the CEHC workshop and this represents the standard for that sector.

In the assessment of the ICEAA Canberra chapter maturity, the benchmarks that have been selected are UK Government and UK, US and European industry.

1.5 This document

This document describes the conduct and results of the CEHC survey of the ICEAA Canberra chapter.



Sections 2 to 7 of this report provide the detailed assessments against the six pillars of the CEHC assessment. For each pillar, the maturity results corresponding to each lower level indicator are presented based on the voting scores at the workshop. For consistency, the results are presented as generic maturity scores (Levels 1 to 5) rather than in terms of the specific questions that were used in the workshop in order to obtain the assessments. In each case, voting results for each benchmark are presented in terms of the percentage of responses that align with each maturity level and as a mean score.

The maturity score is assessed against an optimised business or organisation and their solution has been described using the assumption that they have unlimited resources and endless time to satisfy the maturity indicator that is being reviewed.

The maturity scores for each indicator have been used to derive an overall maturity for projects for each pillar and these are presented in the context of four benchmarks for four categories of organisation (UK Government, UK Industry, US Industry and European Industry). These benchmarks represent the results from previous CEHC assessments that have been undertaken by QinetiQ.

Section 8 defines an improvement plan for ICEAA Canberra chapter concerning specific workshops that can be undertaken to improve the cost engineering maturity of ICEAA in Canberra.

Section 9 provides the overall conclusions and recommendations arising from this assessment.



2 Stakeholders assessment

2.1 Introduction

The Stakeholders assessment is concerned with the effectiveness with which the cost engineering team engages with the different individuals and organisations who have an interest in the cost estimate. The Stakeholders assessment is based on five indicators, two of which were used in this tailored CEHC and which are discussed in the following subsections.

2.2 Effectiveness of engagement with senior management

Engagement with senior management is an important activity underpinning cost engineering since it facilitates high-level ownership of the cost estimates and ensures that factors which are important to the senior management can be reflected sufficiently within the cost engineering process.

The results of the assessment of the members of the ICEAA Canberra chapter relating to engagement with senior management are shown in Figure 2-1. The left hand chart shows the distribution of voting scores as a percentage against the five point maturity scale from Level 1 (Naïve) to Level 5 (optimised) for each of the benchmarks. The right hand chart then illustrates the mean average score per benchmark.

0

1

2

3

4

5

Mea

n a

ve

rag

e s

co

re

Stakeholder: Senior management0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

ICEAA Canberra

UK Ind.

UK Gvnt.

US Ind.

Euro Ind.

Percentage of votes

Level 1 Level 2 Level 3 Level 4 Level 5

Figure 2-1: Senior Management results

It can be seen that there was not vast difference in scores across the workshops, with the ICEAA Canberra chapter marginally below the other workshops.

There were many comments regarding policy; policy, process and reporting is unclear or still being developed; process is driven by a pre-determined outcome. There is a need to engage management and build trust with management. Also there is a need to provide digestible information to management.

QinetiQ would generally require documentary evidence of cost engineering and estimating policies prior to a full CEHC workshop: for this general workshop of the Canberra cost community there is no evidence of any corporate policy.



2.3 Effectiveness of engagement with principal suppliers

This indicator relates to the level to which the cost engineering team engages with industry as part of the process of generating the estimate.

The results of the assessment of the ICEAA Canberra chapter relating to engagement with principal suppliers are shown in Figure 2-2.

0

1

2

3

4

5

Mea

n a

ve

rag

e s

co

re

Stakeholder: Principal suppliers0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

ICEAA Canberra

UK Ind.

UK Gvnt.

US Ind.

Euro Ind.

Percentage of votes


Figure 2-2: Principal supplier engagement results

This question was not answered during the UK Government and European Industry workshops, hence there is no score. Generally, this criteria is not scored as mature with US industry and ICEAA Canberra scoring just above level 2 (Novice) while UK industry score just below. It is evident that more work needs to be done across the cost engineering community generally.

The ICEAA Canberra workshop commented that the scope and boundaries of proposals are ill-defined in their experience. The workshop felt unable to engage industry suppliers due to probity issues or policy, however there was some engagement by exception. Others attendees regularly engage their supply chain outside of tender processes and assist them in benchmarking their business demonstrating that while public bodies found it difficult, some private organisations had a willingness to engage with suppliers.



3 Culture assessment

3.1 Introduction

The Culture assessment is concerned with the degree to which cost engineering are integrated into the overall processes of the project and into the other activities that need to take place to deliver a successful project. The Culture assessment is based on five indicators, one of which was used in this tailored CEHC and which is discussed in the following subsection.

3.2 Integration with corporate processes and procedures

Cost engineering should take place in a manner that is consistent with the general operation of the organisation as articulated in its corporate processes and procedures.

The results of the assessment of the ICEAA Canberra chapter relating to integration of the cost engineering with corporate processes and procedures are summarised in Figure 3-1 below.

0

1

2

3

4

5

Mea

n a

ve

rag

e s

co

re

Culture: Corporate processes0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

ICEAA Canberra

UK Ind.

UK Gvnt.

US Ind.

Euro Ind.

Percentage of votes


Figure 3-1: Result for integration with corporate processes integration results

This question was not answered during the UK Government and European Industry workshops; hence there are no scores for these workshops. US and UK industry judged there maturity between normalised and natural, while the ICEAA cost community was less confident.

The Canberra chapter commented that there was too much informal guidance with no consolidated guidance. In some cases there was experience of conflicting guidance leading to the need for a clear way forward and direction. It was commented that the ICEAA cost community found no independent assurance of guidance (who does it and what qualifications do they have).

Within industry it was common that an organisation would get audited every two years by an independent body with global guidance with bi-annual review of cost estimates.



4 People assessment

4.1 Introduction

The People assessment is concerned with the skills, knowledge and experience of the individuals who undertake cost engineering activities. It is based on nine indicators, three of which were used in this tailored CEHC and which are discussed in the following subsections. The workshop was guided to make the assessments based on the average capabilities of their people noting that there will generally be a mix of capabilities within any community.

4.2 Staff understanding of projects

Cost engineering staff can only generate appropriate cost estimates for a project if they have an appropriate level of understanding of that project.

The results of the assessment of the ICEAA Canberra chapter relating to cost engineers’ understanding of their projects are summarised in Figure 4-1.

0

1

2

3

4

5

Mean a

vera

ge s

core

People: Project understanding0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

ICEAA Canberra

UK Ind.

UK Gvnt.

US Ind.

Euro Ind.

Percentage of votes


Figure 4-1: Results for staff understanding of their projects

All workshops have scored this maturity criterion with the Canberra chapter benchmarking a similar score to UK industry and government.

The ICEAA membership commented that they generally had less effective engagement with project staff and their ability to share project data with cost estimators in a timely fashion was not always good. They commented that, in their experience, estimators are co-located with the project team to improve understanding of their projects. However, estimators needs long term engagement with the project to improve understanding.

4.3 Staff understanding of risk management processes

Staff can only generate appropriate cost estimates for a project if they properly take account of the cost impact of any risks that might occur within that project.

The results of the assessment of the ICEAA Canberra chapter relating to cost engineers’ understanding of risk management processes are summarised in Figure 4-2.



0

1

2

3

4

5

Mean a

vera

ge s

core

People: Risk management

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

ICEAA Canberra

UK Ind.

UK Gvnt.

US Ind.

Euro Ind.

Percentage of votes


Figure 4-2: Results for staff understanding of risk management processes

While the US industry workshop scored the higher maturity, above Natural (level 4), the ICEAA Canberra chapter unfortunately scored the lowest across all the workshops.

It was generally felt by the ICEAA membership that there was no understanding of three point estimating and there was very poor risk estimation guidance. Attendees at the workshop had concerns around how and why risk information will be used and stated that generally it was not part of decision making processes.

There was a worrying lack of knowledge and skills by managers to understand risk management with no clear understanding to link a cost estimate to the quantification of risks due to policies around contingencies. This had resulted in a situation where no clear requirement to quantify and cost risks meant that little maturity existed. Indeed, Monte Carlo analysis was not part of any policy requirement.

4.4 Staff understanding of relevant engineering practices

If staff have a detailed understanding of the engineering aspects of a project then they can reflect this within the generation of the cost estimates, with the effect that this estimate is then more likely to be truly representative of that project.

The results of the assessment of the ICEAA Canberra chapter relating to cost engineers’ understanding of relevant engineering processes are summarised in Figure 4-3.



0

1

2

3

4

5

Mean a

vera

ge s

core

People: Engineering

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

ICEAA Canberra

UK Ind.

UK Gvnt.

US Ind.

Euro Ind.

Percentage of votes


Figure 4-3: Results for staff understanding of relevant engineering processes

All workshops scored between normalised (level 3) and Natural (level 4), while the Canberra chapter was marginally below level 3.

This score would indicate a need for more understanding in terms of engineering particularly if parametric cost estimating is to be adopted. The ability to communicate with engineers on an equal basis to quantify cost drivers will be important.

The cost community at the workshop commented that it is difficult for one person to get across all aspects of engineering.



5 Process assessment

5.1 Introduction

The Process assessment is concerned with the availability of consistent and high quality processes to provide a clear and rigorous basis for the generation of cost estimates. The Process assessment is based on five indicators, two of which were used in this tailored CEHC and which are discussed in the following subsections.

5.2 Management of roles and responsibilities

Cost estimates are often generated by a team of people who will have different roles and responsibilities within the cost engineering activities. It is therefore important that roles and responsibilities are clearly defined so that individuals completely understand what is expected of them and how they will contribute towards the successful completion of the project.

The results of the assessment of the ICEAA Canberra chapter relating to processes for the management of roles and responsibilities are summarised in Figure 5-1.

0

1

2

3

4

5

Mean a

vera

ge s

core

Process: Roles & responsibilities0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

ICEAA Canberra

UK Ind.

UK Gvnt.

US Ind.

Euro Ind.

Percentage of votes


Figure 5-1: Results for processes relating to the management of roles and responsibilities

With the exception of the US industry workshop which did not score this criterion, the maturity is just below Natural (level 4) while the ICEAA Canberra chapter is just below Normalised (level 3) indicating that there is some room for improvement.

The attendees commented that there are defined roles (or a job family) for cost estimation. However, the incumbents are possibly not always suitably qualified, with for example a Project Manager undertaking a cost estimate.

The resource allocated to conduct the role is dependent generally upon workforce pressures, rather than capability, this allocation results in a higher demand for project requiring cost assuring. It is recognised that there is a need for the right cost estimators for the right project.



5.3 The fundamental generation of cost estimates

The fundamental output of cost engineering is the cost estimates that will be used within projects and/or to inform decisions. The confidence in the estimates that are being produced will depend upon the robustness and effectiveness of the processes that have been followed as these have been generated.

The results of the assessment of the ICEAA Canberra chapter relating to processes underpinning the fundamental generation of estimates are summarised in Figure 5-2.

0

1

2

3

4

5

Mean a

vera

ge s

core

Process: Generation of estimates0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

ICEAA Canberra

UK Ind.

UK Gvnt.

US Ind.

Euro Ind.

Percentage of votes


Figure 5-2: Results for process relating to the fundamental generation of estimates

As the fundamental benchmark of the capability to generate a cost estimate this score was particularly worrying to the facilitators, but not the attendees of the workshop. While the four other workshops have scored around level 4 (Natural); the ICEAA Canberra chapter was significantly lower.

The attendees explained that while there has been an improvement in the estimation, a lack of data inhibits good cost estimation. They considered that the organisation was not set-up to improve future estimates and had not tracked lessons learnt over time or benchmarked the organisation’s progress. The membership acknowledged that a lack of skills sets in applying methodologies was a fundamental problem and they recognised the use of various techniques for comparison purposes.



6 Data assessment

6.1 Introduction

The Data assessment is concerned with the availability of consistent and high quality data for use in the generation of cost estimates. The Data assessment is based on seven indicators, two of which were used in this tailored CEHC and which are discussed in the following subsections.

6.2 Capture of historical financial cost data

Historical data is often very useful as the basis for developing future cost estimates, either as the basis of a historic trend analysis (HTA) or developing estimates by analogy. This is only possible if historic cost data is stored in a manner that enables it to be easily reused.

Figure 6-1 summarises the results of the assessment of the ICEAA Canberra chapter relating to the effectiveness with which historical financial cost data is captured.

0

1

2

3

4

5

Mea

n a

ve

rag

e s

co

re

Data: Historical financial data0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

ICEAA Canberra

UK Ind.

UK Gvnt.

US Ind.

Euro Ind.

Percentage of votes


Figure 6-1: Results for capture of historical financial cost data

The maturity of data gathering is particularly worrying for the Canberra chapter as it is considerably lower than the other four workshops that have been used as a comparator. This was discussed at some length by the attendees from ICEAA.

It was agree that there are poor processes for lessons learnt and historical data capture in the cost community, more generally there is a lack of workforce capability.

The attendees considered that there was no such thing as “no capturing of cost data”, but there were poor processes for its capture. Typically, the data is financial and against chart of accounts, which may not be useful for the purposes of cost estimating in the future. There are problems associated with commercial-in-confidence issues in accessing data. It is considered to be a huge task and finding resources to collect data is not easy.

On a positive theme, there was considered to be large amounts of EVM data, but this resides in third party archived applications that cannot be accessed, and so the data cannot be extracted. The workshop recognised that it requires a large amount of work



to collect historical data which can be wasted if it is not re-used and made available to future projects.

The workshop recognised initiatives to engage with other organisations to access historical data. The workshop was aware of the large amounts of data that prime contractors provide to defence that is not properly shared within defence.

6.3 Capture of historical technical and contextual data

Historical data is only useful if the estimator has a complete understanding of the technical and wider context to this data, such as the year in which the expenditure occurred or the quantity of equipment that was purchased.

Figure 6-2 summarises the results of the assessment of the ICEAA Canberra chapter relating to the effectiveness with which historical technical and contextual data is captured.

0

1

2

3

4

5

Mean a

vera

ge s

core

Data: Historic context data

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

ICEAA Canberra

UK Ind.

UK Gvnt.

US Ind.

Euro Ind.

Percentage of votes


Figure 6-2: Results for capture of historical technical and contextual data

Typically the score for the gathering and storage of historical contextual data is lower than that of financial cost data. In the case of the ICEAA cost community in Canberra this maturity score is bordering on level 1 (Naïve).

The workshop did not dwell on this maturity criterion following the detailed discussion regarding financial data, other than to state that the thoughts were mirrored to the financial cost data statements.



7 Tools assessment

7.1 Introduction

The Tools assessment is concerned with the availability of models and other software tools that can be used to assist in the generation of cost estimates. The Tools assessment is based on seven indicators, two of which were used in this tailored CEHC and which are discussed in the following subsections.

7.2 Risk analysis tools

This indicator essentially covers the use of tools to enable Monte Carlo analysis of the impact of uncertainty and risk. This is a very important method that allows the confidence level of an estimate to be determined.

Figure 7-1 summarises the results of the assessment of the ICEAA Canberra chapter relating to the availability of risk analysis tools.

0

1

2

3

4

5

Mean a

vera

ge s

core

Tools: Risk analysis tools

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

ICEAA Canberra

UK Ind.

UK Gvnt.

US Ind.

Euro Ind.

Percentage of votes


Figure 7-1: Results for the availability of risk analysis tools

Again, the ICEAA Canberra membership was not mature when compared to the three benchmarks drawn from previous similar workshops. UK government, European and UK industry all scored above level 3 (Normalised) while the ICEAA Canberra membership considered their maturity to be below level 2 (Novice).

When discussing the e-voting score it became apparent that there was a different of maturity between customer and supplier. The prime contractors must transparently quantity risk and uncertainty because of the pricing and bidding processes within their organisation. They are forced to do it by the customer to enable them to gain a contract.

It was believed that stovepiped risk analysis does not influence cost estimation; where risk analysis was conducted it was not joined up. In terms of budgeting there was contingency around uncertainty but this was not well defined and was very subjective.

When considering the schedule slippage effect on cost, no formal schedule risk analysis was conducted. Normal working practice was to apply contingency after the



cost estimate. It was stated that the Department of Finance mandates certain levels of contingency, which may not align with reality

7.3 Software costing tools

This indicator is concerned with the use of specialist tools to assist in understanding the cost of producing software, given that this is a complex and often a highly risky activity.

Figure 7-2 summarises the results of the assessment of the ICEAA Canberra chapter relating to the availability of software costing tools.

0

1

2

3

4

5

Mean a

vera

ge s

core

Tools: Software costing tools0% 10% 20% 30% 40% 50% 60% 70% 80% 90%100%

ICEAA Canberra

UK Ind.

UK Gvnt.

US Ind.

Euro Ind.

Percentage of votes


Figure 7-2: Results for the availability of software costing tools

Generally the ICEAA Canberra chapter workshop attendees did not recognise the need for software cost estimating. This was reflected in the lower maturity score of the workshop when compared to the four benchmarks selected for comparison.

The workshop understood that function point analysis was sometime used to generate estimates sizing. This positive statement was balanced by a lack of the skill set needed to use tools effectively.

It was recognised that getting software costs through the Department of Finance was challenging given the ‘black box’ nature of the cost tools. There was an acknowledged need for tools that have results that are accepted. This would only happen when the need for funding and resources to train staff was realised together with an education process about parametrics.

It was stated that the customer did not need a tool when they were not writing software, but this sadly missed the capability to judge value for money without the ability to independently calculation the cost of software.

Finally, it was recognised that software cost estimating tools need to be accepted across government and industry to have an impact; ideally through the recognition of a tool champion.



8 Improvement plan

8.1 Introduction

This section provides an outline plan for improving the cost engineering capability of the ICEAA Canberra chapter membership. The plans are designed to bridge the gap between the currently assessed state of the membership and the four benchmarks used for comparison.

This plan is presented in terms of a set of workshop themes that could be delivered by the ICEAA chapter. The actions have a source which links back to the assessment criteria and comments which underpin the action to provide traceability.

8.2 Stakeholders

8.2.1 Senior management workshop

A theme to explore the requirements of senior management from the cost community and establish a dialogue with senior management regarding what the cost community can offer decision makers.

Source: 2.2

8.2.2 Cost engineering with supplier workshop

This workshop needs to be conducted outside of a commercial or contractual event. The theme would explore the needs for the customer and the provision of costs from suppliers with the aim of finding common ground. This could include cost breakdown structures, data sharing and common tools.

Source: 2.3

8.3 Culture

8.3.1 Corporate guidance workshop

An exploratory workshop to share the best practices in corporate guidance within government and industry. Considering the benefit of detailed, comprehensive guidance versus the policy of high level, outline guidance.

Source: 3.2

8.4 People

8.4.1 Risk management workshop

To ensure a quantified risk register is good quality there should be a risk management workshop. This should deal with topics such as risk identification, risk registers, quantification of the probability and impact, justification of the probability and impact, risk analysis, Monte Carlo analysis, correlation and three point estimating / uncertainty.

Source: 4.3



8.5 Process

8.5.1 Cost engineering practitioner workshop

A workshop on the theme of roles and responsibilities of a cost practitioner. This theme would explore the means of earning a cost engineering passport. This workshop would provide and test understanding in all the competencies needed by ICEAA to enable confidence in any cost estimate produced.

Source: 5.2

8.5.2 Cost estimating methodologies workshop

There should be an ICEAA workshop themed on the advantages and disadvantages of different cost estimating methodologies: analogous, parametric and analytical.

Source: 5.3

8.6 Data

8.6.1 Data warehouse and knowledge management workshop

A workshop focused on the theme of gathering cost and technical data, normalisation and application. The theme would explore best practice in data warehousing and appointment of knowledge managers as gatekeepers of quality data.

Source: 6.2, 6.3

8.7 Tools

8.7.1 Risk analysis workshop

This workshop would explore the merits of Monte Carlo analysis when applied to risk management. It would include definitions of probability distributions, correlation, risk and uncertainty.

Source: 7.2

8.7.2 Software estimating workshop

A focused workshop of the benefit of cost models that generate independent cost estimates for software projects. Realising that the software content of projects is increasing this workshop would consider software sizing methodologies; SLOC, function points, object points and use cases.

Source: 7.3

8.8 Improvement roadmap

Each improvement workshop will take the ICEAA Canberra chapter a different amount of time to prepare and will supply different levels of benefit. The definitions below will be used during the benefit/time analysis.



8.8.1 Improvement criteria

The benefit indications are rough order of magnitude assessments of the likely performance improvements considering such elements as availability, cost reduction, schedule impact, process, efficiencies etc. The definitions used to determine the benefit indications are:

High – Greatly improved maturity

Medium – Improved maturity

Low – Small increase in the level of maturity

The time indications are rough order of magnitude to assist ICEAA in their decision making and determination of which workshop to take forward. The definitions used to determine the time indications are:

Long – would take longer to achieve than a one-day workshop

Medium – Achievable within a one-day workshop

Short – Achievable within a half-day workshop

8.8.2 Benefit/Time Analysis

Using the definitions above, the workshop themes have been allocated a benefit and time indicator.

ID Improvement Benefit Time

8.2.1 Senior management workshop L S

8.2.2 Cost engineering with supplier workshop L L

8.3.1 Corporate guidance workshop M M

8.4.1 Risk management workshop L M

8.5.1 Cost engineering practitioner workshop L L

8.5.2 Cost estimating methodologies workshop H M

8.6.1 Data warehouse and knowledge management workshop

H M

8.7.1 Risk analysis workshop H M

8.7.2 Software estimating workshop M M

Table 8-1: Workshop themes with their associated benefit and time indicators

8.8.3 Prioritisation of implementation actions

It is possible to prioritise the workshop themes to determine those that are likely to provide high benefits within a short timescale and shown in Table 8-1.



Time

Short Medium Long

Benefit

Low

8.2.1

8.5.1

8.4.1 8.2.2

Medium

8.3.1

8.7.2

High

8.5.2

8.6.1

8.7.1

Table 8-2: Prioritisation of the workshop themes

8.8.4 Timeline of implementation actions

It is possible to schedule the workshop themes and to plan the forthcoming events into a programme such as the one outlined in Table 8-3.

Initiative Schedule

Up to Apr 17 Apr 17 to Sep 17 Oct 17 and beyond

Stakeholder

-Senior management workshop

-Cost engineering with supplier workshop

Culture

-Corporate guidance workshop

People

-Risk management workshop

Process

-Cost engineering practitioner workshop

-Cost estimating methodologies workshop

Data

-Data warehouse and knowledge management workshop

Schedule to be determined by the ICEAA Canberra

chapter committee.



Initiative Schedule

Up to Apr 17 Apr 17 to Sep 17 Oct 17 and beyond

Tools

-Risk analysis workshop

-Software estimating workshop

Table 8-3: Example implementation schedule for the workshop themes

8.9 Barriers to Change

It is a normal reaction of groups or individuals to place barriers in the way of change. During any suggestion of a change of working practice or alternative way of working, barriers will be identified to try to divert or halt the progress of a change initiative or program.

If a series of ICEAA workshops are planned to improve the community maturity, this human reaction needs to be managed to avoid it from derailing the process. Table 8-4 provides a number of reactions to change and strategies for overcoming them.

Types of Negativity What to do about them

Rational: Misunderstanding of details of plan, belief that change is unnecessary, disbelief in planned change’s effectiveness, expectation of negative consequences

Explain plan with greater clarity and detail.

Project what would happen if the change program was not introduced.

Involve everybody in quality-improvement teams to demonstrate effectiveness of managed change.

Institute a bottom-up program for reorganising systems and processes.

Personal: Fear of job loss, anxiety about the future, resentment at implied criticism of performance, fear of interference from above.

Stress much-improved job prospects for the future for everyone.

Present plans for improvements that people are likely to find positive and exciting.

Accept management responsibility for past failures.

Present a scenario showing the anticipated benefits of the main change.

Emotional: Active and / or passive resistance to change in general, lack of involvement, apathy towards initiative, shock, mistrust of motives behind change

Show, with examples, why the old ways no longer work.

Stage a series of meetings to communicate details of the change agenda.

Demonstrate that the new policy is not merely “a flavour of the month”.

Explain the reasons for the change, and promise involvement.

Be completely honest, and answer all questions.

Table 8-4 - Dealing with negative reaction to change



9 Conclusions and recommendations

9.1 Conclusions

A CEHC survey of the ICEAA Canberra chapter has been undertaken and has been successful in identifying the strengths and weaknesses associated with the cost estimating capability of the cost community represented by the attendees.

The CEHC assessment has demonstrated that while some benchmarks are comparable with previous workshops (e.g. Figure 2-2, Figure 4-1) providing confidence in the CEHC technique; some have identified significant gaps (e.g. Figure 6-1, Figure 7-1) indicating a need for action.

The electronic voting was used successfully to stimulate comments from the attendees. The live display of the voting acted as a catalyst for the attendees to reflect upon the scores and suggest why the voting had a particular pattern.

These comments have been recorded in the body of this report and have led to the improvement initiatives that have been identified in the recommendations section.

9.2 Recommendations

During the workshops a number of workshop themes were identified that have related to the ICEAA Canberra chapter.

These workshop themes have resulted in a proposed improvement plan of actions that has the potential to take the ICEAA members from their current maturity state to the upper quartile of maturity when compared with their peers.



10 List of references

[1] Tracey Clavell, email “RE: ICEAA Canberra event - international guest speaker Dale Shermon - Cost Engineering Health Check”, dated 6 Feb 2017.

[2]



11 List of abbreviations

3PE Three Point Estimate

BOE Basis of Estimate

CBS Cost Breakdown Structure

CEHC Cost Engineering Health Check

CER Cost Estimating Relationship

EVM Earned value management

HTA Historical trend analysis

ICE Independent Cost Estimate

ICEAA International Cost Estimating and Analysis Association

IDR Independent design review

KBE Knowledge Based Estimating

UK United Kingdom

US United States



Report documentation page

Originator’s Document Number QINETIQ/17/00688/1.0

Originator’s Name and Location

Dale Shermon, QinetiQ Bristol, Building 240, The Close, Bristol Business Park, Coldharbour Lane, Bristol, BS16 1FJ

Customer Contract Number and Period Covered

n/a

Customer Sponsor’s Post/Name and Location

Tracey Clavell

Document Protective Marking and any other markings

Date of issue Pagination No. of references

UNMARKED February 2017 Cover + 32 1

Title

International Cost Estimator and Analysis Association (ICEAA) Canberra chapter

Cost Engineering Health Check (CEHC)

Translation / Conference details (if translation give foreign title / if part of conference then give conference particulars)

n/a

Title Protective Marking QinetiQ Proprietary

Authors Dale Shermon

Downgrading Statement N/A

Secondary Release Limitations

N/A

Announcement Limitations N/A

Keywords / Descriptors Cost Engineering Health Check, ICEAA, Canberra

Abstract

This report describes the Cost Engineering Health Check (CEHC) that QinetiQ has undertaken of the ICEAA Canberra chapter.

A CEHC is a structured method of assessing the cost estimation capability of an organisation. It allows strengths and weaknesses to be understood, as well as providing an overall score that can be compared with that of other organisations. This report describes the detailed results of the assessment that has been carried out, as well as an outline plan for actions that can be undertaken to develop further the cost engineering capability of ICEAA members.

Abstract Protective Marking: UNMARKED


Blank page

international cost estimating and analysis association

Documents