astm e2936 standard guide for contractor self assessment for u.s. government property management...

ASTM E2936 Standard Guide for Contractor Self Assessment for U.S. Government Property Management

Systems

Brandon Kriner, CPPM CFHarris Corporation

Background• FAR 52.245-1 requires contractors to self-assess, but no

specific guidance is provided in the clause• There is much confusion and debate on Contractor Self

Assessment (CSA)• What constitutes an “acceptable” CSA?• What should be the role of CSA in a Government audit or

PMSA?• The guide is intended to compile best practices from both a

contractor and Government point of view- not a step-by-step manual- every environment is different

• (Hence a standard “guide” and not a “practice”)

Task Group• Brandon Kriner, Harris Corporation (Lead)• Alex Barenblitt, BSL Company• Cinda Brockman, A2B Tracking• Eric Fassett, Northrop Grumman Corporation• Bill Franklin, Noblis, Inc.• Doug Goetz, GP Consultants• Mike Showers, NASA• Rick Shultz, JHU Applied Physics Laboratory• Tamra Zahn, The Boeing Company

Scope• The standard is intentionally narrow in focus- specifically

addresses the requirements of FAR 52.245-1• Many of the concepts could be applied to other types of

asset management self-assessments, audits, surveillances, reviews, etc.

• The FAR Government Property focus is intended to directly address the widespread challenges associated with CSA implementation and acceptance since 2007

Significance and Use• The guide provides a foundation for minimum effective internal

assessment of a contractor’s Government property management system

• Your CSA may go farther/deeper- probably already aligned with the standard!

• The guide is a menu, not a recipe• A contractor may use all or parts of the guide in accordance

with its procedures and operating environment• Self assessment should be used to identify and correct potential

deficiencies or areas of risk independent of a Government audit• Also provides insight into continuous improvement opportunities

Significance and Use• Self assessments are not a replacement for PMSAs or

other external audits- they are distinct requirements• A contractor is required to self-assess in order to proactively

address issues• The Government has a fiduciary responsibility to ensure that

its property is adequately managed by contractors• The results of CSAs should be made available to Government

auditors for potential inclusion in their audits or reports• Self disclosure of potential issues (and evidence of

corrective action) may preempt CARs (C. Williams Letter 9/13)

• Self disclosure of areas of strength may minimize the time spent on a PMSA

Independence and Objectivity• To the extent possible, a CSA should provide a level of

objectivity similar to that of a PMSA• Individuals performing CSA should be independent when

possible• The contractor’s procedures should address independence

• People should not be assessing their own work• Example: calibration lab performs calibration; Property

Management assessing % of on-schedule calibration of assets

Procedures• Contractors should clearly describe their CSA in their

procedures• The following concepts should be addressed:

• Methodology used (examples):• Government PMSA criteria• ASTM E2452 EMPM• ILPs/Customary Commercial Practices• Balanced Scorecard/CMMI

• In general, the closer you are to the Government’s methodology, the greater the chance of your CSA results being incorporated

Procedures

• The processes/outcomes to be reviewed– These may be the “10 outcomes/15 functions”

or other processes/contractual requirements

• The organizational scope of the CSA– Business units, sites, other sub-divisions to

which the assessment applies

Procedures

• A “defect” should be defined• The differences between minor, major and

critical defects should be defined in the context of the business environment

• Corrective action requirements for defects should be established

Defects- E2936 Definitions• Defect, n – a condition in which a functional

segment, a sample item or sample item element of a property control system contains one or more deficiencies (E 2315).

• Definitions are loosely aligned with DFARS 252.245-7003 Contractor Business System Deficiencies (aka the BSR)

Defects- E2936 Definitions• Minor defect – a defect that is administrative in nature,

non-systemic and would have no material outcome for the control of Government property.

• Major defect – a significant, but not systemic defect that may affect the control of government property, possibly increasing the risk to the Government.

• Critical defect – a significant and systemic defect that would have a material effect on contract performance or cause concern for the reliability of the information provided by the property management system.

Defects- Examples• Minor defect – Characters transposed in a model

number; asset location one room off (no material impact on contract performance or management information)

• Major defect – A $500,000 test set has been improperly stored, leading to damage (significant; not systemic)

• Critical defect – The contractor automatically disposes of all Government property as scrap at its discretion without gaining Government approval or reporting to the Government (significant and systemic)

Risk Assessment• Contractors should perform a risk

assessment in planning a CSA• The frequency of a CSA or parts of a CSA

should be based upon the risk assessment– Direct resources to areas that present risk;

don’t overdo low-risk areas

Process Tests• Contractors should establish process tests

that provide sufficient evidence to credibly evaluate the effectiveness and risk level of the property system

• Process tests may be quantitative (metrics, statistical sampling) or qualitative

• Support documentation is critical• The standard suggests processes to test

Population Selection• Sample data are observed in order to

estimate attributes of the entire population• Transaction-based population

– Driven by actions over a set period of time, e.g. all receipts of GP over the past year

• Attribute-based population– Based on common characteristics, e.g.

condition of storage areas

Population Selection• Processes may have more than one

population– Acquisition of GFP vs. CAP– Sensitive property– ABC or Low Risk Property (ASTM E2811)

Population Selection• Some populations may be used to test

multiple processes– Records and Utilization– Acquisition and Receiving

Sampling• Statistical Sampling: use of random

statistical tests to estimate the characteristics of a complete population

• Judgment Sampling: non-random, non-probability; auditor selects items based on knowledge and experience

• Purposive Sampling: selecting specific items based on knowledge of a situation

Sampling• Contractors must identify the statistical

sampling plan to be used– DCMA 90%,95%,97% Double Sampling Plans– ASTM E2234 AQL 6.5 Single or Double

Sampling Plans– Others

• The statistical sampling plan used should be determined by risk and other factors

Sampling• Statistical sampling populations should be

randomly generated using automated tools– http://www.randomizer.org– Excel’s sampling tool under the Data Analysis

add-in

• The standard contains several statistical sampling tables in the appendix

Evaluation of Samples• Defects should be analyzed from both a

quantitative and qualitative perspective– Quantitative- based on the acceptance rates

in the statistical sampling program– Qualitative- the relative significance and

materiality of the defects• Even quantitative “failures” may be immaterial!• $2.48 cents out of a population of $500,000

Significance• Defined in GAGAS as the relative importance of

a matter within the context in which it is being considered– Magnitude in relation to overall system– Nature and effect of the defect– Relevance and impact on overall contract

performance

Significance• Business Systems Rule Definition: • DFARS 252.242-7005: “Significant deficiency”

means a shortcoming in the system that materially affects the ability of officials of the Department of Defense to rely upon information produced by the system that is needed for management purposes.

• This is a high bar that should not be hastily applied.

Corrective Actions and Plans• Contractors should take corrective action

to resolve issues and mitigate risk as they become known during the CSA.

• The cost and burden of corrective actions should be commensurate with the significance and risk presented to the Government

Corrective Actions and Plans• Significant risks (critical or major defects) should

be addressed through a formal written CAP• Identify steps taken to identify and analyze the

root cause, mitigate the risk, the resources required, and the specific timeline for implementation

• Minor defects should be corrected at the lowest effective responsible level of contractor personnel

Get the Standard• Join ASTM: http://www.astm.org• $75 for individual membership- provides

access to all 27 active standards under E53 Committee on Asset Management

• Your organization may already have an organizational membership