leading indicators for es
TRANSCRIPT
Steven R. WoodburyDOE Office of Nuclear Safety
and Environmental Policy
Leading Indicators for ES&H Performance:
Measuring How Well Our Management System Is Performing
ISM Best Practices Workshop September 12, 2006
2
Overview
The mix of ES&H performance measures changes as an ES&H program matures.
DOE currently uses many lagging measures. Our ‘next level’ of performance measurement
should focus on how well key elements of our ES&H management system are performing.
There are instructive examples in the literature of best practices on which we can build.
But nobody says this is easy.
3
Three Levels of Performance Measurement
Grace Wever identifies three types of performance measures that organizations may employ:1. Operational indicators (lagging) For example:
• waste generation• accident/injury rates• regulatory violations2
2. Identifying underlying management system deficiencies (leading) For example, the quality of: • training• audits• communications• documentation• review processes
4
Three Levels of Performance Measurement (continued)
3. Identifying gaps in internal competencies and external relationships (leading)For example, • Organization begins to measure:
– Impacts of products, operations, or services on public health and the environment
– EHS research and development– Capital investments– Stakeholder and customer partnerships
• Organization strives to link financial, human resource, operation, EHS, and other goals and measures
“The mix of an organization's performance indicators tends to reflect the maturity of its EHS program.”
Grace Wever, Strategic Environmental Management, pp 158-9
5
Importance of Leading Indicators
The importance of developing good leading indicators goes beyond “It’s hard to steer by looking in the rearview mirror.”
The importance goes beyond continuous improvement “We should do what leading organizations are doing.”
We need to be concerned because even good performance, on good lagging indicators, can’t assure us that risks are really being controlled.
6
Outcome Measures are Not Sufficient
“When safety is good and injury and loss rates are low, then [outcome] measurements are not sufficient to provide adequate feedback for managing safety.
For operations where there may be potential for severe accidents, the likelihood of such events must be extremely low. This means that the absence of very unlikely events is not, of itself, a sufficient indicator of good safety management.”
--European Process Safety Center, 1996, p. 3.
7
Outcome Measures are Not Sufficient
“Health and safety differs from many areas measured by managers because success results in the absence of an outcome (injuries or ill health) rather than a presence. But a low injury or ill-health rate, even over a period of years, is no guarantee that risks are being controlled and will not lead to injuries or ill health in the future. This is particularly true in organizations where there is a low probability of accidents but where major hazards are present.”
– A Guide to Measuring Health & Safety Performance Health & Safety Executive (UK), p 5
8
Some Best Practices in “Next Level”
Performance Measurement
Guide to Measuring Health & Safety Performance (British Health and Safety Executive)
Total Quality Environmental Management Matrix (Grace Wever, Council of Great Lakes Industries)
Positive Performance Indicators (Australia, Department of Employment and Workplace Relations)
General Electric Company ProSmart software
(Chemical Process Safety Council)
9
Health & Safety Executive (UK)
Guide discusses performance measurement in the context of a health and safety management system
Discusses the limitations of traditional “lagging” measures
Provides a systematic approach to deriving measures which link to the H&S management system
10
Why Measure Performance?
“The primary purpose of measuring H&S performance is to provide information on the progress and current status of the strategies, processes and activities used by an organization to control risks to health and safety”
– A Guide to Measuring Health & Safety Performance Health & Safety Executive (UK), p 7
11
HSE Guide – a Balanced Approach
Performance measurement should be based on a balanced approach, which addresses:
Input Monitoring the scale, nature and distribution of hazards created
by the organization’s activities
Process Active monitoring of the adequacy, development, implementation
and deployment of the H&S management system
Outcomes Reactive monitoring of adverse outcomes resulting in injuries, ill
health, loss and accidents with the potential to cause injuries ill health or loss (‘measures of failures’)
12
HSE Guide – Measure the Management System
The H&S management system is the process which turns uncontrolled hazards to controlled risks
The key elements of• Policy• Organizing• Planning and implementation• Measuring performance• Audit and review
all need to be in place to control risks effectively The performance measurement system must cover
each element of the H&S management system
13
HSE Guide –What and How to Measure
Measuring elements of the H&S management system should cover three aspects:• Capability• Implementation• Deployment
The measurement process can gather information through:• Direct observation of conditions and people’s behavior• Talking to people to elicit facts and their experiences as well
as gauging their views and opinions• Examining written reports, documents and records
14
HSE Guide – Observations
Performance measurement is approached in the context of the health and safety management system
Guide acknowledges the need to measure outcomes – but this is not sufficient:• ‘measures of failure’• doesn’t guarantee that risks are really controlled
Approach is not limited to Health and Safety – completely applicable to management across ES&H
Focus is on the performance of key management system elements
Measures are largely qualitative Provides a clear framework for thinking about ES&H
management and performance measurement
15
Total Quality Environmental Management Matrix
TQEM identifies 7 performance categories, based on Baldrige Criteria most relevant to EHS management, as well as the ISO 14001 standard
Identifies performance levels (1–10) (based on descriptive text)
Detailed self-assessment questionnaire is provided to support evaluation.
Within a category, one cannot score at a ‘higher’ level unless performance is fully satisfactory at all ‘lower’ levels
Developed in the 1990’s by Grace Wever, and the Council of Great Lakes Industries
Seven Categories
Ten
Per
form
ance
Lev
els
16
Total Quality Environmental Management Matrix – Categories
Seven categories (different weights) Leadership (15%) Information and Analysis (7.5%) Strategic Planning (7.5%) Human Resource
Development (10%) Process Management (15%) Environment, Health & Safety
Results (30%) Customer/Stakeholder
Satisfaction (15%)
17
TQEM Performance Levels – Environmental Health and Safety Results
Example: ‘ES&H Results’ Category
10. Benchmarking shows unit is ‘best-in-class’
7. EHS improvements contribute to financial and business improvements
5. Positive improvement trends in units’ key EHS measures
4. EHS measures reviewed and improved at least annually
3. EHS results compared with objectives and targets; used to improve effectiveness of management systems and performance
1. EHS performance measures identified; baseline data and trends collected
EH
&S
Re
su
lts
18
TQEM Performance Levels – Strategic Planning
Example: ‘Strategic Planning’ Category
9. Information on competitors’ EHS strategies used to improve units’ strategies and performance
8. EHS integrated into long and short-term business plans for all of unit’s products, processes and services.
6. EHS plans and deployment consistently aligned at all levels of the unit
4. Long and short-term plans that include EHS objectives are reviewed and improved at least annually. Key measures include EHS measures
1. Long-term and short-term planning process used that addresses EHS needs; annual operating plan includes EHS management needs
Str
ate
gic
Pla
nn
ing
19
TQEM Performance Levels – Human Resource Development
Example: ‘HR Development’ Category
9. EHS needs fully integrated into unit’s HR development plan. Training/education for EHS staff include key business knowledge
7. Employees proactively initiate activities to improve EHS performance
5. Measures of EHS training effectiveness in place; employees with potential to impact EHS are competent to perform EHS responsibilities
3. All employees have received appropriate EHS training; employees aware of EHS compliance requirements.
1. EHS training needs identified, resources committed.
Hu
man
Re
sou
rce
D
ev
elo
pm
en
t
20
TQEM Matrix – Observations
Not limited to environment – completely applicable to management across ES&H
System is based on explicit, but qualitative, criteria for ES&H management
Focus is on qualitative measures of performance in key management categories
Published matrix is fully defined (not open-ended) Matrix was developed more than 10 years ago –
it should be updated on the basis of additional experience
Its importance is as a conceptual framework; the specific content can be adapted
21
Positive PerformanceIndicators
‘Positive Performance Indicators’ (PPIs) focus on assessing how successfully an organization is performing by monitoring the processes that provide good OHS outcomes.
PPIs are to be developed in addition to traditional ‘outcome’ indicators.
PPI program was developed by the Australian Government’s Department of Employment and Workplace Relations
22
Positive Performance Indicators – Based on Quality Management
PPI is based on a Quality Management approach. PPIs may be Inputs (Key Activities)
• Measures of actions or initiatives undertaken to improve OHS• Provide indicators of commitment and effort, but not of
effectiveness• Can provide useful information on participation, leadership and
communication Processes (Monitoring Key Risks)
• Identify key risks• Identify key contributors to outcomes of concern• Develop measures to monitor behaviors and practices
Outputs (Milestones)• Achievement of objectives • Progress toward achievement of higher-level OHS goals and
targets
23
Positive Performance Indicators – Part of a Management System
PPIs are only useful if an organization has a systematic approach to management of OHS
Many variations exist, but all have the following principles:• Commitment and policy• Planning• Implementation• Measurement and evaluation• Review and improvement
24
Positive Performance Indicators – Categories of PPIs
Commitment and policyMeasures demonstrated commitment to improve OHS performance
PlanningMeasures what procedures are in place to eliminate workplace injury
and disease Implementation
Measures the capability and support mechanisms that are necessary to achieve OHS objectives and targets
Measurement and EvaluationMeasures the extent to which workplace health and safety is monitored
and evaluated so that issues can be identified and corrective action taken
Review and improvementMeasures the effectiveness of the OHS management system, and its
continuing suitability
25
Examples of PPIs – Commitment and Policy
Performance Indicators How to Measure
Evidence of OHS policy statement signed by CEOFrequency and quality of OHS reporting by or to Senior ManagementSenior managers’ performance appraisals include OHSPercentage of workforce and contractors covered by consultation processes and OHS representationRating of effectiveness of employee participation on OHS management
Examination of recordsEmployee questionnaire/survey
26
Examples of PPIs – Implementation
Performance Indicators How to Measure
Percentage of workplace inspections conducted over a specified timeframePercentage of high risks identified over a specified timeframeProportion of items identified through safety walks and inspections that are repeat itemsProportion of reported incidents that do not result in injury compared with those that doPercentage of planned management visits conducted over a specified timeframePercentage of managers and employees that have received OHS training (e.g. induction, job-specific, hazard management, emergency procedures….)
Observation – walk through inspections/ auditsExamination of hazard reportsExamination of hazard logsReview of maintenance logAnalysis of accident and incident reports
27
Positive PerformanceIndicators – Observations
PPIs are not limited to Occupational Health and Safety – completely applicable to management across ES&H
PPIs focus on identifying key elements of the ES&H management system, and measuring how well these key elements are functioning
PPIs are non-prescriptive and open-ended: approach provides great flexibility at the facility and site level
Most PPIs are qualitative, based on assessments, record reviews, surveys, or audits
The Guidance document assumes a fairly basic management system. DOE sites generally have more sophisticated ES&H management systems in place, with the opportunity to develop more sophisticated measures.
28
General Electric –Health and Safety Framework
At the April ISM meeting in Albuquerque, Kurt Krueger described GE’s corporate-wide Health and Safety program
“One standard, one program, one set of metrics for every GE facility around the world – No exceptions!”
GE’s Health and Safety Framework includes a mix of 21• Management system elements
For example:– Training– Accident investigation and follow-up
• Subject matter areas For example:
– Industrial hygiene– Chemical management– Motor vehicle safety
29
General Electric (continued)
Detailed questions
GE-specific guidance & expectations
30
General Electric (continued)
Scorecard ratings are based on • Plant self-assessment (540 questions, twice a year)• Periodic corporate audits
Tied to monthly site operational metrics and supervisor scorecards,• “tailored to site operations”• “designed to drive supervisor behaviors that will find
and fix H&S issues before an accident finds them” GE has a parallel system for assessing
environmental performance GE also continues to track “trailing metrics”
For example:• Recordable injury and illness rates• Lost time injury and illness rates• Permit exceedances• Notices of noncompliance
31
General Electric –Observations
Integrated into GE’s broader management system Reflects a balance between
frequent site self-assessment and reporting, and periodic headquarters audits
GE chose to implement a uniform corporate system. While this might not be DOE’s choice, there is still a lot to learn from the overall framework, and from the measures GE’s chose to include
32
ProSmart – Center for Chemical Process Safety
In 1993, CCPS began a project to measure the effectiveness of process safety management systems
Their objective was“to provide management with the tools for assessing the health of process safety management systems on a real-time basis”
Based around 12 management system elements Extensive set of computer-based questions,
using qualitative and quantitative measures Rolls up responses into a single score or ‘index’
33
CCPS ProSmart – Based on 12 Management System Elements
CCPS identified 12 key management system elements, such as• Accountability• Process knowledge and documentation• Capital project hazard review• Training and performance• Management of change• Incident investigation• Audits and corrective action
ProSmart generates a score, which is continually updated
34
CCPS ProSmart – Based on 12 Management System Elements
35
CCPS ProSmart – Drills Down to Detailed Evaluation Questions
36
ProSmart – Observations
Derived from a quality management approach Focus is on specific management system elements Established for chemical process safety,
but in principle could encompass environment,safety and health
Like TQEM and GE, ProSmart provides a set of detailed assessment questions, for qualitative assessment against established criteria
37
Key Aspects of “Next Level” Performance Measures
“Next Level” performance measures identify and measure how well key elements of the ES&H management system are functioning
Measures tend to involve• Precursor analysis and trending• Observations, assessments and audits
Measures tend to be qualitative rather than just counting events
(This doesn’t mean they can’t be rigorous and replicable)
38
DOE Has Many Pieces in Place
Analysis and trending• Data collection • Trending and analysis• Root cause analysis
Observations, assessments, audits• Contractor self-assessment processes• DOE oversight activities• Annual ISM assessments should systematically
address management system elements• Inspections by the Office of ES&H Inspections (HS-64,
formerly SP-44) address ES&H management system functioning at a high level (ISM core functions)
• Individual IG audits address some management system elements
39
Audits and Inspections
40
ES&H Inspections (colors added)
Define Scope of Work
Analyze the Hazards
Identify and Implement Controls
Perform Work Within Controls
Feedback and Continuous Improvement
(contractor) (DOE)
Essential System Functionality
41
ES&H Evaluations of 17 DOE Sites 2002-2004
Management System Area
Effective Performance Needs ImprovementSignificant
Weakness
Analyze the Hazards 10 7
Develop and Implement Controls
5 11 1
Work within Controls 12 5
Feedback and Improvement 2 15
42
Summary
There is a common thread of best practice, across many different organizations: to move beyond outcome (lagging) indicators, we need to start systematically measuring how well key elements of our management system are performing.
This is not as easy as counting occurrences (injuries, enforcement actions): it must involve walk-throughs, audits, surveys, and other varied approaches.
This ‘next level’ of leading performance measurement complements, and does not replace, outcome (lagging) measures.
Approaches developed for process safety, worker safety, environment, can be applied across ES&H. We need to be eclectic and ecumenical in our thinking.
Nobody has a turnkey system that’s right for DOE. We need to identify and build on the best of these various approaches.
Nobody says it will be easy.
43
Next Steps
Identify key ES&H management system elements• We currently categorized some information (e.g., ORPS
reports; Independent Oversight Assessments) around ISM’s five broad core functions
• Within these core functions, we can identify more specific management system elements, including:
- Identification of requirements - Corrective action management- Hazard identification - Lessons Learned sharing- Communication - Performance Measurement- Training - Worker involvement- Documentation - Audits and assessments- Periodic top management review
44
Next Steps (continued)
Identify the various things we are already doing to assess how well these elements are functioning
Identify and fill gaps as appropriate Organize and present the information in a way which
provides current feedback to management on how effectively key management system elements are functioning at each site and each facility
In Sum: Use the health of these key management system elements as the conceptual and organizing framework to strengthen our ability to know how well we are managing risks
45
In Closing …
“Far better an approximate answer to the right question which may be difficult to frame,
than an exact answer to the wrong questionwhich is always easy to ask.”
– statistician John W. Tukey
46
References
Grace Wever, 1996, Strategic Environmental Management: Using TQEM and ISO 14000 for Competitive Advantage, John Wiley and Sons, NY
European Process Safety Centre (Jacques van Steen, ed.), 1996, Safety Performance Measurement, Institution of Chemical Engineers, Rugby (UK) (distr by Gulf publishing Co., PO Box 2608, Houston, TX 77252-2608)
Health and Safety Executive (United Kingdom), 2001, A Guide to Measuring Health & Safety Performance http://www.hse.gov.uk/opsunit/perfmeas.pdf
Commonwealth of Australia (Department of Employment and Workplace Relations), 2005, Guidance on the Use of Positive Performance Indicators to Improve Workplace Health and Safety http://www.nohsc.gov.au/PDF/Standards/GuidanceNotes/ASCCPPIGuidanceBooklet.pdf
Kurt Krueger, “Creating a Culture of Safety Excellence: The Journey and the Prize,” talk at DOE ISM Champions Workshop, Albuquerque NM, April 2006 http://www.eh.doe.gov/ism/workshops/Creating_a_Culture_of_Safety_Excellence.pdf
George Eckes, The Six Sigma Revolution: How GE and Others Turned Process into Profits, New York: John Wiley and Sons, 2000.
Center for Chemical Process Safety, ProSmart software http://www.aiche.org/CCPS/Publications/Software/ProSmart/index.aspx
47
Contact Information:
Steven R. Woodbury
DOE Office of Nuclear Safety and Environmental Policy
202-586-4371