1 the environmental management program more than cleanup orps safety data normalization joint hss/em...

1

The Environmental Management Program

More Than Cleanup

ORPS Safety Data NormalizationJoint HSS/EM Presentation

Terry KrietzOffice Of Health, Safety and Security

Bob MurrayOffice of Environmental Management

Integrated Safety ManagementBest Practices Workshop

September 12-13, 2006

2

• Background

• Model development approach

• Work hours -- EH’s approach to normalization

• EVMS -- EM’s approach and experience to normalization

• Next steps

ORPS Safety Data Normalization

3

Background

• EM-1 requested EH support the development of methodology to normalize ORPS data as a way to compare safety performance at the project, contractor, and site level

• EH-3 and EM-60 explored traditional and innovative approaches to normalization of available data

• Two models were developed and tested – both are active normalization models today:– Each weights individual ORPS events for severity using the field

assigned Significance Category– Each uses only those events reflective of safety performance

4

Why Normalize?• Places organizations on a even playing field – accounts for

size of effort and incorporates occurrence severity.

• Allows comparison of safety performance site by site, by contractor, and corporately where data is available.

• Normalization can identify areas of needed improvement that may not otherwise be visible -- by making comparisons to a PSO overall performance or a site’s specific performance.

• For the EM adopted model, specific EVMS criteria integrated into the normalization formula will account for project size and performance.

5

Conditions and Restrictions for Developing Normalization Models

• All data must already be available to HQ. – No additional data calls would be made to the field.

• Subjectivity with respect to data input would not be allowed.

• The models were not expected to reflect total safety performance of any organization.

• Restrictions on work hour availability for use as normalizing factor– EM desired monthly work hours, but CAIRS reported quarterly – Actual hours worked under each specific EM project not available at

HQ

• One model would reflect EM’s project based approach to work using the Earned Value Management System (EVMS).

6

Normalization Models Developed and Tested

• Work Hours Model:– Traditional use of work hours as normalization denominator

(e.g., Total Recordable Case Rates)– Data available in ORPS and CAIRS (with some current

limitations)– Safety-related performance only; does not account for

work/schedule performance – Accounts for Severity of ORPS events

• Earned Value Management System (EVMS) Model:– Accounts for size and work performance of organization– Data available in ORPS and IPABS (EM project database)– Accounts for Severity of ORPS events

7

(Σ ORPS Reports Weighted for Severity) x ($10 mil) scale factor

(Σ ACWP) Report Period x (Σ BCWP / Σ BCWS) Report Period

(Σ ORPS Reports Weighted for Severity) x (100,000 work hours) scale factor

(Total work hours for report period)

NS EVMS =

NS work hour =

ACWP = Actual Cost of Work Performed SPI = Schedule Performance Index (BCWP/BCWS)

BCWP = Budgeted Cost of Work Performed BCWS = Budgeted Cost of Work Scheduled

Σ = Summation

Normalization Model Formulas

Work Hour Model

EVMS Model

8

Factors Involved in Work Hour Normalization Model

• Total number of occurrences– Obtained from ORPS

• Occurrence Severity– Severity Scores derived by weighting factor for field assigned

occurrence significance

• Work Hours– Measures size of organization (from CAIRS data)

• Scale Factor– Inserted to address magnitude of work hours (e.g., TRC rates

are per 200,000 work hours)

9

• Suspect/Counterfeit Items - Reporting Criteria 4C(2)

Discovery of any suspect/counterfeit item or material other than office supplies, office equipment, or household products.

• Onsite Legacy Radioactive Waste Contamination – Reporting Criteria 6B(4)

Identification of onsite legacy radioactive contamination greater than 10 times the total contamination values in 10 CFR 835 Appendix D and that is found outside of the following locations: Contamination Areas, High Contamination Areas, Airborne Radioactivity Areas, Radiological Buffer areas, and areas controlled in accordance with 10 CFR 835.1102(c). For tritium, the reporting threshold is 10 times the removable contamination values in 10 CFR Part 835, Appendix D.

• PISA’s – Reporting Criteria 3B(2)

Declaration of a potential inadequacy of the documented safety analysis (PISA) (a potential positive USQ), per 10 CFR 830.203(g).

Occurrences not reflecting Safety Performance are filtered out of normalization calculation

10

Severity Weighting Factor for ORPS Event

• Significance Category 4 = 1 point• Significance Category 3 = 2 points• Significance Category 2 = 4 points• Significance Category 1 = 20 points • Significance Category R = 0 points

– Occurrence information in R reports has been previously captured in other ORPS reports.

• Significance Category OE = 1 to 20 points per OE reporting criteria:

– Operational Emergency not needing further classification = 1 point

– Alert = 2 points

– Site Emergency = 4 points

– General Emergency = 20 points

11

Office of Environment, Safety and Health

Normalization Work Hour Model Initial Test Examples

for “Demonstration Purposes Only”

12

PSO ORPS Normalization Score Trends(Normalized by Work hours, Weighted by Severity)

0

0.5

1

1.5

2

2.5

3

FY2004 Qtr 2 FY2004 Qtr 3 FY2004 Qtr 4 FY2005 Qtr 1 FY2005 Qtr 2 FY2005 Qtr 3 FY2005 Qtr 4 FY2006 Qtr 1 FY2006 Qtr 2 FY2006 Qtr 3

EM Trend

NE Trend

NA TrendFE Trend

EE Trend

RW Trend

SC Trend

Based on Index Normalized per 100,000 Man hours with ORPS Scores Weighted by Significance

Trend Lines: Calculated by a level 2 polynomial of the monthly data points to reflect general Trends over random monthly cycles.

Improving Performance

13

SC Site ORPS Normalization Score Trends - Chart 1 of 2BNL, ANLE, LBL, FNAL Sites

(Normalized per Work Hours, Weighted by Severity)

0

0.25

0.5

0.75

1

1.25

1.5


SC Wide Trend

Argonne East Trend

Brookhaven Trend

Fermi Trend

Lawrence Berkeley Trend


14

SC ORPS Normalization Score Trends - Chart 2 of 2RL, TJNA, ORR, SLAC Sites

(Normalized per Work Hours, Weighted by Severity)

0

0.2

0.4

0.6

0.8

1


SC Wide Trend

Thom. Jeff. Nat. Acc. Trend

Pacific NW Natl Lab Trend

Stanford Acc. Trend Oak Ridge Trend


15

NA Site ORPS Normalization Score Trends

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5


Pantex Trend

Y-12 Trend

KCP Trend

LANL Trend

LLNL Trend

NV Trend

SNLA Trend

NTS Trend

NA Wide Trend

Based on Index Normalized per 100,000 Work hours with ORPS Scores Weighted by Significance

16

SC Site ORPS Normalization Score(Normalized per Work Hours, Weighted by Severity)

0

0.5

1

1.5

2

2.5

3

SC Wide Ames Lab Argonne East Brookhaven Fermi Princeton Law renceBerkeley

Stanford Accel Thom. Jeff. Nat.Acc.

Richland Oak Ridge


SC Average Norm

*

* Value calculated using the average manhours from 1/2004 to 3/2006.


17

FE Site ORPS Normalization Score Trends(Normalized by Work hours, Weighted by Significance)

0

0.5

1

1.5

2

2.5

3


FE Wide Trend

Natl Energy Tech Lab Trend

Strategic Petroleum Reserves Trend


18

NE Site ORPS Normalization Score Trends(Normalized by Work hours, Weighted by Severity)

0

0.5

1

1.5

2

2.5

3


NE Wide Trend

Argonne West Trend

Idaho Trend


Last data FY2005 Qtr 2

Bechtel BWXT ID, LLC. Battelle Energy Alliance, LLC

19

PSO ORPS Normalization ScoresEXAMPLE CONTROL CHART

0

0.5

1

1.5

2

2.5

3

3.5

FY2004 Qtr2

FY2004 Qtr3

FY2004 Qtr4

FY2005 Qtr1

FY2005 Qtr2

FY2005 Qtr3

FY2005 Qtr4

FY2006 Qtr1

FY2006 Qtr2

FY2006 Qtr3

EE

EM

FE

NA

NE

RW

SC

DOE Aver

Warning

UCL

20

Work Hour ModelIssues and Limitations

• CAIRS organizational coding by contractors and subcontractors not always one-to-one fit with facilities/PSO designated in ORPS

– e.g., WSRC hours are not currently broken down between EM and NA– e.g., BNL hours not currently broken down between SC and EM.

• Actual Work Hour data only available quarterly

– Data can be extrapolated from previous quarterly data reported in CAIRS

• Persistent late reporting of hours into CAIRS

21

Office of Environment, Safety and Health

EVMS Normalization Model Initial Test Examples

for “Demonstration Purposes Only”

22

Who in EM Uses the Data

• Senior EM Management

• The Office of Safety Management and Operations (EM-60)

• Other HQ Management and Staff

• Field Management and Staff

• DNFSB

23

How is the Data in EM Being Used

• “Monthly Safety Report.” • Quarterly Performance Reviews (QPRs)

with Field/Project Managers.• EM contractor periodic “check-up.” • One safety performance indicator for EM-

projects and contractors. • One metric for annual site safety

performance goals.

24

∑[ ]

Closer Look at the EM Normalization Formula

NormalizationScore (NS)

Total # of Category 1-4 ORPS events

(SPI)(ACWP) x

Factor for project

performance

Factor for project size or

work accomplished

Factor for severity of ORPS

events

Severity Index( ) ( ) x ($10M)

=

Scale Factor

25

Factors Included In EVMS Normalization

• ACWP – Actual Cost of Work Performed• Measures the size of the project

• SPI – Schedule Performance Indicator• Gives a numerical value to each site for their performance based on

schedule adherence

• Occurrence Severity• Severity scores are designated by ORPS significance and an

assigned weighting factor

• Total Number of Occurrences• Obtained from ORPS

• Scale Factor• Inserted to offset the magnitude of the ACWP

26

Simple Normalization Example

• Site x had one LO/TO (2.C.2) during the month of May. The occurrence was a category 3 (severity index = 2).

• Site x also had several S/CI occurrences and several legacy contamination discoveries.

• The project’s ACWP was $10M for the month and its SPI = 0.98

• The Normalization formula concludes:

∑[ ]2.04

1

(0.98)($10M) x

2( ) ( ) x ($10M)

=

27

Examples of EM Normalized Data

EM Monthly Safety Summary Report

August 2006

Office of Environmental Management Office of Safety Management and Operations, EM-60

28

Field Office/Site Trend LinesORPS Score

(Weighted by Significance, Normalized per EVMS)

0

1

2

3

4

5

6

7

2005Mar

2005Apr

2005May

2005Jun

2005Jul

2005Aug

2005Sep

2005Oct

2005Nov

2005Dec

2006Jan

2006Feb

2006Mar

2006Apr

2006May

2006Jun

2006Jul

Proj.2006Aug

Site A

Site F

Site CSite G

Site H

Site D

Site B

Site E

EM Wide Trend

29

Monthly ORPS Scores by Contractor CorporationOccurrence and EVMS data from Joint-Venture companies are included in each parent’s corporation bar graphs.

0

1

2

3

4

5

6

7

8

9

EM Wide Site A Site B Site C Site D Site E

2006 Jan 2006 Apr 2006 May 2006 Jun 2006 Jul Proj. 2006 Aug

EM Average Norm

30

Field Office EM Wide Site 3 Site 7 Site 8

Site / Month EM Wide Contractor A Contractor B Contractor A Contractor B Contractor C Contractor A Contractor A Contractor B Contractor A Contractor B Contractor A Contractor B Contractor A Contractor A

2005 Mar 1.97 2.98 0.00 0.00 0.92 2.49 4.23 0.41 4.06 3.54 0.00

2005 Apr 2.51 2.15 0.65 1.73 2.87 3.31 4.27 0.00 0.77 2.66 0.00

2005 May 2.20 1.30 5.93 0.76 1.92 4.43 1.25 3.41 1.91 0.72 2.93 0.00

2005 Jun 2.97 9.48 2.40 0.00 5.18 1.44 5.77 4.00 3.99 1.49 0.32 5.20 0.00

2005 Jul 2.86 4.71 4.15 4.18 1.82 1.31 8.84 10.61 5.74 0.61 0.00 4.39 0.71

2005 Aug 3.64 1.99 1.80 0.78 2.62 2.60 8.41 2.65 5.80 4.00 1.49 0.00 5.25 0.00

2005 Sep 2.27 1.57 1.36 0.00 0.00 2.29 3.14 5.10 1.84 2.28 4.70 1.89 1.83 1.32

2005 Oct 1.77 2.28 2.58 0.00 0.00 5.46 7.05 0.00 3.47 0.00 0.91 6.08 3.84 0.00

2005 Nov 2.36 1.29 4.37 0.00 1.01 0.50 3.88 1.21 6.58 4.29 0.00 4.10 1.36 0.00

2005 Dec 2.22 0.00 6.00 0.00 0.00 2.36 4.55 3.81 1.59 0.00 0.43 0.35 3.45 0.99

2006 Jan 2.66 2.95 1.78 0.46 3.54 0.47 8.32 4.99 5.12 0.00 0.37 1.01 3.22 1.63

2006 Feb 4.99 0.25 7.28 2.98 0.00 0.95 2.99 0.00 5.61 3.01 6.57 1.64 3.88 1.64

2006 Mar 1.92 5.04 0.26 0.55 0.00 0.55 1.82 7.77 3.19 3.35 1.72 0.37 2.82 0.00

2006 Apr 1.49 3.23 0.26 5.20 0.00 1.82 3.42 0.00 1.03 0.61 3.28 1.20 1.84 2.67 0.00

2006 May 1.86 5.49 2.96 1.32 5.51 1.01 1.91 2.27 0.92 3.20 0.41 1.56 3.77 1.35 1.05

2006 Jun 1.47 5.32 2.58 3.40 0.00 0.06 0.72 0.63 0.00 3.76 2.86 1.62 1.74 2.26 3.38

2006 Jul 1.65 2.51 1.87 1.09 0.00 0.34 2.51 2.01 0.00 3.36 6.60 0.67 0.00 1.39 2.82

Proj. 2006 Aug 2.10 1.13 4.19 0.00 7.72 0.34 0.69 0.84 6.09 4.61 1.71 0.00 0.00 2.81 0.00

Excellent (< 2.67)Good

(> 2.67 , < 3.92)

Investigate(> 3.92, < 4.55)

Define Corrections(>4.55)

Month is indicated as YELLOW when score is greater than 2 Standard Deviations(> 3.92, < 4.55) above the EM mean.

Month is indicated as GREEN when score is greater than the EM mean and less than 2 Standard Deviations (> 2.67 , < 3.92) above the EM mean.

Month is indicated as Red any time the score is greater than 3 Standard Deviations (>4.55) above the EM mean.

Site 2

Month is indicated as BLUE when score is below the EM mean (< 2.67).

Site 1 Site 4 Site 5 Site 6

Normalized ORPS Score Preliminary August 2006 Dash Board

31

Non Normalized Safety Data

• Specific occurrence categories have control chart plots created to show the statistical significance of performance history and a given data point.

• Helps Analyze Random Data. • A data set of 25~30 data points is needed.• How to create a control chart:

– A plot of the # of occurrences per month is created,– A mean line is calculated and added to the plot,– The upper and lower control limits (UCL/LCL) line is

plotted at a value of 3 standard deviations away from the mean.

32

Control Chart Example 1

Total EM Electrical Occurrences - Control Chart2004 ~ July 2006

0

2

4

6

8

10

12

14

16

Jan-04

Mar-04

May-04

Jul-04

Sep-04

Nov-04

Jan-05

Mar-05

May-05

Jul-05

Sep-05

Nov-05

Jan-06

Mar-06

May-06

Jul-06

Month (Calendar Year)

# O

ccu

rren

ces

Monthly Occurrences Mean UCL UCL Warning (2 Stdv) LCL Warning (2 Stdv)

33

Control Chart Example 2Total EM Occurrences

EM ORPS Score Control Chart

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

2004Jan

2004Feb

2004Mar

2004Apr

2004May

2004Jun

2004Jul

2004Aug

2004Sep

2004Oct

2004Nov

2004Dec

2005Jan

2005Feb

2005Mar

2005Apr

2005May

2005Jun

2005Jul

2005Aug

2005Sep

2005Oct

2005Nov

2005Dec

2006Jan

2006Feb

2006Mar

2006Apr

2006May

2006Jun

Proj.2006Jul

ORPS Score Warning Level (2SD) EM Mean UCL (3SD)

34

Non Normalized Occurrence Data Dash Board

Jul-06

Occurrence By TypeJul 2006

Authorization Basis

ElectricalOffsite Medical

TreatmentNear Miss

Nuclear Criticality

Environmental Release

Conduct Of Operations

Equipment Degradation/

FailureFire Protection

Industrial Operations

OS/IHRadiological

Control

Total EM Occurrences B B B B B B B B B G B B

Oc

cu

rre

nc

es

By

Sit

e

Site 1 B B B G B B B B B B B B

Site 2 G B G B B B B B B B B B

Site 3 B B G B B B B B B B G G

Site 4 B G G G B B G B B G B B

Site 5 B B B Y B B B B B G B B

Site 6 B B B B G B B B B B B B

Site 7 B B G B B G B B B B B B

Site 8 R B B B B B B B B B B B

W Cond FormatColor Trend ArrowRED

YELLOWGREENBLUE Below Mean Value

Consider Corrective Action (2 STDV)

Notable downward (improving) trend in the frenquency of occurrences over the past 6-9 months.

Notable upward (worsening) trend in the frequency of occurrences over 6-9 months

Trend ConditionMeaningTake Corrective Action (3 STDV)

Above Mean Value

EM Occurrence DashboardJuly 2006

35

Uncertainty in Second Degree Polynomial Fits

• The polynomial trend line attempts to filter out data extremes to reflect the general position and direction of a curve.

• The Coefficient of Determination (R2) is a way to mathematically say how close a curve represents the data.

• How well do second degree polynomial fitted curves represent EM normalized data?

36




0

1

2

3

4

5

6

7

2005Mar

2005Apr

2005May

2005Jun

2005Jul

2005Aug

2005Sep

2005Oct

2005Nov

2005Dec

2006Jan

2006Feb

2006Mar

2006Apr

2006May

2006Jun

2006Jul

Proj.2006Aug

Site A

Site F

Site CSite G

Site H

Site D

Site B

Site E

EM Wide Trend

37




R2 = 0.1692

R2 = 0.2345

R2 = 0.2724

R2 = 0.4606

R2 = 0.1225

R2 = 0.0652

R2 = 0.1459

R2 = 0.2553

R2 = 0.2678

0

1

2

3

4

5

6

7

2005Mar

2005Apr

2005May

2005Jun

2005Jul

2005Aug

2005Sep

2005Oct

2005Nov

2005Dec

2006Jan

2006Feb

2006Mar

2006Apr

2006May

2006Jun

2006Jul

Proj.2006Aug

Site A

Site F

Site CSite G

Site H

Site D

Site B

Site E

EM Wide Trend

38


• These R2 values are extremely low.

• Trending analysis should be exercised with care.

• Typical R2 values for engineering and science:– 0.95 – 0.97

• These values can be raised by increasing the order of the polynomial.– For example:

Site Coefficient of Determination R2

EM Wide 0.1692Site A 0.2053Site B 0.0652Site C 0.2345Site D 0.2724Site E 0.2678Site F 0.1459Site G 0.1225Site H 0.2553

39




R2 = 0.1812

0

1

2

3

4

5

6

7

2005Mar

2005Apr

2005May

2005Jun

2005Jul

2005Aug

2005Sep

2005Oct

2005Nov

2005Dec

2006Jan

2006Feb

2006Mar

2006Apr

2006May

2006Jun

2006Jul

Proj.2006Aug

Site G

EM Wide Trend



R2 = 0.1225

0

1

2

3

4

5

6

7

2005Mar

2005Apr

2005May

2005Jun

2005Jul

2005Aug

2005Sep

2005Oct

2005Nov

2005Dec

2006Jan

2006Feb

2006Mar

2006Apr

2006May

2006Jun

2006Jul

Proj.2006Aug

Site G

EM Wide Trend

40

• Due to the random nature of safety occurrences polynomial regression should not be relied on to precisely predict the value of weighted normalized ORPS occurrences.

• Polynomial regression can be used for noting general trends with EM EVMS normalized data.

• EM EVMS normalized data is best suited for populating dash boards and control charts.

Uncertainty in Second Degree Polynomial FitsLets pull it all together …..

41

Next Steps for EM

• Continue to work with EM sites to establish a thorough understanding of the model.

• Present model, with EH, at the ISM conference in Sept 06.

• Refine model to better reflect smaller populations of data.

• Refine model to be a better predictive tool. • Incorporate a refined set of leading/lagging performance

indicators into EM safety data analysis.• Working with HHS, make recommendations to enhance

ORPS and CAIRS to better support EM safety data needs.

• ……….

1 the environmental management program more than cleanup orps safety data normalization joint hss/em...

Documents

orps data

total safety performance

ns work hour

traditional use of work

work hour availability

budgeted cost of work

actual cost of work

normalization modelsall