company confidential registration management committee (rmc) 1 2010 report on oasis data for...

Company Confidential

Registration Management Committee (RMC)

1

2010 Report on Oasis Data for2007-2010

Primary focus on 12 month roll starting in

November 2009 to December 2010(For Open Meeting)

San Diego, CAJanuary 18 – 19, 2011

R. Darrell TaylorRaytheon

San Diego, CAJanuary 18-19, 2011


2

Executive Summary

• Setting a Baseline for the future

• Statistically Based Analysis

• Two-fold Analysis of data

– Trends for the total population

– Trends for the individual CB’s



3

Agenda

• OASIS Entries – Audits

• OASIS Nonconformities

• AS9100 Elements – Quick Overview no change

• Feedback

• Monitoring Model



4

Comparison of Entries 2007 -2010

• Entries:

– 2007 – 5114

– 2008 – 5972 ^14%

– 2009 – 6969 ^17%

– 2010 – 5987

» As of 12/2010

5114

5972

6969

5987

0

1000

2000

3000

4000

5000

6000

7000

2007 2008 2009 2010 YTD

Total Entries

Consistent with Record Entry Delays

Projecting a flat to a slight increase

compared to 2009 year



5

Comparison of Types2007 -2010

2007 3364 888 862

2008 3977 923 1072

2009 4421 1486 1062

2010 4982 1426 823

SUR REC INT

SUR=Surveillance

REC = Recertification

INT = Initial

NOTE : 2010 is still not complete, consistent with previous years

2007 0.66 0.17 0.172008 0.67 0.15 0.182009 0.63 0.21 0.152010 0.69 0.20 0.11

Percentages



6

Standards Entered Into Database

0.0000.0050.0100.0150.0200.0250.0300.035

0.0400.0450.050

2007 2008 2009 2010

AS9120AS9110NBR15100:2004EN9100:2003/ S1EN9120:P2/ S1EN9100:P2/ S1AS9100C

AS9100B:2007- 95%2008 - 95%2009 - 94%2010 -92%

2 AS9100C audits now in the database



7

Summary of OASIS Data

0

0.5

1

1.5

2

2.5

3

3.5

2008 2009 2010

Average # of N/ C per audit

0

0.1

0.2

0.3

2008 2009 2010

2.2

2.4

2.6

2.8

2008 2009 2010

MAJORS

MINORS

Average number of Non Conformities per audit slightly over 3Average number of minors 2.4 per audit

On average a major is written every 5th audit

Minors – 15,814

Majors - 1117



8

AS9100 Nonconformities Top 10

Top Ten 2010 same as 2009 & 2008

0 500 1000 1500 2000 2500 3000

4.3 Conf iguration Management

6.3 Inf ras truc ture

5.4 Planning

4.1 QMS - General Requirements

7.3 Des ign & Dev elopment

7.2 Cus tomer Related Proc es s es

5.6 Management Rev iew

6.2 - Human Res ourc es

8.3 Control of Non-Conf orming Produc t

7.6 Control of Monitor ing and Meas ur ing Equipment

8.5 - Improv ement

8.2- Monitor ing & Meas urement

7.4 - Purc has ing

4.2 Doc umentation Requirements

7.5 - Produc tion and Serv ic e Prov is ion

M ajor

M inor

Will monitor as we transition to AS9100C and determine

if there are any industry related trends



9

Feedback Analysis

2009 average 112 per month, none in December2008 – average 110 per month as of August

0

500

1000

1500

2000

2500

2007 2008 2009 2010YTD

OPENCLOSED



10

ANAB Audit Summary

• 2010

– 6.07 Minors per audit, .4 Majors per audit

• Other Categories monitored:

– Escalated N/C Customer Complaints

– Negative Shift Suspensions (2 Yrs)

– Additional Audits

– N/C from outside assessments



11

Certification Body Statistical Assessment Model

ANAB Audits - 6 Criteria

OASIS Results - 3 Criteria

Feedback - 2 Criteria

General - 1 Criteria

+ 1

Added Old Feedback “OPEN” as separate criteria

ANAB Performance - + 4

Customer Complaints

Suspensions

C/A’s outside of Audits

Extra Audits



12

Threshold Violations

0

2

4

6

8

10

12

14

16

Violations

• RMC & ANAB need to research

– >6

• ANAB to report to RMC

– 4-5

• ANAB to investigate

– 2-3

• No Action

– <2



13

Quantity of CB’s & Threshold Violations 11/2009 – 11/2010

6

16

10

7

0

2

4

6

8

10

12

14

16

<2 2 & 3 4 & 5 >5

Threshold Violations



14

Non - Statistical Driven Metrics

• Feedback (Only 9 CB’s did not have a hit)

– Closed > average 30 Days

– Open > average 30 Days

– OPEN over one year

• Escalated N/C’s (9 CB’s had escalated N/C’s)



15

Questions

Company Confidential


16

Back Up



17

ANAB Audits - Minors

108642

Median

Mean

6.56.05.55.04.5

1st Quartile 3.5000Median 6.00003rd Quartile 7.0000Maximum 11.0000

4.7339 6.4013

5.0000 6.0000

2.0334 3.2480

A-Squared 0.74P-Value 0.050

Mean 5.5676StDev 2.5005Variance 6.2523Skewness 0.140856Kurtosis -0.657909N 37

Minimum 2.0000

Anderson-Darling Normality Test

95% Confidence I nterval for Mean

95% Confidence I nterval for Median

95% Confidence I nterval for StDev95% Confidence I ntervals

Summary for Minors

6543210

99

95

90

80

70

605040

30

20

10

5

1

Ratio Minors

Pe

rce

nt

Mean 2.552StDev 1.275N 37RJ 0.986P-Value >0.100

Probability Plot of Ratio MinorsNormal

Normal Distribution

Identify those CB’s in the top 5 % receiving Minors from

ANAB audits



18

ANAB Audits - Majors

NOT Normal Distribution

2.01.51.00.50.0

Median

Mean

0.50.40.30.20.10.0

1st Quartile 0.00000

Median 0.000003rd Quartile 0.50000Maximum 2.00000

0.16380 0.49837

0.00000 0.33333

0.40801 0.65172

A-Squared 4.09P-Value < 0.005

Mean 0.33108StDev 0.50173Variance 0.25173

Skewness 1.73203Kurtosis 2.69073N 37

Minimum 0.00000





Summary for Ratio Majors

2.01.51.00.50.0-0.5-1.0

99

95

90

80

70

605040

30

20

10

5

1

Ratio Majors

Pe

rce

nt

Mean 0.3311StDev 0.5017N 37RJ 0.966P-Value 0.040

Probability Plot of Ratio MajorsNormal

Identify those CB’s in the top 5 % receiving Majors from

ANAB audits



19

ANAB Audits – Majors (cont.)

Poisson Distribution supports the use of > 2 as threshold

2.01.51.00.50.0

Median

Mean

0.50.40.30.20.10.0



0.16380 0.49837

0.00000 0.33333

0.40801 0.65172




Minimum 0.00000






0.5

0.4

0.3

0.2

0.1

0.0

X

Pro

ba

bili

ty

3

0.0414

0

Distribution PlotPoisson, Mean=0.757



20

ANAB Audits – Ratio Total

Normal Distribution

6543210

99

95

90

80

70

605040

30

20

10

5

1

Ratio TotalP

erc

en

t


Probability Plot of Ratio TotalNormal

6.44.83.21.6

Median

Mean

3.503.253.002.752.502.252.00

1st Quartile 2.0000


2.4294 3.3364

2.0336 3.4832

1.1061 1.7668



Skewness 0.309638Kurtosis -0.423729N 37

Minimum 0.6667





Summary for Ratio Total

Identify those CB’s in the top 5 % receiving

Nonconformities as a ratio to audits performed from ANAB

audits



21

ANAB Audits – Ratio Minors

Normal Distribution

4.83.62.41.2

Median

Mean

3.02.82.62.42.22.0

1st Quartile 1.2917


2.1266 2.9770

2.0000 3.0000

1.0371 1.6565



Skewness 0.290702Kurtosis -0.575196N 37

Minimum 0.6667





Summary for Ratio Minors

6543210

99

95

90

80

70

605040

30

20

10

5

1

Ratio Minors

Pe

rce

nt


Probability Plot of Ratio MinorsNormal

Identify those CB’s in the top 5 % receiving Minors as a ratio to audits performed

from ANAB audits



22

ANAB Audits – Ratio Majors


2.01.51.00.50.0

Median

Mean

0.50.40.30.20.10.0



0.16380 0.49837

0.00000 0.33333

0.40801 0.65172




Minimum 0.00000






2.01.51.00.50.0-0.5-1.0

99

95

90

80

70

605040

30

20

10

5

1

Ratio Majors

Pe

rce

nt

Mean 0.3311StDev 0.5017N 37RJ 0.966P-Value 0.040

Probability Plot of Ratio MajorsNormal

Identify those CB’s in the top 5 % receiving Majors as a ratio to audits performed

from ANAB audits



23

ANAB Audits – Ratio Majors (cont.)

2.01.51.00.50.0

Median

Mean

0.50.40.30.20.10.0



0.16380 0.49837

0.00000 0.33333

0.40801 0.65172




Minimum 0.00000






Poisson Distribution supports the use of >1

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.0

X

Pro

ba

bili

ty2

0.04410

Distribution PlotPoisson, Mean=0.33108



24

ANAB Score Negative Shift

• Formula TY Ratio of Total – LY Ratio of Total

– Trigger count if decrease is > than 1 SD (TY)

• Based on a one standard deviation shift in the negative direction, may be an indicator of a potential issue

TY= This Year

LY = Last Year

SD= Standard Deviation

Identify those CB’s with a greater than one standard

deviation shift in performance in the negative

direction



25

FEEDBACK

• Rule of Thumb threshold > 30 days based on “Closed”

• Rule of Thumb threshold > 30 days based on “Currently OPEN”

• Rule of Thumb any violation of older “OPEN” feedback

Identify CB’s with outstanding feedback responses >30 days



26

ANAB DATA – CB Performance

• Additional ANAB Audits >0

• Escalated Nonconformities > 0

• Customer Complaints > 2 over two years

• Corrective Actions issued outside of Assessment Minors >1, Majors >0

• Suspensions in past 2 years >0

• Outside Assessment Influence



27

OASIS Ratio of Total Findings Modified

Modified criteria excluding outlier

543210

99

95

90

80

70

605040

30

20

10

5

1

Ratio Total_OP

erc

en

t


Probability Plot of Ratio Total_ONormal

54321

Median

Mean

2.82.62.42.22.0

1st Quartile 1.7699Median 2.39053rd Quartile 3.0867Maximum 5.0000

2.1456 2.8086

1.9483 2.7283

0.7948 1.2782


Mean 2.4771StDev 0.9799Variance 0.9602Skewness 0.683738Kurtosis 0.408484N 36

Minimum 0.6522





Summary for Ratio Total_O



28

Ratio Majors/Entries

On average, one Major is written every fifth audit – NOTE: Average down to .21 per audit for 2010 (1/5 audits)

2009Major Histogram

All CRB's

0

5

10

15

20

25

30

0.0 to <= 0.317 0.317 to <= 0.634 0.634 to <= 0.951 0.951 to <= 1.268

Class

# O

bs

erv

ati

on

s

2009Normal Distribution Mean = 0.2642Std Dev = 0.294KS Test p-value = .0663


Totals

2010 – 1117 YTD

2009 -1135

2008 – 976

2007 - 870



29

OASIS Ratio of Majors


1.251.000.750.500.250.00-0.25-0.50

99

95

90

80

70

605040

30

20

10

5

1

Ratio Majors_0

Pe

rce

nt

Mean 0.2106StDev 0.2694N 37RJ 0.828P-Value <0.010

Probability Plot of Ratio Majors_0Normal

1.20.90.60.30.0

Median

Mean

0.300.250.200.150.10



0.12075 0.30043

0.08414 0.17702

0.21912 0.35000




Minimum 0.00000





Summary for Ratio Majors_0

Identify CB’s in the bottom 5% writing majors



30

OASIS Ratio of Majors - Modified

Normal Distribution after a transformation

210-1-2

99

95

90

80

70605040

30

20

10

5

1

Ratio Majors_0

Perc

ent

Goodness of Fit Test

AD = 0.303 P-Value = 0.556

Probability Plot for Ratio Majors_0

After Johnson transformation

Normal



31

Ratio Minors/Entries

On average, 2.8 Minors are written every audit, down to 2.47 minors per audit

2009Minor Histogram

All CRB's

0

2

4

6

8

10

12

0.786 to <=1.465

1.465 to <=2.144

2.144 to <=2.823

2.823 to <=3.502

3.502 to <=4.181

4.181 to <=4.859

4.859 to <=5.538

Class

# O

bs

erv

ati

on

s



Totals

2010 – 15,814 YTD

2009 -19,414

2008 -17,393

2007 –14,087



32

OASIS Ratio of Minors

Normal Distribution

5.64.84.03.22.41.60.8

Median

Mean

2.82.62.42.22.01.8

1st Quartile 1.5947


2.0212 2.6932

1.7428 2.6773

0.8196 1.3092




Minimum 0.5652





Summary for Ratio Minors_O

543210

99

95

90

80

70

605040

30

20

10

5

1

Ratio Total_O

Pe

rce

nt


Probability Plot of Ratio Total_ONormal

Identify CB’s in the bottom 5% writing minors



33

AS9100 Nonconformities Quantities

major minor sum7.5 Production and Service Provision 145 2669 28144.2 Documentation Requirements 72 2088 21607.4 Purchasing 106 1864 19708.2 Monitoring and Measurement 163 1594 17578.5 Improvement 129 1106 12357.6 Control of Monitoring and Measuring Equipment 69 996 10658.3 Control of Nonconforming Product 57 675 7326.2 Human Resources 37 638 6755.6 Management Review 70 499 5697.2 Customer Related Processes 18 433 4514.1 General Requirements (Quality Management System) 18 271 2897.3 Design and Development 16 211 227

NO CHANGE



34

AS9100 Nonconformities Quantities (cont.)

major minor sum5.4 Planning (Management Responsibility) 15 182 1976.3 Infrastructure 9 153 1627.1 Planning of Product Realization 3 102 1058.4 Analysis of Data 7 86 934.3 Configuration Management 3 89 926.4 Work Environment 3 71 745.5 Responsibility, Authority and Communication 0 64 645.3 Quality Policy 2 49 515.1 Management Commitment 6 19 258.1 General (Measurement, Analysis and Improvement) 3 15 185.2 Customer Focus 3 13 166.1 Provision of Resources 2 9 11

NO CHANGE



35

Time – Line, Data Presentation

JAN MAR JUL SEP

First 6 Months

Summary

Previous Year

Summary

RULE VIOLATORS



36

0 50 100 150 200 250 300 350

6.1 Provision of Resources

5.2 Customer Focus

8.1 General (Measurement, Analysis and Improvement)

5.1 Management Commitment

5.3 Quality Policy

5.5 Responsibility, Authority and Communication

6.4 Work Environment

4.3 Configuration Management

8.4 Analysis of Data

7.1 Planning of Product Realization

6.3 Infrastructure

5.4 Planning (Management Responsibility)

7.3 Design and Development

4.1 General Requirements (Quality Management System)

Major Minor

company confidential registration management committee (rmc) 1 2010 report on oasis data for...

Documents