the importance of the process tolerance for the effectiveness of any calibration management system

The Importance Of Process Tolerance

For the effectiveness of the calibration management system

By: Ricardo Vilmenay Instrumentation & Calibration Consultant ASQ Certified-CCT

1/28/2013

Table of Contents

ABSTRACT........................................................................................................................................... ii

TERMS AND DEFINITIONS..............................................................................................................1

WHAT IS THE PROCESS TOLERANCE?.....................................................................................4

THE IMPORTANCE OF HAVING ESTABLISHED A PROCESS TOLERANCE....................4

DEVIATION METRICS (OOT’S).......................................................................................................5

TYPES OF PROCESS TOLERANCES...........................................................................................6

HOW TO CALCULATE THE PROCESS TOLERANCE..............................................................6

THE IMPORTANCE TO MOVE FROM INSTRUMENTS MANUFACTURER’S SPECS TO REAL PROCESS NEEDS..................................................................................................................8

CONCLUSION...................................................................................................................................14

REFERENCES...................................................................................................................................15

i

ABSTRACT

This document explains the risk and consequences of maintaining the calibration tolerances of all the instrumentation enrolled in the calibration management system based only on the manufacturer's specifications, and how the process tolerance (error budget) can help to improve the quality of the calibration management system.

ii

The Importance Of Process Tolerance 2013

TERMS AND DEFINITIONS

For the purposes of this document, the following terms and definitions apply:

Accuracy (of a measuring instrument)

Is a qualitative indication of the ability of a measuring instrument to give responses close to a recognized and accepted standard value. Accuracy is a design specification and may be verified during calibration. Accuracy ratings normally include the combined effects of linearity, hysteresis and repeatability (precision).

Calibration

Is the comparison of the unit under test with specified tolerances, but of unverified accuracy against a certified standard or test equipment of greater accuracy in order to detect, report, and minimize by adjustment any deviations from the tolerance limits or any other variation in the accuracy of the instrument being compared. Calibration is performed according to a specified documented calibration procedure, under a set of specified and controlled measurements conditions, and with a specified and controlled measurement system.

Calibration Management System Is a process of the quality management system that includes management of the calibration, tracking, use and control of process instrumentation used to determine conformance to requirements or used in manufacturing supporting activities. The program also include the measuring and test equipment (M&TE) or calibration standards.

Calibration Procedure Is a controlled document that provides a validated method for evaluating and verifying the essential performance characteristics, specifications, or tolerances for the item being calibrated.

Instrument Tolerance (IT)The acceptable variation in the instrument indication for a given input, for which no adjustment to the instrument is required unless another requirement be specified (e.g. Adjustment Tolerance). Instrument calibration tolerance is derived from the requirements of the process and the manufacturer’s stated accuracy for the instrument. The calibration tolerance must be significantly less than the process tolerance.

Loop Tolerance

Page 1


The acceptable variation in the measurements of the loop for a given input, for which no adjustment to the loop is required unless another requirement be specified (e.g. Adjustment Tolerance). Is derived from the combination of all the error sources identified in the system components.

Measuring & Test Equipment (M &TE) or Calibration StandardCertified instrument with traceability to national or international standards used as true value to compare the output readings of the unit under test.

Out of Tolerance (OOT)The state when the “As-Found” value for a critical instrument is outside of the process tolerance. If process tolerances have not been established for the particular instrument or system, the instrument tolerance (IT) is used as the limit for declaring an “Out-of-Tolerance” condition.

Process Tolerance (PT)Is the range of values within which a process can operate and still be considered in control. The potential exist for the quality of the product to be affected if the process operates outside this range. Is used as a maximum allowable deviation on an instrument or system from the standard’s value before declaring an out of tolerance condition (OOT).The Process Tolerance must be broader than the instrument tolerance. Process tolerances must be established for critical instruments only, and must reside in the Computerized Calibration Management System (CCMS).

Proven Acceptable Range (PAR)The range of values within which a process or utility can operate and still be considered in control (see process tolerance).

Root-Sum of Squares (RSS)Is the square root of the sum of the squares of the elements of a data set. RSS can be used to calculate the aggregate accuracy of a measurement when the accuracies of the all the measuring devices are known. The average accuracy is not merely the arithmetic average of the accuracies (or uncertainties), nor is it the sum of them.

Setpoint The setpoint is the desired value of the process variable or controlled variable. Is where you would like the process variable to be. As a rule of thumb, the set point will be the 50% of the PAR range unless otherwise noted.

Page 2


Unit Under Test (UUT)The device being compared or calibrated against a certified standard of known accuracy.

Test Accuracy Ratio (TAR)Is the ratio of the accuracy tolerance of the unit under test (UUT) to the accuracy tolerance of the calibration standard used. As good practices, the calibration management system aims to achieve a TAR of ≥4:1.The TAR must be calculated using identical parameters and units for the UUT and the calibration standard.

Page 3


WHAT IS THE PROCESS TOLERANCE?

The process tolerance is the range of values within which a process can operate and still be considered in control. The potential exist for the quality of the product to be affected if the process operates outside this range.

Figure 1

THE IMPORTANCE OF HAVING ESTABLISHED A PROCESS TOLERANCE

Once the process has its tolerance established, can be controlled and monitored within the proven acceptable range (PAR) as well as, helping to establish a rational valid in the event of an investigation by an "Out of Tolerance" condition. Additional, the process tolerance is the key factor to achieve the implementation of an effective calibration management system in both aspects, technical and administrative. Once the process tolerance (PT) is determined, you should be able to:

Determine the accuracy ratio between process and control loop (TAR). Establish the control loop error budget. Assign the loop instruments calibration tolerances based on processes needs

(Cal. Tol. > Mfg specs). Increase the probabilities of "In- Tolerance Conditions" of the instruments or

loop system. Avoid default process deviations (PT ≠ IT). Avoid out of tolerance conditions (OOT’s). Avoid buying unnecessarily expensive calibration standards.

Page 4


Relieve the calibration program.

DEVIATION METRICS (OOT’S)

A very typical case in regulated industry is not having established the process tolerances (PT) and therefore as it is a compulsory parameter within the requirements of the quality management system, uses the same value of the instrument tolerances (IT) as the value for process tolerances (PT=IT). This practice has the consequence that when an instrument or system is found out of its established tolerance during calibration, by default must be declared at the same time that the process is out of tolerance too (OOT). This generates an immediate investigation to evaluate a possible impact on the quality of the product manufactured using the measurements of such instrument or system during the period since the last approved calibration until the date of the incident (current calibration). Normally once the entire investigation is completed it is concluded that there was no impact with the deviation found (which is what is expected since the deviation to the process was generated using the instrument tolerance as an acceptance criteria and not the actual process tolerance).

The investigation is closed with all necessary documentation for these effects but, although having determined that there was no impact on product quality (false-rejects), is another deviation that increases the metrics within the company. This occurs over and over again and deviations metrics increasing more and more, especially if the calibration management system is not fully refined. This puts at risk the company to observations during an audit when is find such a high percentage of deviations in processes. To counter this, the company will argue that in each alleged deviation were did it all the investigations of rigor based on the quality management system demonstrating that there was no impact on any of them. After this argument the most common is that the auditor then declares "Out of Control" the system or systems audited by the high percentage of deviations found, and based on the six (6) key points within any calibration management system:

1. Or the tolerances are not well defined (i.e., instruments, process).

2. Or has wrong calibrations frequencies or needs a revision.

3. Or the calibration SOPs are not well written.

4. Or the technicians are not well trained in the calibration procedures.

5. Or the calibration standards are not appropriate (TAR).

6. Or the period of useful life of the instruments is expired.

Page 5


Usually is one of them or the combination of some. There is a misconception where it is believed that as long as all deviations generated (no matter how many) are answered and well documented and even more, if it is determined that there was no impact, will not be any problem in an audit. Really is just the opposite, independently that all are investigated, answered, excellently documented and finally declared that there was no impact on the quality of the final product; the truth is, that a high volume of deviations is a clear signal that a system has problems.

TYPES OF PROCESS TOLERANCES

There are four (4) types of process tolerances:

1. Two sided-symmetric tolerances

2. Two sided-non-symmetrical tolerances

3. One sided-NMT tolerances

4. One sided- NLT tolerances

A two-sided tolerance has an upper and a lower Proven Acceptable Range limit and one-sided has only one limit, not more than (NMT), or not less than (NLT).

HOW TO CALCULATE THE PROCESS TOLERANCE

In order to determine the process tolerance, is necessary the information of the Proven Acceptable Range (PAR) and Process Setpoint obtained from a reliable source (e.g. validation technical services, process data flow, manufacturing documentation, etc.). In the case of the two-sided tolerances, the plus/minus (±) value is simply the calculation of the PAR Upper Limit minus setpoint (PARUL – SP) and PAR Lower Limit minus setpoint (PARLL – SP). In the one-sided tolerances the calculation is the PAR Limit minus setpoint. Example of two sided-symmetric tolerances:

The manufacturing ticket for a specific process gives the following information:

PAR = 2-8 °C Setpoint = 5 °C

The process tolerance is then:

Upper tolerance (+): 8 °C – 5 °C = +3 °C

Page 6


Lower tolerance (-): 2 °C – 5 °C = -3 °C

This rewritten: process setpoint = 5 °C ±3 °C

Example of two sided-non-symmetrical tolerances:

PAR = 2-9 °C Setpoint = 5 °C

The process tolerance is then:

Upper tolerance: 9 °C – 5 °C = +4 °CLower tolerance: 2 °C – 5 °C = -3 °C

Rewritten: Setpoint = 5 °C +4 °C/-3 °C (in non-symmetrical tolerances, the lower value of the limits is used as the worst case for the accuracy ratios analysis).

Example of one sided-NMT tolerances:

PAR = not more than (NMT) 0.1 μs/cm Setpoint = 0.05 μs/cm

Then, the process tolerance is: PAR limit – Setpoint

= 0.1 μs/cm - 0.05 μs/cm = 0.05 μs/cm

Rewritten: Setpoint = 0.05 μs/cm +0.05 μs/cm

Example of one sided-NLT tolerances:

PAR = not less than (NLT) 80 °C Setpoint = 85 °C

Then, the process tolerance is PAR limit – Setpoint:

= 80 °C – 85 °C = -5 °C

Rewritten: Setpoint = 85 °C -5 °C

Page 7


THE IMPORTANCE TO MOVE FROM INSTRUMENTS MANUFACTURER’S SPECS TO REAL PROCESS NEEDS

Normally the industry practice is to use the manufacturer's accuracy specifications as acceptance criteria fixed in the calibration management system. This eventually begins to create compliance problems in calibrations generating deviations which increase the metrics and create quality audit observations. Manufacturing specifications must be seen and used as the starting point in technical assessments when: is specifying instrumentation for a projects and/or design, replacement of instruments, to define calibration standards needed for the calibration management program or, as a first calibration tolerance in the calibration system until a technical assessment is conducted for this purpose. The risks of using the manufacturer's specifications as acceptance criteria permanent in the calibration management system are the following:

The instrumentation of nowadays is a high tech instrumentation and in some cases the specifications are equals or exceeds the specifications of the measurement & test equipments or calibration standards. This makes it difficult to meet the policies of TARs of the company and with FDA-CFR regulations and others quality standards.

With use, wear and time the instruments lose their original performance characteristics (e.g. accuracy, precision, stability, hysteresis, linearity, etc.).

There is no standardization among manufacturers in how to state the performance specifications of the instruments and therefore can be misunderstood and be assigned a wrong calibration tolerance to the instruments and/or systems.

The factory test conditions are not necessarily the same to the process conditions where it will be used these instruments.

Some manufacturers use sales gimmicks in their specifications in order to compete with other brands and models. Can be omitted factors in the variability of the measures of an instrument.

When using the manufacturer's specifications as acceptance criteria permanently in the calibration management system, will be a matter of time that calibration deviations begin to occur frequently because the instruments may not be kept (drift) within the accepted and established range of values (tolerance) for the reasons mentioned above.

Page 8


Many would argue, for that we have the calibrations department! But it’s not necessarily. It's the same as saying, that because there are a fire station in the town well equipped and with firefighters well trained, we want that a fire occurs every day, I'm sure no one wants that. The same thing happens with the calibration program, having to be adjusted each time it is calibrate an instrument or system because it was found out of tolerance is fire off all the time and this again is an indication that something is wrong. Calibration and calibration adjustments are two different activities of the same process, calibration (as found values) is the compliance part, and an adjustment (as left values) is the corrective. We have to convert the calibration management system as far as possible in a compliance program (surveillance and prevention) and not only in a corrective one. The calibrations department must have a qualified staff that in the event of finding an instrument or system out of its tolerance, they can make the necessary adjustments to correct the problem but, calibration adjustments (corrective part) should be the exception not the rule.

There are two news, one good and one bad. The good one is that with the specifications of today instrumentation our processes are monitored and controlled with a high degree of quality measures. The bad news is that we have problems to comply with an acceptable TAR (≥ 4:1) in our calibrations setup when comparing the manufacturer specifications of the instruments to be calibrated (UUT) with the specifications of our calibration standards. The question is, do you really need to have as much accuracy in the instrumentation of any particular process? Should I stay with a ratio of perhaps 20:1, 10:1 when with 4:1 is enough and we are in compliance? Precisely, an effective way to work with this problem is to assign calibration tolerances to the instruments based on the real needs of our processes, in other words, using any excess of 4:1 in the error budget of the systems (process tolerance between control loop). Remember that you can assign a calibration tolerance to any instrument higher than the manufacturer's specifications, all that the system error budget allows (process tolerance). What you must not do is never assign a calibration tolerance lower than the manufacturer's original specification. To carry out this type of assessment, the first thing you need is to have clearly established the processes tolerances (PT). The goal of establish the process tolerance is to determine the accuracy ratios for the complete chain of measurements based in the process error budget. As shown in figure 2, the process tolerance is the cornerstone of the whole calibration management system. If this section is not solid and well implemented, the whole system is weakened.

Page 9


Figure 2

The main idea of this pyramid is to present the concept of traceability and accuracy ratios as well as, the calibration tolerances cascade rule (from highest to lowest, where the process tolerance is the highest) from the point of view of the calibration management system within regulated industry. Generally, with demonstrate that our chain of measurements reaches some external laboratory certified as ISO 17025, is sufficient evidence to meet the traceability requirement. Obviously, the general concept of traceability in the field of metrology and calibrations which uses also the pyramid as a symbol contains other steps and always comes up to some national or international laboratory.

HOW TO USE THE EXCESS OF "TAR" (> 4:1)

As mention before, the minimum requirements to comply satisfactorily with any calibration program in regulated industry, with quality standards (e.g. ANSI/NCSL Z540.1-1994, R2002), and therefore with any quality audit is to have a TAR of at least 4:1 between the unit under test (UUT) and the collective accuracy of calibration standards. This could be translated at industrial metrology level as to have a TAR of at least 4:1 between our processes and the instrumentation control systems, as well as, between the instrumentation and the calibration standards. Therefore, why not takes advantage of the ratio between the process and loop (everything exceeding 4:1), and relieves the calibration management system distributing this excess between the

Page 10


components of the loop while maintaining an acceptable ratio between the process and the control loop in order to:

Increase the probability of "In-Tolerance-Condition" between calibration intervals.

Avoids calibration deviations (Cal Alert, OOT’s), and dramatically reduces

deviations metrics.

Qualify much easier existing calibration standards.

Reduce the need to buy more expensive calibration standards

With a 4:1 ratio, we have enough confidence level and are maintained a False- Accept Risk of 0.8% (consumer’s risk) and False-Reject Risk of 1.5% (producer’s risk). In addition, we are adopting good calibration practices and we are complying very well with the FDA-CFRs requirements or any other quality standards (e.g. ISO, NCSLI, etc.). For example, we are going to analyze the results of the next temperature loop:

ASSESSMENT RESULTS

Process Tolerance

Loop Tolerance

Cal. Setup Tolerance

Process to Loop (TAR)

Loop to Cal. Setup(TAR)

±5.0°C ±0.41°C ±0.046°C ≈12:1 ≈9:1Table 1

After the assessment, the TAR shows an approximate ratio of 12:1 between the process and the control loop, which means three (3) times the minimum acceptable ratio (4:1), which is equivalent to 200% of excess. The actual loop components accuracy (at 4 significant figures):

Sensor: ±0.3464°C

Temp. Transmitter: ±0.05201°C

A.I. card: ±0.2088°C

Loop Tolerance = ±√ (0.34642 + 0.052012 + 0.20882) = ±0.41°C (at 2 significant figures)

To take advantage of this excess and to determine the new calibration tolerances for the loop components, we make the process of "RSS" in reverse by using the following steps:

Page 11


Note: the following procedure is based on the assumption that both the process tolerance and the loop tolerance and its components all are at 95% of confidence level (k= 2). Otherwise, for industrial processes at least a type B uncertainty assessment should be conducted to obtain the results. The important thing is that the calibration tolerances of the instruments in the calibration management system are established at least to 95% of confidence level and that all the TAR/TUR calculations have been done at the same confidence level.

1. Divide the process tolerance by four (4) to find the new value of loop tolerance with a ratio of 4:1:

Process tolerance = ±5 ºC

= 5/4 = 1.25 (loop error budget)

New Loop Tolerance = ±1.25 ºC

2. Raise this value squared:

(1.25)2 = 1.56 ºC

3. It is divided by the number of components of the loop if you want to distribute evenly or divided into different percentages assigning most to the component(s) most likely to fail or better accuracy (lower error). In this case was assigned 50% to the temperature sensor (higher probability of failure), 30% to the temperature transmitter (component of lower error) and 20% to the analog card:

50 %(1.56 ºC) = ±0.78 ºC (Sensor)

30 %(1.56 ºC) = ±0.47 ºC (Temp. transmitter)

20 %(1.56 ºC) = ±0.31 ºC (Analog card)

Note: if you decide to distribute the error budget at different percentages, be careful to not assign a percentage (%) that result in a tolerance value lower than the original.

4. Takes out the square root of each component:

√0.78=0.88 °C (Sensor)

Page 12


√0.47=0.69 °C (Temp. transmitter)

√0.31=0.56 ° C (Analog card)If we introduce these values into the RSS formula is supposed to give us the value of the loop tolerance calculated in step #1. It is possible that as a result we get some approximate value due to the propagation of error created by the rounding. That’s why as a good practices, it is important to keep at least four significant figures in the intermediate calculations and two at the final calculations.

Loop Tolerance = ± √ (0.882 + 0.692 + 0.562 ) = ±1.25 °C

The new recommended values for the calibration management system (CCMS) are:

CCMS TOLERANCES RECOMMENDED

Loop Tolerance Sensor Transmitter A.I. CardProcess to Loop (TAR)

Loop to Cal. Setup(TAR)

±1.25 °C ±0.88 °C ±0.69 °C ±0.56 °C 4:1 ≈27:1Table 2

Benefits:

The sensor is increasing from 0.3464 °C to 0.88 °C (154% of change).

The transmitter is increasing from 0.05201 °C to 0.69 °C (1,227% of change).

The analog card is increasing from 0.2088 °C to 0.56 °C (168% of change).

The TAR between loop and the calibration setup is increasing from 9:1 to 27:1

(200% of change).

With every increase of tolerance that we achieve in the loop components, we are increasing the probability of finding our systems within specifications. The deviations due to out of tolerance systems are reduced, while we remain in compliance with an acceptable TAR (4:1). As you can see, the key is to have an established process tolerance where we can clearly define the error budget of the process and control loop to comply with a TAR of 4:1 between them. If we know how much is our process error budget, we can determine how far we can allow our instruments deviate without jeopardizing our processes. Better yet, we can move the calibration tolerances of all the instrumentation from the manufacturer's original specifications to the real needs of our processes.

Page 13


CONCLUSION

Having the processes tolerances clearly defined not only helps to establish the calibration tolerance of the instruments based on the real process needs if not, that it is the most practical and convenient way to implement the calibration management system and the best path to achieve a quality system and in compliance. More now, that recent FDA audits are aiming to that the process systems are taking into account the uncertainty of the instrumentation in charge to control and monitor these processes for the alarm limits settings as well as, for all the calibration activities. For these technical assessments, is necessary to carry out the accuracy ratios in the measurement chain (e.g. process to loop and loop to calibration standards), and the process tolerance is the key factor in the success of these evaluations.

We know that we will never get a 100% perfect system, it does not exist, just knowing that from the point of view of metrology we will never know the true value of any measure, is something frustrating. But, if we can take our calibration management system to its finest (i.e. perfection within imperfection), that we can reduce the calibration deviations percentage as low as possible and that deviations are arising due to the only factor that often cannot be prevented by any system; "malfunction" (if something has the probability to fail will surely fail, is a matter of time), then we can declare that our calibration management system is highly effective, reliable and of high quality.

Page 14


REFERENCES

ANSI/NCSL Z540-1-1994 (R2002). Calibration Laboratories and Measuring and Test Equipment-General Requirements. Boulder Colorado : NCSL International, 1994. 1-58464-003-0.

Bucher, Jay L., The Metrology Handbook. Milwaukee : ASQ, 2004. 0-87389-620-3.

Fluke., Calibration: Philosophy in Practice, Second Edition. s.l. : Fluke, 1994. 0-9638650-0-5.

Murrill, Paul W., Fundamentals of Process Control Theory Third Edition. s.l. : ISA, 2000. 1-55617-683-x.

21 CFR PART 211 -- Current Good Manufacturing Practice For Finished Pharmaceuticals., Sec. 211.68 Automatic, mechanical, and electronic equipment.

21 CFR PART 820 -- Quality System Regulation., Sec. 820.72 Inspection, measuring, and test equipment.

ISO 10012:2003 -- Measurement Management Systems., Requirements for measurement processes and measuring equipment.

Page 15

the importance of the process tolerance for the effectiveness of any calibration management system

Documents