1

Proficiency Testing (PT)

– a tool to improve laboratory performance

Sciencefor a safer world

3rd GCC Laboratory Proficiency Conference

Dubai, 24-25 March 2015

Overview

PT as a quality tool

Purpose, types and benefits of PT

PT participation strategy

Selecting fit for purpose PT schemes

Interpretation of PT Results

3

Proficiency Testing

- as a quality tool

Measurements

Measurement

Health

Security

Trade

Compliance Decisions

Reliable measurements

Reliable decisions

Decision on

measurement

Report on

measurement

The measurement cycle

Client issue

Client

Define issue

Client

Sampling

Analysis

Analyst

Interface

Decision on result

Evaluation

Data evaluation

AnalystValidated

procedures

Method

Validation

Calibration

Training

IQC

Quality Assurance System

?Measurement

SystemInput Output

RMsPT

Fit-for-purpose analysis requires:

Careful development and validation

of methods

Accurate calibration of instruments

and appropriate equipment

Qualified and competent staff

Comparability with measurements

made in other laboratories

Well defined QC and QA procedures

Independent evidence of performance

CRMs

Traceable Standards

RMs & PT

RMs

RMs & QC

PT

8

Proficiency Testing

- purpose, types and benefits

Purpose of PT

The primary aim of proficiency testing is:

“To provide the infrastructure for a laboratory to monitor and

improve the quality of its routine analytical measurements”

• PT provides an objective evidence of the competence of

the participant which can be used to

– Improve the performance of the participant

– Give confidence in the participant’s ability to perform a

specific measurement

• PT provides information on other aspects of the

management system

– Reception/treatment of the sample, treatment of the data,

result reporting etc

Types of PT

• Various types of PT available – based on one or more

elements of four different categories:

Qualitative

Quantitative

Interpretive

1

Descriptive – nominal or ordinal scale

Numeric – interval or ratio scale

No measurement involved

Types of PT

• Various types of PT available – based on one or more

elements of four different categories:

Qualitative

Quantitative

Interpretive

Pre-measurement

Measurement

Post-measurement

1 2

Types of PT

• Various types of PT available – based on one or more

elements of four different categories:

Qualitative

Quantitative

Interpretive

Sequential

Simultaneous

Pre-measurement

Measurement

Post-measurement

1 32

Monitor

PerformanceReport

Sequential participation

scheme structure

PT

Provider

Test

Item

Test

Result

Test

Result

Test

Result

Monitor

Performance

Monitor

Performance

Lab 1

Lab 2

Lab 3

PT

P

r

o

v

i

d

e

r

Evaluation

of

Results

Lab 1

Lab 2

Lab 3

NMi

Simultaneous scheme structure

PT

Provider

Test

Items

Participating

Laboratories

Test

Results

PT

Provider

Evaluation

of

Results

Report

Participating

Laboratories

Monitoring

Performance

Types of PT

• Various types of PT available – based on one or more

elements of four different categories:

Qualitative

Quantitative

Interpretive

Single

Continuous

Sequential

Simultaneous

Pre-measurement

Measurement

Post-measurement

1 32 4

Use of PT by laboratories

• Basic use of PT for a laboratory is:

– to assess its performance for the conduct of specific

measurements or calibrations

• Results and information received from participation will

provide laboratories with:

– Confirmation that the laboratory’s performance is

satisfactory

– An indication that there are potential problems and that

corrections should be made

• However, the use of PT should be much wider

– Many additional benefits from PT participation

Benefits of PT to laboratories

Benefits

of PT

Third Party

Demonstration

Method/ Procedure

Evaluation

Performance

Comparison

Monitoring

Trends

Independent

Check

Training and

Education

Maximising the benefits of PT

• To obtain the optimum benefit from PT participation,

laboratories should:

– treat PT scheme test items as routine samples (i.e. do not

give them any special treatment)

– vary the analyst testing the test items

– vary the instrumentation used for the test items

• To learn from PT participation, laboratories should:

– use the scheme results to give feedback to laboratory

procedures and staff; both positive and negative

– use the scheme results constructively

– be honest; the scheme is there to help

– remember that PT is education not regulation

19

Proficiency Testing

- Participation Strategy

Strategy of PT participation

• All laboratories need to develop an adequate PT

participation strategy

• A laboratory should decide on an appropriate level and

frequency of participation

– Level: number of specific proficiency tests in which to participate

– Frequency: How often the laboratory will participate in each of

the specific test identified

• This will be specific to the laboratory’s circumstances

• Guidance provided by the European co-operation for

Accreditation (EA) EA-4/18: 2010

– Referenced in ILAC-P9: 2010

Level of PT Participation

• Consider areas of technical competence based on:

• Measurement technique e.g. ICP-MS, GC-MS

• Property to be measured e.g. PAHs, Fat, hardness

• Products to be tested e.g. soil, milk, human serum

• An area of technical competence may encompass

several products, properties and/or measurement

techniques

• The laboratory must be able to demonstrate equivalence

within each area of technical competence

21

Level and frequency of PT participation

• Important to consider a number of different aspects

• Other QA measures implemented:- Use of CRMS

- Comparisons by independent techniques

- Participation in validation or

characterisation studies

- Use of internal QC measures

- Other laboratory comparisons

• Level of risk:

- No measurements undertaken

- Turnover of technical staff

- Staff experience/knowledge

- Known stability/instability of

measurement technique

- Significance and final use of data

•Different types of PTs can be used

• Difficulty of PT participation e.g. technical characteristics of the measurement,

low number of laboratories etc

• Legislative requirements for frequency of type of participation

23

Proficiency Testing

- selecting fit for purpose schemes

Fit for purpose PT schemes

• Need to decide which scheme is fit for purpose

– Often a scheme will not have a perfect fit, so need to find the

best fitness for purpose

• A number of factors to consider

– Test Items

– Distribution

– Participants

– Results

– Reports

– PT Provider

Selection factors

• Test Item– Matrix (real/simulated)

– Parameters

– Concentrations

• Distribution– Dates available

– Frequency

– Flexibility

• Participants– National/International

– Language

– Methods

• Results– Deadlines

– Reporting mechanism

– Statistical approach

• Reports– Speed

– Information/Language

– Format

• PT Provider– Experience

– Scope

– Competence

26

Proficiency Testing

- Interpretation of results

Performance evaluation

• Limited value unless performance is evaluated

• Evaluate after each round and also over time

• Interpretation of PT performance concerns all

management levels

• PT is not about “passing” or “failing” – it is about learning

from the results

Review of single PT round

performance

• Results of each PT round should be evaluated

regardless of performance

– A satisfactory result may not mean good performance

– Unsatisfactory performance in one round does not indicate

that the laboratory is not competent

• Important to consider a number of factors:

– The overall results in the round

– Test method performance

– Test item factors

– Bi-modal result distribution

– PT Scheme factors

Trends in PT performance

• Trends are valuable in preventive and corrective actions,

and root cause analysis

– Performance over time

– Common groups of determinands

– Performance of methods

– Performance of analysts

Need for an investigation

• All laboratories will occasionally have unsatisfactory or

questionable PT results

• A laboratory should establish its own criteria for when an

investigation is necessary

• The depth of investigation will depend on a number of

factors:

– Criticality of the method

– Frequency of unsatisfactory results

– Evidence of a bias

Causes of unsatisfactory performance

• Can be analytical and non-analytical

• Both are equally serious

– the result reported is the important factor for a laboratory

• Analytical Errors

– Calibration

– Instrument problems

– Extraction/clean-up

– Interferences

– Method performance

– Analyst

• Non-Analytical Errors

– Calculation

– Transcription

– Units

– Reporting format

32

Proficiency Testing

- in conclusion

PT – powerful quality tool

PT is an essential tool to all laboratories making measurements

PT participation brings many benefits

PT enables measurements to be monitored and improved

PT is a very powerful quality assurance tool underpinning laboratory accreditation

PT - drives laboratory quality

PT Reliable

Measurements

Reliable

Decisions