This project has been funded in whole or in part with Federal funds from the Division of AIDS (DAIDS), National Institute of Allergy and Infectious Diseases, National Institutes of Health, Department of Health and Human Services, under contract No. HHSN272201200009C, entitled NIAID HIV and Other Infectious Diseases Clinical Research Support Services (CRSS).

Verification of Performance Specifications

An Advanced View of Method Validation

Version 5.0, August 2012

2

Identify test classifications Define what each validation experiment details for

testing methods Discuss what is recommended to perform each of the

validation experiments for testing methods Recognize how to evaluate data obtained from each of

the validation experiments

Objectives

3

A rapid Human Immunodeficiency Virus (HIV) test would likely be classified as a:

A. High complexity, modified assayB. Moderate complexity, unmodified assayC. Food and Drug Administration (FDA)-approved,

modified assayD. Waived, FDA-approved, unmodified assay

Pre-Assessment Question #1

4

The precision of a test method gives information related to the method’s:

A. Systematic errorB. Comparison of results to a reference methodC. ReproducibilityD. Likelihood of being affected by hemolysis, lipemia and

icterus E. Both A and B

Pre-Assessment Question #2

5

When transferring reference intervals of 20 specimens used, what is the minimum number that must fall within manufacturer’s reference intervals?

A. 20B. 18 C. 16D. 15

Pre-Assessment Question #3

6

Which linear regression equation component gives information regarding constant bias?

A. yB. xC. m (slope)D. b (intercept)

Pre-Assessment Question #4

7

Selecting a Method Evaluate diagnostic tests

Characteristics of testing methods

References: Technical literature and manufacturer’s information

Select method of analysis Validate method performance Implement method Perform tests with appropriate

Quality Control (QC) and External Quality Assurance (EQA)

Method Validation

Why must we validate?

When should we validate?

What should we validate?

8

What is method validation?

9

Why is validation important? Division of Acquired Immunodeficiency Syndrome

(DAIDS) requirement How important is it that the results produced by the

testing method are reliable? Shouldn’t the laboratory know the level of performance

of an adopted test method?

Method Validation (cont’d)

Tests to Validate

Waived

Non-waived

• Unmodified FDA-approved

• Modified and/or Non-FDA-approved

10

11

Vendor Publications http://www.fda.gov/MedicalDevices/

ProductsandMedicalProcedures/InVitroDiagnostics/LabTest/ucm126079.htm

FDA Approval Resources

12

What would you consider to be the complexity, per Clinical Laboratory Improvement Amendments (CLIA), of the glucose assay in the workbook?

A. WaivedB. ModerateC. High

Skill Check

13

What would you consider to be the complexity of a rapid urine pregnancy assay?

A. WaivedB. ModerateC. High

Skill Check

14

What would you consider to be the complexity of performing a manual white cell differential using a stained whole blood smear?

A. WaivedB. ModerateC. High

Skill Check

15

Before you begin: Be sure you are familiar with the test method before

starting Know what to expect from the method (package insert,

discussions with technical assistance, and field service representatives)

Do not include results outside of stated reportable ranges

Predict your findings; establish limits/evaluation criteria

Method Validation

16

Terms for Discussion

Central Tendency

Dispersion

17

Terms for Discussion (cont’d)Va

lues

Run

18

Some error is expected Examples

Error must be managed Understanding Defining specifications of allowable error Measurement

Error in Test Methods

19

Total Error of Testing System

Total Allowable Error• CLIA Guidelines per analyte• Other Guidelines

Systematic Error

Random Error Total Error

20

Error Assessment

In either direction,

unpredictable

In one direction, cause results to be high or low

Combined effect

Systematic Error(SE)

Random Error (RE)

Total Error(TE)

21

Low End Performance Standards Recommendations derived from upper portion of

reportable range are more difficult to achieve at lower concentrations

Maximum Total Error Allowed Considered to be 30% by David Rhoads, except for

amplification methods

Total Error Considerations

22

Systematic Error Slope/Proportional error Intercept/Constant error Bias

Random Error Mean Standard deviation (SD) Coefficient of variation (CV)

Systematic and Random Errors

23

Tools for Use

Spreadsheets with

calculationsValidation Software

(Westgard, Analyze-It, EP Evaluator)

Statistical calculators, graph paper

Data-Crunching

Tools

24

One quantitative test taken through the validation process

One qualitative method taken through the validation process

How We Will Work Through This Module

Reportable Range

Precision

Accuracy

Reference Intervals

Sensitivity

Specificity

Elements of Validation

25

Repeat testing over short and long term (one day and 20 days, respectively)

20 samples of same material (typically two levels; e.g., Glucose at 50 and 300 mg/dL)

Standard solutions Control materials Pools (short term only)

Precision Definition: Reproducibility Gives information related to random error

Introduction

What is needed

How we perform

the testing

26

27

Precision: How We Evaluate the Data

Mean Standard deviation (SD) Coefficient of Variation (CV)

Short term: 0.25 of allowable total error Long term: 0.33 of allowable total error

Calculate the following:

What amount of random error is allowable, based on CLIA criteria?

28

Link for: Clinical Laboratory Improvement Amendments (CLIA) College of American Pathologists (CAP) Royal College of Pathologists of Australasia (RCPA) Others

http://www.dgrhoads.com/db2004/ae2004.php

Allowable Total Error Database

29

Precision: Levey-Jennings (LJ) ChartsVa

lues

Run

30

Precision: How We Evaluate the Data

Mean SD CV: More commonly used, allows for

easier comparison

How do we compare to manufacturer’s data?

31

Precision ExampleMean of Level 1 Glucose

CLIA Total Allowable Error

Total Allowable Error Level 1 Glucose

Random error allowed:

90 mg/dL

6 mg/dL or ± 10%

0.1 x 90 = 9 mg/dL

0.25 x total allowable

0.25 x 9 mg/dL

2.25 mg/dL

0.33 x total allowable

0.33 x 9 mg/dL

2.97 mg/dL

Long-term precision

Short-term precision

32

Work with Levey-Jennings graph and data Work with mean and standard deviation to calculate a

coefficient of variation, as well as a mean and a coefficient of variation to calculate a standard deviation

Determine if precision data is acceptable

Activity

Accuracy Definition: How close to the true value Comparison of methods Gives information related to systematic error Potential conflicts on interpretation of results

(reference values)

Introduction

40 different specimens Cover reportable range of method Quality versus quantity

What is needed

Duplicate measurements of each specimen on each method

Minimum of five days, prefer over 20 (since replicate testing is same)

How we perform the

testing

33

Accuracy: How We Evaluate the Data

Graph the Data:

Test method on Y-axisReference (comparative) method on X-axisShows analytical range of data, linearity of response over range and relationship between methods

Real time Difference plot

Comparison plot Calculate y = mx + b

b represents constant error

m represents proportional error

34

35

Visual Inspection for AccuracyTe

st M

etho

d

Reference Method

Intercept

(x1, y1)

(x2, y2)Slope = (y2- y1) / (x2- x1)

36

Slope: Usually not significantly different from 1 Intercept: Not significantly different from 0 Significant difference with Medical Decision Points

Accuracy: How We Evaluate the Data

37

Slope Measure of proportional bias

m = (y1-y2)/(x1-x2) or “rise/run” Slope greater than 1 means the Y (Test) values are

generally higher than the X (Comparative) values Slope of 1.11 means the Y (Test) values are on

average 11% higher than the X (Comparative) values

Calculate Appropriate Statistics

38

Intercept of the Line Measure of constant bias between two methods

Y (Test) value at the point where the line crosses the Y axis

If Y intercept is 12, then all Y (Test) values are at least 12 units higher than the X (Comparative) values

Calculate Appropriate Statistics (cont'd)

39

Accuracy

What type of bias do you see?

Accuracy (cont’d)

Constant Bias Proportional Bias

40

41

Can a linear regression formula offer predictive value in relation to method comparisons?

A. YesB. No

Skill Check

42

Create graph based on sample set Determine slope from best-fit line Determine Y-intercept from best-fit line Explain the relationship between comparative and test

results

Activity

CLSI recommends four measurements of each specimen; three are sufficient

Series of samples of known concentrations (e.g., standard solutions, EQA linearity sets)

Series of known dilutions of highly elevated specimen or spiked specimens; EQA specimens

At least four levels (five preferred)

Reportable Range / Linearity Definition: Lowest and highest test results that

are reliable Especially important with two point calibrations Analytical Measurement Range (AMR) and

derived Clinical Reportable Range (CRR)

Introduction

What is needed

How we perform the

testing

43

44

Reportable Range:How We Evaluate the Data

Measured values on Y-axis versus Known or assigned values on X-axis

Plot mean values of:

Compare with expected values (typically provided by manufacturer)

Visually inspect, draw best-fit line, estimate reportable range

45

Reportable Range Activity

AssignedValue

Experimental Results

Average Rep #1 Rep #2 Rep #3 Rep #4

10.0 ____ 11.0 10.0 11.0 10.0

100.0 ____ 99.0 103.0 103.0 101.0

300.0 ____ 303.0 305.0 304.0 306.0

500.0 ____ 505.0 506.0 505.0 506.0

800.0 ____ 740.0 741.0 744.0 742.0

46

Reportable Range Activity (cont'd)

AssignedValue

Experimental Results

Average Rep #1 Rep #2 Rep #3 Rep #4

10.0 10.5 11.0 10.0 11.0 10.0

100.0 101.5 99.0 103.0 103.0 101.0

300.0 304.5 303.0 305.0 304.0 306.0

500.0 505.5 505.0 506.0 505.0 506.0

800.0 741.8 740.0 741.0 744.0 742.0

Reportable Range Activity (cont'd)Linearity Scatter Plot

0.0

100.0

200.0

300.0

400.0

500.0

600.0

700.0

800.0

0 100 200 300 400 500 600 700 800 900

Assigned Concentrations (units)

Rec

over

ed V

alue

s (M

eans

)

47

AMR vs. CRR

Analytical Measurement Range (AMR)

Linearity

Clinically Reportable Range (CRR)

Allows for dilution or other preparatory steps beyond routine

48

49

If you do not have enough specimen to perform a dilution, upon which reportable range component must you rely?

A. AMRB. CRRC. Neither A or BD. Both A and B

Skill Check

50

Utilizing the marketing materials from the two chemistry linearity kits in your handouts:

1. Determine which kit would be more appropriate for use with the chemistry assay you chose earlier

2. Explain your reasoning

Linearity Materials

51

Given your choice of linearity kits, you perform your AMR experiments by performing four replicates of each level of known concentration solution. The data you obtain is displayed on the next slide.

1. Review data; record any initial observations2. Graph data on supplied graph paper3. Determine your assay’s AMR

Graph Activity

Level Rep 1 Rep 2 Rep 3 Rep 4

1 24 23 25 24

2 196 197 171 194

3 359 360 358 361

4 530 532 529 535

5 700 695 702 709

Linearity Experiment Results

52

Activity

Using an Excel spreadsheet, create a graph and calculate

linear regression statistics from the data provided

53

54

Rep 1 Rep 2 Rep 3 Rep 4Lab's

AverageKnownConc

24 23 25 24 24 25

196 197 171 194 195.7 200

359 360 358 361 360 375

530 532 529 535 532 550

700 695 702 709 702 725

55

0100200300400500600700800

0 200 400 600 800

Reco

vere

d

Known Concentration

AMR Verfication

56

Your medical director, in consultation with clinicians, determines that for proper study participant care the Clinically Reportable Range (CRR) for glucose is15 – 1400 mg/dL

Given your linearity experiment results and the package insert, devise a dilution protocol to be contained within our Glucose SOP

Dilution Protocols

57

Given your AMR, CRR, and dilution protocol, how would you handle the following analyzer results?

1. 12 mg/dL2. 800 mg/dL3. 1600 mg/dL

Reportable Results

58

Reference Intervals Definition: Normal range in healthy population Used for diagnosis/clinical interpretation of

resultsIntroduction

Pre-defined “normal” criteria for screening purposes

Transferring: 20 “normal” individuals’ specimens Establishing: 120 “normal” individuals’ specimens

What is needed

Perform testing on all samples Document results

How we perform the

testing

Transferring Establishing 18 of 20 must

fall within manufacturer’s ranges

Calculate mean and SD of data for each group

Reference Intervals = mean ± 2 SD (if Gaussian Distribution only, otherwise, additional calculations recommended)

59

Reference Intervals:How We Evaluate the Data

60

Activity

Determine if assay is eligible for transference of reference intervals

Review a sample set of data to determine if transference may be performed; if not, determine next step(s)

Sensitivity Definition: Lowest reliable value; lower

limit of detection, especially of interest in drug testing and tumor markers

Different terminologies used by different manufacturers

Introduction

Blank solutions Spiked samples

What is needed

20 replicate measurements over short or long term, depending on focus

How we perform the

testing

61

Sensitivity: How We Evaluate the Data

Lower Limit of Detection (LLD): Mean of the blank

sample, plus two or three SD of blank

sample

Biological Limit of Detection:

LLD plus two or three times SD of

spiked sample with concentration of detection limit

Functional Sensitivity:

Mean concentration for spiked sample whose CV = 20%; lowest limit where quantitative data is

reliable

Three methods used:

62

63

Activity

Using the manufacturer’s package inserts, find the related information for sensitivity. How was it

calculated?

Specificity Definition: Determination of how well a method

measures the analyte of interest accompanied by potential interfering materials

Introduction

Standard solutions, participant specimens or pools

Interferer solutions (standard solutions, if possible; otherwise, pools or specimens) added at high concentrations

What is needed

Duplicate measurementsHow we

perform the testing

64

Specificity: How We Evaluate the Data

Tabulate results for pairs of samples (dilution and interferent)

Calculate means for each (dilution and interferent)

Calculate the differences Calculate the average interference

of all specimens tested at a given concentration of interference

65

66

Compare diagnosis Assume comparative (reference) method is accurate Determine the following:

True Positives, True negatives False Positives, False negatives

Calculate sensitivity and specificity and compare to manufacturer

Qualitative Assays

67

Negative and Positive Quality Controls Use QC materials recommended by manufacturer for

verification purposes Determine validity of other results, e.g., method

comparisons Evaluate failed runs if they occur during verification

process

Qualitative Assays: Control of Validation

68

How is it performed? Runs of specimens with analyte concentrations near

the cutoff point Three specimens, one at cutoff, one just below cutoff,

and one just above cutoff (± 20% recommended) Replicate measurements of each of three specimens

(20 each, minimum) How is it evaluated?

Determine percentage of positives and negatives for each specimen

Evaluate cutoff, as well as other two specimens

Qualitative Methods: Precision

69

How is it performed? Specimens typical of population (to be tested in future

use of method) 50 positive specimens and 50 negative specimens

recommended; minimum 20 each Performed over 10 to 20 days

How is it evaluated? Discrepant results near cutoff? Most often sensitivity and specificity used to describe

performance

Accuracy/Method Comparisons

70

Qualitative MethodsComparative or Reference

Method Result

Positive Negative

Test Method Result

Positive True Positive False Positive Positive Predictive Value

Negative False Negative True NegativeNegative

PredictiveValue

Sensitivity Specificity

False Positive Rate - False Positives divided by total number of Negatives

False Negative Rate - False Negatives divided by total number of Positives

True vs. False

71

Qualitative Methods (cont'd) Comparative or Reference

Method Result

Positive Negative

Test Method Result

Positive True Positive False Positive Positive Predictive Value

Negative False Negative True NegativeNegative

PredictiveValue

Sensitivity Specificity

Sensitivity = 100 x True Positives divided by (True Positives + False Negatives)

Specificity = 100 x True Negatives divided by (True Negatives + False Positives)

72

Qualitative Methods (cont'd)Comparative or Reference

Method Result

Positive Negative

Test Method Result

Positive True Positive False Positive Positive Predictive Value

Negative False Negative True NegativeNegative

PredictiveValue

Sensitivity Specificity

Predictive Values - Operation of a test on a mixed population of Positive and Negatives

A property of the test and the population; and affected by prevalence of Positives

Positive Predictive Value = True Positives divided by (True Positives + False

Positives) Negative Predictive Value = True Negatives divided by

(True Negatives + False Negatives)

73

High Diagnostic Value 100% Sensitivity 100% Specificity

What happens if True Positive rate is equal to the False Positive rate?

Evaluation Criteria

74

Activity

Estimate sensitivity and specificity of a qualitative method given a data set.

75

Activity (cont’d)

Create a validation plan for a quantitative assay to be performed in your laboratory.

76

Now that you have completed this module, you should be able to:

Identify test classifications Define what each validation experiment details for

testing methods Discuss what is recommended to perform each of the

validation experiments for testing methods Recognize how to evaluate data obtained from each of

the validation experiments

In Closing

77

A rapid HIV test would likely be classified as a:

A. High complexity, modified assayB. Moderate complexity, unmodified assayC. FDA-approved, modified assayD. Waived, FDA-approved, unmodified assay

Post-Assessment Question #1

78

The precision of a test method gives information related to the method’s:

A. Systematic errorB. Comparison of results to a reference methodC. ReproducibilityD. Likelihood of being affected by hemolysis, lipemia and

icterusE. Both A and B

Post-Assessment Question #2

79

When transferring reference intervals of 20 specimens used, what is the minimum number that must fall within manufacturer’s reference intervals?

A. 20B. 18C. 16D. 15

Post-Assessment Question #3

80

Which linear regression equation component gives information regarding constant bias?

A. yB. xC. m (slope)D. b (intercept)

Post-Assessment Question #4

81

DAIDS Good Clinical Laboratory Practice (GCLP) Guidelines. www.westgard.com. Validation of Qualitative Methods. 42 CFR § 493.1253. College of American Pathologists Commission on Laboratory Accreditation,

Accreditation Checklists, April 2006. Westgard, James O. Basic Method Validation 2nd Edition. Madison, WI: Westgard

QC, Inc., 2003. Clinical and Laboratory Standards Institute. User Protocol for Evaluation of

Qualitative Test Performance; Approved Guideline. NCCLS document EP12-A. Clinical and Laboratory Standards Institute, Wayne, PA USA, 2002.

Clinical and Laboratory Standards Institute. Evaluation of Precision. Performance of Quantitative Measurement Methods. NCCLS document EP5-A2.

Clinical and Laboratory Standards Institute, Wayne, PA USA, 2004. Clinical and Laboratory Standards Institute. User verification of Performance for

Precision and Trueness. CLSI document EP15-A2. Clinical and Laboratory Standards Institute, Wayne, PA USA, 2005.

References

Wrap Up

82